Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEA] Implement lore framework to support all operators. #10987

Closed
liurenjie1024 opened this issue Jun 6, 2024 · 2 comments · Fixed by #11084
Closed

[FEA] Implement lore framework to support all operators. #10987

liurenjie1024 opened this issue Jun 6, 2024 · 2 comments · Fixed by #11084
Assignees
Labels
feature request New feature or request

Comments

@liurenjie1024
Copy link
Collaborator

Is your feature request related to a problem? Please describe.
We want to implement a lore framework to support all operators.

Describe the solution you'd like
We need to figure out a way to allow user to tell us the operator id at runtime, e.g. we call it lore_id. The lore_id should be determinstic when given same spark configration, spark sql, and input data. Then in the second run we will dump the operators' input data, meta data(e.g. plan information) so that we can replay it in local. Ideally, we will also dump nsight tracing utilizing work here: #10870

Describe alternatives you've considered
No.

Additional context
No.

@liurenjie1024 liurenjie1024 added feature request New feature or request ? - Needs Triage Need team to review and classify labels Jun 6, 2024
@liurenjie1024 liurenjie1024 self-assigned this Jun 6, 2024
@binmahone
Copy link
Collaborator

binmahone commented Jun 7, 2024

with #10999, we can start to ues LORE at customer site for simple cases like GpuAggregateExec. I can think of these remaining issues to address:

  1. target Exec must be GpuExec, target Exec must have a child and it must be GpuExec
  2. only UnaryLike GpuExec is suppported now (Join not supported yet)
  3. target Exec's RDD paritions must be a 1:1 mapping with child Exec's RDD. e.g. GpuCoaleseExec(CoaleseRDD) not suppport now.
  4. cannot deal with cases where input has no columns (e.g. select count(*) )
  5. can only dump to executor local disk (/tmp/lore/)
  6. test case required

The list might be incomplete.

@liurenjie1024
Copy link
Collaborator Author

target Exec must be GpuExec, target Exec must have a child and it must be GpuExec

I think we only need to care about GpuExec?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants