Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug] msg: Zstd only supports the same type of bytebuffer decompression #1358

Open
2 of 3 tasks
zuston opened this issue Dec 7, 2023 · 7 comments
Open
2 of 3 tasks

[Bug] msg: Zstd only supports the same type of bytebuffer decompression #1358

zuston opened this issue Dec 7, 2023 · 7 comments

Comments

@zuston
Copy link
Member

zuston commented Dec 7, 2023 •
edited
Loading

Code of Conduct

Search before asking

  • I have searched in the issues and found no similar issues.

Describe the bug

image

Affects Version(s)

master

Uniffle Server Log Output

No response

Uniffle Engine Log Output

No response

Uniffle Server Configurations

No response

Uniffle Engine Configurations

No response

Additional context

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!
@zuston
Copy link
Member Author

zuston commented Dec 8, 2023

Do you know the managedBuffer is not direct buffer when getting memory by netty? @leixm

@leixm
Copy link
Contributor

leixm commented Dec 8, 2023

Do you know the managedBuffer is not direct buffer when getting memory by netty? @leixm

Does this happen when netty is turned on or off? RssShuffleDataIterator line164 will keep uncompressedData the same type as rawData. Why does the above exception still appear?

@zuston
Copy link
Member Author

zuston commented Dec 8, 2023

Do you know the managedBuffer is not direct buffer when getting memory by netty? @leixm

Does this happen when netty is turned on or off? RssShuffleDataIterator line164 will keep uncompressedData the same type as rawData. Why does the above exception still appear?

Netty is on.

@jerqi
Copy link
Contributor

jerqi commented Dec 8, 2023 •
edited
Loading

Is this issue related to spark.rss.client.off.heap.memory.enable?

@zuston
Copy link
Member Author

zuston commented Dec 8, 2023

I found sometimes the bytebuffer is not direct after coverting from netty direct bytebuf using bytebuf.nioBuffer()

@zuston
Copy link
Member Author

zuston commented Dec 8, 2023 •
edited
Loading

image

@leixm

zuston added a commit to zuston/incubator-uniffle that referenced this issue Dec 8, 2023
@zuston
[apache#1358] fix(spark): pre-check bytebuffer whether is direct befo…
e15a1c6 
…re uncompress
@zuston zuston mentioned this issue Dec 8, 2023
Merged
zuston added a commit to zuston/incubator-uniffle that referenced this issue Dec 13, 2023
@zuston
[apache#1358] fix(spark): pre-check bytebuffer whether is direct befo…
29fd1d9 
…re uncompress
@jerqi
Copy link
Contributor

jerqi commented Dec 18, 2023

image

@leixm

@LuciferYang Do you know more about this point?

zuston added a commit that referenced this issue Dec 19, 2023
@zuston
[#1358] fix(spark): pre-check bytebuffer whether is direct before unc…
56490d4 
…ompress (#1360)

### What changes were proposed in this pull request?

precheck bytebuffer whether is direct before uncompress, and rebuild the temp bytebuffer.

### Why are the changes needed?

For #1358 . But this PR is not the fundamental solution

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Existing tests.
zuston added a commit to zuston/incubator-uniffle that referenced this issue Jan 18, 2024
@zuston
[apache#1358] fix(spark): pre-check bytebuffer whether is direct befo…
4db2396 
…re uncompress (apache#1360)

### What changes were proposed in this pull request?

precheck bytebuffer whether is direct before uncompress, and rebuild the temp bytebuffer.

### Why are the changes needed?

For apache#1358 . But this PR is not the fundamental solution

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Existing tests.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@jerqi @zuston @leixm