Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug] server memory leak with netty #1355

Closed
3 tasks done
zuston opened this issue Dec 7, 2023 · 1 comment · Fixed by #1357
Closed
3 tasks done

[Bug] server memory leak with netty #1355

zuston opened this issue Dec 7, 2023 · 1 comment · Fixed by #1357

Comments

@zuston
Copy link
Member

zuston commented Dec 7, 2023 •
edited
Loading

Code of Conduct

Search before asking

  • I have searched in the issues and found no similar issues.

Describe the bug

image

Affects Version(s)

master

Uniffle Server Log Output

No response

Uniffle Engine Log Output

No response

Uniffle Server Configurations

rss.rpc.server.type            GRPC_NETTY
rss.server.netty.port          21100

Uniffle Engine Configurations

spark.rss.client.type         GRPC_NETTY

Additional context

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!
@zuston
Copy link
Member Author

zuston commented Dec 7, 2023

PTAL @leixm

@zuston zuston linked a pull request Dec 7, 2023 that will close this issue
[#1335] fix(server)(netty): release bytebuf explicitly when requiredId is expired and cache failed #1357
Merged
rickyma added a commit to rickyma/incubator-uniffle that referenced this issue Jan 15, 2024
@rickyma
[apache#1355] fix(client): Netty client will leak when decoding respo…
a3fbd4c 
…nses
rickyma added a commit to rickyma/incubator-uniffle that referenced this issue Jan 15, 2024
@rickyma
[apache#1355] fix(client): Netty client will leak when decoding respo…
d64e308 
…nses
rickyma added a commit to rickyma/incubator-uniffle that referenced this issue Jan 16, 2024
@rickyma
[apache#1355] fix(client): Netty client will leak when decoding respo…
971c63c 
…nses
rickyma added a commit to rickyma/incubator-uniffle that referenced this issue Jan 16, 2024
@rickyma
[apache#1355] fix(client): Netty client will leak when decoding respo…
d82a0fa 
…nses
rickyma added a commit to rickyma/incubator-uniffle that referenced this issue Jan 16, 2024
@rickyma
[apache#1355] fix(client): Netty client will leak when decoding respo…
2b77d46 
…nses
rickyma added a commit to rickyma/incubator-uniffle that referenced this issue Jan 16, 2024
@rickyma
[apache#1355] fix(client): Netty client will leak when decoding respo…
55bcb03 
…nses
rickyma added a commit to rickyma/incubator-uniffle that referenced this issue Jan 16, 2024
@rickyma
[apache#1355] fix(client): Netty client will leak when decoding respo…
ee27f6a 
…nses
rickyma added a commit to rickyma/incubator-uniffle that referenced this issue Jan 16, 2024
@rickyma
[apache#1355] fix(client): Netty client will leak when decoding respo…
d2d7794 
…nses
jerqi pushed a commit that referenced this issue Jan 19, 2024
@rickyma
[#1355] fix(client): Netty client will leak when decoding responses (#…
04d6854 
…1455)

### What changes were proposed in this pull request?

The current code logic is that the `ByteBuf` is only released when `msg.body() == null`. However, when `msg.body != null`, `msg.body().byteBuf()` returns a `NettyManagedBuffer.EMPTY_BUFFER`, and the `ByteBuf` is not released in this case, resulting in a memory leak issue every time decoding an RPC response from ShuffleServer.
Over time, if the Spark Job runs long enough and there are enough requests, it will eventually cause a significant memory leak on the client side (Spark Executor).
The modifications to the other code are mainly for readability and enhanced protection, and will not cause any side effects.

### Why are the changes needed?

To fix the memory leak issue in the Netty client when decoding RPC responses.
For [#1359](#1359)

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Rerun successfully and no more Netty leak logs.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
1 participant
@zuston