Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Improvement] Avoid removeResources for multiple times #456

Closed
2 of 3 tasks
xianjingfeng opened this issue Jan 4, 2023 · 2 comments · Fixed by #459
Closed
2 of 3 tasks

[Improvement] Avoid removeResources for multiple times #456

xianjingfeng opened this issue Jan 4, 2023 · 2 comments · Fixed by #459

Comments

@xianjingfeng
Copy link
Member

Code of Conduct

Search before asking

  • I have searched in the issues and found no similar issues.

What would you like to be improved?

If removeResources take to much time. NPE will be thrown as follows.

[ERROR] 2022-12-30 00:43:28,403 clearResourceThread ShuffleTaskManager lambda$new$3 - Exception happened when clear resource for expired application
java.lang.NullPointerException
        at org.apache.uniffle.server.ShuffleTaskManager.removeResources(ShuffleTaskManager.java:479)
        at org.apache.uniffle.server.ShuffleTaskManager.lambda$new$3(ShuffleTaskManager.java:130)
        at java.lang.Thread.run(Thread.java:745)

How should we improve?

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!
@zuston
Copy link
Member

zuston commented Jan 6, 2023

Nice catch, could you help fix this? If not, you could give some points to fix and mark it as good first issue.

@xianjingfeng xianjingfeng mentioned this issue Jan 9, 2023
Merged
@jerqi jerqi linked a pull request Jan 9, 2023 that will close this issue
[ISSUE-456] Avoid removeResources for multiple times #459
Merged
zuston pushed a commit that referenced this issue Jan 10, 2023
@xianjingfeng
[ISSUE-456] Avoid removing resources for multiple times (#459)
3f166f4 
### What changes were proposed in this pull request?
If `Resource` had been removed, avoid removing twice.

### Why are the changes needed?
When some appIds' removeResource took too much time, the `expiredAppCleanupExecutorService` in ShuffleTaskManager would check and detect the same appId is expired multiple times. Therefore the 
same appId might be added to `expiredAppIdQueue` multiple times. This PR fixes #456 

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
UT
@zuston
Copy link
Member

zuston commented Jan 10, 2023

Fixed by #459

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[ISSUE-456] Avoid removeResources for multiple times xianjingfeng/incubator-uniffle
2 participants
@zuston @xianjingfeng