[FEATURE] [TEZ] Support remote spill in merge stage. #1231
Comments
3 tasks
zhengchenyu
added a commit
to zhengchenyu/incubator-uniffle
that referenced
this issue
Oct 18, 2023
This was referenced Oct 18, 2023
jerqi
pushed a commit
that referenced
this issue
Oct 19, 2023
### What changes were proposed in this pull request? Add new merger manager for remote spill. ### Why are the changes needed? In a cloud scenario, the storage of the machine that performs the job is very small. The merge phase often uses disk. The remote spill allows remote shuffle service job to use as little disk as possible. Fix: #1231 ### Does this PR introduce _any_ user-facing change? | config | description | | --- | --- | | tez.rss.reduce.remote.spill.enable | Whether to enable remote spill | | tez.rss.remote.spill.storage.path | the remote spill path | | rss.reduce.remote.spill.replication | the replication of remote fs | | rss.reduce.remote.spill.retries | try times for remote fs | ### How was this patch tested? unit test, integration test, tez examples on cluster.
qijiale76
pushed a commit
to qijiale76/incubator-uniffle
that referenced
this issue
Jan 17, 2024
add integration test
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Code of Conduct
Search before asking
Describe the feature
In a cloud scenario, the storage of the machine that performs the job is very small. The merge phase often uses disk. The remote spill allows remote shuffle service job to use as little disk as possible.
Motivation
No response
Describe the solution
No response
Additional context
No response
Are you willing to submit PR?
The text was updated successfully, but these errors were encountered: