Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feature Request] Support deploy multiple shuffle servers in a single node #77

Closed
xianjingfeng opened this issue Jul 26, 2022 · 9 comments · Fixed by #166
Closed

[Feature Request] Support deploy multiple shuffle servers in a single node #77

xianjingfeng opened this issue Jul 26, 2022 · 9 comments · Fixed by #166

Comments

@xianjingfeng
Copy link
Member

We have 1.2T memory per host, so we need deploy multiple shuffle servers in a single node.
But current partition assignment policy is not suitable to do that.
Because partition maybe assign to a same host and client maybe fail if one host down
@zuston
Copy link
Member

zuston commented Jul 26, 2022

As i know, partition assignment policy could be implemented by yourself.

@xianjingfeng
Copy link
Member Author

As i know, partition assignment policy could be implemented by yourself.

Yes, but not only assignment policy need to be change, such as start script, i will create a pr for this

@jerqi
Copy link
Contributor

jerqi commented Jul 27, 2022

I'm also curious why we need to modify start script?

@xianjingfeng
Copy link
Member Author

I'm also curious why we need to modify start script?

start script will process existence

@xianjingfeng
Copy link
Member Author

If our plan is deploy on k8s, this issue should close?

@jerqi
Copy link
Contributor

jerqi commented Jul 27, 2022

K8S Deployment can't solve this issue.

@colinmjj
Copy link

I'm also curious why we need to modify start script?

start script will process existence

Yes, this is the limit with current implementation

@jerqi
Copy link
Contributor

jerqi commented Jul 28, 2022

K8S Deployment can't solve this issue.

@xianjingfeng I think you can continue this.

@xianjingfeng
Copy link
Member Author

K8S Deployment can't solve this issue.

@xianjingfeng I think you can continue this.

ok

@xianjingfeng xianjingfeng mentioned this issue Aug 18, 2022
Merged
@jerqi jerqi linked a pull request Aug 22, 2022 that will close this issue
Support deploy multiple shuffle servers in a single node #166
Merged
jerqi pushed a commit that referenced this issue Sep 27, 2022
@xianjingfeng
Support deploy multiple shuffle servers in a single node (#166)
f1cb43f 
### What changes were proposed in this pull request?
1.Sufflle server with same ip will not be assigned to same partition
2.Check whether port is in use in start script of shuffle server

### Why are the changes needed?
If we have a lot of  memory(more than 1T) per host, so we need deploy multiple shuffle servers in a single node. #77 

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
already added
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Support deploy multiple shuffle servers in a single node xianjingfeng/incubator-uniffle
4 participants
@colinmjj @jerqi @zuston @xianjingfeng