Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feature Request] Deploy Uniffle on Kubernetes #48

Closed
jerqi opened this issue Jul 11, 2022 · 10 comments
Closed

[Feature Request] Deploy Uniffle on Kubernetes #48

jerqi opened this issue Jul 11, 2022 · 10 comments
Assignees
@zuston @wangao1236 @czy006

Comments

@jerqi
Copy link
Contributor

jerqi commented Jul 11, 2022

It's good to have a Kubernetes operator to help us deploy Uniffle on Kubernetes. We can automate our system operations. This issue track the pull requests about Kubernetes operator.
@jerqi jerqi mentioned this issue Jul 26, 2022
Merged
@jerqi
Copy link
Contributor Author

jerqi commented Jul 26, 2022

@wangao1236 Do you want to help us?

@wangao1236
Copy link
Collaborator

wangao1236 commented Jul 26, 2022 •
edited
Loading

Thanks, I'll submit some PRs about dockerfile and operator of Uniffe.

jerqi added a commit that referenced this issue Jul 27, 2022
@jerqi
[ISSUE-48] [Feature] Init Kubernetes operator directory (#75)
229b63b 
### What changes were proposed in this pull request?
To solve issue #48, First step. we add Kubernetes operator profile and directory. We add two scripts build-operator.sh and test-operator.sh. We can use these scripts to build and test our operator. When we use the command `mvn compile -Pkubernetes`, we can build the operator. When we use the command `mvn test -Pkubernetes`, we can test the operator.

### Why are the changes needed?
We hope use Kubernetes to deploy Uniffle.

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
No need.
@czy006
Copy link

czy006 commented Jul 27, 2022

I want to join in this issuses dev,I am currently working as an Apache Flink Operator developer, and I believe my experience will be helpful to our project @jerqi

@jerqi
Copy link
Contributor Author

jerqi commented Jul 27, 2022

I want to join in this issuses dev,I am currently working as an Apache Flink Operator developer, and I believe my experience will be helpful to our project @jerqi

Thanks, you're welcome. We have developed this feature in our internal company for a period time. And @wangao1236 is my colleague. He will share our design document and give you some tasks. And you can also review his pull request and give us some advice.

@jerqi jerqi pinned this issue Jul 27, 2022
@jerqi jerqi unpinned this issue Jul 27, 2022
@jerqi jerqi pinned this issue Jul 27, 2022
@wangao1236
Copy link
Collaborator

I have written a design document, please help to review it: https://docs.google.com/document/d/1dlTvC0I-1WDNS6MOZp_0KB6mR3qr-EUiEJ11eeI828g/edit#heading=h.e0k4tcq0il5a
@jerqi @czy006

@jerqi
Copy link
Contributor Author

jerqi commented Aug 1, 2022

@wangao1236 Great work!

@zuston
Copy link
Member

zuston commented Aug 1, 2022

I have some experience in operator development(our internal elastic Yarn operator), maybe i can join this ticket

@wangao1236
Copy link
Collaborator

I have some experience in operator development(our internal elastic Yarn operator), maybe i can join this ticket

Welcome~

@zuston
Copy link
Member

zuston commented Aug 1, 2022

Left some comments. Thanks for your great work @wangao1236.

Besides, i want to know which language will be used when implementing this operator? Java or Golang.
Do we have a POC? Maybe it benefit us to understand your design.

@wangao1236
Copy link
Collaborator

Left some comments. Thanks for your great work @wangao1236.

Besides, i want to know which language will be used when implementing this operator? Java or Golang. Do we have a POC? Maybe it benefit us to understand your design.

We use golang to develop it. Sorry, there is currently no public POC environment.

@frankliee frankliee unpinned this issue Aug 5, 2022
@wangao1236 wangao1236 mentioned this issue Aug 5, 2022
Merged
@jerqi jerqi pinned this issue Aug 7, 2022
jerqi pushed a commit that referenced this issue Aug 10, 2022
@wangao1236
[ISSUE-48][FEATURE] Add Uniffle Dockerfile (#132)
18663ed 
### What changes were proposed in this pull request?
To solve issue #48 ，I add the Dockerfile for Uniffle.

### Why are the changes needed?
Support K8S Operator

### Does this PR introduce _any_ user-facing change?
Yes, we will add the document later

### Does this PR introduce _any_ user-facing change?
Yes, we will add the document later
@frankliee frankliee unpinned this issue Aug 15, 2022
@wangao1236 wangao1236 mentioned this issue Aug 18, 2022
Merged
@jerqi jerqi pinned this issue Aug 20, 2022
jerqi pushed a commit that referenced this issue Aug 22, 2022
@wangao1236
[ISSUE-48][FEATURE]: Init Operator Directory (#161)
0f5a42a 
### What changes were proposed in this pull request?
for issue #48 
I init the operator first, and I will submit a following pr of crd. 

### Why are the changes needed?
Support to deploy Uniffle on K8S

### Does this PR introduce _any_ user-facing change?
We will write the docs later.

### How was this patch tested?
No need
@wangao1236 wangao1236 mentioned this issue Aug 22, 2022
Merged
@jerqi jerqi linked a pull request Aug 22, 2022 that will close this issue
[ISSUE-48][FEATURE][FOLLOW UP] Add RemoteShuffleService CRD #175
Merged
@jerqi jerqi removed a link to a pull request Aug 22, 2022
[ISSUE-48][FEATURE][FOLLOW UP] Add RemoteShuffleService CRD #175
Merged
jerqi pushed a commit that referenced this issue Aug 25, 2022
@wangao1236
[ISSUE-48][FEATURE][FOLLOW UP] Add RemoteShuffleService CRD (#175)
c25e5a1 
### What changes were proposed in this pull request?
for issue #48 
I add a crd object called RemoteShuffleService this time, and I will submit the controller of the crd in next pr.

### Why are the changes needed?
Support K8S

### Does this PR introduce _any_ user-facing change?
Yes,  we will add the doc later.

### How was this patch tested?
No need
@wangao1236 wangao1236 mentioned this issue Aug 26, 2022
Merged
jerqi pushed a commit that referenced this issue Sep 1, 2022
@wangao1236
[ISSUE-48][FEATURE][FOLLOW UP] Add webhook component (#188)
d931e8b 
### What changes were proposed in this pull request?
for issue #48 
I add webhook module this time, maybe some logic is missing unit test, I will add in the next PR.

### Why are the changes needed?
Support K8S


### Does this PR introduce _any_ user-facing change?
Yes, We will add the doc later..

### How was this patch tested?
UT
@wangao1236 wangao1236 mentioned this issue Sep 6, 2022
Merged
jerqi pushed a commit that referenced this issue Sep 6, 2022
@wangao1236
[ISSUE-48][FEATURE][FOLLOW UP] Generate informer and lister for crd (#…
07f70ed 
…202)

### What changes were proposed in this pull request?
for issue #48 
I add some codes of crd's informer and lister generated by client-gen, and I will add controller module in the next pr.

### Why are the changes needed?
Support K8S

### Does this PR introduce _any_ user-facing change?
We will add the doc later

### How was this patch tested?
No need
@wangao1236 wangao1236 mentioned this issue Sep 13, 2022
Merged
jerqi pushed a commit that referenced this issue Sep 20, 2022
@wangao1236
[ISSUE-48][FEATURE][FOLLOW UP] Add controller component (#214)
e6b4260 
### What changes were proposed in this pull request?
For issue #48 
I add some codes about controller module, and I'll add more unit tests in next PR.

### Why are the changes needed?
Support K8S


### Does this PR introduce _any_ user-facing change?
Yes, we will add the doc later


### How was this patch tested?
Manual test
@jerqi jerqi closed this as completed Sep 20, 2022
@jerqi jerqi unpinned this issue Sep 20, 2022
@wangao1236 wangao1236 mentioned this issue Sep 22, 2022
Merged
jerqi pushed a commit that referenced this issue Sep 23, 2022
@wangao1236
[ISSUE-48][FEATURE][FOLLOW UP] Add yaml of components and crd exampes (…
1b281c4 
…#236)

### What changes were proposed in this pull request?
For issue #48 
I add some yaml files of components and crd exampes.

### Why are the changes needed?
Add yaml configuration


### Does this PR introduce _any_ user-facing change?
No


### How was this patch tested?
No need.
@wangao1236 wangao1236 mentioned this issue Sep 28, 2022
Merged
jerqi pushed a commit that referenced this issue Oct 8, 2022
@wangao1236
[ISSUE-48][FEATURE][FOLLOW UP] add unit test for validating rss objec…
4408359 
…ts (#248)

### What changes were proposed in this pull request?
For issue #48 
I add some unit test about validating rss objects in webhook components.

### Why are the changes needed?
Support K8S

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
Just test
@wangao1236 wangao1236 mentioned this issue Oct 12, 2022
Merged
jerqi pushed a commit that referenced this issue Oct 13, 2022
@wangao1236
[ISSUE-48][FEATURE][FOLLOW UP] add docs for operator (#261)
8be8390 
### What changes were proposed in this pull request?
For issue #48 
I add docs about design and usage for operator.

### Why are the changes needed?
Add doc for operator

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
Just doc
@jerqi jerqi mentioned this issue Nov 19, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@zuston zuston

@wangao1236 wangao1236

@czy006 czy006

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@jerqi @zuston @wangao1236 @czy006