dm-operator/: support start a new dm cluster with dm-masters and dm-workers #3146

lichunzhu · 2020-08-21T02:40:42Z

What problem does this PR solve?

#2868

What is changed and how does it work?

Support deploying dm cluster with masters and workers.

Check List

Tests

Unit test
E2E test
Stability test
Manual test (add detailed scripts or steps below)

Deploy dm-cluster with manifests/dm/dm-cluster.yaml:

kubectl get pods, dm-master and dm-worker pods start correctly

basic-dm-master-0                  1/1     Running   0          7m27s
basic-dm-master-1                  1/1     Running   0          7m27s
basic-dm-master-2                  1/1     Running   1          7m27s
basic-dm-worker-0                  1/1     Running   1          7m8s
basic-dm-worker-1                  1/1     Running   1          7m8s
basic-dm-worker-2                  1/1     Running   0          7m8s

kubectl get dc, can return dm-cluster info correctly

NAME    READY   MASTER                     STORAGE   READY   DESIRE   WORKER                     STORAGE   READY   DESIRE   AGE
basic   True    pingcap/dm:v2.0.0-beta.2   1Gi       3       3        pingcap/dm:v2.0.0-beta.2   1Gi       3       3        7m52s

kubectl port-forward svc/basic-dm-master, can forward dm-master service correctly.

basic-dm-master        ClusterIP   10.107.238.92    <none>        8261/TCP              8m8s
basic-dm-master-peer   ClusterIP   None             <none>        8291/TCP              8m8s
basic-dm-worker-peer   ClusterIP   None             <none>        8262/TCP              7m49s

kubectl describe dc basic:

Spec:
  Discovery:
    Address:          http://basic-discovery.demo:10261
  Enable PV Reclaim:  false
  Image Pull Policy:  IfNotPresent
  Master:
    Base Image:  pingcap/dm
    Config:
    Replicas:         3
    Storage Size:     1Gi
  Pv Reclaim Policy:  Delete
  Tls Cluster:
  Version:  v2.0.0-beta.2
  Worker:
    Base Image:  pingcap/dm
    Config:
    Replicas:      3
    Storage Size:  1Gi
Status:
  Conditions:
    Last Transition Time:  2020-08-26T02:31:53Z
    Last Update Time:      2020-08-26T02:31:53Z
    Message:               DM cluster is fully up and running
    Reason:                Ready
    Status:                True
    Type:                  Ready
  Master:
    Image:  pingcap/dm:v2.0.0-beta.2
    Leader:
      Client URL:            http://0.0.0.0:8261
      Health:                true
      Id:                    14863793955117218053
      Last Transition Time:  2020-08-26T02:31:45Z
      Name:                  basic-dm-master-0
    Members:
      Basic - Dm - Master - 0:
        Client URL:            http://0.0.0.0:8261
        Health:                true
        Id:                    14863793955117218053
        Last Transition Time:  2020-08-26T02:31:45Z
        Name:                  basic-dm-master-0
      Basic - Dm - Master - 1:
        Client URL:            http://0.0.0.0:8261
        Health:                true
        Id:                    10916830957434093435
        Last Transition Time:  2020-08-26T02:31:45Z
        Name:                  basic-dm-master-1
      Basic - Dm - Master - 2:
        Client URL:            http://0.0.0.0:8261
        Health:                true
        Id:                    14896049602475078108
        Last Transition Time:  2020-08-26T02:31:52Z
        Name:                  basic-dm-master-2
    Phase:                     Normal
    Stateful Set:
      Collision Count:      0
      Current Replicas:     3
      Current Revision:     basic-dm-master-59697d9554
      Observed Generation:  1
      Ready Replicas:       3
      Replicas:             3
      Update Revision:      basic-dm-master-59697d9554
      Updated Replicas:     3
    Synced:                 true
  Worker:
    Image:  pingcap/dm:v2.0.0-beta.2
    Members:
      Basic - Dm - Worker - 0 . Basic - Dm - Worker - Peer : 8262:
        Addr:                  basic-dm-worker-0.basic-dm-worker-peer:8262
        Last Transition Time:  2020-08-26T02:31:52Z
        Name:                  basic-dm-worker-0.basic-dm-worker-peer:8262
        Stage:                 free
      Basic - Dm - Worker - 1 . Basic - Dm - Worker - Peer : 8262:
        Addr:                  basic-dm-worker-1.basic-dm-worker-peer:8262
        Last Transition Time:  2020-08-26T02:31:52Z
        Name:                  basic-dm-worker-1.basic-dm-worker-peer:8262
        Stage:                 free
      Basic - Dm - Worker - 2 . Basic - Dm - Worker - Peer : 8262:
        Addr:                  basic-dm-worker-2.basic-dm-worker-peer:8262
        Last Transition Time:  2020-08-26T02:31:51Z
        Name:                  basic-dm-worker-2.basic-dm-worker-peer:8262
        Stage:                 free
    Phase:                     Normal
    Stateful Set:
      Collision Count:      0
      Current Replicas:     3
      Current Revision:     basic-dm-worker-5bfd664759
      Observed Generation:  1
      Ready Replicas:       3
      Replicas:             3
      Update Revision:      basic-dm-worker-5bfd664759
      Updated Replicas:     3
    Synced:                 true

Code changes

Has Go code change

Does this PR introduce a user-facing change?:

NONE

…ator into defineDMSpec

codecov-commenter · 2020-08-21T02:48:04Z

Codecov Report

Merging #3146 into master will decrease coverage by 2.97%.
The diff coverage is 1.78%.

@@            Coverage Diff             @@
##           master    #3146      +/-   ##
==========================================
- Coverage   43.66%   40.69%   -2.98%     
==========================================
  Files         159      167       +8     
  Lines       17101    18365    +1264     
==========================================
+ Hits         7467     7473       +6     
- Misses       8985    10244    +1259     
+ Partials      649      648       -1

Flag	Coverage Δ
#unittest	`40.69% <1.78%> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

…o supportStartDMCluster

manifests/dm/dm-cluster.yaml

pkg/apis/pingcap/v1alpha1/defaulting/dmcluster.go

pkg/apis/pingcap/v1alpha1/dmcluster.go

DanielZhangQD · 2020-08-25T06:47:42Z

pkg/apis/pingcap/v1alpha1/dmcluster.go

+		if version == nil {
+			version = &dc.Spec.Version
+		}
+		image = fmt.Sprintf("%s:%s", baseImage, *version)


What if *version=""?

Use "latest" in 161b862

pkg/apis/pingcap/v1alpha1/dmcluster.go

pkg/apis/pingcap/v1alpha1/register.go

pkg/apis/pingcap/v1alpha1/validation/validation.go

pkg/controller/dmcluster/dm_cluster_condition_updater.go

DanielZhangQD · 2020-08-25T07:31:17Z

pkg/controller/dmcluster/dm_cluster_control.go

+	//   - upgrade the dm-master cluster
+	//   - scale out/in the dm-master cluster
+	//   - failover the dm-master cluster
+	if err := dcc.masterMemberManager.Sync(dc); err != nil {


Can we just log and append the error here and continue to sync the worker? WDYT @weekface @cofyc

pkg/controller/dmcluster/dm_cluster_controller.go

pkg/label/label.go

weekface · 2020-08-25T08:34:11Z

manifests/dm/dm-cluster.yaml

+  version: v2.0.0-beta.2
+  pvReclaimPolicy: Delete
+  discovery:
+    address: "http://basic-discovery.demo.svc:10261"


Why we need to config the discovery address?

Because dmcluster won't start another discovery service for itself. It will just reuse tidb cluster's discovery address.

pkg/manager/member/dm_worker_member_manager.go

pkg/manager/member/dm_master_member_manager.go

pkg/manager/member/dm_worker_member_manager.go

pkg/manager/member/pd_member_manager.go

pkg/manager/member/template.go

pkg/manager/member/utils.go

pkg/util/crdutil.go

examples/dm/README.md

pkg/apis/pingcap/v1alpha1/defaulting/dmcluster.go

pkg/apis/pingcap/v1alpha1/dmcluster.go

pkg/manager/member/dm_master_member_manager.go

DanielZhangQD · 2020-08-25T12:39:41Z

pkg/manager/member/dm_worker_member_manager.go

+	if !dc.MasterIsAvailable() {
+		return controller.RequeueErrorf("DMCluster: %s/%s, waiting for dm-master cluster running", ns, dcName)
+	}


I think it's OK to remove this check if dm-worker can handle the case where dm-master service is unavailable.

If dm-master doesn't have more than replicas/2 members alive, it can't offer service. I think we can keep it?

pkg/manager/member/dm_worker_member_manager.go

DanielZhangQD · 2020-08-25T12:49:13Z

When adding logic about the upgrade, scale, and failover, please consider the case in #2886.

examples/dm/README.md

pkg/controller/dmcluster/dm_cluster_controller.go

…tidb-operator into supportStartDMCluster

DanielZhangQD

Rest LGTM

pkg/controller/dmcluster/dm_cluster_controller.go

examples/dm/README.md

DanielZhangQD

LGTM

cofyc

LGTM

lichunzhu · 2020-08-26T05:42:32Z

/test pull-e2e-kind

lichunzhu added 22 commits August 5, 2020 14:36

add dmclusters CRD

08a8ae4

resolve conflicts

5cb82ac

Merge branch 'master' into defineDMSpec

bcf58ea

address comment

55a6365

Merge branch 'master' into defineDMSpec

0aae8c0

address comments

f146be1

Merge branch 'defineDMSpec' of https://github.com/lichunzhu/tidb-oper…

94c427b

…ator into defineDMSpec

delete monitor ref

4f10a51

generate dmcluster client

a020492

address comments

3edfaa2

Merge branch 'master' into defineDMSpec

74aca39

address comment

2c1bec5

tmp commit

fbe26f3

resolve conflict

d85a9fc

merge master

a9da15f

remove dm package

ba0f518

fix bugs

7a51c07

fix bug

3ec6c86

support start dm-master and dm-worker in cluster

8122c71

fix some bugs

a38bb0e

merge master branch and resolve conflicts

1e7efd8

fix ut

850035c

lichunzhu added 7 commits August 24, 2020 19:38

fix dm-master start

cd3f5b4

fix dm-worker start bug

f4a641d

Merge branch 'master' of https://github.com/pingcap/tidb-operator int…

5a51cbe

…o supportStartDMCluster

add more column info

be4fd5e

Merge branch 'master' of https://github.com/pingcap/tidb-operator int…

1e4f2ac

…o supportStartDMCluster

Merge branch 'master' into supportStartDMCluster

daf81f7

fix ut

f7c70bd

DanielZhangQD reviewed Aug 25, 2020

View reviewed changes

weekface reviewed Aug 25, 2020

View reviewed changes

DanielZhangQD reviewed Aug 25, 2020

View reviewed changes

lichunzhu added 3 commits August 25, 2020 18:05

address comments

161b862

Merge branch 'master' into supportStartDMCluster

35a69d3

regenerate code

d3da808

DanielZhangQD reviewed Aug 25, 2020

View reviewed changes

DanielZhangQD requested a review from lonng August 25, 2020 12:59

lichunzhu added 2 commits August 25, 2020 23:55

address comments

8e52fbb

Merge branch 'master' into supportStartDMCluster

cefe7b8

DanielZhangQD reviewed Aug 26, 2020

View reviewed changes

examples/dm/README.md Outdated Show resolved Hide resolved

pkg/controller/dmcluster/dm_cluster_controller.go Outdated Show resolved Hide resolved

lichunzhu added 3 commits August 26, 2020 10:37

fix import cycle problem

f1b6029

Merge branch 'supportStartDMCluster' of https://github.com/lichunzhu/…

7726669

…tidb-operator into supportStartDMCluster

fix check

84a6309

DanielZhangQD reviewed Aug 26, 2020

View reviewed changes

pkg/controller/dmcluster/dm_cluster_controller.go Outdated Show resolved Hide resolved

examples/dm/README.md Outdated Show resolved Hide resolved

address comments

5ab015d

DanielZhangQD approved these changes Aug 26, 2020

View reviewed changes

ti-srebot added the status/LGT1 label Aug 26, 2020

cofyc approved these changes Aug 26, 2020

View reviewed changes

ti-srebot added status/LGT2 and removed status/LGT1 labels Aug 26, 2020

ti-srebot approved these changes Aug 26, 2020

View reviewed changes

lichunzhu merged commit d6edf59 into pingcap:master Aug 26, 2020

lichunzhu deleted the supportStartDMCluster branch August 26, 2020 07:06

lichunzhu restored the supportStartDMCluster branch August 26, 2020 11:44

lichunzhu deleted the supportStartDMCluster branch August 26, 2020 11:44

lichunzhu restored the supportStartDMCluster branch August 26, 2020 11:44

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

dm-operator/: support start a new dm cluster with dm-masters and dm-workers #3146

dm-operator/: support start a new dm cluster with dm-masters and dm-workers #3146

lichunzhu commented Aug 21, 2020 •

edited

Loading

codecov-commenter commented Aug 21, 2020 •

edited

Loading

DanielZhangQD Aug 25, 2020

lichunzhu Aug 25, 2020

DanielZhangQD Aug 25, 2020

weekface Aug 25, 2020

lichunzhu Aug 25, 2020

DanielZhangQD Aug 25, 2020

lichunzhu Aug 25, 2020

DanielZhangQD commented Aug 25, 2020

DanielZhangQD left a comment

DanielZhangQD left a comment

cofyc left a comment

lichunzhu commented Aug 26, 2020

dm-operator/: support start a new dm cluster with dm-masters and dm-workers #3146

dm-operator/: support start a new dm cluster with dm-masters and dm-workers #3146

Conversation

lichunzhu commented Aug 21, 2020 • edited Loading

What problem does this PR solve?

What is changed and how does it work?

Check List

Does this PR introduce a user-facing change?:

codecov-commenter commented Aug 21, 2020 • edited Loading

Codecov Report

DanielZhangQD Aug 25, 2020

Choose a reason for hiding this comment

lichunzhu Aug 25, 2020

Choose a reason for hiding this comment

DanielZhangQD Aug 25, 2020

Choose a reason for hiding this comment

weekface Aug 25, 2020

Choose a reason for hiding this comment

lichunzhu Aug 25, 2020

Choose a reason for hiding this comment

DanielZhangQD Aug 25, 2020

Choose a reason for hiding this comment

lichunzhu Aug 25, 2020

Choose a reason for hiding this comment

DanielZhangQD commented Aug 25, 2020

DanielZhangQD left a comment

Choose a reason for hiding this comment

DanielZhangQD left a comment

Choose a reason for hiding this comment

cofyc left a comment

Choose a reason for hiding this comment

lichunzhu commented Aug 26, 2020

lichunzhu commented Aug 21, 2020 •

edited

Loading

codecov-commenter commented Aug 21, 2020 •

edited

Loading