Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CDC changefeed stucks after killing PD owner #8877

Closed
fubinzh opened this issue Apr 28, 2023 · 2 comments · Fixed by #8884
Closed

CDC changefeed stucks after killing PD owner #8877

fubinzh opened this issue Apr 28, 2023 · 2 comments · Fixed by #8884
Labels
affects-6.1 This bug affects the 6.1.x(LTS) versions. affects-6.5 This bug affects the 6.5.x(LTS) versions. affects-7.1 This bug affects the 7.1.x(LTS) versions. area/ticdc Issues or PRs related to TiCDC. severity/critical type/bug The issue is confirmed as a bug.

Comments

@fubinzh
Copy link

fubinzh commented Apr 28, 2023

What did you do?

  1. ti-operator cluster with 36 tikv, 3 cdc, 3 pd, create changefeed to sync to kafka
  2. running workload, throughtput ~50MB/s
  3. kill pd owner

What did you expect to see?

cdc lag < 1min

What did you see instead?

cdc changefeed stucks

image

image

Versions of the cluster

[root@upstream-ticdc-0 log]# /cdc version
Release Version: v7.1.0-rc.0
Git Commit Hash: 4d633bf
Git Branch: heads/refs/tags/v7.1.0-rc.0
UTC Build Time: 2023-04-21 10:14:01
Go Version: go version go1.20.3 linux/amd64
Failpoint Build: false
@fubinzh fubinzh added area/ticdc Issues or PRs related to TiCDC. type/bug The issue is confirmed as a bug. labels Apr 28, 2023
@fubinzh
Copy link
Author

fubinzh commented Apr 28, 2023

/severity critical

@asddongmen asddongmen mentioned this issue May 5, 2023
Merged
@asddongmen asddongmen added affects-6.1 This bug affects the 6.1.x(LTS) versions. affects-6.5 This bug affects the 6.5.x(LTS) versions. affects-7.1 This bug affects the 7.1.x(LTS) versions. and removed may-affects-6.1 may-affects-6.5 may-affects-7.1 labels May 6, 2023
@ti-chi-bot ti-chi-bot bot closed this as completed in #8884 May 6, 2023
ti-chi-bot bot pushed a commit that referenced this issue May 6, 2023
@asddongmen
capture (ticdc): fix processor exit unexpectedly when some pd node fa…
648fe8a 
…il (#8884)

ref #8868, close #8877
This was referenced May 6, 2023
Merged
Merged
@ti-chi-bot ti-chi-bot mentioned this issue May 6, 2023
Merged
@nongfushanquan
Copy link
Contributor

/area ticdc

@ti-chi-bot ti-chi-bot bot added the area/ticdc Issues or PRs related to TiCDC. label May 8, 2023
ti-chi-bot bot pushed a commit that referenced this issue May 8, 2023
@ti-chi-bot
capture (ticdc): fix processor exit unexpectedly when some pd node fa…
09c43f4 
…il (#8884) (#8900)

ref #8868, close #8877
ti-chi-bot bot pushed a commit that referenced this issue May 15, 2023
@ti-chi-bot
capture (ticdc): fix processor exit unexpectedly when some pd node fa…
0287295 
…il (#8884) (#8898)

ref #8868, close #8877
@nongfushanquan nongfushanquan mentioned this issue May 25, 2023
Merged
16 tasks
@nongfushanquan nongfushanquan mentioned this issue Jun 9, 2023
Merged
17 tasks
ti-chi-bot bot pushed a commit that referenced this issue Jun 13, 2023
@ti-chi-bot
pd, changefeed (ticdc): fix pd related issues (#8884, #8813, #9106, #…
1ea99d5 
…9174) (#8901)

ref #8868, close #8877
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
affects-6.1 This bug affects the 6.1.x(LTS) versions. affects-6.5 This bug affects the 6.5.x(LTS) versions. affects-7.1 This bug affects the 7.1.x(LTS) versions. area/ticdc Issues or PRs related to TiCDC. severity/critical type/bug The issue is confirmed as a bug.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

capture (ticdc): fix processor exit unexpectedly when some pd node fail asddongmen/tiflow
3 participants
@fubinzh @asddongmen @nongfushanquan