Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

typing replication stream starts from stream_id 0 on reconnect #7907

Closed
richvdh opened this issue Jul 20, 2020 · 2 comments · Fixed by #7959
Closed

typing replication stream starts from stream_id 0 on reconnect #7907

richvdh opened this issue Jul 20, 2020 · 2 comments · Fixed by #7959

Comments

@richvdh
Copy link
Member

richvdh commented Jul 20, 2020

After the replication stream dropped, the replication handler streamed all 5 million typing updates ever, which took 10 minutes:

2020-07-20 09:27:10,630 - synapse.replication.tcp.handler - 593 - INFO -  - Lost replication connection; streams now disconnected: {'backfill', 'caches', 'typing', 'events'}
2020-07-20 09:27:13,624 - synapse.replication.tcp.handler - 471 - INFO - replication-POSITION-10159 - Handling 'POSITION typing typing-1 5018044'
2020-07-20 09:27:13,625 - synapse.replication.tcp.handler - 505 - INFO - replication-POSITION-10159 - Fetching replication rows for 'typing' between 0 and 5018044
@richvdh
Copy link
Member Author

richvdh commented Jul 20, 2020

I suspect this is a recent bug in the work done to split typing out of master. would be good to fix it before release.

@clokep
Copy link
Member

clokep commented Jul 23, 2020

This was likely broken by #7869.

erikjohnston added a commit that referenced this issue Jul 27, 2020
@erikjohnston
Fix typing replication not being handled on master
38a3e02 
Handling of incoming typing stream updates from replication was not
hooked up on master, effecting set ups where typing was handled on a
different worker.

This is really only a problem if the master process is also handling
sync requests, which is unlikely for those that are at the stage of
moving typing off.

The other observable effect is that if a worker restarts or a
replication connect drops then the typing worker will issue a
`POSITION typing`, triggering master process to try and stream *all*
typing updates from position 0.

Fixes #7907
@erikjohnston erikjohnston mentioned this issue Jul 27, 2020
Merged
erikjohnston added a commit that referenced this issue Jul 27, 2020
@erikjohnston
Fix typing replication not being handled on master (#7959)
84d099a 
Handling of incoming typing stream updates from replication was not
hooked up on master, effecting set ups where typing was handled on a
different worker.

This is really only a problem if the master process is also handling
sync requests, which is unlikely for those that are at the stage of
moving typing off.

The other observable effect is that if a worker restarts or a
replication connect drops then the typing worker will issue a
`POSITION typing`, triggering master process to try and stream *all*
typing updates from position 0.

Fixes #7907
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix typing replication not being handled on master matrix-org/synapse
2 participants
@clokep @richvdh