fix: drop replication slot when db deletes wal segment #154
Conversation
|
Yeah I agree with this approach for our current use-case. If we ever want to use this as an auditing tool however, we'll have to re-engineer this. Perhaps we can add this to the FAQ's in the readme. Something like:
I think as part of our re-engineering or realtime/sockets/workflows we can come up with some solutions here |
Yep, totally agree!
Yep, good idea! Let me go add to readme and i'll ping you for feedback |
README.md
Outdated
|
|
||
| 1. Postgres database runs out of disk space due to Write-Ahead Logging (WAL) buildup, which will crash the database and prevent Realtime server from streaming replication and broadcasting changes. | ||
| 2. Realtime server will crash due to a larger replication lag than available memory, forcing the creation of a new replication slot and resetting streaming replication to read from the latest WAL data. | ||
| 3. When Realtime server falls too far behind for any reason, for example disconnecting from database as WAL continues to build up, then database will delete WAL segments the server still needs to read from, for example after reconnecting. |
There was a problem hiding this comment.
Looks great @w3b6x9 - I would just update some of the "will"s to "can"s. eg:
which can crash the database
There was a problem hiding this comment.
@kiwicopple yep, good call
|
🎉 This PR is included in version 0.14.1 🎉 The release is available on GitHub release Your semantic-release bot 📦🚀 |
fix Api.decrypt_extensions_data
What kind of change does this PR introduce?
Bug fix
What is the current behavior?
Database deletes a WAL segment that Realtime server still needs to replay. Realtime is caught restarting in a loop as it cannot find the WAL segment.
What is the new behavior?
Database deletes a WAL segment that Realtime server still needs to replay. Realtime recognizes this, drops the replication slot, and on restart a new replication slot is created. All is normal again.
Additional context
Re-creating a standby (Realtime server) that falls behind is a suggested solution (see https://www.2ndquadrant.com/en/blog/postgresql-9-4-slots for more details)
There's no good way to test this live so we'll just have to see if it resolves the issue after deploying.