feat(cdc): Prepare the self hosted environment for the Change Data Capture pipeline #938

fpacifici · 2021-04-29T05:45:16Z

This is an RFC on how to set up Change Data Capture.
We will use Change Data Capture to stream WAL updates from postgres into clickhouse so that features like issue search will be able to join event data and metadata (from postgres) through Snuba.

This requires the followings:

A logical replicaiton plugin to be installed in postgres (https://github.com/getsentry/wal2json)
A service to run that streams from the replication log to Kafka (https://github.com/getsentry/cdc)
Datasets in Snuba.

This PR is preparing postgres to stream updates via the replication log.
The idea is to

download the the replication log plugin binary during install.sh
mount a volume with the binary when starting postgres
providing a new entrypoint to postgres that ensures everything is correctly configured.

There is a difference between how this is set up and how we do the same in the development environment.
In the development environment we download the library from the entrypoint itself and store it in a persistent volume, so we do not have to download it every time.
Unfortunately this does not work here as the postgres image is postgres:9.6 while it is postgres:9.6-alpine. This one does not come with either wget or curl. I don't think installing that in the entrypoint would be a good idea, so the download happens in install.sh. I actually think this way is safer so we never depend on connectivity for postgres to start properly.

install/install-wal2json.sh

BYK

Looks good! Left a few comments to verify certain things. Otherwise I think we can merge this.

docker-compose.yml

BYK · 2021-04-30T18:53:51Z

docker-compose.yml

+      - type: bind
+        read_only: true
+        source: ./postgres/init_hba.sh
+        target: /docker-entrypoint-initdb.d/init_hba.sh


With this script in place, do we still need line 83 above?

POSTGRES_HOST_AUTH_METHOD: "trust"

Also, since we now control the entrypoint, does it have to live under this specific directory? Because if not, then why not just mount ./postgres under /opt/sentry and use everything from there? This would reduce the number of bind mounts to 1 and would avoid file-mounts which we had issues with sometimes.

We still need POSTGRES_HOST_AUTH_METHOD: "trust". That is not about replication while the changes to pg_hba are only about the replication connections.

POSTGRES_HOST_AUTH_METHOD changes the standard connections we use to run queries.

BYK · 2021-04-30T18:59:06Z

install/install-wal2json.sh

+
+if [[ $WAL2JSON_VERSION == "latest" ]]; then
+    VERSION=$(
+        wget "https://api.github.com/repos/getsentry/wal2json/releases/latest" -O - |


We haven't relied on the availability of wget so far (with the exception of our experimental --minimize-downtime flag). We haven't received any complaints about that but just wanted to double check on whether this is a safe assumption to make or not.

What bout curl ?

The same. The only assumptions we make are bash, docker, docker-compose, sed, awk, and grep. I think we can find/use a curl or wget docker image to be safe :)

Turns out I ~~don't~~ didn't have wget. 😊

▶ Downloading and installing wal2json ... install-wal2json.sh: line 13: wget: command not found An error occurred, caught SIGERR on line 20 Cleaning up... $

Looks like macOS does ship with curl. If we can't find a Docker image easily I think curl would be a safe choice. Do you know of any Linux that doesn't have it?

Now relying on curlimages/curl

install/install-wal2json.sh

postgres/postgres-entrypoint.sh

BYK · 2021-05-03T09:02:00Z

install/install-wal2json.sh

+
+if [[ $WAL2JSON_VERSION == "latest" ]]; then
+    VERSION=$(
+        wget "https://api.github.com/repos/getsentry/wal2json/releases/latest" -O - |


The same. The only assumptions we make are bash, docker, docker-compose, sed, awk, and grep. I think we can find/use a curl or wget docker image to be safe :)

install.sh

install/install-wal2json.sh

postgres/postgres-entrypoint.sh

scripts/bump-version.sh

chadwhitacre · 2021-05-03T15:11:24Z

install/install-wal2json.sh

+
+if [[ $WAL2JSON_VERSION == "latest" ]]; then
+    VERSION=$(
+        wget "https://api.github.com/repos/getsentry/wal2json/releases/latest" -O - |


Turns out I ~~don't~~ didn't have wget. 😊

▶ Downloading and installing wal2json ... install-wal2json.sh: line 13: wget: command not found An error occurred, caught SIGERR on line 20 Cleaning up... $

chadwhitacre · 2021-05-03T15:14:18Z

install/install-wal2json.sh

+
+if [[ $WAL2JSON_VERSION == "latest" ]]; then
+    VERSION=$(
+        wget "https://api.github.com/repos/getsentry/wal2json/releases/latest" -O - |


Looks like macOS does ship with curl. If we can't find a Docker image easily I think curl would be a safe choice. Do you know of any Linux that doesn't have it?

install/install-wal2json.sh

fpacifici · 2021-05-06T00:43:10Z

Ok, we have a problem.
I just noticed that that wal2json.so depends on libc, but postgres alpine (on the dev environment and where we build the binary) has libc.musl instead.

On postgres-alpine

bash-5.1# ldd /wal2json/latest.so 
        /lib/ld-musl-x86_64.so.1 (0x7effc8ba9000)
        libc.musl-x86_64.so.1 => /lib/ld-musl-x86_64.so.1 (0x7effc8ba9000)

On postgres

root@1bca1c71f616:/usr/lib/postgresql/9.6/lib# ldd wal2json.so 
        linux-vdso.so.1 (0x00007ffd30348000)
        libc.musl-x86_64.so.1 => not found

So questions @BYK :

What is the reason we use postgres-alpine in dev and postgres in self hosted ?
What if, on self hosted, we cloned the sentry wal2json repo and built the library from source during the install process instead of using the binary?

I doubt linking libc statically would be a good idea, I don't even think it is possible as the build process is orchestrated by postgres.

BYK · 2021-05-06T07:18:55Z

What is the reason we use postgres-alpine in dev and postgres in self hosted ?

Probably someone chose "the lighter" version on dev and we stuck with it. For self-hosted, we tend to use the most common distributions, which is not alpine :)

What if, on self hosted, we cloned the sentry wal2json repo and built the library from source during the install process instead of using the binary?

That sounds like a step backwards and would introduce a lot of complexity that should live inside the wal2json repo. I've noticed that we use the postgres-alpine base for the builder Docker image. I'm pretty sure if we switch that up to postgres we wouldn't have issues.

This means now wal2json produces two binaries based on build args again:

For libc.musl w/ Alpine
For regular libc

Thoughts?

fpacifici · 2021-05-06T15:11:49Z

Yes using postgres standard is supposed to work. I did not test it, but there should be no reason for that.

Sure I can deliver two binaries in the release, I was not convinced distributing multiple binaries per required implementation of libc would be a common practice. But that may be just because I never noticed it.

fpacifici · 2021-05-06T22:34:09Z

Building two binaries is possible, and it works correctly when importing the right one here.
getsentry/wal2json#12

fpacifici added 4 commits April 28, 2021 18:33

Set up pg_hba

7b5074d

Download wal2json

66844f8

Fix symlink

57b2d1a

Pin version when releasing

47e95d6

fpacifici requested a review from BYK April 29, 2021 05:45

fpacifici commented Apr 29, 2021

View reviewed changes

install/install-wal2json.sh Outdated Show resolved Hide resolved

fpacifici added 2 commits April 28, 2021 22:51

Change variable name

dbc94c6

Fix script

d82b216

fpacifici requested a review from a team April 30, 2021 16:09

BYK reviewed Apr 30, 2021

View reviewed changes

Address review comments

8844e8d

fpacifici requested a review from BYK April 30, 2021 22:52

BYK approved these changes May 3, 2021

View reviewed changes

chadwhitacre reviewed May 3, 2021

View reviewed changes

fpacifici added 6 commits May 4, 2021 18:11

Do not rely on having wget

9d2f8a7

Fix path

a31b8e1

Ensure the user can write

66d0b0c

Ensure the user can write

47906d9

Ensure the user can write

2f62595

Try by redirecting output

b2a48a9

fpacifici commented May 5, 2021

View reviewed changes

install/install-wal2json.sh Show resolved Hide resolved

nit

b10d57c

BYK approved these changes May 5, 2021

View reviewed changes

BYK changed the title ~~feat(cdc) Prepare the self hosted environment for the Change Data Capture pipeline~~ feat(cdc): Prepare the self hosted environment for the Change Data Capture pipeline May 5, 2021

fpacifici added 2 commits May 13, 2021 17:09

Change lib name

9e3a60a

Merge branch 'master' into feat/cdc_onprem

a5488d4

fpacifici merged commit 8dc8460 into master May 25, 2021

fpacifici deleted the feat/cdc_onprem branch May 25, 2021 00:51

github-actions bot locked and limited conversation to collaborators Jun 9, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(cdc): Prepare the self hosted environment for the Change Data Capture pipeline #938

feat(cdc): Prepare the self hosted environment for the Change Data Capture pipeline #938

fpacifici commented Apr 29, 2021

BYK left a comment

BYK Apr 30, 2021

fpacifici Apr 30, 2021

BYK Apr 30, 2021

fpacifici Apr 30, 2021

BYK May 3, 2021

chadwhitacre May 3, 2021

chadwhitacre May 3, 2021

fpacifici May 5, 2021

BYK May 3, 2021

chadwhitacre May 3, 2021

chadwhitacre May 3, 2021

fpacifici commented May 6, 2021 •

edited

Loading

BYK commented May 6, 2021

fpacifici commented May 6, 2021

fpacifici commented May 6, 2021 •

edited

Loading

feat(cdc): Prepare the self hosted environment for the Change Data Capture pipeline #938

feat(cdc): Prepare the self hosted environment for the Change Data Capture pipeline #938

Conversation

fpacifici commented Apr 29, 2021

BYK left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fpacifici commented May 6, 2021 • edited Loading

BYK commented May 6, 2021

fpacifici commented May 6, 2021

fpacifici commented May 6, 2021 • edited Loading

fpacifici commented May 6, 2021 •

edited

Loading

fpacifici commented May 6, 2021 •

edited

Loading