Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

vreplication: error out if source table is mentioned twice in Materialize #8845

Closed
derekperkins opened this issue Sep 17, 2021 · 0 comments · Fixed by #14184
Closed

vreplication: error out if source table is mentioned twice in Materialize #8845

derekperkins opened this issue Sep 17, 2021 · 0 comments · Fixed by #14184

Comments

@derekperkins
Copy link
Member

If you reference the same source table twice in the same Materialize workflow (and presumably other streams, though untested), the second reference doesn't get populated, but transitions into RUNNING without throwing an error. On Slack, @sougou confirmed that this is how it functions.

it's very likely that vreplication assumes that the source tables are distinct (per stream), yeah. the copy_state supports only one unique table per stream
https://vitess.slack.com/archives/CMKTCUYNQ/p1631687493154700

I'm guessing it's a deeper issue to fix to allow an arbitrary number of references to a source table, which would be the preferred solution. In the meantime, we should check for it and throw an error before executing so that people aren't getting incorrectly materialized data.

Example materialize statement

vtctlclient -server localhost:15999 Materialize '{
  "workflow": "billing__usage__vrepl",
  "source_keyspace": "workspaces",
  "target_keyspace": "workspaces",
  "table_settings": [
    {
      "target_table": "billing__usage__vrepl",
      "source_expression": "select workspace_id as account_id, date(from_unixtime(requested)) as requested_date, date(created_at) as created_date, 2 as type, count(*) as usage from workspaces_rankings__pulls where is_priority=1 group by account_id, requested_date, created_date, type"
    },
    {
      "target_table": "billing__usage__vrepl",
      "source_expression": "select workspace_id as account_id, date(from_unixtime(requested)) as requested_date, date(created_at) as created_date, 1 as type, count(*) as usage from workspaces_rankings__pulls where is_priority=0 group by account_id, requested_date, created_date, type"
    }
  ],
  "tablet_types": "MASTER"
}'

Binary version

11.0.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Done
Status: Done
Development

Successfully merging a pull request may close this issue.

3 participants