Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CT-2419] Ensure process_refs is called for nodes referring to versioned models #7369

Closed
gshank opened this issue Apr 17, 2023 · 2 comments · Fixed by #7439
Closed

[CT-2419] Ensure process_refs is called for nodes referring to versioned models #7369

gshank opened this issue Apr 17, 2023 · 2 comments · Fixed by #7439
Assignees
Labels
bug Something isn't working model_versions
Milestone

Comments

@gshank
Copy link
Contributor

gshank commented Apr 17, 2023

If a node refs a versioned model but does not specify a version, the ref will use the latest version. But if a new latest version is added to the ref'd model and nothing changes in the model with the ref, the depends_on unique_id will not be updated.

Create a test case for this situation and update the code in core/dbt/parser/manifest.py "process_refs". We could check for refs to versioned models or just process all refs (not just models with an updated "created_at" timestamp. Processing refs is pretty fast, though it might be useful to compare the time it takes to do it both ways.

@github-actions github-actions bot changed the title Ensure process_refs is called for nodes referring to versioned models [CT-2419] Ensure process_refs is called for nodes referring to versioned models Apr 17, 2023
@jtcohen6 jtcohen6 added this to the v1.5.x milestone Apr 18, 2023
@MichelleArk MichelleArk self-assigned this Apr 19, 2023
@gshank
Copy link
Contributor Author

gshank commented Apr 21, 2023

I think this may not actually be a real problem, since referencing nodes are scheduled for re-parsing.

@MichelleArk
Copy link
Contributor

MichelleArk commented Apr 22, 2023

reproduction case:

---models/schema.yml
models:
  - name: my_model
    latest_version: 0
    versions:
      - v: 1
      - v: 0
---models/depends_on_version.sql
SELECT * FROM {{ ref('my_model') }}
  1. initial run, without partial parsing - depends on v0 (latest) as expected
dbt --no-partial-parse --quiet ls --select depends_on_version --output json --output-keys unique_id depends_on
{"unique_id": "model.jaffle_shop.depends_on_version", "depends_on": {"macros": [], "nodes": ["model.jaffle_shop.my_model.v0"]}}
  1. updating latest_version to 1 in schema.yml + subsequent runs with partial parsing - does not depend on v1 (latest) as expected
dbt --quiet ls --select depends_on_version --output json --output-keys unique_id depends_on
{"unique_id": "model.jaffle_shop.depends_on_version", "depends_on": {"macros": [], "nodes": ["model.jaffle_shop.my_model.v0"]}}
  1. proceeding to run depends_on_version leads to:
dbt run --select depends_on_version
23:36:59  Running with dbt=1.6.0-a1
23:37:00  Found 3 models, 0 tests, 0 snapshots, 0 analyses, 552 macros, 0 operations, 0 seed files, 0 sources, 0 exposures, 0 metrics, 0 groups
23:37:00  
23:37:02  Concurrency: 1 threads (target='dev')
23:37:02  
23:37:02  1 of 1 START sql view model dbt_marky.depends_on_version ....................... [RUN]
23:37:02  1 of 1 ERROR creating sql view model dbt_marky.depends_on_version .............. [ERROR in 0.02s]
23:37:02  
23:37:02  Finished running 1 view model in 0 hours 0 minutes and 1.85 seconds (1.85s).
23:37:02  
23:37:02  Completed with 1 error and 0 warnings:
23:37:02  
23:37:02  Compilation Error in model depends_on_version (models2/depends_on_version.sql)
23:37:02    dbt was unable to infer all dependencies for the model "depends_on_version".
23:37:02    This typically happens when ref() is placed within a conditional block.
23:37:02    
23:37:02    To fix this, add the following hint to the top of the model "depends_on_version":
23:37:02    
23:37:02    -- depends_on: {{ ref('my_model') }}
23:37:02  
23:37:02  Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working model_versions
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants