Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Env vars + partial parsing #3885

Closed
jtcohen6 opened this issue Sep 15, 2021 · 1 comment · Fixed by #4101, #4162, #4212 or #4236
Closed

Env vars + partial parsing #3885

jtcohen6 opened this issue Sep 15, 2021 · 1 comment · Fixed by #4101, #4162, #4212 or #4236
Assignees
Labels
1.0.0 Issues related to the 1.0.0 release of dbt partial_parsing

Comments

@jtcohen6
Copy link
Contributor

jtcohen6 commented Sep 15, 2021

If an env var value needed at parse time (input to ref/source/config or yaml property) has changed, and partial parsing is enabled, dbt will not re-parse affected nodes and is likely to yield incorrect results.

Reproduction case: gist

  • Non-solution (better than nothing): Continue to document this in known partial parsing limitations; add to env var docs as well. Instruct users to manually trigger a full re-parse (--no-partial-parse) when they change an env var value needed at parse time.
  • Simpler solution: Store a dictionary of all available env vars in the internal manifest (partial_parse.msgpack). If any env var values change, trigger a full re-parse, and say that a change in env vars is the reason. Note: I'm not sure how this would work in deployment environments that set contextual env vars, such as dbt Cloud scheduled runs.
  • Complex, good solution: For each node in the graph, statically analyze calls to the env_var macro, and then store those env vars in depends_on for that node a separate map object, similar to how we handle docs blocks. Then, if we detect a change in env var values, we can trigger re-parse for only those nodes which depend on that env var, plus (if the node is a macro) any nodes that depend on the macro. (It would be great to do this for --vars someday as well.)
@jtcohen6 jtcohen6 added the 1.0.0 Issues related to the 1.0.0 release of dbt label Oct 20, 2021
gshank added a commit that referenced this issue Oct 26, 2021
* [#3885] Handle env_vars in partial parsing

* Comment method to build env_vars_to_source_files
@jtcohen6 jtcohen6 reopened this Oct 26, 2021
@jtcohen6
Copy link
Contributor Author

jtcohen6 commented Oct 26, 2021

@gshank Up to you whether we keep this issue open, or close it and open a new one for env vars in other files (.yml files, dbt_project.yml, profiles.yml)

gshank added a commit that referenced this issue Nov 8, 2021
…ange (#4162)

* [#3885] Partially parse when environment variables in schema files
change

* Add documentation for test kwargs

* Add test and fix for schema configs with env_var
gshank added a commit that referenced this issue Nov 8, 2021
* [#3885] Skip partial parsing if project env vars change

* Support env_vars in the profile
iknox-fa pushed a commit that referenced this issue Feb 8, 2022
* [#3885] Handle env_vars in partial parsing

* Comment method to build env_vars_to_source_files

automatic commit by git-black, original commits:
  73af9a5
iknox-fa pushed a commit that referenced this issue Feb 8, 2022
…ange (#4162)

* [#3885] Partially parse when environment variables in schema files
change

* Add documentation for test kwargs

* Add test and fix for schema configs with env_var

automatic commit by git-black, original commits:
  bda70c9
iknox-fa pushed a commit that referenced this issue Feb 8, 2022
* [#3885] Skip partial parsing if project env vars change

* Support env_vars in the profile

automatic commit by git-black, original commits:
  3e9da06
iknox-fa pushed a commit that referenced this issue Feb 8, 2022
* [#3885] Handle env_vars in partial parsing

* Comment method to build env_vars_to_source_files

automatic commit by git-black, original commits:
  73af9a5
  dc65118
iknox-fa pushed a commit that referenced this issue Feb 8, 2022
* [#3885] Handle env_vars in partial parsing

* Comment method to build env_vars_to_source_files

automatic commit by git-black, original commits:
  307d47e
  73af9a5
  e2cd45b
iknox-fa pushed a commit that referenced this issue Feb 8, 2022
* [#3885] Handle env_vars in partial parsing

* Comment method to build env_vars_to_source_files

automatic commit by git-black, original commits:
  22db36c
  307d47e
  73af9a5
  e3fb923
iknox-fa pushed a commit that referenced this issue Feb 8, 2022
…ange (#4162)

* [#3885] Partially parse when environment variables in schema files
change

* Add documentation for test kwargs

* Add test and fix for schema configs with env_var

automatic commit by git-black, original commits:
  070e13d
  6c6649f
  bda70c9
iknox-fa pushed a commit that referenced this issue Feb 8, 2022
* [#3885] Handle env_vars in partial parsing

* Comment method to build env_vars_to_source_files

automatic commit by git-black, original commits:
  32c5598
  73af9a5
iknox-fa pushed a commit that referenced this issue Feb 8, 2022
* [#3885] Handle env_vars in partial parsing

* Comment method to build env_vars_to_source_files

automatic commit by git-black, original commits:
  73af9a5
  a5ce658
iknox-fa pushed a commit that referenced this issue Feb 8, 2022
…ange (#4162)

* [#3885] Partially parse when environment variables in schema files
change

* Add documentation for test kwargs

* Add test and fix for schema configs with env_var

automatic commit by git-black, original commits:
  16e055a
  bda70c9
iknox-fa pushed a commit that referenced this issue Feb 8, 2022
…ange (#4162)

* [#3885] Partially parse when environment variables in schema files
change

* Add documentation for test kwargs

* Add test and fix for schema configs with env_var

automatic commit by git-black, original commits:
  bda70c9
  dc65118
iknox-fa pushed a commit that referenced this issue Feb 8, 2022
…ange (#4162)

* [#3885] Partially parse when environment variables in schema files
change

* Add documentation for test kwargs

* Add test and fix for schema configs with env_var

automatic commit by git-black, original commits:
  bda70c9
  f344166
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment