-
Notifications
You must be signed in to change notification settings - Fork 1.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[CT-1556] Smarter handling of --vars
in partial parsing
#6323
Comments
--vars
in partial parsing--vars
in partial parsing
This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please comment on the issue or else it will be closed in 7 days. |
Although we are closing this issue as stale, it's not gone forever. Issues can be reopened if there is renewed community interest. Just add a comment to notify the maintainers. |
If possible, can we separate this handling from parsing? What I am thinking is that parsing is going from everything in project file -> a representation, then in actual runtime, we apply configuration to the representation and do the execution. |
@ChenyuLInx Supportive of this line of thinking! The biggest caveats here is that During parsing, we could store pointers to those variables, and then conditionally reevaluate them just before each execution. That feels similar to the approach described in this issue (partial parsing), though with some subtle differences in implementation. |
Yeah, there's a difference between vars that are needed at parse time and vars can be resolved at compilation/execution time. Maybe we need some use cases to help think through the different situations. Vars in configs have to be resolved at parse time. Vars in plain sql could be delayed. I'm not sure how we could distinguish between them. |
@gshank Do you know if the partial parsing manifest ( I'm thinking:
|
This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please comment on the issue or else it will be closed in 7 days. |
We use the cli --vars to pass in airflow datetime variables. These change on each run, so we can't partial parse. |
Just like #3885, but for CLI
--vars
.This would require us to capture, at parse time, which files depend on which
--vars
, via calls to the Jinja{{ var() }}
function. That would also include macros that callvar()
, and are then called by models / other macros in turn.For Python models, if we introduce a built-in
dbt.var()
function, we'd want to do the same. We're already doing something similar for configs, to powerconfig.get()
at runtime.Whenever the
--vars
change, instead of triggering a full re-parse, we'd schedule just the files that depend on thevar
for re-parsing. Of course, if thevar
is used for a configuration withindbt_project.yml
, that could still affect many many nodes.The text was updated successfully, but these errors were encountered: