Add incremental materializations #226
Conversation
There was a problem hiding this comment.
@stu-k Thanks for pushing up the initial PR!
As discussed during our chat yesterday, there are (I believe) two cases in which we do not create a "temp" table before merging:
mergestrategy withon_schema_change: ignoreinsert_overwritestrategy with "static"partitionsconfig
In those cases, dbt templates the model {{ sql }} directly into the merge statement. That won't work for Python, so we really do need to submit the Python job and save the results to a table before the merge takes place. We also need to make sure that table is cleaned up after the merge, either by altering it to set it to expire within X hours (our standard approach is 12), or drop it explicitly at the end of the materialization.
|
@jtcohen6 Apologies I didn't mean to add you as a reviewer, this is still in progress, I'm just using the pr to run the CI. |
|
@stu-k Oh whoops, sorry! Keep on keepin' on! |
4a070ff to
b2763d9
Compare
b2763d9 to
1f4799c
Compare
| {%- call statement('main') -%} | ||
| {{ build_sql }} | ||
| {% endcall %} | ||
| Also, why does not either drop_relation or adapter.drop_relation work here?! |
There was a problem hiding this comment.
Should we remove this comment?
Description
Implements the incremental materialization for bigquery. Nearly directly copied from Ian's work in spark