DuneSQL Migration: new macros, new dunesql CI, and translated + legacy evms_blocks model #3570
Conversation
dec3bc2 to
13a8682
Compare
.github/workflows/dbt_slim_ci.yml
Outdated
|
|
||
| - name: dbt test incremental model(s) if applicable | ||
| if: env.INC_MODEL_COUNT > 0 | ||
| run: "dbt test --select state:modified,config.materialized:incremental --exclude tag:prod_exclude --defer --state ." | ||
| run: "dbt test $PROFILE --select state:modified,config.materialized:incremental,$TAG --exclude $EXCLUDEtag:prod_exclude --defer --state ." |
There was a problem hiding this comment.
| run: "dbt test $PROFILE --select state:modified,config.materialized:incremental,$TAG --exclude $EXCLUDEtag:prod_exclude --defer --state ." | |
| run: "dbt test $PROFILE --select state:modified,config.materialized:incremental,$TAG --exclude ${EXCLUDE}tag:prod_exclude --defer --state ." |
|
taking a few notes here to prep for merge, mostly for myself to follow:
|
…fests The existing logic for spark manifest is unchanged, we still upload the spark manifest to the same location. We run the same logic to upload the dunesql manifest to a different s3 manifest.
This reverts commit 17dab72.
|
notes post-merge critical path
not critical path, but maybe nice-to-have:
potential clean-up:
|
|
regarding permissions bug, this is the PR from the s3 permissions I initially set up on the original manifest bucket. have we made a similar pr for the new bucket |
|
Hey @couralex6, I'm looking into adding Celo to the tables in the 
|
we are still working on an updated contribution guide that will include most of these details for yourself and others. if you're looking to get ahead, here are some answers:
we will ensure to look closely in review on new PRs, then can use findings to help improve contribution guide |
Ah alright makes sense! I made a PR to add Celo but looks like it passed tests without needing me to touch anything new, lmk if there's something new that ci tests dont account for! #3640 |
will follow up on that PR when i can 🙏 |
…y evms_blocks model (duneanalytics#3570) * create schema macro for trino * trino create table * Adds a translated spell to test trino selector * changes tag name * Adds exclude tag names from CI workflow definition * Adds missing comma * Run alter table on every run * Adds delta_prod as database for source * DBT macro to override source database * Only add delta_prod to source if we are running on trino * Updates expose spell and mark as spell macros * Jonas' comments * revert uniswap changes * Adds dbt trino to pipfile * alter view fix * adds dunesql tag to evms_blocks model * use tags instead of env var in macros to detect dunesql models * Adds legacy evms_block file which contains the original spark model * Add dbt matrix strategy for spark and dunesql * Add alias macro * Use alias macro in evms blocks * Fix select/excludes * Run sequentially * Fix exclude tag * Test fix for broken spark selector * fixing bug where compile on spark had an empty select flag * Refactor dbt compile * Updates commit manifest action to generate both dunsql and spark manifests The existing logic for spark manifest is unchanged, we still upload the spark manifest to the same location. We run the same logic to upload the dunesql manifest to a different s3 manifest. * Download manifest from correct location for dunesql runs * simplifies copy manifest logic * only run dunesql * Fix commit manifest target location * dunesql_check CI fix * Only select state:new,tag:dunesql * Move comma inside of $TAG. state:modified != state:modified, * Removes 'or True' from expose_spell and mark_as_spell macros prior to merging * Test CI run on different s3 location * Revert "Test CI run on different s3 location" This reverts commit 17dab72. * prod s3 bucket * trailing / * Prod location again --------- Co-authored-by: André Monteiro <andre@dune.com> Co-authored-by: Jonas Irgens Kylling <jonas@dune.xyz>
Trino macro prior to trino migration