[dagster-dbt] Upgrade dagster-dbt to be compatible with dbt 1.5.x #13907
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Currently blocked on: dbt-labs/dbt-core#7465, although should be possible to work around |
rexledesma
reviewed
Apr 26, 2023
python_modules/libraries/dagster-dbt/dagster_dbt/cloud/asset_defs.py
Outdated
Show resolved
Hide resolved
|
Ok avoided the |
|
Not sure if this is beneficial for this PR at all, but wanted to put this on your radar (coming out in v1.5): https://docs.getdbt.com/reference/programmatic-invocations I know dagster-dbt does a lot of stuff behind the scenes with the CLI, so this might help clean things up a bit |
@tha23rd Owen mentioned this internally to our team! Yeah this will be such a great QOL cleanup for our integration. |
rexledesma
approved these changes
Apr 27, 2023
17fdbf8 to
66dbb73
Compare
66dbb73 to
6549e3e
Compare
|
Current dependencies on/for this PR: This comment was auto-generated by Graphite. |
rexledesma
added a commit
that referenced
this pull request
Apr 27, 2023
…3907) ## Summary & Motivation Fixes: #13898 The changes: - `--no-use-color` is deprecated (`--no-use-colors` is allowed on all versions 1.0+) - The dbt selection string parsing logic relies on global state being set (I know, I know...), and the way that this global state is set has been changed (previously, they were attributes directly on the module, now they're all wrapped into a single GLOBAL_FLAGS object) - Nasty hack to deal with dbt-core shifting from `argparse` to `click`. We were using the argument parsing functionality to parse dbt Cloud job commands, so I found a somewhat similar entrypoint to work with. - Ensure that the current working directory when executing dbt commands is the project directory. This handles the issue linked below, where dbt now writes output files to the cwd rather than the project directory. This is not an issue if the cwd *is* the project directory ;) - Update some tests to be a bit more flexible to subtly different output formats (generally this is just testing stuff in the DbtCliOutput object, which we don't particularly expect or need to be very stable) - Some errors got reworded, so tests needed to be updated In general, the `select_unique_ids_from_manifest` function access a lot of things it potentially shouldn't touch from the dbt internals, and there's no guarantee similar issues don't crop up in the future. The hope is that there is eventually a dbt-native python entrypoint for this functionality, at which point in time we can remove all this ugly stuff. ## How I Tested These Changes Added a dbt_15X tox env, which is pinned to the specific rcs that were available to me. Once the real `dbt 1.5.x` comes out, this should be unpinned --------- Co-authored-by: Rex Ledesma <rex@elementl.com>
takeru911
pushed a commit
to takeru911/dagster
that referenced
this pull request
Apr 29, 2023
…gster-io#13907) ## Summary & Motivation Fixes: dagster-io#13898 The changes: - `--no-use-color` is deprecated (`--no-use-colors` is allowed on all versions 1.0+) - The dbt selection string parsing logic relies on global state being set (I know, I know...), and the way that this global state is set has been changed (previously, they were attributes directly on the module, now they're all wrapped into a single GLOBAL_FLAGS object) - Nasty hack to deal with dbt-core shifting from `argparse` to `click`. We were using the argument parsing functionality to parse dbt Cloud job commands, so I found a somewhat similar entrypoint to work with. - Ensure that the current working directory when executing dbt commands is the project directory. This handles the issue linked below, where dbt now writes output files to the cwd rather than the project directory. This is not an issue if the cwd *is* the project directory ;) - Update some tests to be a bit more flexible to subtly different output formats (generally this is just testing stuff in the DbtCliOutput object, which we don't particularly expect or need to be very stable) - Some errors got reworded, so tests needed to be updated In general, the `select_unique_ids_from_manifest` function access a lot of things it potentially shouldn't touch from the dbt internals, and there's no guarantee similar issues don't crop up in the future. The hope is that there is eventually a dbt-native python entrypoint for this functionality, at which point in time we can remove all this ugly stuff. ## How I Tested These Changes Added a dbt_15X tox env, which is pinned to the specific rcs that were available to me. Once the real `dbt 1.5.x` comes out, this should be unpinned --------- Co-authored-by: Rex Ledesma <rex@elementl.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary & Motivation
Fixes: #13898
The changes:
--no-use-coloris deprecated (--no-use-colorsis allowed on all versions 1.0+)argparsetoclick. We were using the argument parsing functionality to parse dbt Cloud job commands, so I found a somewhat similar entrypoint to work with.In general, the
select_unique_ids_from_manifestfunction access a lot of things it potentially shouldn't touch from the dbt internals, and there's no guarantee similar issues don't crop up in the future. The hope is that there is eventually a dbt-native python entrypoint for this functionality, at which point in time we can remove all this ugly stuff.How I Tested These Changes
Added a dbt_15X tox env, which is pinned to the specific rcs that were available to me. Once the real
dbt 1.5.xcomes out, this should be unpinned