-
Notifications
You must be signed in to change notification settings - Fork 240
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Check existing relation type when renaming to __dbt_backup #16
Comments
@jtcohen6 this is good to know! We dynamically check the relation type in the spark__drop_relation implementation -- sounds like we should do something similar in the rename too |
@drewbanin Got it, makes sense. A related thought, though possibly specific to Databricks:
Renaming a table can be a very slow/expensive operation, on par with building the intermediate table. Edit: Realizing this is a much longer conversation |
@jtcohen6 I also penned a response before also realizing that there's a lot to say here......... Lmk the next time you're free for an espresso :) |
There was a short-term resolution for this in #17: do not alter + rename tables because it is an expensive process. There will be a fix for the larger issue—some macros need to know current relation types, and we must adjudicate which—in #50: get all existing relation types at the start of the run via smarter metadata query. I'm going to close this issue given both developments. |
commit 8e1a8f4 Merge: 28909f2 908d9fe Author: dparent1 <65294587+dparent1@users.noreply.github.com> Date: Thu Feb 2 13:22:23 2023 -0500 Merge pull request dbt-labs#16 from Fokko/fd-revert-some-changes Revert some stuff commit 908d9fe Merge: b286fd0 f877d1e Author: Fokko Driesprong <fokko@tabular.io> Date: Thu Feb 2 10:00:18 2023 -0800 Merge branch 'main' of https://github.com/dbt-labs/dbt-spark into fd-revert-some-changes commit b286fd0 Author: Fokko Driesprong <fokko@tabular.io> Date: Wed Feb 1 12:00:21 2023 -0800 Revert some stuff I noticed that two Spark tests are failing, so tried to revert some of the changes that seemed related. They are now passing and also my Iceberg dbt project is still running fine commit 28909f2 Merge: 3ba0a72 3c03d9a Author: Dan Parent <65294587+dparent1@users.noreply.github.com> Date: Mon Jan 30 15:05:00 2023 -0500 Merge branch 'main' into dparent1/iceberg - in impl.py changed: List[agate.Row] to AttrDict This fixed the pre-commit error after merging in changes from main. commit 3c03d9a Merge: 10c9dac 4d179e0 Author: dparent1 <65294587+dparent1@users.noreply.github.com> Date: Mon Jan 30 14:31:18 2023 -0500 Merge branch 'dbt-labs:main' into main commit 3ba0a72 Merge: 83f2d61 10c9dac Author: Dan Parent <65294587+dparent1@users.noreply.github.com> Date: Mon Jan 23 16:39:22 2023 -0500 Merge branch 'main' into dparent1/iceberg - Fixing up merge conflicts around Exception/Error types - Putting back accidental newline removal in tox.ini commit 10c9dac Merge: 3a7ca7c 39800e0 Author: dparent1 <65294587+dparent1@users.noreply.github.com> Date: Mon Jan 23 15:25:29 2023 -0500 Merge branch 'dbt-labs:main' into main commit 83f2d61 Merge: beac4b4 3a7ca7c Author: Dan Parent <65294587+dparent1@users.noreply.github.com> Date: Mon Jan 9 11:12:28 2023 -0500 Merge branch 'main' into dparent1/iceberg commit 3a7ca7c Merge: cad11e2 d9a3d76 Author: dparent1 <65294587+dparent1@users.noreply.github.com> Date: Mon Jan 9 11:02:55 2023 -0500 Merge branch 'dbt-labs:main' into main commit beac4b4 Merge: 29610a5 6e6daf5 Author: dparent1 <65294587+dparent1@users.noreply.github.com> Date: Tue Dec 20 16:24:54 2022 -0500 Merge pull request dbt-labs#12 from Fokko/fd-comments Cleanup based on comments commit 6e6daf5 Author: Fokko Driesprong <fokko@tabular.io> Date: Tue Dec 20 21:49:01 2022 +0100 Cleanup based on comments commit 29610a5 Merge: 34c5549 cad11e2 Author: Dan Parent <65294587+dparent1@users.noreply.github.com> Date: Tue Dec 20 13:26:20 2022 -0500 Merge branch 'main' into dparent1/iceberg Fixed conflicts: dbt/adapters/spark/impl.py dbt/adapters/spark/relation.py dbt/include/spark/macros/materializations/incremental/incremental.sql dbt/include/spark/macros/materializations/incremental/strategies.sql Updated tox.ini removing my own addition of allowlist_externals, main has it set now. commit cad11e2 Merge: 37307d1 512b3d0 Author: dparent1 <65294587+dparent1@users.noreply.github.com> Date: Tue Dec 20 12:50:27 2022 -0500 Merge branch 'dbt-labs:main' into main commit 34c5549 Author: Dan Parent <65294587+dparent1@users.noreply.github.com> Date: Thu Dec 8 10:47:25 2022 -0500 Allowing the use of /bin/bash by tox tox on certain platforms will complain that /bin/bash is not allowed to be used. I'm allowing it to be used with this change. commit 3ffcf5b Author: Dan Parent <65294587+dparent1@users.noreply.github.com> Date: Thu Dec 8 10:36:57 2022 -0500 Removed use of ParsedSourceDefinition, add iceberg - upstream dbt changed breaking the use of ParsedSourceDefinition, using SourceDefinition appears to work instead - Added in change to include iceberg in adapters.sql: macro spark__alter_column_comment commit edb6f01 Merge: 65eac08 37307d1 Author: Dan Parent <65294587+dparent1@users.noreply.github.com> Date: Thu Dec 8 10:02:36 2022 -0500 Merge changes in main to bring branch up to date commit 37307d1 Merge: d482d66 9511847 Author: dparent1 <65294587+dparent1@users.noreply.github.com> Date: Thu Dec 8 09:22:16 2022 -0500 Merge branch 'dbt-labs:main' into main commit 65eac08 Author: Dan Parent <65294587+dparent1@users.noreply.github.com> Date: Thu Dec 8 09:05:48 2022 -0500 Backing out previous merge which broke unit tests commit 513293b Merge: 294b7fb b936b7a Author: dparent1 <65294587+dparent1@users.noreply.github.com> Date: Thu Dec 8 08:55:36 2022 -0500 Merge pull request dbt-labs#10 from Fokko/fd-enable-docs Add Iceberg to the list commit b936b7a Author: Fokko Driesprong <fokko@tabular.io> Date: Wed Dec 7 19:04:24 2022 +0100 Add Iceberg to the list commit 294b7fb Merge: 8d3984f d282b21 Author: dparent1 <65294587+dparent1@users.noreply.github.com> Date: Mon Nov 28 09:55:51 2022 -0500 Merge pull request dbt-labs#9 from Fokko/fd-fix-incremental-runs Fix incremental runs commit d282b21 Author: Fokko Driesprong <fokko@tabular.io> Date: Fri Nov 25 15:27:14 2022 +0100 Fix incremental runs commit d482d66 Author: dparent1 <65294587+dparent1@users.noreply.github.com> Date: Mon Nov 21 16:05:10 2022 -0500 Set up CI with Azure Pipelines [skip ci] commit 8d3984f Merge: 93d6b97 b759267 Author: Dan Parent <65294587+dparent1@users.noreply.github.com> Date: Mon Nov 7 14:35:22 2022 -0500 Merge branch 'main' into dparent1/iceberg commit 93d6b97 Merge: 402be01 7f233b1 Author: Dan Parent <65294587+dparent1@users.noreply.github.com> Date: Thu Sep 29 09:58:24 2022 -0400 Merge branch 'main' into dparent1/iceberg commit 402be01 Merge: 50f7b94 80dc029 Author: Dan Parent <65294587+dparent1@users.noreply.github.com> Date: Wed Sep 28 09:02:05 2022 -0400 Merge branch 'main' into dparent1/iceberg commit 50f7b94 Author: Dan Parent <65294587+dparent1@users.noreply.github.com> Date: Tue Sep 27 14:18:53 2022 -0400 Removing the is_iceberg check it is not needed Upon further investigation this check is not needed since self.database will not be set. commit bbf79e4 Merge: 6764e55 ebd011e Author: Dan Parent <65294587+dparent1@users.noreply.github.com> Date: Mon Sep 12 11:21:35 2022 -0400 Merge branch 'main' into dparent1/iceberg commit 6764e55 Author: Dan Parent <65294587+dparent1@users.noreply.github.com> Date: Fri Aug 19 14:13:58 2022 -0400 Adding changelog entry. commit ee8c7b0 Author: Dan Parent <65294587+dparent1@users.noreply.github.com> Date: Fri Aug 19 13:45:24 2022 -0400 Renaming macro and method name On second look I wasn't happy with my name choices for macro name and method, hopefully what I have now makes more sense. [CT-276] Apache Iceberg Support dbt-labs#294 commit 6f96705 Author: Dan Parent <65294587+dparent1@users.noreply.github.com> Date: Fri Aug 19 10:09:02 2022 -0400 Setting _schema rather than replacing it [CT-276] Apache Iceberg Support dbt-labs#294 The _schema variable was used for non-iceberg tables but was being overridden by work for iceberg v2 tables. I've made it so the iceberg condition will set _schema rather than blanket changing the schema for all providers. commit 35bc12e Author: Dan Parent <65294587+dparent1@users.noreply.github.com> Date: Tue Aug 16 12:51:53 2022 -0400 Adding in additional support for iceberg v2 tables Found a way to identify iceberg tables given that spark returns an error when trying to execute "SHOW TABLE EXTENDED..." See https://issues.apache.org/jira/browse/SPARK-33393 Instead of show table extended a "DESCRIBE EXTENDED" is performed to retrieve the provider information. This allows for identification of iceberg through an is_iceberg member variable. Allow for multiple join conditions to allow for mutliple columns to make a row distinct Use is_iceberg everywhere handling iceberg tables differs from other sources of data.
If a dbt model already exists as a view and we are replacing it with a table, dbt will run
which returns the following error:
I think this is because Redshift/Postgres allow using
alter table
on views.The text was updated successfully, but these errors were encountered: