From a7f4f51fe1d9883150f01f623e83f1ab197efcbb Mon Sep 17 00:00:00 2001 From: Joel Labes Date: Wed, 23 Feb 2022 15:53:49 +1300 Subject: [PATCH 01/10] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index bab3a503..f22c3809 100644 --- a/README.md +++ b/README.md @@ -67,7 +67,7 @@ For compatibility details between versions of dbt-core and dbt-utils, [see this [Materializations](#materializations): - [insert_by_period](#insert_by_period-source) ---- +---- ### Schema Tests #### equal_rowcount ([source](macros/schema_tests/equal_rowcount.sql)) This schema test asserts the that two relations have the same number of rows. From 168396fb2babd889ed2acba6fb8242563fd4bf27 Mon Sep 17 00:00:00 2001 From: Joel Labes Date: Mon, 28 Feb 2022 15:51:08 +1300 Subject: [PATCH 02/10] Mutually excl range examples in disclosure triangle --- README.md | 99 +++++++++++++++++++++++++++++-------------------------- 1 file changed, 52 insertions(+), 47 deletions(-) diff --git a/README.md b/README.md index f22c3809..709640b3 100644 --- a/README.md +++ b/README.md @@ -377,53 +377,58 @@ models: partition_by: customer_id gaps: allowed ``` - -**Understanding the `gaps` argument:** -Here are a number of examples for each allowed `gaps` argument. -* `gaps: not_allowed`: The upper bound of one record must be the lower bound of -the next record. - -| lower_bound | upper_bound | -|-------------|-------------| -| 0 | 1 | -| 1 | 2 | -| 2 | 3 | - -* `gaps: allowed` (default): There may be a gap between the upper bound of one -record and the lower bound of the next record. - -| lower_bound | upper_bound | -|-------------|-------------| -| 0 | 1 | -| 2 | 3 | -| 3 | 4 | - -* `gaps: required`: There must be a gap between the upper bound of one record and -the lower bound of the next record (common for date ranges). - -| lower_bound | upper_bound | -|-------------|-------------| -| 0 | 1 | -| 2 | 3 | -| 4 | 5 | - -**Understanding the `zero_length_range_allowed` argument:** -Here are a number of examples for each allowed `zero_length_range_allowed` argument. -* `zero_length_range_allowed: false`: (default) The upper bound of each record must be greater than its lower bound. - -| lower_bound | upper_bound | -|-------------|-------------| -| 0 | 1 | -| 1 | 2 | -| 2 | 3 | - -* `zero_length_range_allowed: true`: The upper bound of each record can be greater than or equal to its lower bound. - -| lower_bound | upper_bound | -|-------------|-------------| -| 0 | 1 | -| 2 | 2 | -| 3 | 4 | +
+Additional `gaps` and `zero_length_range_allowed` examples + + **Understanding the `gaps` argument:** + + Here are a number of examples for each allowed `gaps` argument. + * `gaps: not_allowed`: The upper bound of one record must be the lower bound of + the next record. + + | lower_bound | upper_bound | + |-------------|-------------| + | 0 | 1 | + | 1 | 2 | + | 2 | 3 | + + * `gaps: allowed` (default): There may be a gap between the upper bound of one + record and the lower bound of the next record. + + | lower_bound | upper_bound | + |-------------|-------------| + | 0 | 1 | + | 2 | 3 | + | 3 | 4 | + + * `gaps: required`: There must be a gap between the upper bound of one record and + the lower bound of the next record (common for date ranges). + + | lower_bound | upper_bound | + |-------------|-------------| + | 0 | 1 | + | 2 | 3 | + | 4 | 5 | + + **Understanding the `zero_length_range_allowed` argument:** + Here are a number of examples for each allowed `zero_length_range_allowed` argument. + * `zero_length_range_allowed: false`: (default) The upper bound of each record must be greater than its lower bound. + + | lower_bound | upper_bound | + |-------------|-------------| + | 0 | 1 | + | 1 | 2 | + | 2 | 3 | + + * `zero_length_range_allowed: true`: The upper bound of each record can be greater than or equal to its lower bound. + + | lower_bound | upper_bound | + |-------------|-------------| + | 0 | 1 | + | 2 | 2 | + | 3 | 4 | + +
#### sequential_values ([source](macros/schema_tests/sequential_values.sql)) This test confirms that a column contains sequential values. It can be used From 5717b109d358844e0f085565b93e74cbe469ecb0 Mon Sep 17 00:00:00 2001 From: Joel Labes Date: Thu, 3 Mar 2022 14:24:19 +1300 Subject: [PATCH 03/10] Fix union_relations error when no include/exclude provided * Fix union_relations error when no include/exclude provided (#509) * Update CHANGELOG.md --- CHANGELOG.md | 4 ++++ macros/sql/union.sql | 2 +- 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 67b690e7..f346c55b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,7 @@ +# dbt-utils v0.8.2 +## Fixes +- Fix union_relations error from [#473](https://github.com/dbt-labs/dbt-utils/pull/473) when no include/exclude parameters are provided ([#505](https://github.com/dbt-labs/dbt-utils/issues/505), [#509](https://github.com/dbt-labs/dbt-utils/pull/509)) + # dbt-utils v0.8.1 ## New features diff --git a/macros/sql/union.sql b/macros/sql/union.sql index 009a765a..a7bf1d95 100644 --- a/macros/sql/union.sql +++ b/macros/sql/union.sql @@ -61,7 +61,7 @@ {%- set ordered_column_names = column_superset.keys() -%} - {%- if not column_superset.keys() -%} + {% if (include | length > 0 or exclude | length > 0) and not column_superset.keys() %} {%- set relations_string -%} {%- for relation in relations -%} {{ relation.name }} From 3c83bf40a6735b74532afa38fc60ae2c8a8087cd Mon Sep 17 00:00:00 2001 From: Joel Labes Date: Thu, 10 Mar 2022 16:35:23 +1300 Subject: [PATCH 04/10] Add to_condition to relationships where --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 709640b3..f707a376 100644 --- a/README.md +++ b/README.md @@ -310,6 +310,7 @@ models: to: ref('other_model_name') field: client_id from_condition: id <> '4ca448b8-24bf-4b88-96c6-b1609499c38b' + to_condition: created_date >= '2020-01-01' ``` #### mutually_exclusive_ranges ([source](macros/schema_tests/mutually_exclusive_ranges.sql)) From b000d8b7609104f90b4ae05fea96051f8aeb4200 Mon Sep 17 00:00:00 2001 From: Jamie Rosenberg Date: Mon, 14 Mar 2022 15:46:38 +1100 Subject: [PATCH 05/10] very minor nit - update "an new" to "a new" (#519) --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index f707a376..114c12c4 100644 --- a/README.md +++ b/README.md @@ -772,7 +772,7 @@ from {{ ref('my_model') }} This macro unions together an array of [Relations](https://docs.getdbt.com/docs/writing-code-in-dbt/class-reference/#relation), even when columns have differing orders in each Relation, and/or some columns are missing from some relations. Any columns exclusive to a subset of these -relations will be filled with `null` where not present. An new column +relations will be filled with `null` where not present. A new column (`_dbt_source_relation`) is also added to indicate the source for each record. **Usage:** From d48e35ea4b07a5776f5dc46d99d1dbc0698ce321 Mon Sep 17 00:00:00 2001 From: Luis Leon <98919783+luisleon90@users.noreply.github.com> Date: Mon, 21 Mar 2022 15:58:51 -0400 Subject: [PATCH 06/10] Typed materialized views as views --- macros/sql/get_relations_by_pattern.sql | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/macros/sql/get_relations_by_pattern.sql b/macros/sql/get_relations_by_pattern.sql index 9325a883..b772face 100644 --- a/macros/sql/get_relations_by_pattern.sql +++ b/macros/sql/get_relations_by_pattern.sql @@ -19,7 +19,7 @@ database=database, schema=row.table_schema, identifier=row.table_name, - type=row.table_type + type = row.table_type if row.table_type != ‘materialized view’ else ‘view’ ) -%} {%- do tbl_relations.append(tbl_relation) -%} {%- endfor -%} From 6ef4df5e820d9af5842c32bf9ebb3431c2607c72 Mon Sep 17 00:00:00 2001 From: Luis Leon <98919783+luisleon90@users.noreply.github.com> Date: Mon, 21 Mar 2022 16:52:07 -0400 Subject: [PATCH 07/10] Update get_relations_by_pattern.sql --- macros/sql/get_relations_by_pattern.sql | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/macros/sql/get_relations_by_pattern.sql b/macros/sql/get_relations_by_pattern.sql index b772face..b247f321 100644 --- a/macros/sql/get_relations_by_pattern.sql +++ b/macros/sql/get_relations_by_pattern.sql @@ -19,7 +19,7 @@ database=database, schema=row.table_schema, identifier=row.table_name, - type = row.table_type if row.table_type != ‘materialized view’ else ‘view’ + type = row.table_type if row.table_type != 'materialized view' else 'materializedview' ) -%} {%- do tbl_relations.append(tbl_relation) -%} {%- endfor -%} From 03a2cd5afc810dbd96c1cc5a2c64b4d775d5757a Mon Sep 17 00:00:00 2001 From: Luis Leon <98919783+luisleon90@users.noreply.github.com> Date: Tue, 22 Mar 2022 20:25:06 -0400 Subject: [PATCH 08/10] Moving fix from get_tables_by_pattern_sql reverting changes to this file to add a fix to the macro get_tables_by_pattern_sql --- macros/sql/get_relations_by_pattern.sql | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/macros/sql/get_relations_by_pattern.sql b/macros/sql/get_relations_by_pattern.sql index b247f321..aed8bd60 100644 --- a/macros/sql/get_relations_by_pattern.sql +++ b/macros/sql/get_relations_by_pattern.sql @@ -19,7 +19,7 @@ database=database, schema=row.table_schema, identifier=row.table_name, - type = row.table_type if row.table_type != 'materialized view' else 'materializedview' + type = row.table_type ) -%} {%- do tbl_relations.append(tbl_relation) -%} {%- endfor -%} From 3932f152121bc1a733a5f09652a15067b9cc097c Mon Sep 17 00:00:00 2001 From: Luis Leon <98919783+luisleon90@users.noreply.github.com> Date: Tue, 22 Mar 2022 20:26:30 -0400 Subject: [PATCH 09/10] removing quoting from table_type removing quoting from table_type as this was causing an error when calling this macro within get_tables_by_pattern_sql --- macros/sql/get_table_types_sql.sql | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/macros/sql/get_table_types_sql.sql b/macros/sql/get_table_types_sql.sql index e3f86884..c16275e9 100644 --- a/macros/sql/get_table_types_sql.sql +++ b/macros/sql/get_table_types_sql.sql @@ -8,7 +8,7 @@ when 'EXTERNAL TABLE' then 'external' when 'MATERIALIZED VIEW' then 'materializedview' else lower(table_type) - end as "table_type" + end as table_type {% endmacro %} From a700eb62c11f6af2a03009321b98a76511084296 Mon Sep 17 00:00:00 2001 From: Luis Leon <98919783+luisleon90@users.noreply.github.com> Date: Tue, 22 Mar 2022 20:30:11 -0400 Subject: [PATCH 10/10] calling get_table_types_sql for materialized views calling get_table_types_sql macro to handle materialized views in sources. --- macros/sql/get_tables_by_pattern_sql.sql | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/macros/sql/get_tables_by_pattern_sql.sql b/macros/sql/get_tables_by_pattern_sql.sql index 93f3c6a6..4d5a8fc9 100644 --- a/macros/sql/get_tables_by_pattern_sql.sql +++ b/macros/sql/get_tables_by_pattern_sql.sql @@ -30,10 +30,7 @@ select distinct table_schema, table_name, - case table_type - when 'BASE TABLE' then 'table' - else lower(table_type) - end as table_type + {{ dbt_utils.get_table_types_sql() }} from {{ adapter.quote(database) }}.{{ schema }}.INFORMATION_SCHEMA.TABLES where lower(table_name) like lower ('{{ table_pattern }}')