From a7f4f51fe1d9883150f01f623e83f1ab197efcbb Mon Sep 17 00:00:00 2001 From: Joel Labes Date: Wed, 23 Feb 2022 15:53:49 +1300 Subject: [PATCH 01/11] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index bab3a503..f22c3809 100644 --- a/README.md +++ b/README.md @@ -67,7 +67,7 @@ For compatibility details between versions of dbt-core and dbt-utils, [see this [Materializations](#materializations): - [insert_by_period](#insert_by_period-source) ---- +---- ### Schema Tests #### equal_rowcount ([source](macros/schema_tests/equal_rowcount.sql)) This schema test asserts the that two relations have the same number of rows. From 168396fb2babd889ed2acba6fb8242563fd4bf27 Mon Sep 17 00:00:00 2001 From: Joel Labes Date: Mon, 28 Feb 2022 15:51:08 +1300 Subject: [PATCH 02/11] Mutually excl range examples in disclosure triangle --- README.md | 99 +++++++++++++++++++++++++++++-------------------------- 1 file changed, 52 insertions(+), 47 deletions(-) diff --git a/README.md b/README.md index f22c3809..709640b3 100644 --- a/README.md +++ b/README.md @@ -377,53 +377,58 @@ models: partition_by: customer_id gaps: allowed ``` - -**Understanding the `gaps` argument:** -Here are a number of examples for each allowed `gaps` argument. -* `gaps: not_allowed`: The upper bound of one record must be the lower bound of -the next record. - -| lower_bound | upper_bound | -|-------------|-------------| -| 0 | 1 | -| 1 | 2 | -| 2 | 3 | - -* `gaps: allowed` (default): There may be a gap between the upper bound of one -record and the lower bound of the next record. - -| lower_bound | upper_bound | -|-------------|-------------| -| 0 | 1 | -| 2 | 3 | -| 3 | 4 | - -* `gaps: required`: There must be a gap between the upper bound of one record and -the lower bound of the next record (common for date ranges). - -| lower_bound | upper_bound | -|-------------|-------------| -| 0 | 1 | -| 2 | 3 | -| 4 | 5 | - -**Understanding the `zero_length_range_allowed` argument:** -Here are a number of examples for each allowed `zero_length_range_allowed` argument. -* `zero_length_range_allowed: false`: (default) The upper bound of each record must be greater than its lower bound. - -| lower_bound | upper_bound | -|-------------|-------------| -| 0 | 1 | -| 1 | 2 | -| 2 | 3 | - -* `zero_length_range_allowed: true`: The upper bound of each record can be greater than or equal to its lower bound. - -| lower_bound | upper_bound | -|-------------|-------------| -| 0 | 1 | -| 2 | 2 | -| 3 | 4 | +
+Additional `gaps` and `zero_length_range_allowed` examples + + **Understanding the `gaps` argument:** + + Here are a number of examples for each allowed `gaps` argument. + * `gaps: not_allowed`: The upper bound of one record must be the lower bound of + the next record. + + | lower_bound | upper_bound | + |-------------|-------------| + | 0 | 1 | + | 1 | 2 | + | 2 | 3 | + + * `gaps: allowed` (default): There may be a gap between the upper bound of one + record and the lower bound of the next record. + + | lower_bound | upper_bound | + |-------------|-------------| + | 0 | 1 | + | 2 | 3 | + | 3 | 4 | + + * `gaps: required`: There must be a gap between the upper bound of one record and + the lower bound of the next record (common for date ranges). + + | lower_bound | upper_bound | + |-------------|-------------| + | 0 | 1 | + | 2 | 3 | + | 4 | 5 | + + **Understanding the `zero_length_range_allowed` argument:** + Here are a number of examples for each allowed `zero_length_range_allowed` argument. + * `zero_length_range_allowed: false`: (default) The upper bound of each record must be greater than its lower bound. + + | lower_bound | upper_bound | + |-------------|-------------| + | 0 | 1 | + | 1 | 2 | + | 2 | 3 | + + * `zero_length_range_allowed: true`: The upper bound of each record can be greater than or equal to its lower bound. + + | lower_bound | upper_bound | + |-------------|-------------| + | 0 | 1 | + | 2 | 2 | + | 3 | 4 | + +
#### sequential_values ([source](macros/schema_tests/sequential_values.sql)) This test confirms that a column contains sequential values. It can be used From 5717b109d358844e0f085565b93e74cbe469ecb0 Mon Sep 17 00:00:00 2001 From: Joel Labes Date: Thu, 3 Mar 2022 14:24:19 +1300 Subject: [PATCH 03/11] Fix union_relations error when no include/exclude provided * Fix union_relations error when no include/exclude provided (#509) * Update CHANGELOG.md --- CHANGELOG.md | 4 ++++ macros/sql/union.sql | 2 +- 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 67b690e7..f346c55b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,7 @@ +# dbt-utils v0.8.2 +## Fixes +- Fix union_relations error from [#473](https://github.com/dbt-labs/dbt-utils/pull/473) when no include/exclude parameters are provided ([#505](https://github.com/dbt-labs/dbt-utils/issues/505), [#509](https://github.com/dbt-labs/dbt-utils/pull/509)) + # dbt-utils v0.8.1 ## New features diff --git a/macros/sql/union.sql b/macros/sql/union.sql index 009a765a..a7bf1d95 100644 --- a/macros/sql/union.sql +++ b/macros/sql/union.sql @@ -61,7 +61,7 @@ {%- set ordered_column_names = column_superset.keys() -%} - {%- if not column_superset.keys() -%} + {% if (include | length > 0 or exclude | length > 0) and not column_superset.keys() %} {%- set relations_string -%} {%- for relation in relations -%} {{ relation.name }} From 3c83bf40a6735b74532afa38fc60ae2c8a8087cd Mon Sep 17 00:00:00 2001 From: Joel Labes Date: Thu, 10 Mar 2022 16:35:23 +1300 Subject: [PATCH 04/11] Add to_condition to relationships where --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 709640b3..f707a376 100644 --- a/README.md +++ b/README.md @@ -310,6 +310,7 @@ models: to: ref('other_model_name') field: client_id from_condition: id <> '4ca448b8-24bf-4b88-96c6-b1609499c38b' + to_condition: created_date >= '2020-01-01' ``` #### mutually_exclusive_ranges ([source](macros/schema_tests/mutually_exclusive_ranges.sql)) From b000d8b7609104f90b4ae05fea96051f8aeb4200 Mon Sep 17 00:00:00 2001 From: Jamie Rosenberg Date: Mon, 14 Mar 2022 15:46:38 +1100 Subject: [PATCH 05/11] very minor nit - update "an new" to "a new" (#519) --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index f707a376..114c12c4 100644 --- a/README.md +++ b/README.md @@ -772,7 +772,7 @@ from {{ ref('my_model') }} This macro unions together an array of [Relations](https://docs.getdbt.com/docs/writing-code-in-dbt/class-reference/#relation), even when columns have differing orders in each Relation, and/or some columns are missing from some relations. Any columns exclusive to a subset of these -relations will be filled with `null` where not present. An new column +relations will be filled with `null` where not present. A new column (`_dbt_source_relation`) is also added to indicate the source for each record. **Usage:** From ffe0bb4cd2dea555fcaf0a5894f12436c3920209 Mon Sep 17 00:00:00 2001 From: brid-moynihan Date: Wed, 16 Mar 2022 09:53:37 +0000 Subject: [PATCH 06/11] Updated Rreferences to 'schema test' in README along with small improvements to test descriptions. Updates were also carried out in folder structure and integration README --- README.md | 81 ++++++++++--------- integration_tests/README.md | 4 +- .../models/datetime/test_date_spine.sql | 2 +- .../schema.yml | 0 .../test_equal_column_subset.sql | 0 .../test_equal_rowcount.sql | 0 .../test_fewer_rows_than.sql | 0 .../test_recency.sql | 0 .../models/sql/test_generate_series.sql | 2 +- .../accepted_range.sql | 0 .../at_least_one.sql | 0 .../cardinality_equality.sql | 0 .../equal_rowcount.sql | 0 .../equality.sql | 0 .../expression_is_true.sql | 0 .../fewer_rows_than.sql | 0 .../mutually_exclusive_ranges.sql | 0 .../not_accepted_values.sql | 0 .../not_constant.sql | 0 .../not_null_proportion.sql | 0 .../recency.sql | 0 .../relationships_where.sql | 0 .../sequential_values.sql | 0 .../test_not_null_where.sql | 0 .../test_unique_where.sql | 0 .../unique_combination_of_columns.sql | 0 26 files changed, 47 insertions(+), 42 deletions(-) rename integration_tests/models/{schema_tests => generic_tests}/schema.yml (100%) rename integration_tests/models/{schema_tests => generic_tests}/test_equal_column_subset.sql (100%) rename integration_tests/models/{schema_tests => generic_tests}/test_equal_rowcount.sql (100%) rename integration_tests/models/{schema_tests => generic_tests}/test_fewer_rows_than.sql (100%) rename integration_tests/models/{schema_tests => generic_tests}/test_recency.sql (100%) rename macros/{schema_tests => generic_tests}/accepted_range.sql (100%) rename macros/{schema_tests => generic_tests}/at_least_one.sql (100%) rename macros/{schema_tests => generic_tests}/cardinality_equality.sql (100%) rename macros/{schema_tests => generic_tests}/equal_rowcount.sql (100%) rename macros/{schema_tests => generic_tests}/equality.sql (100%) rename macros/{schema_tests => generic_tests}/expression_is_true.sql (100%) rename macros/{schema_tests => generic_tests}/fewer_rows_than.sql (100%) rename macros/{schema_tests => generic_tests}/mutually_exclusive_ranges.sql (100%) rename macros/{schema_tests => generic_tests}/not_accepted_values.sql (100%) rename macros/{schema_tests => generic_tests}/not_constant.sql (100%) rename macros/{schema_tests => generic_tests}/not_null_proportion.sql (100%) rename macros/{schema_tests => generic_tests}/recency.sql (100%) rename macros/{schema_tests => generic_tests}/relationships_where.sql (100%) rename macros/{schema_tests => generic_tests}/sequential_values.sql (100%) rename macros/{schema_tests => generic_tests}/test_not_null_where.sql (100%) rename macros/{schema_tests => generic_tests}/test_unique_where.sql (100%) rename macros/{schema_tests => generic_tests}/unique_combination_of_columns.sql (100%) diff --git a/README.md b/README.md index 114c12c4..01f8adad 100644 --- a/README.md +++ b/README.md @@ -9,7 +9,7 @@ For compatibility details between versions of dbt-core and dbt-utils, [see this ---- ## Contents -**[Schema tests](#schema-tests)** +**[Generic tests](#generic-tests)** - [equal_rowcount](#equal_rowcount-source) - [fewer_rows_than](#fewer_rows_than-source) - [equality](#equality-source) @@ -68,9 +68,9 @@ For compatibility details between versions of dbt-core and dbt-utils, [see this - [insert_by_period](#insert_by_period-source) ---- -### Schema Tests -#### equal_rowcount ([source](macros/schema_tests/equal_rowcount.sql)) -This schema test asserts the that two relations have the same number of rows. +### Generic Tests +#### equal_rowcount ([source](macros/generic_tests/equal_rowcount.sql)) +Asserts that two relations have the same number of rows. **Usage:** ```yaml @@ -84,8 +84,8 @@ models: ``` -#### fewer_rows_than ([source](macros/schema_tests/fewer_rows_than.sql)) -This schema test asserts that this model has fewer rows than the referenced model. +#### fewer_rows_than ([source](macros/generic_tests/fewer_rows_than.sql)) +Asserts that the respective model has fewer rows than the model being compared. Usage: ```yaml @@ -98,8 +98,8 @@ models: compare_model: ref('other_table_name') ``` -#### equality ([source](macros/schema_tests/equality.sql)) -This schema test asserts the equality of two relations. Optionally specify a subset of columns to compare. +#### equality ([source](macros/generic_tests/equality.sql)) +Asserts the equality of two relations. Optionally specify a subset of columns to compare. **Usage:** ```yaml @@ -115,8 +115,13 @@ models: - second_column ``` -#### expression_is_true ([source](macros/schema_tests/expression_is_true.sql)) -This schema test asserts that a valid sql expression is true for all records. This is useful when checking integrity across columns, for example, that a total is equal to the sum of its parts, or that at least one column is true. +#### expression_is_true ([source](macros/generic_tests/expression_is_true.sql)) +Asserts that a valid SQL expression is true for all records. This is useful when checking integrity across columns. +Examples: + +- Verify an outcome based on the application of basic alegbraic operations between columns. +- Verify the length of a column. +- Verify the truth value of a column. **Usage:** ```yaml @@ -163,8 +168,8 @@ models: condition: col_a = 1 ``` -#### recency ([source](macros/schema_tests/recency.sql)) -This schema test asserts that there is data in the referenced model at least as recent as the defined interval prior to the current timestamp. +#### recency ([source](macros/generic_tests/recency.sql)) +Asserts that a timestamp column in the reference model contains data that is at least as recent as the defined date interval. **Usage:** ```yaml @@ -179,8 +184,8 @@ models: interval: 1 ``` -#### at_least_one ([source](macros/schema_tests/at_least_one.sql)) -This schema test asserts if column has at least one value. +#### at_least_one ([source](macros/generic_tests/at_least_one.sql)) +Asserts if a column has at least one value. **Usage:** ```yaml @@ -194,8 +199,8 @@ models: - dbt_utils.at_least_one ``` -#### not_constant ([source](macros/schema_tests/not_constant.sql)) -This schema test asserts if column does not have same value in all rows. +#### not_constant ([source](macros/generic_tests/not_constant.sql)) +Asserts if a column does not have the same value in all rows. **Usage:** ```yaml @@ -209,8 +214,8 @@ models: - dbt_utils.not_constant ``` -#### cardinality_equality ([source](macros/schema_tests/cardinality_equality.sql)) -This schema test asserts if values in a given column have exactly the same cardinality as values from a different column in a different model. +#### cardinality_equality ([source](macros/generic_tests/cardinality_equality.sql)) +Asserts if values in a given column have exactly the same cardinality as values from a different column in a different model. **Usage:** ```yaml @@ -226,8 +231,8 @@ models: to: ref('other_model_name') ``` -#### unique_where ([source](macros/schema_tests/test_unique_where.sql)) -This test validates that there are no duplicate values present in a field for a subset of rows by specifying a `where` clause. +#### unique_where ([source](macros/generic_tests/test_unique_where.sql)) +Asserts that there are no duplicate values present in a field for a subset of rows by specifying a `where` clause. *Warning*: This test is no longer supported. Starting in dbt v0.20.0, the built-in `unique` test supports a `where` config. [See the dbt docs for more details](https://docs.getdbt.com/reference/resource-configs/where). @@ -244,8 +249,8 @@ models: where: "_deleted = false" ``` -#### not_null_where ([source](macros/schema_tests/test_not_null_where.sql)) -This test validates that there are no null values present in a column for a subset of rows by specifying a `where` clause. +#### not_null_where ([source](macros/generic_tests/test_not_null_where.sql)) +Asserts that there are no null values present in a column for a subset of rows by specifying a `where` clause. *Warning*: This test is no longer supported. Starting in dbt v0.20.0, the built-in `not_null` test supports a `where` config. [See the dbt docs for more details](https://docs.getdbt.com/reference/resource-configs/where). @@ -262,8 +267,8 @@ models: where: "_deleted = false" ``` -#### not_null_proportion ([source](macros/schema_tests/not_null_proportion.sql)) -This test validates that the proportion of non-null values present in a column is between a specified range [`at_least`, `at_most`] where `at_most` is an optional argument (default: `1.0`). +#### not_null_proportion ([source](macros/generic_tests/not_null_proportion.sql)) +Asserts that the proportion of non-null values present in a column is between a specified range [`at_least`, `at_most`] where `at_most` is an optional argument (default: `1.0`). **Usage:** ```yaml @@ -278,8 +283,8 @@ models: at_least: 0.95 ``` -#### not_accepted_values ([source](macros/schema_tests/not_accepted_values.sql)) -This test validates that there are no rows that match the given values. +#### not_accepted_values ([source](macros/generic_tests/not_accepted_values.sql)) +Asserts that there are no rows that match the given values. Usage: ```yaml @@ -294,8 +299,8 @@ models: values: ['Barcelona', 'New York'] ``` -#### relationships_where ([source](macros/schema_tests/relationships_where.sql)) -This test validates the referential integrity between two relations (same as the core relationships schema test) with an added predicate to filter out some rows from the test. This is useful to exclude records such as test entities, rows created in the last X minutes/hours to account for temporary gaps due to ETL limitations, etc. +#### relationships_where ([source](macros/generic_tests/relationships_where.sql)) +Asserts the referential integrity between two relations (same as the core relationships assertions) with an added predicate to filter out some rows from the test. This is useful to exclude records such as test entities, rows created in the last X minutes/hours to account for temporary gaps due to ETL limitations, etc. **Usage:** ```yaml @@ -313,9 +318,9 @@ models: to_condition: created_date >= '2020-01-01' ``` -#### mutually_exclusive_ranges ([source](macros/schema_tests/mutually_exclusive_ranges.sql)) -This test confirms that for a given lower_bound_column and upper_bound_column, -the ranges of between the lower and upper bounds do not overlap with the ranges +#### mutually_exclusive_ranges ([source](macros/generic_tests/mutually_exclusive_ranges.sql)) +Asserts that for a given lower_bound_column and upper_bound_column, +the ranges between the lower and upper bounds do not overlap with the ranges of another row. **Usage:** @@ -431,9 +436,9 @@ models: -#### sequential_values ([source](macros/schema_tests/sequential_values.sql)) -This test confirms that a column contains sequential values. It can be used -for both numeric values, and datetime values, as follows: +#### sequential_values ([source](macros/generic_tests/sequential_values.sql)) +Asserts that a column contains sequential values. It can be used +for both numeric values and datetime values, as follows: ```yml version: 2 @@ -459,8 +464,8 @@ seeds: * `interval` (default=1): The gap between two sequential values * `datepart` (default=None): Used when the gaps are a unit of time. If omitted, the test will check for a numeric gap. -#### unique_combination_of_columns ([source](macros/schema_tests/unique_combination_of_columns.sql)) -This test confirms that the combination of columns is unique. For example, the +#### unique_combination_of_columns ([source](macros/generic_tests/unique_combination_of_columns.sql)) +Asserts that the combination of columns is unique. For example, the combination of month and product is unique, however neither column is unique in isolation. @@ -495,8 +500,8 @@ An optional `quote_columns` argument (`default=false`) can also be used if a col ``` -#### accepted_range ([source](macros/schema_tests/accepted_range.sql)) -This test checks that a column's values fall inside an expected range. Any combination of `min_value` and `max_value` is allowed, and the range can be inclusive or exclusive. Provide a `where` argument to filter to specific records only. +#### accepted_range ([source](macros/generic_tests/accepted_range.sql)) +Asserts that a column's values fall inside an expected range. Any combination of `min_value` and `max_value` is allowed, and the range can be inclusive or exclusive. Provide a `where` argument to filter to specific records only. In addition to comparisons to a scalar value, you can also compare to another column's values. Any data type that supports the `>` or `<` operators can be compared, so you could also run tests like checking that all order dates are in the past. diff --git a/integration_tests/README.md b/integration_tests/README.md index 243af411..4f9f0131 100644 --- a/integration_tests/README.md +++ b/integration_tests/README.md @@ -26,14 +26,14 @@ Where possible, targets are being run in docker containers (this works for Postg ### Creating a new integration test -This directory contains an example dbt project which tests the macros in the `dbt-utils` package. An integration test typically involves making 1) a new seed file 2) a new model file 3) a schema test. +This directory contains an example dbt project which tests the macros in the `dbt-utils` package. An integration test typically involves making 1) a new seed file 2) a new model file 3) a generic test to assert anticipated behaviour. For an example integration tests, check out the tests for the `get_url_parameter` macro: 1. [Macro definition](https://github.com/fishtown-analytics/dbt-utils/blob/master/macros/web/get_url_parameter.sql) 2. [Seed file with fake data](https://github.com/fishtown-analytics/dbt-utils/blob/master/integration_tests/data/web/data_urls.csv) 3. [Model to test the macro](https://github.com/fishtown-analytics/dbt-utils/blob/master/integration_tests/models/web/test_urls.sql) -4. [A schema test to assert the macro works as expected](https://github.com/fishtown-analytics/dbt-utils/blob/master/integration_tests/models/web/schema.yml#L2) +4. [A generic test to assert the macro works as expected](https://github.com/fishtown-analytics/dbt-utils/blob/master/integration_tests/models/web/schema.yml#L2) Once you've added all of these files, you should be able to run: diff --git a/integration_tests/models/datetime/test_date_spine.sql b/integration_tests/models/datetime/test_date_spine.sql index 93cd07f1..fa4ae52b 100644 --- a/integration_tests/models/datetime/test_date_spine.sql +++ b/integration_tests/models/datetime/test_date_spine.sql @@ -1,6 +1,6 @@ -- snowflake doesn't like this as a view because the `generate_series` --- call creates a CTE called `unioned`, as does the `equality` schema test. +-- call creates a CTE called `unioned`, as does the `equality` generic test. -- Ideally, Snowflake would be smart enough to know that these CTE names are -- different, as they live in different relations. TODO: use a less common cte name diff --git a/integration_tests/models/schema_tests/schema.yml b/integration_tests/models/generic_tests/schema.yml similarity index 100% rename from integration_tests/models/schema_tests/schema.yml rename to integration_tests/models/generic_tests/schema.yml diff --git a/integration_tests/models/schema_tests/test_equal_column_subset.sql b/integration_tests/models/generic_tests/test_equal_column_subset.sql similarity index 100% rename from integration_tests/models/schema_tests/test_equal_column_subset.sql rename to integration_tests/models/generic_tests/test_equal_column_subset.sql diff --git a/integration_tests/models/schema_tests/test_equal_rowcount.sql b/integration_tests/models/generic_tests/test_equal_rowcount.sql similarity index 100% rename from integration_tests/models/schema_tests/test_equal_rowcount.sql rename to integration_tests/models/generic_tests/test_equal_rowcount.sql diff --git a/integration_tests/models/schema_tests/test_fewer_rows_than.sql b/integration_tests/models/generic_tests/test_fewer_rows_than.sql similarity index 100% rename from integration_tests/models/schema_tests/test_fewer_rows_than.sql rename to integration_tests/models/generic_tests/test_fewer_rows_than.sql diff --git a/integration_tests/models/schema_tests/test_recency.sql b/integration_tests/models/generic_tests/test_recency.sql similarity index 100% rename from integration_tests/models/schema_tests/test_recency.sql rename to integration_tests/models/generic_tests/test_recency.sql diff --git a/integration_tests/models/sql/test_generate_series.sql b/integration_tests/models/sql/test_generate_series.sql index a943cf6c..11370b7b 100644 --- a/integration_tests/models/sql/test_generate_series.sql +++ b/integration_tests/models/sql/test_generate_series.sql @@ -1,6 +1,6 @@ -- snowflake doesn't like this as a view because the `generate_series` --- call creates a CTE called `unioned`, as does the `equality` schema test. +-- call creates a CTE called `unioned`, as does the `equality` generic test. -- Ideally, Snowflake would be smart enough to know that these CTE names are -- different, as they live in different relations. TODO: use a less common cte name diff --git a/macros/schema_tests/accepted_range.sql b/macros/generic_tests/accepted_range.sql similarity index 100% rename from macros/schema_tests/accepted_range.sql rename to macros/generic_tests/accepted_range.sql diff --git a/macros/schema_tests/at_least_one.sql b/macros/generic_tests/at_least_one.sql similarity index 100% rename from macros/schema_tests/at_least_one.sql rename to macros/generic_tests/at_least_one.sql diff --git a/macros/schema_tests/cardinality_equality.sql b/macros/generic_tests/cardinality_equality.sql similarity index 100% rename from macros/schema_tests/cardinality_equality.sql rename to macros/generic_tests/cardinality_equality.sql diff --git a/macros/schema_tests/equal_rowcount.sql b/macros/generic_tests/equal_rowcount.sql similarity index 100% rename from macros/schema_tests/equal_rowcount.sql rename to macros/generic_tests/equal_rowcount.sql diff --git a/macros/schema_tests/equality.sql b/macros/generic_tests/equality.sql similarity index 100% rename from macros/schema_tests/equality.sql rename to macros/generic_tests/equality.sql diff --git a/macros/schema_tests/expression_is_true.sql b/macros/generic_tests/expression_is_true.sql similarity index 100% rename from macros/schema_tests/expression_is_true.sql rename to macros/generic_tests/expression_is_true.sql diff --git a/macros/schema_tests/fewer_rows_than.sql b/macros/generic_tests/fewer_rows_than.sql similarity index 100% rename from macros/schema_tests/fewer_rows_than.sql rename to macros/generic_tests/fewer_rows_than.sql diff --git a/macros/schema_tests/mutually_exclusive_ranges.sql b/macros/generic_tests/mutually_exclusive_ranges.sql similarity index 100% rename from macros/schema_tests/mutually_exclusive_ranges.sql rename to macros/generic_tests/mutually_exclusive_ranges.sql diff --git a/macros/schema_tests/not_accepted_values.sql b/macros/generic_tests/not_accepted_values.sql similarity index 100% rename from macros/schema_tests/not_accepted_values.sql rename to macros/generic_tests/not_accepted_values.sql diff --git a/macros/schema_tests/not_constant.sql b/macros/generic_tests/not_constant.sql similarity index 100% rename from macros/schema_tests/not_constant.sql rename to macros/generic_tests/not_constant.sql diff --git a/macros/schema_tests/not_null_proportion.sql b/macros/generic_tests/not_null_proportion.sql similarity index 100% rename from macros/schema_tests/not_null_proportion.sql rename to macros/generic_tests/not_null_proportion.sql diff --git a/macros/schema_tests/recency.sql b/macros/generic_tests/recency.sql similarity index 100% rename from macros/schema_tests/recency.sql rename to macros/generic_tests/recency.sql diff --git a/macros/schema_tests/relationships_where.sql b/macros/generic_tests/relationships_where.sql similarity index 100% rename from macros/schema_tests/relationships_where.sql rename to macros/generic_tests/relationships_where.sql diff --git a/macros/schema_tests/sequential_values.sql b/macros/generic_tests/sequential_values.sql similarity index 100% rename from macros/schema_tests/sequential_values.sql rename to macros/generic_tests/sequential_values.sql diff --git a/macros/schema_tests/test_not_null_where.sql b/macros/generic_tests/test_not_null_where.sql similarity index 100% rename from macros/schema_tests/test_not_null_where.sql rename to macros/generic_tests/test_not_null_where.sql diff --git a/macros/schema_tests/test_unique_where.sql b/macros/generic_tests/test_unique_where.sql similarity index 100% rename from macros/schema_tests/test_unique_where.sql rename to macros/generic_tests/test_unique_where.sql diff --git a/macros/schema_tests/unique_combination_of_columns.sql b/macros/generic_tests/unique_combination_of_columns.sql similarity index 100% rename from macros/schema_tests/unique_combination_of_columns.sql rename to macros/generic_tests/unique_combination_of_columns.sql From f92669621dbd7781aca6fd77fba0dcb895f9c1d0 Mon Sep 17 00:00:00 2001 From: brid-moynihan Date: Wed, 16 Mar 2022 09:54:12 +0000 Subject: [PATCH 07/11] Updated references to 'schema test' in Changelog --- CHANGELOG.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f346c55b..679ac957 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -87,12 +87,12 @@ ## Features -- Add `not_null_proportion` schema test that allows the user to specify the minimum (`at_least`) tolerated proportion (e.g., `0.95`) of non-null values ([#411](https://github.com/dbt-labs/dbt-utils/pull/411)) +- Add `not_null_proportion` generic test that allows the user to specify the minimum (`at_least`) tolerated proportion (e.g., `0.95`) of non-null values ([#411](https://github.com/dbt-labs/dbt-utils/pull/411)) ## Under the hood - Allow user to provide any case type when defining the `exclude` argument in `dbt_utils.star()` ([#403](https://github.com/dbt-labs/dbt-utils/pull/403)) -- Log whole row instead of just column name in 'accepted_range' schema test to allow better visibility into failures ([#413](https://github.com/dbt-labs/dbt-utils/pull/413)) +- Log whole row instead of just column name in 'accepted_range' generic test to allow better visibility into failures ([#413](https://github.com/dbt-labs/dbt-utils/pull/413)) - Use column name to group in 'get_column_values ' to allow better cross db functionality ([#407](https://github.com/dbt-labs/dbt-utils/pull/407)) # dbt-utils v0.7.1 @@ -149,7 +149,7 @@ If you were relying on the position to match up your optional arguments, this ma ## Features * Add new argument, `order_by`, to `get_column_values` (code originally in [#289](https://github.com/fishtown-analytics/dbt-utils/pull/289/) from [@clausherther](https://github.com/clausherther), merged via [#349](https://github.com/fishtown-analytics/dbt-utils/pull/349/)) * Add `slugify` macro, and use it in the pivot macro. :rotating_light: This macro uses the `re` module, which is only available in dbt v0.19.0+. As a result, this feature introduces a breaking change. ([#314](https://github.com/fishtown-analytics/dbt-utils/pull/314)) -* Add `not_null_proportion` schema test that allows the user to specify the minimum (`at_least`) tolerated proportion (e.g., `0.95`) of non-null values +* Add `not_null_proportion` generic test that allows the user to specify the minimum (`at_least`) tolerated proportion (e.g., `0.95`) of non-null values ## Under the hood * Update the default implementation of concat macro to use `||` operator ([#373](https://github.com/fishtown-analytics/dbt-utils/pull/314) from [@ChristopheDuong](https://github.com/ChristopheDuong)). Note this may be a breaking change for adapters that support `concat()` but not `||`, such as Apache Spark. @@ -160,18 +160,18 @@ If you were relying on the position to match up your optional arguments, this ma ## Fixes -- make `sequential_values` schema test use `dbt_utils.type_timestamp()` to allow for compatibility with db's without timestamp data type. [#376](https://github.com/fishtown-analytics/dbt-utils/pull/376) from [@swanderz](https://github.com/swanderz) +- make `sequential_values` generic test use `dbt_utils.type_timestamp()` to allow for compatibility with db's without timestamp data type. [#376](https://github.com/fishtown-analytics/dbt-utils/pull/376) from [@swanderz](https://github.com/swanderz) # dbt-utils v0.6.5 ## Features * Add new `accepted_range` test ([#276](https://github.com/fishtown-analytics/dbt-utils/pull/276) [@joellabes](https://github.com/joellabes)) * Make `expression_is_true` work as a column test (code originally in [#226](https://github.com/fishtown-analytics/dbt-utils/pull/226/) from [@elliottohara](https://github.com/elliottohara), merged via [#313](https://github.com/fishtown-analytics/dbt-utils/pull/313/)) -* Add new schema test, `not_accepted_values` ([#284](https://github.com/fishtown-analytics/dbt-utils/pull/284) [@JavierMonton](https://github.com/JavierMonton)) +* Add new generic test, `not_accepted_values` ([#284](https://github.com/fishtown-analytics/dbt-utils/pull/284) [@JavierMonton](https://github.com/JavierMonton)) * Support a new argument, `zero_length_range_allowed` in the `mutually_exclusive_ranges` test ([#307](https://github.com/fishtown-analytics/dbt-utils/pull/307) [@zemekeneng](https://github.com/zemekeneng)) -* Add new schema test, `sequential_values` ([#318](https://github.com/fishtown-analytics/dbt-utils/pull/318), inspired by [@hundredwatt](https://github.com/hundredwatt)) +* Add new generic test, `sequential_values` ([#318](https://github.com/fishtown-analytics/dbt-utils/pull/318), inspired by [@hundredwatt](https://github.com/hundredwatt)) * Support `quarter` in the `postgres__last_day` macro ([#333](https://github.com/fishtown-analytics/dbt-utils/pull/333/files) [@seunghanhong](https://github.com/seunghanhong)) * Add new argument, `unit`, to `haversine_distance` ([#340](https://github.com/fishtown-analytics/dbt-utils/pull/340) [@bastienboutonnet](https://github.com/bastienboutonnet)) -* Add new schema test, `fewer_rows_than` (code originally in [#221](https://github.com/fishtown-analytics/dbt-utils/pull/230/) from [@dmarts](https://github.com/dmarts), merged via [#343](https://github.com/fishtown-analytics/dbt-utils/pull/343/)) +* Add new generic test, `fewer_rows_than` (code originally in [#221](https://github.com/fishtown-analytics/dbt-utils/pull/230/) from [@dmarts](https://github.com/dmarts), merged via [#343](https://github.com/fishtown-analytics/dbt-utils/pull/343/)) ## Fixes * Handle booleans gracefully in the unpivot macro ([#305](https://github.com/fishtown-analytics/dbt-utils/pull/305) [@avishalom](https://github.com/avishalom)) @@ -245,7 +245,7 @@ enabling users of community-supported database plugins to add or override macro specific to their database ([#267](https://github.com/fishtown-analytics/dbt-utils/pull/267)) * Use `add_ephemeral_prefix` instead of hard-coding a string literal, to support database adapters that use different prefixes ([#267](https://github.com/fishtown-analytics/dbt-utils/pull/267)) -* Implement a quote_columns argument in the unique_combination_of_columns schema test ([#270](https://github.com/fishtown-analytics/dbt-utils/pull/270) [@JoshuaHuntley](https://github.com/JoshuaHuntley)) +* Implement a quote_columns argument in the unique_combination_of_columns generic test ([#270](https://github.com/fishtown-analytics/dbt-utils/pull/270) [@JoshuaHuntley](https://github.com/JoshuaHuntley)) ## Quality of life * Remove deprecated macros `get_tables_by_prefix` and `union_tables` ([#268](https://github.com/fishtown-analytics/dbt-utils/pull/268)) From f4d25e7250a5986abcccfcc8fa3d881672d5dcea Mon Sep 17 00:00:00 2001 From: brid-moynihan Date: Wed, 16 Mar 2022 10:02:53 +0000 Subject: [PATCH 08/11] updated changelog with changes to documentation and fproject file structure --- CHANGELOG.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 679ac957..8e5e5e90 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,7 +1,10 @@ # dbt-utils v0.8.2 -## Fixes +## Fixes - Fix union_relations error from [#473](https://github.com/dbt-labs/dbt-utils/pull/473) when no include/exclude parameters are provided ([#505](https://github.com/dbt-labs/dbt-utils/issues/505), [#509](https://github.com/dbt-labs/dbt-utils/pull/509)) +## Quality of life +- Updated references to 'schema test' in project file structure and documentation referred to in [#485](https://github.com/dbt-labs/dbt-utils/issues/485) + # dbt-utils v0.8.1 ## New features From 437aa33733bc42ba942b9184c468ac1d043bf4d9 Mon Sep 17 00:00:00 2001 From: Brid Moynihan Date: Fri, 25 Mar 2022 09:08:14 +0000 Subject: [PATCH 09/11] Apply suggestions from code review Update macro descriptions to be "asserts that" Co-authored-by: Joel Labes --- README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 01f8adad..0a76fbc0 100644 --- a/README.md +++ b/README.md @@ -185,7 +185,7 @@ models: ``` #### at_least_one ([source](macros/generic_tests/at_least_one.sql)) -Asserts if a column has at least one value. +Asserts that a column has at least one value. **Usage:** ```yaml @@ -200,7 +200,7 @@ models: ``` #### not_constant ([source](macros/generic_tests/not_constant.sql)) -Asserts if a column does not have the same value in all rows. +Asserts that a column does not have the same value in all rows. **Usage:** ```yaml @@ -215,7 +215,7 @@ models: ``` #### cardinality_equality ([source](macros/generic_tests/cardinality_equality.sql)) -Asserts if values in a given column have exactly the same cardinality as values from a different column in a different model. +Asserts that values in a given column have exactly the same cardinality as values from a different column in a different model. **Usage:** ```yaml From 090ad35c519076c37296ac801f0bebd6b1e06b92 Mon Sep 17 00:00:00 2001 From: Joel Labes Date: Mon, 28 Mar 2022 15:14:24 +1300 Subject: [PATCH 10/11] Update CHANGELOG.md --- CHANGELOG.md | 1 + 1 file changed, 1 insertion(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 58b3805b..366a4964 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,7 @@ ## Fixes - Fix union_relations error from [#473](https://github.com/dbt-labs/dbt-utils/pull/473) when no include/exclude parameters are provided ([#505](https://github.com/dbt-labs/dbt-utils/issues/505), [#509](https://github.com/dbt-labs/dbt-utils/pull/509)) +# dbt-utils v0.8.1 ## New features - A cross-database implementation of `any_value()` ([#497](https://github.com/dbt-labs/dbt-utils/issues/497), [#501](https://github.com/dbt-labs/dbt-utils/pull/501)) - A cross-database implementation of `bool_or()` ([#504](https://github.com/dbt-labs/dbt-utils/pull/504)) From 460d528c460d3575295be811da2f96c164ecab92 Mon Sep 17 00:00:00 2001 From: Joel Labes Date: Mon, 28 Mar 2022 15:14:30 +1300 Subject: [PATCH 11/11] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 143c80cb..c9caa8cc 100644 --- a/README.md +++ b/README.md @@ -71,7 +71,7 @@ For compatibility details between versions of dbt-core and dbt-utils, [see this ---- ### Generic Tests #### equal_rowcount ([source](macros/generic_tests/equal_rowcount.sql)) -Asserts the that two relations have the same number of rows. +Asserts that two relations have the same number of rows. **Usage:** ```yaml