Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Model contract configs #59

Merged
merged 15 commits into from
Apr 4, 2023
Merged

Model contract configs #59

Show file tree
Hide file tree
Changes from 13 commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
1ab6d12
initial config in yml entry
dave-connors-3 Mar 6, 2023
f46347a
current model contract spec
dave-connors-3 Mar 6, 2023
0b90473
remove description from groups for now
dave-connors-3 Mar 6, 2023
8793d92
groups for metrics
dave-connors-3 Mar 6, 2023
49a64d4
change everything to defs
dave-connors-3 Mar 7, 2023
f6affa2
Revert "change everything to defs"
dave-connors-3 Mar 7, 2023
46b76ce
Merge branch 'main' into model-contract-configs
Mar 14, 2023
3ac6545
allow contract config in project yml
Mar 14, 2023
f9d25af
resolve conflicts
dave-connors-3 Mar 30, 2023
830c7ae
committing to view diff
dave-connors-3 Mar 30, 2023
f87468d
model version, model level contracts
dave-connors-3 Mar 30, 2023
4204ed4
add require type
dave-connors-3 Mar 30, 2023
a350bde
refs to ref
dave-connors-3 Mar 30, 2023
a6538db
multi-line, unify constraints dict, remove constraints_check
dave-connors-3 Mar 31, 2023
72bce91
update contract in project yml
dave-connors-3 Mar 31, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions schemas/dbt_project.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -254,6 +254,9 @@
"$ref": "#/$defs/boolean_or_jinja_string",
"default": false
},
"+contract": {
"$ref": "#/$defs/boolean_or_jinja_string"
},
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

contract is an object now, containing a single bool enforced (dbt-labs/dbt-core#7184, dbt-labs/dbt-core#7222)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yep, this should be deleted. Per the tests in that second PR the contract object is nested in config and i added it to the model_config block down below

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh this is in the project yml file! will update!

"+database": {
"$ref": "#/$defs/database"
},
Expand Down Expand Up @@ -309,6 +312,9 @@
"$ref": "#/$defs/boolean_or_jinja_string",
"default": false
},
"contract": {
"$ref": "#/$defs/boolean_or_jinja_string"
},
"database": {
"$ref": "#/$defs/database"
},
Expand Down
214 changes: 169 additions & 45 deletions schemas/dbt_yml_files.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -115,6 +115,32 @@
}
}
},
"groups": {
"type": "array",
"items": {
"type": "object",
"required": ["name", "owner"],
"properties": {
"name": {
"type": "string"
},
"owner": {
"type": "object",
"minProperties": 1,
dave-connors-3 marked this conversation as resolved.
Show resolved Hide resolved
"properties": {
"name": {
"type": "string"
},
"email": {
"type": "string"
}
},
"additionalProperties": false
}
},
"additionalProperties": false
}
},
"macros": {
"type": "array",
"items": {
Expand Down Expand Up @@ -183,6 +209,20 @@
]
},
"properties": {
"config": {
"type": "object",
"properties": {
"enabled": {
"$ref": "#/$defs/boolean_or_jinja_string"
},
"treat_null_values_as_zero": {
"$ref": "#/$defs/boolean_or_jinja_string"
},
"group": {
"type": "string"
}
}
},
"calculation_method": {
"type": "string"
},
Expand Down Expand Up @@ -246,62 +286,45 @@
"name": {
"type": "string"
},
"access": {
"type": "string",
"enum": ["private", "protected", "public"]
},
"columns": {
"type": "array",
"items": {
"$ref": "#/$defs/column_properties"
}
},
"config": {
"type": "object",
"properties": {
"grant_access_to": {
"title": "Authorized views",
"type": "array",
"description": "Configuration, specific to BigQuery adapter, used to setup authorized views.",
"items": {
"type": "object",
"required": ["database", "project"],
"properties": {
"database": {
"type": "string"
},
"project": {
"type": "string"
}
},
"additionalProperties": false
}
},
"hours_to_expiration": {
"type": "number",
"description": "Configuration specific to BigQuery adapter used to set an expiration delay (in hours) to a table."
},
"kms_key_name": {
"type": "string",
"description": "Configuration of the KMS key name, specific to BigQuery adapter.",
"pattern": "projects/[a-zA-Z0-9_-]*/locations/[a-zA-Z0-9_-]*/keyRings/.*/cryptoKeys/.*"
},
"labels": {
"title": "Label configs",
"type": "object",
"description": "Configuration specific to BigQuery adapter used to add labels and tags to tables/views created by dbt.",
"patternProperties": {
"^[a-z][a-z0-9_-]{0,62}$": {
"type": "string",
"pattern": "^[a-z0-9_-]{0,63}$"
}
"$ref": "#/$defs/model_configs"
},
"constraints": {
"type": "array",
"items": {
"type": "object",
"required": [
"type"
],
"properties": {
"columns": {
"$ref": "#/$defs/string_or_array_of_strings"
},
"additionalProperties": false
},
"materialized": {
"type": "string"
},
"sql_header": {
"type": "string"
"expression": { "type": "string" },
"type": { "type": "string" },
"name": { "type": "string" },
"warn_unenforced": {
"$ref": "#/$defs/boolean_or_jinja_string"
},
"warn_unsupported": {
"$ref": "#/$defs/boolean_or_jinja_string"
}
}
}
},
"contract": {
"$ref": "#/$defs/boolean_or_jinja_string"
},
"description": {
"type": "string"
},
Expand All @@ -313,6 +336,12 @@
}
}
},
"group": {
"type": "string"
},
"latest_version": {
"type": "number"
},
"meta": {
"type": "object"
},
Expand All @@ -321,6 +350,26 @@
"items": {
"$ref": "#/$defs/tests"
}
},
"versions": {
"type": "array",
"items": {
"type": "object",
"required": ["v"],
"properties": {
"v": { "type": "number" },
dave-connors-3 marked this conversation as resolved.
Show resolved Hide resolved
"config": { "$ref": "#/$defs/model_configs" },
"columns": {
"type": "array",
"items": {
"anyOf": [
{ "$ref": "#/$defs/include_exclude" },
dave-connors-3 marked this conversation as resolved.
Show resolved Hide resolved
{ "$ref": "#/$defs/column_properties" }
]
}
}
}
}
}
},
"additionalProperties": false
Expand Down Expand Up @@ -634,6 +683,12 @@
"name": {
"type": "string"
},
"constraints": {
"$ref": "#/$defs/array_of_strings"
},
"constraints_check": {
dave-connors-3 marked this conversation as resolved.
Show resolved Hide resolved
"type": "string"
},
"data_type": {
"type": "string"
},
Expand Down Expand Up @@ -712,10 +767,79 @@
},
"additionalProperties": false
},
"include_exclude": {
"type": "object",
"properties": {
"include": {
"$ref": "#/$defs/string_or_array_of_strings"
},
"exclude": {
"$ref": "#/$defs/string_or_array_of_strings"
}
}
},
"jinja_string": {
"type": "string",
"pattern": "\\{\\{.*\\}\\}"
},
"model_configs": {
"type": "object",
"properties": {
"contract": {
"type": "object",
"properties": {
"enforced": {
"$ref": "#/$defs/boolean_or_jinja_string"
}
}
},
"grant_access_to": {
"title": "Authorized views",
"type": "array",
"description": "Configuration, specific to BigQuery adapter, used to setup authorized views.",
"items": {
"type": "object",
"required": ["database", "project"],
"properties": {
"database": {
"type": "string"
},
"project": {
"type": "string"
}
},
"additionalProperties": false
}
},
"hours_to_expiration": {
"type": "number",
"description": "Configuration specific to BigQuery adapter used to set an expiration delay (in hours) to a table."
},
"kms_key_name": {
"type": "string",
"description": "Configuration of the KMS key name, specific to BigQuery adapter.",
"pattern": "projects/[a-zA-Z0-9_-]*/locations/[a-zA-Z0-9_-]*/keyRings/.*/cryptoKeys/.*"
},
"labels": {
"title": "Label configs",
"type": "object",
"description": "Configuration specific to BigQuery adapter used to add labels and tags to tables/views created by dbt.",
"patternProperties": {
"^[a-z][a-z0-9_-]{0,62}$": {
"type": "string",
"pattern": "^[a-z0-9_-]{0,63}$"
}
},
"additionalProperties": false
},
"materialized": {
"type": "string"
},
"sql_header": {
"type": "string"
}
}
},
dave-connors-3 marked this conversation as resolved.
Show resolved Hide resolved
"number_or_jinja_string": {
"oneOf": [
{
Expand Down
53 changes: 52 additions & 1 deletion tests/valid/schema.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,50 @@ models:
- unique
- not_null

- name: my_contracted_dbt_model
description: "A dbt model with contracts"
access: public
group: analytics
latest_version: 2
config:
contract:
enforced: true
columns:
- name: id
description: "The primary key for this table"
data_type: int
constraints:
- "not null"
- "unique"
constraints_check: (id > 0)
tests:
- unique
- not_null
versions:
- v: 2
columns:
- include: '*'
exclude: country_name
- name: id # included in addition the '*' set. if customer_id were in the '*' set -> override it
dave-connors-3 marked this conversation as resolved.
Show resolved Hide resolved
description: This is the primary key
data_type: float
- v: 1
config:
alias: dim_customers

- name: my_model_level_contract_model
config:
contract:
enforced: true
constraints:
- type: check
expression: (id > 0)
- type: primary_key
columns: [ id ]
- type: unique
columns: [ color, date_day ]
name: strange_uniqueness_requirement


snapshots:
- name: snapshot_name
Expand All @@ -28,6 +72,13 @@ snapshots:
description: cool column, eh?


# model groups

groups:
- name: analytics
owner:
name: dave

metrics:
- name: new_customers
label: New Customers marked 'paying'
Expand Down Expand Up @@ -61,4 +112,4 @@ metrics:
description: "The number of paid customers using the product"

calculation_method: derived
expression: "{{ metric('new_customers') }} * 2"
expression: "{{ metric('new_customers') }} * 2"