Merge pull request #9 from fivetran/feature/unioned-data

Feature/unioned data with fivetran utils macro

CHANGELOG.md

@@ -0,0 +1,17 @@
+# dbt_klaviyo_source v0.3.0
+
+## Features
+- Allow for multiple sources by unioning source tables across multiple Klaviyo connectors.
+([#8](https://github.com/fivetran/dbt_klaviyo_source/pull/8) & [#9](https://github.com/fivetran/dbt_klaviyo_source/pull/9))
+  - Refer to the [README](https://github.com/fivetran/dbt_klaviyo_source#unioning-multiple-klaviyo-connectors) for more details.
+
+## Under the Hood
+- Unioning: The unioning occurs in the tmp models using the `fivetran_utils.union_data` macro. ([#8](https://github.com/fivetran/dbt_klaviyo_source/pull/8))
+- Unique tests: Because columns that were previously used for unique tests may now have duplicate fields across multiple sources, these columns are combined with the new `source_relation` column for unique tests and tested using the `dbt_utils.unique_combination_of_columns` macro. ([#8](https://github.com/fivetran/dbt_klaviyo_source/pull/8))
+- Source Relation column: To distinguish which source each field comes from, we added a new `source_relation` column in each staging model and applied the `fivetran_utils.source_relation` macro. ([#8](https://github.com/fivetran/dbt_klaviyo_source/pull/8))
+
+## Contributors
+- [@pawelngei](https://github.com/pawelngei) [#8](https://github.com/fivetran/dbt_klaviyo_source/pull/8)
+
+# dbt_klaviyo_source v0.1.0 -> v0.3.0
+Refer to the relevant release notes on the Github repository for specific details for the previous releases. Thank you!

README.md

@@ -26,7 +26,7 @@ Include in your `packages.yml`
 ```yaml
 packages:
   - package: fivetran/klaviyo_source
-    version: [">=0.2.0", "<0.3.0"]
+    version: [">=0.3.0", "<0.4.0"]
 ```
 
 ## Configuration
@@ -44,6 +44,19 @@ vars:
     klaviyo_schema: your_schema_name 
 ```
 
+### Unioning Multiple Klaviyo Connectors
+If you have multiple Klaviyo connectors in Fivetran and would like to use this package on all of them simultaneously, we have provided functionality to do so. The package will union all of the data together and pass the unioned table into the transformations. You will be able to see which source it came from in the `source_relation` column of each model. To use this functionality, you will need to set either (**note that you cannot use both**) the `union_schemas` or `union_databases` variables:
+
+```yml
+# dbt_project.yml
+...
+config-version: 2
+vars:
+  klaviyo_source:
+    union_schemas: ['klaviyo_usa','klaviyo_canada'] # use this if the data is in different schemas/datasets of the same database/project
+    union_databases: ['klaviyo_usa','klaviyo_canada'] # use this if the data is in different databases/projects but uses the same schema name
+```
+
 ### Passthrough Columns
 
 Additionally, this package includes all source columns defined in the macros folder. We highly recommend including custom fields in this package as models now only bring in the standard fields for the `EVENT` and `PERSON` tables. You can add more columns using our passthrough column variables. These variables allow the passthrough fields to be aliased (`alias`) and casted (`transform_sql`) if desired, although it is not required. Datatype casting is configured via a SQL snippet within the `transform_sql` key. You may add the desired SQL snippet while omitting the `as field_name` part of the casting statement - this will be dealt with by the alias attribute - and your custom passthrough fields will be casted accordingly.

dbt_project.yml

@@ -1,5 +1,5 @@
 name: 'klaviyo_source'
-version: '0.2.0'
+version: '0.3.0'
 config-version: 2
 require-dbt-version: ">=0.20.0"
 

docs/catalog.json

@@ -0,0 +1 @@
+{"metadata": {"dbt_schema_version": "https://schemas.getdbt.com/dbt/catalog/v1.json", "dbt_version": "0.21.0", "generated_at": "2021-10-27T17:26:44.966660Z", "invocation_id": "126124d0-113d-4d78-955f-29e3a017afdf", "env": {}}, "nodes": {"seed.klaviyo_source_integration_tests.event": {"metadata": {"type": "BASE TABLE", "schema": "DBT_JAMIE", "name": "EVENT", "database": "DBT_PACKAGE_TESTING", "comment": null, "owner": "DBT_PACKAGE"}, "columns": {"ACCOUNT_ID": {"type": "TEXT", "index": 1, "name": "ACCOUNT_ID", "comment": null}, "REMOTE_IP": {"type": "TEXT", "index": 2, "name": "REMOTE_IP", "comment": null}, "SERVER_NAME": {"type": "TEXT", "index": 3, "name": "SERVER_NAME", "comment": null}, "TIMESTAMP": {"type": "TIMESTAMP_NTZ", "index": 4, "name": "TIMESTAMP", "comment": null}, "USER_AGENT": {"type": "TEXT", "index": 5, "name": "USER_AGENT", "comment": null}, "VISITOR_ID": {"type": "TEXT", "index": 6, "name": "VISITOR_ID", "comment": null}, "_FIVETRAN_SYNCED": {"type": "TIMESTAMP_NTZ", "index": 7, "name": "_FIVETRAN_SYNCED", "comment": null}, "APP_ID": {"type": "NUMBER", "index": 8, "name": "APP_ID", "comment": null}, "NUM_EVENTS": {"type": "NUMBER", "index": 9, "name": "NUM_EVENTS", "comment": null}, "NUM_MINUTES": {"type": "NUMBER", "index": 10, "name": "NUM_MINUTES", "comment": null}, "PAGE_ID": {"type": "TEXT", "index": 11, "name": "PAGE_ID", "comment": null}}, "stats": {"bytes": {"id": "bytes", "label": "Approximate Size", "value": 4096.0, "include": true, "description": "Approximate size of the table as reported by Snowflake"}, "last_modified": {"id": "last_modified", "label": "Last Modified", "value": "2021-10-18 20:04UTC", "include": true, "description": "The timestamp for last update/change"}, "row_count": {"id": "row_count", "label": "Row Count", "value": 5.0, "include": true, "description": "An approximate count of rows in this table"}, "has_stats": {"id": "has_stats", "label": "Has Stats?", "value": true, "include": false, "description": "Indicates whether there are statistics for this table"}}, "unique_id": "seed.klaviyo_source_integration_tests.event"}}, "sources": {}, "errors": null}

docs/run_results.json

@@ -0,0 +1 @@
+{"metadata": {"dbt_schema_version": "https://schemas.getdbt.com/dbt/run-results/v3.json", "dbt_version": "0.21.0", "generated_at": "2021-10-27T17:26:29.603172Z", "invocation_id": "126124d0-113d-4d78-955f-29e3a017afdf", "env": {}}, "results": [{"status": "success", "timing": [{"name": "compile", "started_at": "2021-10-27T17:26:24.881701Z", "completed_at": "2021-10-27T17:26:24.884277Z"}, {"name": "execute", "started_at": "2021-10-27T17:26:24.884498Z", "completed_at": "2021-10-27T17:26:24.884512Z"}], "thread_id": "Thread-1", "execution_time": 0.003580808639526367, "adapter_response": {}, "message": null, "failures": null, "unique_id": "seed.klaviyo_source_integration_tests.campaign"}, {"status": "success", "timing": [{"name": "compile", "started_at": "2021-10-27T17:26:24.885662Z", "completed_at": "2021-10-27T17:26:24.887951Z"}, {"name": "execute", "started_at": "2021-10-27T17:26:24.888154Z", "completed_at": "2021-10-27T17:26:24.888163Z"}], "thread_id": "Thread-1", "execution_time": 0.003165006637573242, "adapter_response": {}, "message": null, "failures": null, "unique_id": "seed.klaviyo_source_integration_tests.event"}, {"status": "success", "timing": [{"name": "compile", "started_at": "2021-10-27T17:26:24.889227Z", "completed_at": "2021-10-27T17:26:24.891288Z"}, {"name": "execute", "started_at": "2021-10-27T17:26:24.891484Z", "completed_at": "2021-10-27T17:26:24.891492Z"}], "thread_id": "Thread-1", "execution_time": 0.0029058456420898438, "adapter_response": {}, "message": null, "failures": null, "unique_id": "seed.klaviyo_source_integration_tests.flow_snowflake"}, {"status": "success", "timing": [{"name": "compile", "started_at": "2021-10-27T17:26:24.892506Z", "completed_at": "2021-10-27T17:26:24.894471Z"}, {"name": "execute", "started_at": "2021-10-27T17:26:24.894652Z", "completed_at": "2021-10-27T17:26:24.894660Z"}], "thread_id": "Thread-1", "execution_time": 0.002759218215942383, "adapter_response": {}, "message": null, "failures": null, "unique_id": "seed.klaviyo_source_integration_tests.integration"}, {"status": "success", "timing": [{"name": "compile", "started_at": "2021-10-27T17:26:24.895610Z", "completed_at": "2021-10-27T17:26:24.897406Z"}, {"name": "execute", "started_at": "2021-10-27T17:26:24.897767Z", "completed_at": "2021-10-27T17:26:24.897775Z"}], "thread_id": "Thread-1", "execution_time": 0.0027527809143066406, "adapter_response": {}, "message": null, "failures": null, "unique_id": "seed.klaviyo_source_integration_tests.metric"}, {"status": "success", "timing": [{"name": "compile", "started_at": "2021-10-27T17:26:24.898727Z", "completed_at": "2021-10-27T17:26:24.900488Z"}, {"name": "execute", "started_at": "2021-10-27T17:26:24.900652Z", "completed_at": "2021-10-27T17:26:24.900659Z"}], "thread_id": "Thread-1", "execution_time": 0.0024950504302978516, "adapter_response": {}, "message": null, "failures": null, "unique_id": "seed.klaviyo_source_integration_tests.person"}, {"status": "success", "timing": [{"name": "compile", "started_at": "2021-10-27T17:26:24.901903Z", "completed_at": "2021-10-27T17:26:24.908640Z"}, {"name": "execute", "started_at": "2021-10-27T17:26:24.908903Z", "completed_at": "2021-10-27T17:26:24.908913Z"}], "thread_id": "Thread-1", "execution_time": 0.007647037506103516, "adapter_response": {}, "message": null, "failures": null, "unique_id": "model.klaviyo_source.stg_klaviyo__campaign_tmp"}, {"status": "success", "timing": [{"name": "compile", "started_at": "2021-10-27T17:26:24.910067Z", "completed_at": "2021-10-27T17:26:24.914370Z"}, {"name": "execute", "started_at": "2021-10-27T17:26:24.914577Z", "completed_at": "2021-10-27T17:26:24.914586Z"}], "thread_id": "Thread-1", "execution_time": 0.0050699710845947266, "adapter_response": {}, "message": null, "failures": null, "unique_id": "model.klaviyo_source.stg_klaviyo__event_tmp"}, {"status": "success", "timing": [{"name": "compile", "started_at": "2021-10-27T17:26:24.915443Z", "completed_at": "2021-10-27T17:26:24.920134Z"}, {"name": "execute", "started_at": "2021-10-27T17:26:24.920338Z", "completed_at": "2021-10-27T17:26:24.920346Z"}], "thread_id": "Thread-1", "execution_time": 0.005439043045043945, "adapter_response": {}, "message": null, "failures": null, "unique_id": "model.klaviyo_source.stg_klaviyo__flow_tmp"}, {"status": "success", "timing": [{"name": "compile", "started_at": "2021-10-27T17:26:24.921276Z", "completed_at": "2021-10-27T17:26:24.927542Z"}, {"name": "execute", "started_at": "2021-10-27T17:26:24.927836Z", "completed_at": "2021-10-27T17:26:24.927847Z"}], "thread_id": "Thread-1", "execution_time": 0.007193088531494141, "adapter_response": {}, "message": null, "failures": null, "unique_id": "model.klaviyo_source.stg_klaviyo__integration_tmp"}, {"status": "success", "timing": [{"name": "compile", "started_at": "2021-10-27T17:26:24.928867Z", "completed_at": "2021-10-27T17:26:24.933850Z"}, {"name": "execute", "started_at": "2021-10-27T17:26:24.934160Z", "completed_at": "2021-10-27T17:26:24.934173Z"}], "thread_id": "Thread-1", "execution_time": 0.005964040756225586, "adapter_response": {}, "message": null, "failures": null, "unique_id": "model.klaviyo_source.stg_klaviyo__metric_tmp"}, {"status": "success", "timing": [{"name": "compile", "started_at": "2021-10-27T17:26:24.935276Z", "completed_at": "2021-10-27T17:26:24.940876Z"}, {"name": "execute", "started_at": "2021-10-27T17:26:24.941191Z", "completed_at": "2021-10-27T17:26:24.941202Z"}], "thread_id": "Thread-1", "execution_time": 0.006639003753662109, "adapter_response": {}, "message": null, "failures": null, "unique_id": "model.klaviyo_source.stg_klaviyo__person_tmp"}, {"status": "success", "timing": [{"name": "compile", "started_at": "2021-10-27T17:26:24.942470Z", "completed_at": "2021-10-27T17:26:25.524030Z"}, {"name": "execute", "started_at": "2021-10-27T17:26:25.524516Z", "completed_at": "2021-10-27T17:26:25.524537Z"}], "thread_id": "Thread-1", "execution_time": 0.7337300777435303, "adapter_response": {}, "message": null, "failures": null, "unique_id": "model.klaviyo_source.stg_klaviyo__campaign"}, {"status": "success", "timing": [{"name": "compile", "started_at": "2021-10-27T17:26:25.677173Z", "completed_at": "2021-10-27T17:26:26.254177Z"}, {"name": "execute", "started_at": "2021-10-27T17:26:26.254569Z", "completed_at": "2021-10-27T17:26:26.254588Z"}], "thread_id": "Thread-1", "execution_time": 0.7620787620544434, "adapter_response": {}, "message": null, "failures": null, "unique_id": "model.klaviyo_source.stg_klaviyo__event"}, {"status": "success", "timing": [{"name": "compile", "started_at": "2021-10-27T17:26:26.440142Z", "completed_at": "2021-10-27T17:26:27.192829Z"}, {"name": "execute", "started_at": "2021-10-27T17:26:27.193099Z", "completed_at": "2021-10-27T17:26:27.193115Z"}], "thread_id": "Thread-1", "execution_time": 0.8928189277648926, "adapter_response": {}, "message": null, "failures": null, "unique_id": "model.klaviyo_source.stg_klaviyo__flow"}, {"status": "success", "timing": [{"name": "compile", "started_at": "2021-10-27T17:26:27.334005Z", "completed_at": "2021-10-27T17:26:27.908608Z"}, {"name": "execute", "started_at": "2021-10-27T17:26:27.909092Z", "completed_at": "2021-10-27T17:26:27.909115Z"}], "thread_id": "Thread-1", "execution_time": 0.7567088603973389, "adapter_response": {}, "message": null, "failures": null, "unique_id": "model.klaviyo_source.stg_klaviyo__integration"}, {"status": "success", "timing": [{"name": "compile", "started_at": "2021-10-27T17:26:28.091458Z", "completed_at": "2021-10-27T17:26:28.682874Z"}, {"name": "execute", "started_at": "2021-10-27T17:26:28.683141Z", "completed_at": "2021-10-27T17:26:28.683157Z"}], "thread_id": "Thread-1", "execution_time": 0.7392308712005615, "adapter_response": {}, "message": null, "failures": null, "unique_id": "model.klaviyo_source.stg_klaviyo__metric"}, {"status": "success", "timing": [{"name": "compile", "started_at": "2021-10-27T17:26:28.831491Z", "completed_at": "2021-10-27T17:26:29.380376Z"}, {"name": "execute", "started_at": "2021-10-27T17:26:29.380637Z", "completed_at": "2021-10-27T17:26:29.380653Z"}], "thread_id": "Thread-1", "execution_time": 0.7060751914978027, "adapter_response": {}, "message": null, "failures": null, "unique_id": "model.klaviyo_source.stg_klaviyo__person"}, {"status": "success", "timing": [{"name": "compile", "started_at": "2021-10-27T17:26:29.538395Z", "completed_at": "2021-10-27T17:26:29.550943Z"}, {"name": "execute", "started_at": "2021-10-27T17:26:29.551214Z", "completed_at": "2021-10-27T17:26:29.551227Z"}], "thread_id": "Thread-1", "execution_time": 0.013705968856811523, "adapter_response": {}, "message": null, "failures": null, "unique_id": "test.klaviyo_source.dbt_utils_unique_combination_of_columns_stg_klaviyo__campaign_campaign_id__source_relation.59158488ff"}, {"status": "success", "timing": [{"name": "compile", "started_at": "2021-10-27T17:26:29.552409Z", "completed_at": "2021-10-27T17:26:29.558837Z"}, {"name": "execute", "started_at": "2021-10-27T17:26:29.559055Z", "completed_at": "2021-10-27T17:26:29.559064Z"}], "thread_id": "Thread-1", "execution_time": 0.007339000701904297, "adapter_response": {}, "message": null, "failures": null, "unique_id": "test.klaviyo_source.dbt_utils_unique_combination_of_columns_stg_klaviyo__event_event_id__source_relation.3778c651d7"}, {"status": "success", "timing": [{"name": "compile", "started_at": "2021-10-27T17:26:29.560123Z", "completed_at": "2021-10-27T17:26:29.566838Z"}, {"name": "execute", "started_at": "2021-10-27T17:26:29.567065Z", "completed_at": "2021-10-27T17:26:29.567073Z"}], "thread_id": "Thread-1", "execution_time": 0.0075609683990478516, "adapter_response": {}, "message": null, "failures": null, "unique_id": "test.klaviyo_source.dbt_utils_unique_combination_of_columns_stg_klaviyo__flow_flow_id__source_relation.015215d481"}, {"status": "success", "timing": [{"name": "compile", "started_at": "2021-10-27T17:26:29.568153Z", "completed_at": "2021-10-27T17:26:29.573622Z"}, {"name": "execute", "started_at": "2021-10-27T17:26:29.573829Z", "completed_at": "2021-10-27T17:26:29.573837Z"}], "thread_id": "Thread-1", "execution_time": 0.006515026092529297, "adapter_response": {}, "message": null, "failures": null, "unique_id": "test.klaviyo_source.dbt_utils_unique_combination_of_columns_stg_klaviyo__integration_integration_id__source_relation.be6158ad21"}, {"status": "success", "timing": [{"name": "compile", "started_at": "2021-10-27T17:26:29.574876Z", "completed_at": "2021-10-27T17:26:29.579917Z"}, {"name": "execute", "started_at": "2021-10-27T17:26:29.580092Z", "completed_at": "2021-10-27T17:26:29.580100Z"}], "thread_id": "Thread-1", "execution_time": 0.005747079849243164, "adapter_response": {}, "message": null, "failures": null, "unique_id": "test.klaviyo_source.dbt_utils_unique_combination_of_columns_stg_klaviyo__metric_metric_id__source_relation.e9f33c04e5"}, {"status": "success", "timing": [{"name": "compile", "started_at": "2021-10-27T17:26:29.581001Z", "completed_at": "2021-10-27T17:26:29.586062Z"}, {"name": "execute", "started_at": "2021-10-27T17:26:29.586232Z", "completed_at": "2021-10-27T17:26:29.586239Z"}], "thread_id": "Thread-1", "execution_time": 0.005776166915893555, "adapter_response": {}, "message": null, "failures": null, "unique_id": "test.klaviyo_source.dbt_utils_unique_combination_of_columns_stg_klaviyo__person_person_id__source_relation.33a4f9ca24"}, {"status": "success", "timing": [{"name": "compile", "started_at": "2021-10-27T17:26:29.587135Z", "completed_at": "2021-10-27T17:26:29.593621Z"}, {"name": "execute", "started_at": "2021-10-27T17:26:29.593794Z", "completed_at": "2021-10-27T17:26:29.593801Z"}], "thread_id": "Thread-1", "execution_time": 0.007213115692138672, "adapter_response": {}, "message": null, "failures": null, "unique_id": "test.klaviyo_source.not_null_stg_klaviyo__person_email.c7094cfe27"}, {"status": "success", "timing": [{"name": "compile", "started_at": "2021-10-27T17:26:29.594636Z", "completed_at": "2021-10-27T17:26:29.601166Z"}, {"name": "execute", "started_at": "2021-10-27T17:26:29.601339Z", "completed_at": "2021-10-27T17:26:29.601347Z"}], "thread_id": "Thread-1", "execution_time": 0.007214069366455078, "adapter_response": {}, "message": null, "failures": null, "unique_id": "test.klaviyo_source.unique_stg_klaviyo__person_email.dc3a98b014"}], "elapsed_time": 7.070431232452393, "args": {"log_format": "default", "write_json": true, "use_experimental_parser": false, "profiles_dir": "/Users/jamie.rodriguez/.dbt", "use_cache": true, "compile": true, "version_check": true, "which": "generate", "rpc_method": "docs.generate"}}

integration_tests/dbt_project.yml

@@ -1,5 +1,5 @@
 name: 'klaviyo_source_integration_tests'
-version: '0.2.0'
+version: '0.3.0'
 config-version: 2
 profile: 'integration_tests'