Skip to content

Commit

Permalink
Merge pull request #18 from fivetran/sept_2023_updates
Browse files Browse the repository at this point in the history
Sept 2023 updates
  • Loading branch information
fivetran-reneeli authored Nov 1, 2023
2 parents de69406 + c4d8486 commit acee59c
Show file tree
Hide file tree
Showing 27 changed files with 110 additions and 172 deletions.
33 changes: 33 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,36 @@
# dbt_klaviyo_source v0.7.0
[PR #18](https://github.com/fivetran/dbt_klaviyo_source/pull/18/) includes updates regarding the [September 2023](https://fivetran.com/docs/applications/klaviyo/changelog#september2023) changes to the Klaviyo connector.

## 🚨 Breaking Changes 🚨:

- As the `integration` table has been deprecated, we removed the `stg_klaviyo__integration` model, and instead have passed the integration columns through `stg_klaviyo__metric`.

- We have removed these deprecated columns from the following tables:

| **Table** | **Column** |
|--------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| CAMPAIGN | is_segmented
| FLOW | customer_filter
| FLOW | trigger

- We have added these columns from the following tables and renamed them:

| **Table** | **Column** | **New Name** |
| :--- | :----: | ---: |
| CAMPAIGN | archived | is_archived
| CAMPAIGN | scheduled | scheduled_at
| FLOW | archived | is_archived
| FLOW | trigger_type | trigger_type
| PERSON | last_event_date | last_event_date
| METRIC | integration_name | integration_name
| METRIC | integration_category | integration_category

For more information on the fields, refer to [our docs](https://fivetran.github.io/dbt_klaviyo_source/#!/model/model.klaviyo_source).

## Under the Hood:
- We removed the Snowflake-specific logic in place for passing through the `trigger` field in the Flow table as it was a reserved word. Now that the `trigger` field has been deprecated, we have also removed the associated logic in the package.
- We removed the not_null test for `email` in the `stg_klaviyo__person` model. This is because for the most recent schema, the only primary key is `person_id` rather than `email`, and `email` may not be present.

# dbt_klaviyo_source v0.6.0
## 🚨 Breaking Changes 🚨:
- This change is made breaking due to impact on incremental models in the downstream transformation package.
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ Include the following klaviyo_source package version in your `packages.yml` file
```yaml
packages:
- package: fivetran/klaviyo_source
version: [">=0.6.0", "<0.7.0"]
version: [">=0.7.0", "<0.8.0"]
```
## Step 3: Define database and schema variables
By default, this package runs using your destination and the `klaviyo` schema. If this is not where your Klaviyo data is (for example, if your Klaviyo schema is named `klaviyo_fivetran`), add the following configuration to your root `dbt_project.yml` file:
Expand Down
2 changes: 1 addition & 1 deletion dbt_project.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
name: 'klaviyo_source'
version: '0.6.0'
version: '0.7.0'
config-version: 2
require-dbt-version: [">=1.0.0", "<2.0.0"]

Expand Down
2 changes: 1 addition & 1 deletion docs/catalog.json

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion docs/manifest.json

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion docs/run_results.json

Large diffs are not rendered by default.

10 changes: 5 additions & 5 deletions integration_tests/ci/sample.profiles.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,13 +16,13 @@ integration_tests:
pass: "{{ env_var('CI_REDSHIFT_DBT_PASS') }}"
dbname: "{{ env_var('CI_REDSHIFT_DBT_DBNAME') }}"
port: 5439
schema: klaviyo_source_integration_tests_1
schema: klaviyo_source_integration_tests_2
threads: 8
bigquery:
type: bigquery
method: service-account-json
project: 'dbt-package-testing'
schema: klaviyo_source_integration_tests_1
schema: klaviyo_source_integration_tests_2
threads: 8
keyfile_json: "{{ env_var('GCLOUD_SERVICE_KEY') | as_native }}"
snowflake:
Expand All @@ -33,7 +33,7 @@ integration_tests:
role: "{{ env_var('CI_SNOWFLAKE_DBT_ROLE') }}"
database: "{{ env_var('CI_SNOWFLAKE_DBT_DATABASE') }}"
warehouse: "{{ env_var('CI_SNOWFLAKE_DBT_WAREHOUSE') }}"
schema: klaviyo_source_integration_tests_1
schema: klaviyo_source_integration_tests_2
threads: 8
postgres:
type: postgres
Expand All @@ -42,13 +42,13 @@ integration_tests:
pass: "{{ env_var('CI_POSTGRES_DBT_PASS') }}"
dbname: "{{ env_var('CI_POSTGRES_DBT_DBNAME') }}"
port: 5432
schema: klaviyo_source_integration_tests_1
schema: klaviyo_source_integration_tests_2
threads: 8
databricks:
catalog: "{{ env_var('CI_DATABRICKS_DBT_CATALOG') }}"
host: "{{ env_var('CI_DATABRICKS_DBT_HOST') }}"
http_path: "{{ env_var('CI_DATABRICKS_DBT_HTTP_PATH') }}"
schema: klaviyo_source_integration_tests_1
schema: klaviyo_source_integration_tests_2
threads: 8
token: "{{ env_var('CI_DATABRICKS_DBT_TOKEN') }}"
type: databricks
13 changes: 4 additions & 9 deletions integration_tests/dbt_project.yml
Original file line number Diff line number Diff line change
@@ -1,19 +1,19 @@
name: 'klaviyo_source_integration_tests'
version: '0.6.0'
version: '0.7.0'
config-version: 2
profile: 'integration_tests'


vars:
klaviyo_campaign_identifier: "campaign"
klaviyo_event_identifier: "event"
flow: "{{ fivetran_utils.snowflake_seed_data('flow') }}"
klaviyo_flow_identifier: "flow"
klaviyo_integration_identifier: "integration"
klaviyo_person_identifier: "person"
klaviyo_metric_identifier: "metric"
klaviyo__event_pass_through_columns: []
klaviyo__person_pass_through_columns: []
klaviyo_schema: klaviyo_source_integration_tests_1
klaviyo_schema: klaviyo_source_integration_tests_2

seeds:
klaviyo_source_integration_tests:
Expand All @@ -24,9 +24,4 @@ seeds:
+column_types:
PHONE_NUMBER: "{{ 'string' if target.type in ('bigquery', 'spark', 'databricks') else 'varchar' }}"
flow:
+enabled: "{{ true if target.type != 'snowflake' else false }}"
+quote_columns: "{{ true if target.type == 'redshift' else false }}"
+column_types:
TRIGGER: "{{ 'string' if target.type in ('bigquery', 'spark', 'databricks') else 'varchar' }}"
flow_snowflake:
+enabled: "{{ true if target.type == 'snowflake' else false }}"
+quote_columns: "{{ true if target.type == 'redshift' else false }}"
6 changes: 3 additions & 3 deletions integration_tests/seeds/campaign.csv
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
ID,FROM_NAME,STATUS_ID,IS_SEGMENTED,CAMPAIGN_TYPE,STATUS_LABEL,FROM_EMAIL,SUBJECT,NAME,STATUS,CREATED,UPDATED,SEND_TIME,SENT_AT,_FIVETRAN_DELETED,EMAIL_TEMPLATE_ID,_FIVETRAN_SYNCED
WMddDe,friend,2,FALSE,Regular,Draft,hello@gmail.com,Ready to do stuff?,Jan-2 Email,draft,2021-04-12 23:42:35.000,2021-04-12 23:42:44.000,,,FALSE,YzD79H,2021-04-14 07:45:40.981
WViD3c,friend,1,FALSE,Regular,Sent,hello@gmail.com,say hello,camp,sent,2021-03-16 19:16:37.000,2021-03-17 15:38:59.000,2021-03-17 15:38:28.000,2021-03-17 15:38:59.000,FALSE,Y8hHH4,2021-04-14 07:45:41.005
ID,FROM_NAME,STATUS_ID,CAMPAIGN_TYPE,STATUS_LABEL,FROM_EMAIL,SUBJECT,NAME,STATUS,CREATED,UPDATED,SEND_TIME,SENT_AT,_FIVETRAN_DELETED,EMAIL_TEMPLATE_ID,_FIVETRAN_SYNCED,ARCHIVED,SCHEDULED
WMddDe,friend,2,Regular,Draft,hello@gmail.com,Ready to do stuff?,Jan-2 Email,draft,2021-04-12 23:42:35.000,2021-04-12 23:42:44.000,,,FALSE,YzD79H,2021-04-14 07:45:40.981,false,2021-04-22 23:42:44.000
WViD3c,friend,1,Regular,Sent,hello@gmail.com,say hello,camp,sent,2021-03-16 19:16:37.000,2021-03-17 15:38:59.000,2021-03-17 15:38:28.000,2021-03-17 15:38:59.000,FALSE,Y8hHH4,2021-04-14 07:45:41.005,false,2021-03-22 15:38:59.000
6 changes: 3 additions & 3 deletions integration_tests/seeds/flow.csv
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
ID,NAME,STATUS,CREATED,UPDATED,CUSTOMER_FILTER,TRIGGER,_FIVETRAN_DELETED,_FIVETRAN_SYNCED
X839zt,Welcome SMS - Redesign,live,2020-10-12 22:56:59.000,2021-02-10 18:34:39.000,"{""stanzas"": [ {""criteria"": [{""timeframe"": ""alltime"",""type"": ""customer-in-flow""}]}]}","{""filter"": null,""metric"": {""id"": ""UE7prv"",""name"": ""Consented to Receive SMS""},""type"": ""Metric""}",FALSE,2021-04-14 07:46:13.280
S6uajk,Abandoned Cart - NEW VERSION - Katelyn & DW Working,draft,2020-07-08 20:55:56.000,2020-07-09 16:27:58.000,"{""stanzas"": [{""criteria"": [{""operator"": ""nin"",""region_id"": ""EUROPEAN_UNION"",""type"": ""customer-location""}]}]}","{""filter"": null,""metric"": {""id"": ""NXCwha"",""name"": ""Checkout Started""},""type"": ""Metric""}",FALSE,2021-04-14 07:46:13.285
ID,NAME,STATUS,CREATED,UPDATED,CUSTOMER_FILTER,_FIVETRAN_DELETED,_FIVETRAN_SYNCED,ARCHIVED,TRIGGER_TYPE
X839zt,Welcome SMS - Redesign,live,2020-10-12 22:56:59.000,2021-02-10 18:34:39.000,"{""stanzas"": [ {""criteria"": [{""timeframe"": ""alltime"",""type"": ""customer-in-flow""}]}]}",FALSE,2021-04-14 07:46:13.280,false,Metric
S6uajk,Abandoned Cart - NEW VERSION - Katelyn & DW Working,draft,2020-07-08 20:55:56.000,2020-07-09 16:27:58.000,"{""stanzas"": [{""criteria"": [{""operator"": ""nin"",""region_id"": ""EUROPEAN_UNION"",""type"": ""customer-location""}]}]}",FALSE,2021-04-14 07:46:13.285,false,Added to List
3 changes: 0 additions & 3 deletions integration_tests/seeds/flow_snowflake.csv

This file was deleted.

3 changes: 0 additions & 3 deletions integration_tests/seeds/integration.csv

This file was deleted.

6 changes: 3 additions & 3 deletions integration_tests/seeds/metric.csv
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
ID,NAME,INTEGRATION_ID,CREATED,UPDATED,_FIVETRAN_DELETED,_FIVETRAN_SYNCED
Vx2d6i,Received SMS,0rG4eQ,2020-04-15 15:15:18.000,2020-04-15 15:15:18.000,FALSE,2021-04-14 07:46:12.974
MNdFpi,Cancelled Order,0eMvjm,2019-06-19 19:10:26.000,2019-07-01 15:16:51.000,FALSE,2021-04-14 07:46:12.973
ID,NAME,INTEGRATION_ID,CREATED,UPDATED,_FIVETRAN_DELETED,_FIVETRAN_SYNCED,INTEGRATION_CATEGORY,INTEGRATION_NAME
MNdFpi,Cancelled Order,0eMvjm,2019-06-19 19:10:26.000,2019-07-01 15:16:51.000,FALSE,2021-04-14 07:46:12.973,Email,Mailchimp
Vx2d6i,Received SMS,0rG4eQ,2020-04-15 15:15:18.000,2020-04-15 15:15:18.000,FALSE,2021-04-14 07:46:12.974,ecommerce,Shopify
6 changes: 3 additions & 3 deletions integration_tests/seeds/person.csv
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
ID,UPDATED,CREATED,_FIVETRAN_DELETED,FIRST_NAME,LAST_NAME,ADDRESS_1,ADDRESS_2,TITLE,TIMEZONE,ORGANIZATION,REGION,LONGITUDE,LATITUDE,PHONE_NUMBER,COUNTRY,ZIP,CITY,EMAIL,CUSTOM_OBJECT,CUSTOM_EMAIL,CUSTOM_ACCEPTS_MARKETING,CUSTOM_SHOPIFY_TAGS,_FIVETRAN_SYNCED,CUSTOM_CONSENT_FORM_ID,CUSTOM_SOURCE,CUSTOM_CONSENT_METHOD,CUSTOM_CONSENT,CUSTOM_GIFT_GIVER_NAME,CUSTOM_STATE,CUSTOM_FLOW,CUSTOM_GIFT_RECIPIENT_NAME,CUSTOM_LAST_SIGN_IN,CUSTOM_CONSENT_FORM_VERSION,CUSTOM_CONSENT_TIMESTAMP,CUSTOM_MAIL_CHIMP_RATING,CUSTOM_GIFT_RECIPIENT_EMAIL,CUSTOM_GIFT_OPTIONS,CUSTOM_ADDRESS,CUSTOM_SUBSCRIPTION_EXPIRATION,CUSTOM_CITY,CUSTOM_ADDRESS_LINE_2,CUSTOM_ZIPCODE,CUSTOM_EXPECTED_DATE_OF_NEXT_ORDER,CUSTOM_SMS_ATTENTIVE_SIGNUP,CUSTOM_PHONE,CUSTOM_AFTERPAY_ORDER,CUSTOM_PHONE_NUMBER_REGION,CUSTOM_NAME,CUSTOM_REFERRER_NAME,CUSTOM_REFERRER_EMAIL,CUSTOM_BIRTHDAY,CUSTOM_FIRST_PURCHASE_DATE_,CUSTOM_UNENGAGED,CUSTOM_LANDING_PAGE_TAG,CUSTOM_FITNESS_GOAL,CUSTOM_AGE,CUSTOM_WORKOUT,CUSTOM_QUIZ,CUSTOM_BREAKFAST,CUSTOM_QUIZ_DATA,CUSTOM_RAF_SUBSCRIBE,CUSTOM_SUPPRESS,CUSTOM_SMS_CONSENT,CUSTOM_GIFT_OPTION_DREAMBELT,CUSTOM_GIFT_OPTION_STARTER,CUSTOM_GIFT_OPTION_MACHINE,CUSTOM_LANDINGPAGE_TAG
01F35SYVPX6MS88ARK37864GJX,2021-04-13 14:44:06.000,2021-04-13 14:25:27.000,FALSE,,,,,,America/New_York,,New York,-12.0851,421.6117,+15165555555,United States,,Brooklyn,nobody@hotmail.com,person,nobody@hotmail.com,,,2021-04-14 07:47:10.440,X2GkGj,Redesign - Mobile Email + SMS,Klaviyo Form,,,,,,,190980,,,,,,,,,,,,,,US,,,,,,,,,,,,,,,,,,,,
01F366M1ZERT4HM2Z9R2V9G7BY,2021-04-13 18:09:10.000,2021-04-13 18:06:44.000,FALSE,MArio,Lopez,1600 Penn,,,America/Chicago,,Kansas,,,+15165555555,United States,64224,Jefferson City,mrgoogle@gmail.com,person,mrgoogle@gmail.com,false,[],2021-04-14 07:47:45.376,,,,,,,,,,,,,,,,,,,,05/16/2021,,,,,,,,,2021-04-13,,,,,,,,,,,,,,,
ID,UPDATED,CREATED,_FIVETRAN_DELETED,FIRST_NAME,LAST_NAME,ADDRESS_1,ADDRESS_2,TITLE,TIMEZONE,ORGANIZATION,REGION,LONGITUDE,LATITUDE,PHONE_NUMBER,COUNTRY,ZIP,CITY,EMAIL,CUSTOM_OBJECT,CUSTOM_EMAIL,CUSTOM_ACCEPTS_MARKETING,CUSTOM_SHOPIFY_TAGS,_FIVETRAN_SYNCED,CUSTOM_CONSENT_FORM_ID,CUSTOM_SOURCE,CUSTOM_CONSENT_METHOD,CUSTOM_CONSENT,CUSTOM_GIFT_GIVER_NAME,CUSTOM_STATE,CUSTOM_FLOW,CUSTOM_GIFT_RECIPIENT_NAME,CUSTOM_LAST_SIGN_IN,CUSTOM_CONSENT_FORM_VERSION,CUSTOM_CONSENT_TIMESTAMP,CUSTOM_MAIL_CHIMP_RATING,CUSTOM_GIFT_RECIPIENT_EMAIL,CUSTOM_GIFT_OPTIONS,CUSTOM_ADDRESS,CUSTOM_SUBSCRIPTION_EXPIRATION,CUSTOM_CITY,CUSTOM_ADDRESS_LINE_2,CUSTOM_ZIPCODE,CUSTOM_EXPECTED_DATE_OF_NEXT_ORDER,CUSTOM_SMS_ATTENTIVE_SIGNUP,CUSTOM_PHONE,CUSTOM_AFTERPAY_ORDER,CUSTOM_PHONE_NUMBER_REGION,CUSTOM_NAME,CUSTOM_REFERRER_NAME,CUSTOM_REFERRER_EMAIL,CUSTOM_BIRTHDAY,CUSTOM_FIRST_PURCHASE_DATE_,CUSTOM_UNENGAGED,CUSTOM_LANDING_PAGE_TAG,CUSTOM_FITNESS_GOAL,CUSTOM_AGE,CUSTOM_WORKOUT,CUSTOM_QUIZ,CUSTOM_BREAKFAST,CUSTOM_QUIZ_DATA,CUSTOM_RAF_SUBSCRIBE,CUSTOM_SUPPRESS,CUSTOM_SMS_CONSENT,CUSTOM_GIFT_OPTION_DREAMBELT,CUSTOM_GIFT_OPTION_STARTER,CUSTOM_GIFT_OPTION_MACHINE,CUSTOM_LANDINGPAGE_TAG,LAST_EVENT_DATE
01F35SYVPX6MS88ARK37864GJX,2021-04-13 14:44:06.000,2021-04-13 14:25:27.000,FALSE,,,,,,America/New_York,,New York,-12.0851,421.6117,+15165555555,United States,,Brooklyn,nobody@hotmail.com,person,nobody@hotmail.com,,,2021-04-14 07:47:10.440,X2GkGj,Redesign - Mobile Email + SMS,Klaviyo Form,,,,,,,190980,,,,,,,,,,,,,,US,,,,,,,,,,,,,,,,,,,,,2023-07-18 12:43:52 UTC
01F366M1ZERT4HM2Z9R2V9G7BY,2021-04-13 18:09:10.000,2021-04-13 18:06:44.000,FALSE,MArio,Lopez,1600 Penn,,,America/Chicago,,Kansas,,,+15165555555,United States,64224,Jefferson City,mrgoogle@gmail.com,person,mrgoogle@gmail.com,false,[],2021-04-14 07:47:45.376,,,,,,,,,,,,,,,,,,,,05/16/2021,,,,,,,,,2021-04-13,,,,,,,,,,,,,,,,2023-09-08 09:19:46 UTC
5 changes: 3 additions & 2 deletions macros/get_campaign_columns.sql
Original file line number Diff line number Diff line change
Expand Up @@ -9,15 +9,16 @@
{"name": "from_email", "datatype": dbt.type_string()},
{"name": "from_name", "datatype": dbt.type_string()},
{"name": "id", "datatype": dbt.type_string()},
{"name": "is_segmented", "datatype": "boolean"},
{"name": "name", "datatype": dbt.type_string()},
{"name": "send_time", "datatype": dbt.type_timestamp()},
{"name": "sent_at", "datatype": dbt.type_timestamp()},
{"name": "status", "datatype": dbt.type_string()},
{"name": "status_id", "datatype": dbt.type_string()},
{"name": "status_label", "datatype": dbt.type_string()},
{"name": "subject", "datatype": dbt.type_string()},
{"name": "updated", "datatype": dbt.type_timestamp()}
{"name": "updated", "datatype": dbt.type_timestamp()},
{"name": "archived", "datatype": "boolean" },
{"name": "scheduled", "datatype": dbt.type_timestamp()}
] %}

{{ return(columns) }}
Expand Down
9 changes: 2 additions & 7 deletions macros/get_flow_columns.sql
Original file line number Diff line number Diff line change
Expand Up @@ -8,15 +8,10 @@
{"name": "name", "datatype": dbt.type_string()},
{"name": "status", "datatype": dbt.type_string()},
{"name": "updated", "datatype": dbt.type_timestamp()},
{"name": "customer_filter", "datatype": dbt.type_string()}
{"name": "archived", "datatype": "boolean" },
{"name": "trigger_type", "datatype": dbt.type_string()},
] %}

{% if target.type == 'databricks' %}
{{ columns.append( {"name": "trigger", "datatype": dbt.type_string(), "quote": False} ) }}
{% else %}
{{ columns.append( {"name": "trigger", "datatype": dbt.type_string(), "quote": True} ) }}
{% endif %}

{{ return(columns) }}

{% endmacro %}
13 changes: 0 additions & 13 deletions macros/get_integration_columns.sql

This file was deleted.

2 changes: 2 additions & 0 deletions macros/get_metric_columns.sql
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@
{"name": "created", "datatype": dbt.type_timestamp()},
{"name": "id", "datatype": dbt.type_string()},
{"name": "integration_id", "datatype": dbt.type_string()},
{"name": "integration_category", "datatype": dbt.type_string()},
{"name": "integration_name", "datatype": dbt.type_string()},
{"name": "name", "datatype": dbt.type_string()},
{"name": "updated", "datatype": dbt.type_timestamp()}
] %}
Expand Down
3 changes: 2 additions & 1 deletion macros/get_person_columns.sql
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,8 @@
{"name": "timezone", "datatype": dbt.type_string()},
{"name": "title", "datatype": dbt.type_string()},
{"name": "updated", "datatype": dbt.type_timestamp()},
{"name": "zip", "datatype": dbt.type_string()}
{"name": "zip", "datatype": dbt.type_string()},
{"name": "last_event_date", "datatype": dbt.type_timestamp()}
] %}

{{ fivetran_utils.add_pass_through_columns(columns, var('klaviyo__person_pass_through_columns')) }}
Expand Down
Loading

0 comments on commit acee59c

Please sign in to comment.