Merge pull request #5 from fivetran/feature/v2_updates

Feature/v2 updates
fivetran · Sep 1, 2022 · 9cfb8af · 9cfb8af
2 parents 51090f3 + 5cf9a8e
commit 9cfb8af
Show file tree

Hide file tree

Showing 28 changed files with 751 additions and 428 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,3 +1,31 @@
+# dbt_tiktok_ads v0.2.0
+This PR ([#5](https://github.com/fivetran/dbt_tiktok_ads/pull/5)) applies the following updates to provide further flexibility in your ad reporting. Additionally, these new end models will be leveraged in the respective downstream [dbt_ad_reporting](https://github.com/fivetran/dbt_ad_reporting) models. 
+## 🚨 Breaking Changes 🚨
+- Updates `ad_group_hourly` variable to `ad_group_report_hourly`
+- Removes intermediate models and instead incorporates previous logic into end models
+- Renames the `tikok_ads__ad_adapter` model to `tiktok_ads__url_report` 
+- Removes dependency of models on the `tikok_ads__ad_adapter` model and moves that previous logic into each model, using corresponding hourly reports for each model's granularity (e.g. `tikok_ads__campaign_report` now uses the `campaign_report_hourly`)
+## 🎉 Feature Enhancements 🎉
+- Adds in the following models and respective field descriptions and tests:
+    - `tiktok_ads__account_report`
+    - `tiktok_ads__url_report`
+    - `tiktok_ads__ad_report`
+- Adds additional fields to existing models `ad_group_report` and `campaign_report` that were previously not brought in
+- Inclusion of passthrough metrics:
+  - `tiktok_ads__ad_group_hourly_passthrough_metrics`
+  - `tiktok_ads__ad_hourly_passthrough_metrics`
+  - `tiktok_ads__campaign_hourly_passthrough_metrics`
+> This applies to all passthrough columns within the `dbt_tiktok_ads` package and not just the `tiktok_ads__ad_group_hourly_passthrough_metrics` example.
+```yml
+vars:
+  tiktok_ads__ad_group_hourly_passthrough_metrics:
+    - name: "my_field_to_include" # Required: Name of the field within the source.
+      alias: "field_alias" # Optional: If you wish to alias the field within the staging model.
+```
+- Adds not-null tests to key fields
+- Applies standardization updates 
+- Add enable configs for this specific ad platform, for use in the Ad Reporting rollup package 
+
 # dbt_tiktok_ads v0.1.1
 ## Under the Hood
 - Updated package yml to point to live source 

diff --git a/README.md b/README.md
@@ -1,52 +1,83 @@
-[![Apache License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
-# TikTok Ads 
-
-This package models TikTok Ads data from [Fivetran's connector](https://fivetran.com/docs/applications/tiktok-ads). It uses data in the format described by [this ERD](https://fivetran.com/docs/applications/tiktok-ads#schemainformation).
-
-The main focus of the package is to transform the core ad object tables into analytics-ready models, including an 'ad adapter' model that can be easily unioned in to other ad platform packages to get a single view. This is especially easy using our [Ad Reporting package](https://github.com/fivetran/dbt_ad_reporting).
-
-## Models
-
-This package contains transformation models, designed to work simultaneously with our [Tiktok Ads source package](https://github.com/fivetran/dbt_tiktok_ads_source) and our [multi-platform Ad Reporting package](https://github.com/fivetran/dbt_ad_reporting). A dependency on the source package is declared in this package's `packages.yml` file, so it will automatically download when you run `dbt deps`. The primary outputs of this package are described below.
+<p align="center">
+    <a alt="License"
+        href="https://github.com/fivetran/dbt_tiktok/blob/main/LICENSE">
+        <img src="https://img.shields.io/badge/License-Apache%202.0-blue.svg" /></a>
+    <a alt="dbt-core">
+        <img src="https://img.shields.io/badge/dbt_Core™_version->=1.0.0_,<2.0.0-orange.svg" /></a>
+    <a alt="Maintained?">
+        <img src="https://img.shields.io/badge/Maintained%3F-yes-green.svg" /></a>
+    <a alt="PRs">
+        <img src="https://img.shields.io/badge/Contributions-welcome-blueviolet" /></a>
+</p>
+
+# Tiktok Ads Transformation dbt Package ([Docs](https://fivetran.github.io/dbt_tiktok_ads/))
+# 📣 What does this dbt package do?
+- Produces modeled tables that leverage Tiktok Ads data from [Fivetran's connector](https://fivetran.com/docs/applications/tiktok-ads) in the format described by [this ERD](https://fivetran.com/docs/applications/tiktok-ads#schemainformation) and builds off the output of our [Tiktok Ads source package](https://github.com/fivetran/dbt_tiktok_ads_source).
+- Creates reports on daily marketing performance across various levels of granularity. 
+- Generates a comprehensive data dictionary of your source and modeled Tiktok Ads data through the [dbt docs site](https://fivetran.github.io/dbt_tiktok_ads/).
+
+The following table provides a detailed list of all models materialized within this package by default. 
+> TIP: See more details about these models in the package's [dbt docs site](https://fivetran.github.io/dbt_tiktok_ads/#!/overview?g_v=1&g_e=seeds).
 
 | **model**                    | **description**                                                                                                        |
 | ---------------------------- | ---------------------------------------------------------------------------------------------------------------------- |
-| [tiktok_ads__ad_adapter](https://github.com/fivetran/dbt_tiktok_ads/blob/main/models/tiktok_ads__ad_adapter.sql)                          | Each record represents the daily ad performance, including information about the UTM parameters. |
-| [tiktok_ads__ad_group_report](https://github.com/fivetran/dbt_tiktok_ads/blob/main/models/tiktok_ads__ad_group_report.sql)                     | Each record represents the daily ad performance for each ad group. This also includes additional data on the demographics you want to target. | 
-| [tiktok_ads__campaign_report](https://github.com/fivetran/dbt_tiktok_ads/blob/main/models/tiktok_ads__campaign_report.sql)                     | Each record represents the daily ad performance for each campaign. | 
+| [tiktok_ads__ad_group_report](https://github.com/fivetran/dbt_tiktok_ads/blob/main/models/tiktok_ads__ad_group_report.sql)                     | Each record represents the daily performance for each ad group. This also includes additional data on the demographics you are targeting. | 
+| [tiktok_ads__campaign_report](https://github.com/fivetran/dbt_tiktok_ads/blob/main/models/tiktok_ads__campaign_report.sql)                     | Each record represents the daily performance for each campaign. |
+| [tiktok_ads__advertiser_report](https://github.com/fivetran/dbt_tiktok_ads/blob/main/models/tiktok_ads__advertiser_report.sql)                     | Each record represents the daily performance for each account. |
+| [tiktok_ads__ad_report](https://github.com/fivetran/dbt_tiktok_ads/blob/main/models/tiktok_ads__ad_report.sql)                     | Each record represents the daily performance for each ad. This also includes additional data on the demographics you are targeting. | 
+| [tiktok_ads__url_report](https://github.com/fivetran/dbt_tiktok_ads/blob/main/models/tiktok_ads__url_report.sql)                     | Each record in this table represents the daily performance of URLs at the ad level. This also includes additional data on the demographics you are targeting.
 
-## Installation Instructions
+# 🎯 How do I use the dbt package?
 
-Check [dbt Hub](https://hub.getdbt.com/) for the latest installation instructions, or [read the dbt docs](https://docs.getdbt.com/docs/package-management) for more information on installing packages.
+## Step 1: Prerequisites
+To use this dbt package, you must have the following:
 
-Include in your `packages.yml`
+- At least one Fivetran Tiktok Ads connector syncing data into your destination.
+- A **BigQuery**, **Snowflake**, **Redshift**, **PostgreSQL**, or **Databricks** destination.
 
+## Step 2: Install the package
+Include the following tiktok_ads package version in your `packages.yml` file:
+> TIP: Check [dbt Hub](https://hub.getdbt.com/) for the latest installation instructions or [read the dbt docs](https://docs.getdbt.com/docs/package-management) for more information on installing packages.
 ```yaml
 packages:
   - package: fivetran/tiktok_ads
-    version: [">=0.1.0", "<0.2.0"]
-```
+    version: [">=0.2.0", "<0.3.0"]
 
-## Configuration
+```
 
-By default, this package will look for your TikTok Ads data in the `tiktok_ads` schema of your [target database](https://docs.getdbt.com/docs/running-a-dbt-project/using-the-command-line-interface/configure-your-profile). If this is not where your TikTok Ads data is, please add the following configuration to your 
-`dbt_project.yml` file:
+## Step 3: Define database and schema variables
+By default, this package will look for your TikTok Ads data in the `tiktok_ads` schema of your [target database](https://docs.getdbt.com/docs/running-a-dbt-project/using-the-command-line-interface/configure-your-profile). If this is not where your TikTok Ads data is, you would add the following configuration to your root `dbt_project.yml` file with your custom database and schema names:
 
 ```yml
-# dbt_project.yml
-
-...
-config-version: 2
-
 vars:
     tiktok_ads_schema: your_database_name
     tiktok_ads_database: your_schema_name
 ```
 
 For additional configurations for the source models, visit the [Tiktok Ads source package](https://github.com/fivetran/dbt_tiktok_ads_source).
 
-### Changing the Build Schema
+## (Optional) Step 4: Additional configurations
+<details><summary>Expand for configurations</summary>
+
+### Passing Through Additional Metrics
+By default, this package will select `clicks`, `impressions`, and `cost` from the source reporting tables to store into the staging models. If you would like to pass through additional metrics to the staging models, add the below configurations to your `dbt_project.yml` file. These variables allow for the pass-through fields to be aliased (`alias`) if desired, but not required. Use the below format for declaring the respective pass-through variables:
+
+>**Note** Please ensure you exercised due diligence when adding metrics to these models. The metrics added by default (taps, impressions, and spend) have been vetted by the Fivetran team maintaining this package for accuracy. There are metrics included within the source reports, for example metric averages, which may be inaccurately represented at the grain for reports created in this package. You will want to ensure whichever metrics you pass through are indeed appropriate to aggregate at the respective reporting levels provided in this package.
+
+```yml
+vars:
+    tiktok_ads__ad_group_hourly_passthrough_metrics: 
+      - name: "new_custom_field"
+        alias: "custom_field"
+      - name: "my_other_field"
+    tiktok_ads__ad_hourly_passthrough_metrics:
+      - name: "this_field"
+    tiktok_ads__campaign_hourly_passthrough_metrics:
+      - name: "unique_string_field"
+        alias: "field_id"
+```
 
+### Change the build schema
 By default, this package will build the TikTok Ads staging models within a schema titled (`<target_schema>` + `_stg_tiktok_ads`) and the final TikTok Ads models within a schema titled (`<target_schema>` + `_tiktok_ads`) in your target database. If this is not where you would like your modeled TikTok data to be written to, add the following configuration to your `dbt_project.yml` file:
 
 ```yml
@@ -60,37 +91,51 @@ models:
         +schema: my_new_schema_name # leave blank for just the target_schema
 ```
 
-## Contributions
+
+### Change the source table references
+If an individual source table has a different name than the package expects, add the table name as it appears in your destination to the respective variable:
 
-Don't see a model or specific metric you would have liked to be included? Notice any bugs when installing and running the package? If so, we highly encourage and welcome contributions to this package! 
-Please create issues or open PRs against `master`. See [this Discourse post](https://discourse.getdbt.com/t/contributing-to-a-dbt-package/657) for information on how to contribute to a package.
+> IMPORTANT: See this project's [`dbt_project.yml`](https://github.com/fivetran/dbt_tiktok_ads/blob/main/dbt_project.yml) variable declarations to see the expected names.
 
-## Database Support
+```yml
+vars:
+    tiktok_ads_<default_source_table_name>_identifier: your_table_name 
+```
 
-This package has been tested on BigQuery, Snowflake, Redshift, Postgres, and Databricks.
+</details>
 
-### Databricks Dispatch Configuration
+## (Optional) Step 5: Orchestrate your models with Fivetran Transformations for dbt Core™
+<details><summary>Expand for more details</summary>
 
-dbt `v0.20.0` introduced a new project-level dispatch configuration that enables an "override" setting for all dispatched macros. If you are using a Databricks destination with this package, you will need to add the following (or a variation of the following) dispatch configuration within your `dbt_project.yml`. This is required in order for the package to accurately search for macros within the `dbt-labs/spark_utils` and then the `dbt-labs/dbt_utils` packages, respectively.
+Fivetran offers the ability for you to orchestrate your dbt project through [Fivetran Transformations for dbt Core™](https://fivetran.com/docs/transformations/dbt). Learn how to set up your project for orchestration through Fivetran in our [Transformations for dbt Core setup guides](https://fivetran.com/docs/transformations/dbt#setupguide).
 
+</details>
+
+# 🔍 Does this package have dependencies?
+This dbt package is dependent on the following dbt packages. Please be aware that these dependencies are installed by default within this package. For more information on the following packages, refer to the [dbt hub](https://hub.getdbt.com/) site.
+> IMPORTANT: If you have any of these dependent packages in your own `packages.yml` file, we highly recommend that you remove them from your root `packages.yml` to avoid package version conflicts.
+
 ```yml
-# dbt_project.yml
+packages:
+    - package: fivetran/tiktok_ads_source
+      version: [">=0.2.0", "<0.3.0"]
 
-dispatch:
-  - macro_namespace: dbt_utils
-    search_order: ['spark_utils', 'dbt_utils']
+    - package: fivetran/fivetran_utils
+      version: [">=0.3.0", "<0.4.0"]
+
+    - package: dbt-labs/dbt_utils
+      version: [">=0.8.0", "<0.9.0"]
 ```
+# 🙌 How is this package maintained and can I contribute?
+## Package Maintenance
+The Fivetran team maintaining this package _only_ maintains the latest version of the package. We highly recommend you stay consistent with the [latest version](https://hub.getdbt.com/fivetran/tiktok_ads/latest/) of the package and refer to the [CHANGELOG](https://github.com/fivetran/dbt_tiktok_ads/blob/main/CHANGELOG.md) and release notes for more information on changes across versions.
+
+## Contributions
+A small team of analytics engineers at Fivetran develops these dbt packages. However, the packages are made better by community contributions! 
 
-## Resources:
-- Provide [feedback](https://www.surveymonkey.com/r/DQ7K7WW) on our existing dbt packages or what you'd like to see next
-- Have questions, feedback, or need help? Book a time during our office hours [using Calendly](https://calendly.com/fivetran-solutions-team/fivetran-solutions-team-office-hours) or email us at solutions@fivetran.com
-- Find all of Fivetran's pre-built dbt packages in our [dbt hub](https://hub.getdbt.com/fivetran/)
-- Learn how to orchestrate your models with [Fivetran Transformations for dbt Core™](https://fivetran.com/docs/transformations/dbt)
-- Learn more about Fivetran overall [in our docs](https://fivetran.com/docs)
-- Check out [Fivetran's blog](https://fivetran.com/blog)
-- Learn more about dbt [in the dbt docs](https://docs.getdbt.com/docs/introduction)
-- Check out [Discourse](https://discourse.getdbt.com/) for commonly asked questions and answers
-- Join the [chat](http://slack.getdbt.com/) on Slack for live discussions and support
-- Find [dbt events](https://events.getdbt.com) near you
-- Check out [the dbt blog](https://blog.getdbt.com/) for the latest news on dbt's development and best practices
+We highly encourage and welcome contributions to this package. Check out [this dbt Discourse article](https://discourse.getdbt.com/t/contributing-to-a-dbt-package/657) on the best workflow for contributing to a package!
 
+# 🏪 Are there any resources available?
+- If you have questions or want to reach out for help, please refer to the [GitHub Issue](https://github.com/fivetran/dbt_tiktok_ads/issues/new/choose) section to find the right avenue of support for you.
+- If you would like to provide feedback to the dbt package team at Fivetran or would like to request a new dbt package, fill out our [Feedback Form](https://www.surveymonkey.com/r/DQ7K7WW).
+- Have questions or want to just say hi? Book a time during our office hours [on Calendly](https://calendly.com/fivetran-solutions-team/fivetran-solutions-team-office-hours) or email us at solutions@fivetran.com.
diff --git a/dbt_project.yml b/dbt_project.yml
@@ -1,21 +1,23 @@
 name: 'tiktok_ads'
-version: '0.1.1'
+version: '0.2.0'
 config-version: 2
 require-dbt-version: [">=1.0.0", "<2.0.0"]
 
 models:
   tiktok_ads:
     +schema: tiktok_ads
     +materialized: table
-    intermediate:
-      +materialized: ephemeral
 vars:
   tiktok_ads:
-    ad_history: "{{ ref('stg_tiktok_ads__ad_history') }}"
+    ad_group_report_hourly: "{{ ref('stg_tiktok_ads__ad_group_report_hourly') }}"
     ad_group_history: "{{ ref('stg_tiktok_ads__ad_group_history') }}"
+    campaign_report_hourly: "{{ ref('stg_tiktok_ads__campaign_report_hourly') }}"
     campaign_history: "{{ ref('stg_tiktok_ads__campaign_history') }}"
-    advertiser: "{{ ref('stg_tiktok_ads__advertiser') }}"
     ad_report_hourly: "{{ ref('stg_tiktok_ads__ad_report_hourly') }}"
-    campaign_report_hourly: "{{ ref('stg_tiktok_ads__campaign_report_hourly') }}"
-    ad_group_hourly: "{{ ref('stg_tiktok_ads__ad_group_report_hourly') }}"
+    ad_history: "{{ ref('stg_tiktok_ads__ad_history') }}"
+    advertiser: "{{ ref('stg_tiktok_ads__advertiser') }}"
+
+  tiktok_ads__ad_group_hourly_passthrough_metrics: []
+  tiktok_ads__ad_hourly_passthrough_metrics: []
+  tiktok_ads__campaign_hourly_passthrough_metrics: []
 
diff --git a/docs/catalog.json b/docs/catalog.json
diff --git a/docs/index.html b/docs/index.html
diff --git a/docs/manifest.json b/docs/manifest.json
diff --git a/docs/run_results.json b/docs/run_results.json
diff --git a/integration_tests/ci/sample.profiles.yml b/integration_tests/ci/sample.profiles.yml
@@ -16,14 +16,14 @@ integration_tests:
       pass: "{{ env_var('CI_REDSHIFT_DBT_PASS') }}"
       dbname: "{{ env_var('CI_REDSHIFT_DBT_DBNAME') }}"
       port: 5439
-      schema: tiktok_ads_integration_tests
+      schema: tiktok_ads_integration_tests_1
       threads: 8
     bigquery:
       type: bigquery
       method: service-account
       keyfile: "{{ env_var('GCLOUD_SERVICE_KEY_PATH') }}"
       project: 'dbt-package-testing'
-      schema: tiktok_ads_integration_tests
+      schema: tiktok_ads_integration_tests_1
       threads: 8
     snowflake:
       type: snowflake
@@ -33,12 +33,12 @@ integration_tests:
       role: "{{ env_var('CI_SNOWFLAKE_DBT_ROLE') }}"
       database: "{{ env_var('CI_SNOWFLAKE_DBT_DATABASE') }}"
       warehouse: "{{ env_var('CI_SNOWFLAKE_DBT_WAREHOUSE') }}"
-      schema: tiktok_ads_integration_tests
+      schema: tiktok_ads_integration_tests_1
       threads: 8
     spark:
       type: spark
       method: http
-      schema: tiktok_ads_integration_tests
+      schema: tiktok_ads_integration_tests_1
       host: "{{ env_var('CI_SPARK_DBT_HOST') }}"
       organization: "{{ env_var('CI_SPARK_DBT_ORGANIZATION') }}"
       token: "{{ env_var('CI_SPARK_DBT_TOKEN') }}"
@@ -54,7 +54,7 @@ integration_tests:
       password: "{{ env_var('CI_POSTGRES_DBT_PASS') }}"
       port: 5432
       dbname: "{{ env_var('CI_POSTGRES_DBT_DATABASE') }}"
-      schema: tiktok_ads_integration_tests
+      schema: tiktok_ads_integration_tests_1
       threads: 8
       keepalives_idle: 0
       sslmode: prefer