Skip to content

Commit

Permalink
[CI/CD] Introduce workflows to test dbt installations (#64)
Browse files Browse the repository at this point in the history
* Add workflows to test different dbt installations methods;
* Add `main` and `homebrew` workflows implementation and dummy implementation for other workflows;
  • Loading branch information
alexander-smolyakov authored Jan 18, 2023
1 parent b37524a commit 8d93437
Show file tree
Hide file tree
Showing 5 changed files with 326 additions and 0 deletions.
51 changes: 51 additions & 0 deletions .github/workflows/test-dbt-installation-docker.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
# **what?**

# **why?**

# **when?**
# This reusable workflow can be called or started manually
# by specifying the package name

name: dbt Installation - Docker Integration Tests

on:
workflow_dispatch:
inputs:
package_name:
description: "Package name to test (i.e. dbt-postgres)"
required: true
type: string
workflow_call:
inputs:
package_name:
required: true
type: string
secrets:
SLACK_WEBHOOK_URL:
description: Slack app webhook url
required: true

# no permissions are needed for this workflow
permissions: {}

jobs:
docker-integration-test:
runs-on: ubuntu-latest

steps:
- name: "Output Package Name - ${{ inputs.package_name }}"
run: |
echo ${{ inputs.package_name }}
slack-notification:
name: "Post Scheduled Run Failures"
needs: docker-integration-test
if: ${{ failure() && github.event_name == 'schedule' }}

uses: dbt-labs/dbt-release/.github/workflows/slack-post-notification.yml@main
with:
status: "failure"
notification_title: "Docker nightly integration test failed for - ${{ inputs.package_name }}"

secrets:
SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}
82 changes: 82 additions & 0 deletions .github/workflows/test-dbt-installation-homebrew.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
# **what?**
# This workflow installs the latest version of dbt adapter from homebrew.
# It then runs 'dbt --version' to verify the installation was successful.
# If it fails for the scheduled runs, it will post to our team alerts channel.

# **why?**
# This is a simple way to test all adapter installations at a single
# time. It allows us to test them on a schedule as well to check for
# any breaking dependency changes that might happen and alert us on it.

# **when?**
# This reusable workflow can be called or started manually
# by specifying the package name

name: dbt Installation - Homebrew Integration Tests

on:
workflow_dispatch:
inputs:
package_name:
description: "Package name to test (i.e. dbt-postgres)"
required: true
type: string
workflow_call:
inputs:
package_name:
required: true
type: string
secrets:
SLACK_WEBHOOK_URL:
description: Slack app webhook url
required: true

# no permissions are needed for this workflow
permissions: {}

jobs:
homebrew-integration-test:
runs-on: macos-11

steps:
- name: "Prepare Brew"
# brew upgrade generates some symlink update warnings that cause failures without the || true.
# They're just that the symlink wasn't updated which is okay to ignore.
# This does cause other errors to be ignored but for this case that's safe since the brew install
# step would likely fail at that point.
run: |
brew update
brew upgrade || true
- name: "[DEBUG] Show List Of Installed Homebrew Packages"
run: |
brew list
- name: "Tap dbt-labs/dbt Repository To Homebrew Formulae"
run: |
brew tap dbt-labs/dbt
- name: "[DEBUG] Show List Of All Currently Tapped Repositories"
run: |
brew tap
- name: "Brew Install ${{ inputs.package_name }}"
run: |
brew install --verbose ${{ inputs.package_name }}
- name: "Verify ${{ inputs.package_name }} Version"
run: |
dbt --version
slack-notification:
name: "Post Scheduled Run Failures"
needs: homebrew-integration-test
if: ${{ failure() && github.event_name == 'schedule' }}

uses: dbt-labs/dbt-release/.github/workflows/slack-post-notification.yml@main
with:
status: "failure"
notification_title: "Homebrew nightly integration test failed for - ${{ inputs.package_name }}"

secrets:
SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}
91 changes: 91 additions & 0 deletions .github/workflows/test-dbt-installation-main.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
# **what?**

# **why?**

# **when?**
# This workflow will run on a schedule every night and also can be
# manually invoked.

name: dbt Installation Tests

on:
workflow_dispatch:

# run this once per night to ensure no regressions
# schedule:
# - cron: "0 9,13,18 * * *" # 9:00, 13:00, 18:00 UTC

jobs:
dbt-installation-homebrew:
if: ${{ always() }}
strategy:
fail-fast: false
matrix:
package:
["dbt-postgres", "dbt-redshift", "dbt-snowflake", "dbt-bigquery"]

uses: .github/workflows/test-dbt-installation-homebrew.yml
with:
package_name: ${{ matrix.package }}
secrets:
SLACK_WEBHOOK_URL: ${{ secrets.SLACK_TEST_CHANNEL }}

dbt-installation-pip:
if: ${{ always() }}
strategy:
fail-fast: false
matrix:
package:
[
"dbt-postgres",
"dbt-redshift",
"dbt-snowflake",
"dbt-bigquery",
"dbt-spark",
]

uses: .github/workflows/test-dbt-installation-pip.yml
with:
package_name: ${{ matrix.package }}
secrets:
SLACK_WEBHOOK_URL: ${{ secrets.SLACK_TEST_CHANNEL }}

dbt-installation-docker:
if: ${{ always() }}
strategy:
fail-fast: false
matrix:
package:
[
"dbt-postgres",
"dbt-redshift",
"dbt-snowflake",
"dbt-bigquery",
"dbt-spark",
]

uses: .github/workflows/test-dbt-installation-docker.yml
with:
package_name: ${{ matrix.package }}
secrets:
SLACK_WEBHOOK_URL: ${{ secrets.SLACK_TEST_CHANNEL }}

dbt-installation-source:
if: ${{ always() }}
strategy:
fail-fast: false
matrix:
package:
[
"dbt-postgres",
"dbt-redshift",
"dbt-snowflake",
"dbt-bigquery",
"dbt-spark",
]

uses: .github/workflows/test-dbt-installation-source.yml
with:
package_name: ${{ matrix.package }}
secrets:
SLACK_WEBHOOK_URL: ${{ secrets.SLACK_TEST_CHANNEL }}
51 changes: 51 additions & 0 deletions .github/workflows/test-dbt-installation-pip.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
# **what?**

# **why?**

# **when?**
# This reusable workflow can be called or started manually
# by specifying the package name

name: dbt Installation - pip Integration Tests

on:
workflow_dispatch:
inputs:
package_name:
description: "Package name to test (i.e. dbt-postgres)"
required: true
type: string
workflow_call:
inputs:
package_name:
required: true
type: string
secrets:
SLACK_WEBHOOK_URL:
description: Slack app webhook url
required: true

# no permissions are needed for this workflow
permissions: {}

jobs:
pip-integration-test:
runs-on: ubuntu-latest

steps:
- name: "Output Package Name - ${{ inputs.package_name }}"
run: |
echo ${{ inputs.package_name }}
slack-notification:
name: "Post Scheduled Run Failures"
needs: pip-integration-test
if: ${{ failure() && github.event_name == 'schedule' }}

uses: dbt-labs/dbt-release/.github/workflows/slack-post-notification.yml@main
with:
status: "failure"
notification_title: "pip nightly integration test failed for - ${{ inputs.package_name }}"

secrets:
SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}
51 changes: 51 additions & 0 deletions .github/workflows/test-dbt-installation-source.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
# **what?**

# **why?**

# **when?**
# This reusable workflow can be called or started manually
# by specifying the package name

name: dbt Installation - Source Integration Tests

on:
workflow_dispatch:
inputs:
package_name:
description: "Package name to test (i.e. dbt-postgres)"
required: true
type: string
workflow_call:
inputs:
package_name:
required: true
type: string
secrets:
SLACK_WEBHOOK_URL:
description: Slack app webhook url
required: true

# no permissions are needed for this workflow
permissions: {}

jobs:
source-integration-test:
runs-on: ubuntu-latest

steps:
- name: "Output Package Name - ${{ inputs.package_name }}"
run: |
echo ${{ inputs.package_name }}
slack-notification:
name: "Post Scheduled Run Failures"
needs: source-integration-test
if: ${{ failure() && github.event_name == 'schedule' }}

uses: dbt-labs/dbt-release/.github/workflows/slack-post-notification.yml@main
with:
status: "failure"
notification_title: "Source nightly integration test failed for - ${{ inputs.package_name }}"

secrets:
SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}

0 comments on commit 8d93437

Please sign in to comment.