Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Metric Parsing #66

Closed
wants to merge 23 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
b2ff85a
large refactor to use click as cli interface, improve and further abs…
Sep 26, 2021
3032328
ephemeral model skip should be debug level like rest
Sep 26, 2021
e47ad64
full implementation of yaml config support in tandem to env var suppo…
Sep 26, 2021
95a1810
add config load func to utils
Sep 26, 2021
9754511
add config global, override option process method to include config l…
Sep 26, 2021
81ef87d
utility function to list env vars, validate in use env vars, inject e…
Sep 26, 2021
2ab00ca
last linting and typing updates and more robust type handler for conf…
Sep 26, 2021
3f8952f
specify encoding for all file handlers
Sep 26, 2021
8b76066
explicitly specify lru cache size
Sep 26, 2021
6377f81
remove incorrect underscore in kwarg
Sep 26, 2021
55d3508
update interface to inherit from config objects
Sep 29, 2021
92dd9b1
little comment for clarity
Oct 3, 2021
e21f1aa
readme updates to programmatic invocation and config
Oct 3, 2021
c61027b
fix typo
Oct 3, 2021
4357ff7
fixed same type
Oct 3, 2021
45ae898
remove uneeded sentence
Oct 3, 2021
7b2384e
add small note for manual config yml create and insight into layout.
Oct 3, 2021
191cd49
rst needs a break between code-block decl and code
Oct 3, 2021
8dfb5f1
peg based metric parser utilizing pyparsing
z3z1ma Oct 3, 2021
9f59df5
include meta field in dataclass model for metrics
z3z1ma Oct 3, 2021
a2df2dd
include meta field in dataclass model for metrics
z3z1ma Oct 3, 2021
6acd281
add metabase client method for metric sync
z3z1ma Oct 3, 2021
bfd7adc
metrics function integrated into cli
z3z1ma Oct 3, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
112 changes: 82 additions & 30 deletions README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -309,6 +309,53 @@ You can control this behavior with two arguments:
* ``--metabase_sync_timeout`` - number of seconds to wait and re-check data model before
giving up

Configuration
-------------

.. code-block:: shell

dbt-metabase config

Using the above command, you can enter an interactive configuration session where you can cache default selections
for arguments. This creates a ``config.yml`` in ~/.dbt-metabase. This is particularly useful for arguments which are repeated on every invocation like metabase_user, metabase_host,
metabase_password, dbt_manifest_path, etc.

In addition, there are a few injected env vars that make deploying dbt-metabase in a CI/CD environment simpler without exposing
secrets. Listed below are acceptable env vars which correspond to their CLI flags:

* ``DBT_DATABASE``
* ``DBT_PATH``
* ``DBT_MANIFEST_PATH``
* ``MB_USER``
* ``MB_PASS``
* ``MB_HOST``
* ``MB_DATABASE``

If any one of the above is present in the environment, the corresponding CLI flag is not needed unless overriding
the environment value. In the absence of a CLI flag, dbt-metabase will first look to the environment for any
env vars to inject, then we will look to the config.yml for cached defaults.

A ``config.yml`` can be created or updated manually as well if needed. The only
requirement is that it must be located in ~/.dbt-metabase. The layout is as follows:

.. code-block:: yaml

config:
dbt_database: reporting
dbt_manifest_path: /home/user/dbt/target/manifest.json
metabase_database: Reporting
metabase_host: reporting.metabase.io
metabase_user: user@source.co
metabase_password: ...
metabase_use_http: false
metabase_sync: true
metabase_sync_timeout: null
dbt_schema_excludes:
- development
- testing
dbt_excludes:
- test_monday_io_site_diff

Programmatic Invocation
-----------------------

Expand All @@ -317,23 +364,10 @@ line. But if you prefer to call it from your code, here's how to do it:

.. code-block:: python

import dbtmetabase
from dbtmetabase.models.interface import MetabaseInterface, DbtInterface

# Collect Args the Build Configs #
##################################

metabase_config = MetabaseConfig(
host=metabase_host,
user=metabase_user,
password=metabase_password,
use_http=metabase_use_http,
verify=metabase_verify,
database=metabase_database,
sync_skip=metabase_sync_skip,
sync_timeout=metabase_sync_timeout,
)

dbt_config = DbtConfig(
# Instantiate dbt interface
dbt = DbtInterface(
path=dbt_path,
manifest_path=dbt_manifest_path,
database=dbt_database,
Expand All @@ -343,24 +377,42 @@ line. But if you prefer to call it from your code, here's how to do it:
excludes=dbt_excludes,
)

# Propagate models to Metabase #
################################
# Load models
dbt_models, aliases = dbt.parser.read_models(
dbt_config=dbt.get_config(),
include_tags=dbt_include_tags,
docs_url=dbt_docs_url,
)

dbtmetabase.models(
metabase_config=metabase_config,
dbt_config=dbt_config,
dbt_docs_url=dbt_docs,
dbt_include_tags=include_tags,
# Instantiate Metabase interface
metabase = MetabaseInterface(
host=metabase_host,
user=metabase_user,
password=metabase_password,
use_http=metabase_use_http,
verify=metabase_verify,
database=metabase_database,
sync=metabase_sync,
sync_timeout=metabase_sync_timeout,
)

# Parse exposures from Metabase into dbt yml #
##############################################
# Load client
metabase.prepare_metabase_client(dbt_models)

# Propagate models to Metabase
metabase.client.export_models(
database=metabase.database,
models=dbt_models,
aliases=aliases,
)

dbtmetabase.exposures(
metabase_config=metabase_config,
dbt_config=dbt_config,
output_path=output_path,
output_name=output_name,
# Parse exposures from Metabase into dbt schema yml
metabase.client.extract_exposures(
models=dbt_models,
output_path=output_path,
output_name=output_name,
include_personal_collections=include_personal_collections,
collection_excludes=collection_excludes,
)


Expand Down
Loading