add prefix and suffix arguments to star macro #436
Conversation
|
Hey @fivetran-jamie, this makes sense to me! Happy to welcome it into the fold. Could you add another test in the same vein as the existing one (model.yml, model, seed) to verify that they are behaving the way you expect? dbt 1.0.0-rc1 is imminent, and so we'll be cutting utils 0.7.4 and 0.8.0 in the next week or so. Would love to sneak this in as well! |
|
hey @joellabes happy to do so! i'm a little confused by the test files though (this is my first dbt-utils PR 😅 ) -- are there guidelines somewhere on what to add? |
|
@fivetran-jamie There aren't (yet) sorry! In short, there's another dbt project inside of this repo called In this case, you'd want to do something along these lines:
- name: test_star_prefix_suffix
tests:
- dbt_utils.equality:
compare_model: ref('data_star_prefix_suffix_expected')Then when you push it up, the CI will check everything for you. if it fails, you can't actually see why 😢 just @ me and I'll check it out for you. Hope that gets you started! |
|
@fivetran-jamie three outta four ain't bad! The CI error is: Looks like a casing issue to me? (shakes fist at Snowflake) |
|
darn you snowflake...trying out the target-conditional casing used in https://github.com/dbt-labs/dbt-utils/blob/main/integration_tests/models/sql/test_pivot.sql |
|
huzzah! |
I hate it but I love it 💀 it feels very backwards to change the test to pass the reality, but the fact that there's prior art puts my mind a bit more at ease. I'll give it a test locally before I properly merge it in, but this is looking good! thank you 🌟 |
|
yah it didn't feel good to add that lol -- happy to investigate other solutions if needed, just came across this one first 🤷♂️ also thanks for laying out everything so clearly! |
|
hm would it be best to automatically capitalize the prefix + suffix on snowflake warehouses? ie {% macro star(from, relation_alias=False, except=[], prefix='', suffix='') -%}
{{ return(adapter.dispatch('star', 'dbt_utils')(from, relation_alias, except, prefix, suffix)) }}
{% endmacro %}
{% macro default__star(from, relation_alias=False, except=[], prefix='', suffix='') -%}
{%- do dbt_utils._is_relation(from, 'star') -%}
{%- do dbt_utils._is_ephemeral(from, 'star') -%}
{#-- Prevent querying of db in parsing mode. This works because this macro does not create any new refs. #}
{%- if not execute -%}
{{ return('') }}
{% endif %}
{%- set include_cols = [] %}
{%- set cols = adapter.get_columns_in_relation(from) -%}
{%- set except = except | map("lower") | list %}
{%- for col in cols -%}
{%- if col.column|lower not in except -%}
{% do include_cols.append(col.column) %}
{%- endif %}
{%- endfor %}
{% set prefix_cased = prefix | upper if target.type == 'snowflake' else prefix %} -- <--- here
{% set suffix_cased = suffix | upper if target.type == 'snowflake' else suffix %} -- <--- here
{%- for col in include_cols %}
{%- if relation_alias %}{{ relation_alias }}.{% else %}{%- endif -%}{{ adapter.quote(col)|trim }} as {{ adapter.quote(prefix_cased ~ col ~ suffix_cased)|trim }}
{%- if not loop.last %},{{ '\n ' }}{% endif %}
{%- endfor -%}
{%- endmacro %}
|
One day I'd like to flesh out the test suite to cover more variations of quoting behaviour, but that would require another warehouse connection etc so I'm not in a rush to do that 😬 |
Hi,
|
This is a:
masterdev/branchdev/branchDescription & motivation
Just added optional
prefixandsuffixarguments to the star macro, so that the output fields will all be renamed as prefix ~ name ~ suffix (very similar to thepivotmacro).This is helpful to me in that I can more easily know where columns came from. Morewover, I have a particular case where I am using
startwice in the same CTE, and the columns included in each source relation that I'm joining/selecting from can have custom names. If there's any overlap between the two sources, we'll see anambiguous columnerror without any prefix or suffix to distinguish themChecklist
star()source) is this just adding the arguments to the dispatch line?