ref and source overrides not working when importing models from another project

[NicolasGuary]

Hello !
I found a bug in the ref() (and potentially also in source()) macro override. Here's the setup to reproduce :

• Create a dbt project project_a in which you create model_a
• Then, create a dbt project project_b in which :
  • you create model_b that uses model_a
  • you add the dependency to dbt-unit-testing in packages.yml
  • you add the macro override for ref() and source()
• In project_a import project_b as a package, depending on your setup, it will look like :

  packages:
    - local: "../../../project_b"
  

Now, if you try to perform a dbt ls command from project_a, you'll have the following error :

Compilation Error
  Model 'model.project_b.model_b' (models/model_b.sql) depends on a node named 'model_a' in package or project 'project_b' which was not found

But if you remove the macro override for ref() it will work just fine.

-> The issue is that now it simply performs a look-up in the project in which the model is defined and not in the project that imports the model which contains the model in this case

[NicolasGuary]

This seems to work, but not sure if 100% correct :

{% macro ref(project_or_package, model_name) %}
  {% if running_unit_test() %}
    {% set project_or_package, model_name = setup_project_and_model_name(project_or_package, model_name) %}
    {% set node_version = kwargs["version"] | default (kwargs["v"]) %}
    {% set node = {"package_name": project_or_package, "name": model_name, "version": node_version} %}
    {{ return (ref_cte_name(node)) }}
  {% else %}
    {{ return (builtins.ref(project_or_package, **kwargs)) }}
  {% endif %}
{% endmacro %}

[NicolasGuary]

The issue might also come from this line :

dbt-unit-testing/macros/utils.sql

Line 77 in 49f92aa

selectattr('package_name', 'equalto', node.package_name) |

It will throw an error as long as a used model is not from the current package.
I used selectattr('package_name', 'in', [node.package_name, 'my_custom_dbt_project']) and it allowed me to test using a ref from my_custom_dbt_project in another dbt project

[psousa50]

Hi @NicolasGuary,
could you please clarify your scenario? You mentioned that in project_a you import project_b, but then you have a model_b in project_b that uses model_a in project_a. Is this right?

[NicolasGuary]

Hello @psousa50 yes exactly!

[psousa50]

I'm a bit confused. If project A imports project B you should be able to use models from project B in project A, not the other way around. I'm sorry if I misunderstood what you said

[NicolasGuary]

Imagine you have a project "common" with models that should be shared by a set of different dbt projects for instance, you have a "custom_project_a" importing common.

You define unit tests in common project (as this is where the model is implemented) but one of the common models uses a model from custom_project_a (so any custom project is supposed to have this model but that's not the point here).

The unit test on the common model will fail because it won't be able to resolve the node from custom_project_a using the custom ref() macro + as the package name for this node won't be "common" it will throw an error.

I don't know if that's clear but if you try this setup locally with your package you'll be able to reproduce

[psousa50]

Oh ok, now it's clear, thank you! 🙂

Let me try to find a way to fix this, not sure if it's possible, we'll see.

Thank you again for your report.

[psousa50]

Please take a look at the latest release (0.4.0) and check if it fixes this issue.

Thank you!

[NicolasGuary]

It seems to have worked indeed thank's!

[NicolasGuary]

@rantibi have you tried using the latest version (https://github.com/EqualExperts/dbt-unit-testing/releases) and set the correct ref and source overrides as described in the README (https://github.com/EqualExperts/dbt-unit-testing?tab=readme-ov-file#important-requirement)

[rantibi]

@NicolasGuary yes, it didn't help.
I found that removing this line solves the issue, but I am not sure that this is the right fix, I didn't dive into the code more then that yet.

(BTW: I accidentally deleted my original question, maybe you have it in your mail and you can repost it? if not, I will write it down again, thanks)

[rantibi]

reposting my question again as I accidentally deleted it:

I have a related issue:
I have common project with model named model1.
I have my_project project that depends on common.

When I am trying to mock ref as follow:

{% call dbt_unit_testing.mock_ref('model1', {"input_format": "csv"}) %}

I am getting the following error:

 Compilation Error in test my_test (tests/unit/my_test.sql)
  Node my_project.model1 not found.
  
  > in macro raise_error (macros/utils.sql)
  > called by macro model_node (macros/utils.sql)
  > called by macro graph_node (macros/utils.sql)
  > called by macro build_mocks_and_expectations (macros/tests.sql)
  > called by macro build_configuration_and_test_queries (macros/tests.sql)
  > called by macro test (macros/tests.sql)
  > called by test my_test (tests/unit/my_test.sql)

I found that removing this line solves the issue, but I am not sure that this is the right fix and if it doesn't breaks other things, I didn't dive into the code more then that yet.

[psousa50]

Hi @rantibi, your test file is in common or in my_project? And in which are you running the dbt test command?

[rantibi]

@psousa50 my_project for both

[psousa50]

In that case, you should use the two-parameter version of mock_ref, much like the ref macro:

{% call dbt_unit_testing.mock_ref('common', 'model1', {"input_format": "csv"}) %}

Please try it and let me know if it works

Thanks

[rantibi]

I have tried that, but I am getting the same error

[psousa50]

Hi @rantibi ,

Sorry for the late response, but my time has been a bit limited lately.

Are you still experiencing this issue? If so, could you please share more details about your project? That would greatly help in understanding what's happening.

[cdiniz]

Closing due to inactivity. @rantibi we are open to discussing this further if you still need it.