[CT-2176] [Feature] Report bytes processed for tests

[bruno-szdl]

Is this your first time submitting a feature request?

• I have read the expectations for open source contributors
• I have searched the existing issues, and I could not find an existing issue for this feature
• I am requesting a straightforward extension of existing dbt-bigquery functionality, rather than a Big Idea better suited to a discussion

Describe the feature

As @elyobo commented in this closed issue #14 (comment), it would be very nice to see the bytes processed by running tests in BigQuery.

Tests can consume a lot of bytes, especially when running tests like unique and not_null without where, where they scan the whole table.

Since BQ billing is based on these bytes, you can get surprised by an expensive bill because you have no idea of the costs of your tests when developing. So, it would be fantastic to see the cost of each test in the logs as we see for the models.

Describe alternatives you've considered

Currently, the information of bytes_processed for models is taken from total_bytes_processed from the query_job object.
https://github.com/dbt-labs/dbt-bigquery/blob/main/dbt/adapters/bigquery/connections.py#L491

I am assuming the query_job information is queried from INFORMATION_SCHEMA.JOBS. I checked it and jobs that run tests also return a value for total_bytes_processed.

I don't understand why in run_results.json tests don't return the bytes processed, as I didn't find any restriction in this code
https://github.com/dbt-labs/dbt-bigquery/blob/main/dbt/adapters/bigquery/connections.py
Maybe I am looking at the wrong code.

Anyway, the information is available and we already do this for models, so I think we can do this for tests.

Who will this benefit?

People working with dbt in BigQuery as they can monitor better the costs caused by dbt tests.

Are you interested in contributing this feature?

Yes, I am interested. I am not sure where the information for test results is written.

Anything else?

No response

[jtcohen6]

Good news! Starting in v1.5 (currently available as a beta prerelease), this information will also be available for tests and source freshness checks, thanks to this contribution:

• add adapter_response for test dbt-core#6645

You can try it out!

pip install dbt-bigquery~=1.5.0b1

[bruno-szdl]

great! I am trying to better monitor the costs of BigQuery projects of our clients and that was exactly what I needed! Thanks @jtcohen6 for showing this contribution.

I will close this issue now.