[CT-1725] [CT-1722] [CT-1721] [Bug] 1.4.0-b1 Bigquery Incremental Query breaks when ending in comment

[thejens]

Is this a new bug in dbt-core?

• I believe this is a new bug in dbt-core
• I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

In 1.4.0-b1 when running:

test_model.sql:

{{- config(materialized='incremental') -}}
SELECT
  '123' AS test
-- Test

The compiled query becomes:

        
    

    

    merge into `[REDACTED]` as DBT_INTERNAL_DEST
        using (SELECT
  '123' AS test
-- Test) as DBT_INTERNAL_SOURCE
        on (FALSE)

    

    when not matched then insert
        (`test`)
    values
        (`test`)


    

As you see, the --Test comment row also comments out parts of the merge statement, this happens with or without a trailing newline after the final comment

Expected Behavior

The newlines are persisted and the commend won't influence the wrapping merge statement

Steps To Reproduce

Create a .sql file ending with a comment, configure it to incremental, run twice to trigger an incremental run

Relevant log output

`Syntax error: Expected ")" but got keyword ON at [12:9]`

Environment

- OS: Debian GNU/Linux 11 (bullseye)
- Python: 3.11.1
- dbt: 1.4.0b1

Which database adapter are you using with dbt?

No response

Additional Context

No response

[dbeatty10]

Thanks for reporting this @thejens !

This looks like an issue with Jinja whitespace Control. i.e., there's probably {%- instead of {% somewhere (and/or -%} instead of -%}).

I was able to reproduce the same thing as you in dbt-bigquery.

Suspected PRs

dbt-bigquery has had a few PRs recently related to incremental materializations, and I'm wondering if it might have been introduced in one of them:

• Support for ingestion time partition table on BigQuery as incremental materialization dbt-bigquery#136
• Optimize insert_overwrite incremental strategy with WRITE_TRUNCATE / Partition copy dbt-bigquery#167
• Feature/incremental predicates support dbt-bigquery#284

The last one has an associated PR in dbt-core:

• Feature/support incremental predicates #5702

Originally assuming that the bug was introduced in dbt-bigquery, I moved the issue into that repo. But we haven't yet ruled out that the bug is in dbt-core, so I'll transfer the issue back for now.

How we should test for this

• We should add a comment to the end of incremental models in integration tests to catch any regressions in the future

[leahwicz]

Closed with this fix dbt-labs/dbt-bigquery#458