[CT-656] [Feature] Allow for merge exclude columns

[dave-connors-3]

Is this your first time opening an issue?

• I have read the expectations for open source contributors

Describe the Feature

Right now, for incremental models using the merge strategy, we allow users to specify a specific list of merge_update_columns to include in the merge statement.

For wide, denormalized models, it's likely that users may want/need to exclude only a couple of columns, and update all the rest. Right now, this would require the user to manually specify a potentially long list of all columns that need to be updated, rather than a shorter, more straightforward list of columns not to update.

Once incremental materializations are simplified in the effort to create more streamlined materialization code, this seems like a simple quality of life enhancement!

Describe alternatives you've considered

Create a custom version of get_merge_sql to allow for an additional config merge_exclude_columns to achieve this behavior.

Who will this benefit?

This will be beneficial for users who have wide incremental models, and may need tight controls over the merge behavior on those tables.

Are you interested in contributing this feature?

heck yeah

Anything else?

This was mentioned in the original issue for merge_update_columns

[jtcohen6]

@dave-connors-3 Okay, I buy it :) thanks for opening to help us keep track of the request, since we've gotten it before!

The incremental materialization code is getting away from us, and I hesitate to add more complexity there in its current state. The good news is that we're soon to meaningfully refactor it, starting with incremental "strategies" in core and our adapter plugins. We're going to be freezing net-new code additions in the meantime. That should clear the path for a straightforward addition like this one, or like support for arbitrary incremental_predicates (#3293/#4546). At that point, this can be either a line of code that we write, or a good first issue. I can't guarantee that we ourselves will manage to do it before v1.2, but I'm adding it to the milestone anyway, if just to signal that we're on a linear path in its direction.

As discussed in #5215, I don't believe we should support or encourage any merge statement logic more complex than:

• match criteria: unique_key, "lookback window" predicates to speed up performance
• do update these columns with new data
• don't update these columns with new data