Implement insert_overwrite in s3

[mattsmith123]

It appears that the current implementation of the insert_overwrite incremental strategy is to CTAS a temp table and the do an INSERT INTO from there. This seems to have the unwanted side effect of an extra read and write of the result set. Do I have that wrong? Is there something I don't have configured correctly?

Has there been any discussion about a mode that would use S3 to move the data over to the table (basically rename the files) and then just add the partitions? I would maybe be willing to look into implementing it if there is interest and it does not go against the design principles of the project.

Thanks for a great project. This really lets us take advantage of dbt to clean up so really unwieldy queries.

[nicor88]

Hey @mattsmith123 what you discover it's correct. We first persist the data in a tmp table, then we use it to perform the insert from it.
Of course what you raise is right, we process the data multiple times, that could lead to additional costs.

Pretty much https://github.com/dbt-athena/dbt-athena/issues/455 address what you described, feel free to add your thoughts there, and we IMHO should consider to close this discussion and add all in the mentioned issue.

And yes, if you are willing to contribute, I'm more than happy to review and make all the necessary steps to add such feature.

[mattsmith123]

Awesome. Thanks for the reply.