Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Snapshot strategies: newline for subquery #6780

Merged
merged 12 commits into from
Apr 11, 2023
Merged

Conversation

RobbertDM
Copy link
Contributor

@RobbertDM RobbertDM commented Jan 30, 2023

resolves #6781

Description

If you don't add these, the ) subq will be appended to the last line of the snapshot query, which could be a comment line.

Checklist

@RobbertDM RobbertDM requested a review from a team as a code owner January 30, 2023 14:58
@RobbertDM RobbertDM requested a review from VersusFacit January 30, 2023 14:58
@cla-bot
Copy link

cla-bot bot commented Jan 30, 2023

Thanks for your pull request, and welcome to our community! We require contributors to sign our Contributor License Agreement and we don't seem to have your signature on file. Check out this article for more information on why we have a CLA.

In order for us to review and merge your code, please submit the Individual Contributor License Agreement form attached above above. If you have questions about the CLA, or if you believe you've received this message in error, please reach out through a comment on this PR.

CLA has not been signed by users: @RobbertDM

@RobbertDM
Copy link
Contributor Author

CLA: signed again with case sensitive GitHub username this time 🤔

@dbeatty10
Copy link
Contributor

dbeatty10 commented Jan 30, 2023

Thanks for opening this PR @RobbertDM !

A bookkeeping piece (which will resolve the other failing CI check you are seeing):

  • Could you run changie new as described here so we can generate a changelog entry and credit you?

Speaking of CI, we'll want to:

  • add a test similar to this so that we can catch any regressions in the future. Ideally, there would be a pre-existing test model to which a comment can be appended as a postfix.

@dbeatty10 dbeatty10 added Team:Adapters Issues designated for the adapter area of the code ready_for_review Externally contributed PR has functional approval, ready for code review from Core engineering cla:yes labels Jan 30, 2023
@RobbertDM RobbertDM requested a review from a team as a code owner January 30, 2023 22:15
@RobbertDM RobbertDM requested a review from iknox-fa January 30, 2023 22:15
@VersusFacit
Copy link
Contributor

Hey @RobbertDM, thanks for your ping. I think Doug's right to make sure we have a test for this case. We have snapshot tests in tests/functional/simple_snapshot. Basically, add to our snapshot tests a new or existing file that describes the case where the model has your (pre-merge) failing use case. Should be as simple as writing a SQL definition and copying the format for a test class.

@RobbertDM
Copy link
Contributor Author

Thanks for the guidance. I added a test file but I have little idea of what I am doing, so please review carefully :-)

@VersusFacit
Copy link
Contributor

Awesome!! Thanks for the fast turnaround. It's basically all but there. You'll have to follow flake's directions of nixing unneeded names and such. It's perfectly fine you went with the conventions you saw. I think they pretty much look good for this case. We'll see with the adapter test results on your CI here.

@dbeatty10
Copy link
Contributor

Thanks again for working on this PR @RobbertDM !

TL;DR

I think you'll ultimately need to add lines like the following:

    seeds__seed_csv,

and:

    @pytest.fixture(scope="class")
    def seeds(self):
        return {"seed.csv": seeds__seed_csv}

Do you want to give that a try?

Please let us know if you run into any problems and we'll try to help out.

More detail

I took a peek at the error during CI:

18:37:17  Completed with 1 error and 0 warnings:
18:37:17  
18:37:17  Database Error in snapshot snapshot_actual (snapshots/snapshot.sql)
18:37:17    relation "test16758814328907864877_test_comment_ending_snapshot.seed" does not exist
18:37:17    LINE 21:     select * from dbt.test16758814328907864877_test_comment_...

It's trying to snapshot a table named dbt.test16758814328907864877_test_comment_ending_snapshot.seed, but it's not finding that table. So I think the main thing to do is make sure it exists.

I didn't confirm, but I suspect that you need build the seed table with something like the following two pieces:

  1. seeds__seed_newcol_csv,
    seeds__seed_csv,
  2. @pytest.fixture(scope="class")
    def seeds(self):
    return {"seed_newcol.csv": seeds__seed_newcol_csv, "seed.csv": seeds__seed_csv}

I'm assuming the seed_newcol portions in the examples above are not applicable to your case, so you can remove them (but also wouldn't hurt to leave them in initially just to make sure all the CI is working after copy-pasting).

@RobbertDM
Copy link
Contributor Author

RobbertDM commented Feb 27, 2023

Hey @dbeatty10 thanks a lot for the detailed help, I wanted to give up after getting nothing useful from CI.
Where did you get that CI output?
I looked at this https://github.com/dbt-labs/dbt-core/actions/runs/4127200194 but that doesn't give me your output.
E: oh, there's also https://github.com/dbt-labs/dbt-core/actions/runs/4127200189

@dbeatty10
Copy link
Contributor

Happy to help @RobbertDM !

I saw that error listed here.

It sounds like you looked at this top one:

image

And I scrolled down a bit and dug into one of these (which might all be basically identical):

image

@dbeatty10
Copy link
Contributor

@RobbertDM

The CI isn't quite happy yet. I accidentally steered you wrong based on your test setup 😅 . If you update def seeds(self) to def seeds(), it might make the CI happy 🤞

@RobbertDM
Copy link
Contributor Author

Yeah sorry, I wasn't thinking, just copy pasted :s

@Fleid Fleid added the pr_tracked PR is tracked in the Adapters Backlog (ask Florian) label Mar 13, 2023
iknox-fa

This comment was marked as outdated.

@iknox-fa iknox-fa self-requested a review April 6, 2023 22:40
@iknox-fa
Copy link
Contributor

iknox-fa commented Apr 7, 2023

@RobbertDM
I made a few small tweaks to your test case for you. You had a few issues:

  • Trying to combine a class-based approach with a function-based approach, and pytest was having none of it.
  • Not finding the seed table because you hadn't actually populated it by running the SQL file
  • Needed to run snapshot to test your fix, not run, and crucially, you need to do it twice because the bug only happened when the snapshot table already exists in the DW.
  • A bunch of unneeded fixtures. They didn't hurt anything, but they weren't being used.

Should be all set now. :)

(I also added a comment to the macro since we don't want someone coming along and cleaning up that whitespace you added at later date)

@VersusFacit
Mea Culpa for jumping in on an adapter ticket. I got tagged for review somehow and got halfway through before I realized I probably shouldn't have been. 🤓

@iknox-fa iknox-fa dismissed their stale review April 7, 2023 14:53

Added code, requires another reviewer

@iknox-fa iknox-fa removed their request for review April 7, 2023 14:53
@VersusFacit VersusFacit merged commit b450a57 into dbt-labs:main Apr 11, 2023
@VersusFacit
Copy link
Contributor

Huge thanks to everyone for keeping us alive and getting it across the finish line

@RobbertDM
Copy link
Contributor Author

Woah, thanks a lot for all the help and explanation everyone. I'll definitely come back to this PR if I make another one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cla:yes pr_tracked PR is tracked in the Adapters Backlog (ask Florian) ready_for_review Externally contributed PR has functional approval, ready for code review from Core engineering Team:Adapters Issues designated for the adapter area of the code
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[CT-1946] [Bug] dbt snapshot fails if the last line is a comment
5 participants