Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: dbt incremental model takes extremely long to finish when run against large table #211

Closed
1 task done
mxmarg opened this issue Jan 19, 2024 · 2 comments
Closed
1 task done

[Bug]: dbt incremental model takes extremely long to finish when run against large table #211

mxmarg opened this issue Jan 19, 2024 · 2 comments
Labels
bug Something isn't working

Comments

@mxmarg
Copy link
Contributor

mxmarg commented Jan 19, 2024

Is there an existing issue for this?

  • I have searched the existing issues

Current Behavior

Description see Dremio-internal JIRA: https://dremio.atlassian.net/browse/DX-86623

Expected Behavior

No response

Steps To Reproduce

No response

Environment

- OS:
- dbt-dremio:
- Dremio Software:
- Dremio Cloud:

Relevant log output

No response
@mxmarg mxmarg added the bug Something isn't working label Jan 19, 2024
@mxmarg
Copy link
Contributor Author

mxmarg commented Jan 19, 2024

#212

@ravjotbrar ravjotbrar mentioned this issue Jan 25, 2024
Merged
@mxmarg
Copy link
Contributor Author

mxmarg commented Jan 26, 2024

Issue has been fixed by #214

@mxmarg mxmarg closed this as completed Jan 26, 2024
ravjotbrar added a commit that referenced this issue Jan 29, 2024
@ravjotbrar
Ensure adapter does not fetch actual records from materialization (#214)
ef025e7 
### Summary

During the second run of an incremental materialization, the adapter
would try and fetch records from the temporary table it is creating.
This made the adapter hang if the table was large enough.

### Description
Made the fetch configuration false by default. Also changed the
run_query macro to use call_statement, as run_query will always fetch
the data that is returned by the sql query. Thanks to @mxmarg for the
initial iteration of this fix.

### Related Issue

#211
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Milestone
No milestone
Development

No branches or pull requests
1 participant
@mxmarg