Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add node type codes to more events + more hook log data #4378

Merged
merged 6 commits into from
Dec 3, 2021

Conversation

emmyoop
Copy link
Member

@emmyoop emmyoop commented Dec 1, 2021

on-run-start/end hooks were not getting node_status, node_started_at, node_finished_at populated in node_status of logs. Fixed.

Made it a little more clear when we use the _message method in AdapterResponse for logging purposes.

Fixed up a few logging codes to point to nodes.

@gshank
Copy link
Contributor

gshank commented Dec 1, 2021

We don't need to prefix the 'started_at' and 'finished_at' keys with 'dbt_internal_' anymore since it's now not in the config dict. Could we change that back?

@@ -1928,8 +1928,8 @@ class PrintHookStartLine(InfoLevel, Cli, File, NodeInfo):
index: int
total: int
truncate: bool
report_node_data: Any # TODO use ParsedHookNode here
code: str = "Z032"
report_node_data: ParsedHookNode
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I couldn't figure out how to get these mypy checks passing either!

core/dbt/task/run.py:361: error: Argument "report_node_data" to "PrintHookStartLine" has incompatible type "dbt.contracts.graph.parsed.ParsedHookNode"; expected "dbt.events.stubs.ParsedHookNode"

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Take a close look at that error. There's two ParsedHookNodes: one is a type stub and one is a real class. We need to make sure we're not using the type stub, and delete it if we can.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So I am using the stub type. But I"m also referencing ParsedHookNode on other events that don't trigger the error. Then I'm using the stub type for ParsedModelNode on 16 events that don't trigger this. Why does it only trigger sometimes?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

probably because mypy doesn't run on all the files so some of the call sites don't get caught.

@nathaniel-may
Copy link
Contributor

Status gets hard coded to 'OK' and then if the hook contains SQL, the status becomes an instance of AdapterResponse for logging purposes. Is this intended?

I think we can make a better choice than this. What would you like to do here?

@emmyoop
Copy link
Member Author

emmyoop commented Dec 1, 2021

@nathaniel-may I'm honestly thinking now it's on purpose. It's actually consistent with how it gets handled on other log lines. The second line is the one I'm speaking of specifically.

16:10:15 | [ info  ] | 1 of 1 START hook: jaffle_shop.on-run-start.0................................... [RUN]
16:10:15 | [ info  ] | 1 of 1 OK hook: jaffle_shop.on-run-start.0...................................... [SELECT 1 in 0.00s]
16:10:15 | [ info  ] |
16:10:15 | [ info  ] | Concurrency: 4 threads (target='dev')
16:10:15 | [ info  ] |
16:10:15 | [ info  ] | 1 of 7 START view model dbt_testing.env_var_model............................... [RUN]
16:10:15 | [ info  ] | 2 of 7 START view model dbt_testing.even_id_table............................... [RUN]
16:10:15 | [ info  ] | 3 of 7 START table model dbt_testing.my_first_dbt_model......................... [RUN]
16:10:15 | [ error ] | 1 of 7 ERROR creating view model dbt_testing.env_var_model...................... [ERROR in 0.12s]
16:10:15 | [ info  ] | 3 of 7 OK created table model dbt_testing.my_first_dbt_model.................... [SELECT 2 in 0.16s]
16:10:15 | [ info  ] | 2 of 7 OK created view model dbt_testing.even_id_table.......................... [CREATE VIEW in 0.16s]

AdapterResponse has a __str__ method so that it just returns the _message. I'm going to add a comment but the behavior (I believe?) is correct.

@nathaniel-may
Copy link
Contributor

Maybe there's a better way to model this behavior so it's more obviously intentional. I'm ok leaving that for later unless you're particularly motivated now.

@emmyoop emmyoop force-pushed the er/cleanup-node-logging branch from ed98b37 to 2f0fa95 Compare December 2, 2021 16:46
@emmyoop emmyoop marked this pull request as ready for review December 2, 2021 20:04
@emmyoop emmyoop force-pushed the er/cleanup-node-logging branch from 2f0fa95 to e2a2753 Compare December 2, 2021 22:54
@leahwicz leahwicz force-pushed the er/cleanup-node-logging branch from 791564a to abcb909 Compare December 3, 2021 00:11
ParsedHookNode,
ParsedModelNode,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The only conflict I fixed myself with the rebase was removing _ReferenceKey from here since @nathaniel-may added it above in his PR

@leahwicz leahwicz merged commit b3039fd into main Dec 3, 2021
@leahwicz leahwicz deleted the er/cleanup-node-logging branch December 3, 2021 00:25
leahwicz pushed a commit that referenced this pull request Dec 3, 2021
* add node type codes to more events + more hook log

* minor fixes

* renames started/finished keys

* made process more clear

* fixed errors

* Put back report_node_data in fresshness.py

Co-authored-by: Gerda Shank <gerda@dbtlabs.com>
leahwicz added a commit that referenced this pull request Dec 3, 2021
* add node type codes to more events + more hook log

* minor fixes

* renames started/finished keys

* made process more clear

* fixed errors

* Put back report_node_data in fresshness.py

Co-authored-by: Gerda Shank <gerda@dbtlabs.com>

Co-authored-by: Emily Rockman <emily.rockman@dbtlabs.com>
Co-authored-by: Gerda Shank <gerda@dbtlabs.com>
iknox-fa pushed a commit that referenced this pull request Feb 8, 2022
* add node type codes to more events + more hook log

* minor fixes

* renames started/finished keys

* made process more clear

* fixed errors

* Put back report_node_data in fresshness.py

Co-authored-by: Gerda Shank <gerda@dbtlabs.com>

automatic commit by git-black, original commits:
  b3039fd
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants