CT-2414: Add graph summaries to target directory output

[peterallenwebb]

resolves #7357

Description

Adds a new graph summary output file named graph_summary.json to the target directory output.

Checklist

• I have read the contributing guide and understand what's expected of me
• I have signed the CLA
• I have run this code in development and it appears to resolve the stated issue
• This PR includes tests, or tests are not required/relevant for this PR
• I have opened an issue to add/update docs, or docs changes are not required/relevant for this PR
• I have run changie new to create a changelog entry

[github-actions]

Thank you for your pull request! We could not find a changelog entry for this change. For details on how to document a change, see the contributing guide.

[dbeatty10]

To keep the number of artifacts to a minimum, could we just write out graph_with_test_edges.json?

From there, consumers could filter out any resource types they don't want.

I know we started to discuss that idea here, but now that you have this pull request, it's probably useful to move that conversation to here.

[dbeatty10]

@peterallenwebb Would you be open to just a single JSON artifact that is a superset of both artifacts?

That way, there's just one artifact to produce/consumer, and the end user can filter out any nodes they want to ignore.

Some benefits is that it would make the docs and implementation more simple.

[peterallenwebb]

@dbeatty10 I am just getting back to working on this, and not disregarding your earlier request, I promise. Using a single file will complicate the implementation, especially since the tests are added on one code path but not another, so I'm trying to figure out how to keep both the implementation and the output simple.

I think I am going to spin the snowplow stats off as a separate case, though.

[dbeatty10]

No worries @peterallenwebb !

Spinning off the snowplow stats seems like nice effort that is complementary.

I was mainly wondering what would be lost if you just dropped this line:

linker.write_graph_summary("linked", out_stream, manifest)

Is it because graph_with_test_edges.json would not be a simple superset of graph.json?

[peterallenwebb]

@dbeatty10 There is some debate about whether the algorithm for adding the test edges is completely correct, and we know it has performance issues that we'd like to address, so my idea here is to make sure we know exactly what the graph looks like before and after that algorithm runs. That way we'll have a known starting point for refining and/or optimizing that algorithm.

I'm going to take one more pass at making this simpler.

[iknox-fa]

LGTM-- I bet we'll want to make some tweaks at some point (esp wrt test edges), but this seems like a very sane starting place.

[aranke]

Can we write a test here?

[peterallenwebb]

Added a test as suggested and included the invocation_id in the summary so that it could more easily be correlated with logging events and other artifacts of the same dbt invocation.

[peterallenwebb]

@boxysean You may find this interesting.