Fix parsing dbt ls outputs that contain JSONs that are not dbt nodes

[tatiana]

This change makes Cosmos more resilient, allowing it to be used even when JSONs do not represent dbt nodes in the dbt ls output.

Context

An Astronomer customer raised a P1 incident, mentioning they could no longer run their Cosmos-powered DAGs.

They were using Cosmos 1.5.0, and the issue was observed whenever DAGs were deployed using Astro deploy --dags, even if they only had whitespace as a difference.

The DAGs could no longer be parsed, raising an exception similar to:

    File /usr/local/lib/python3.11/site-packages/cosmos/dbt/graph.py, line 135, in parse_dbt_ls_output
        unique_id=node_dict[unique_id]
    KeyError: 'unique_id'

Explanation

The customer recently changed their dbt project, adding print debug statements to one of their dbt macros.

This caused the dbt ls output to contain lines that were valid JSON but were not valid dbt nodes, as observed in:

11:20:43  Running with dbt=1.7.6
11:20:45  Registered adapter: bigquery=1.7.2
11:20:45  Unable to do partial parsing because saved manifest not found. Starting full parse.
/***************************/
Values returned by mac_get_values:
{}
/***************************/
{"name": "some_model", "resource_type": "model", "package_name": "some_package", "original_file_path": "models/some_model.sql", "unique_id": "model.some_package.some_model", "alias": "some_model_some_package_1_8_0", "config": {"enabled": true, "alias": "some_model_some_package-1.8.0", "schema": "some_schema", "database": null, "tags": [], "meta": {}, "group": null, "materialized": "view", "incremental_strategy": null, "persist_docs": {}, "post-hook": [], "pre-hook": [], "quoting": {}, "column_types": {}, "full_refresh": null, "unique_key": null, "on_schema_change": "ignore", "on_configuration_change": "apply", "grants": {}, "packages": [], "docs": {"show": true, "node_color": null}, "contract": {"enforced": false, "alias_types": true}, "access": "protected"}, "tags": [], "depends_on": {"macros": [], "nodes": ["source.some_source"]}}"""

Cosmos didn't consider this use case. It assumed if a line was a JSON, it should be a dbt node:

astronomer-cosmos/cosmos/dbt/graph.py

Lines 161 to 185 in 42a397f

	def parse_dbt_ls_output(project_path: Path | None, ls_stdout: str) -> dict[str, DbtNode]:
	"""Parses the output of `dbt ls` into a dictionary of `DbtNode` instances."""
	nodes = {}
	for line in ls_stdout.split("\n"):
	try:
	node_dict = json.loads(line.strip())
	except json.decoder.JSONDecodeError:
	logger.debug("Skipped dbt ls line: %s", line)
	else:
	node = DbtNode(
	unique_id=node_dict["unique_id"],
	resource_type=DbtResourceType(node_dict["resource_type"]),
	depends_on=node_dict.get("depends_on", {}).get("nodes", []),
	file_path=project_path / node_dict["original_file_path"],
	tags=node_dict.get("tags", []),
	config=node_dict.get("config", {}),
	has_freshness=(
	is_freshness_effective(node_dict.get("freshness"))
	if DbtResourceType(node_dict["resource_type"]) == DbtResourceType.SOURCE
	else False
	),
	)
	nodes[node.unique_id] = node
	logger.debug("Parsed dbt resource `%s` of type `%s`", node.unique_id, node.resource_type)
	return nodes

Workaround

If customers updated the macro to print the information in a single line, they'd no longer observe the issue:

Values returned by mac_get_values: {}

We also released 1.5.0rc2 with the change #1295, similar to the one introduced by this PR.

Fix

This change makes Cosmos more resilient to scenarios where dbt ls may output JSON lines that are not valid dbt nodes. It also logs those lines to help troubleshoot. We added a unit test to make sure we continue supporting this use-case.

[netlify]

✅ Deploy Preview for sunny-pastelito-5ecb04 canceled.

Name	Link
🔨 Latest commit	a06266c
🔍 Latest deploy log	https://app.netlify.com/sites/sunny-pastelito-5ecb04/deploys/6723821d5d3bab00086967a5

[pankajkoti]

Wow, what a real quick debug and fix 👏

LGTM.

[tatiana]

Thanks a lot for the rapid review, @pankajkoti - specially considering it's Diwali.