Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Emit telemetry to Scarf during DAG run #1397

Merged
merged 46 commits into from
Dec 20, 2024
Merged
Changes from 1 commit
Commits
Show all changes
46 commits
Select commit Hold shift + click to select a range
3eb5d08
Add telemetry module and tests
tatiana Dec 17, 2024
6e0c6f5
Emit telemetry via Airflow listener
tatiana Dec 17, 2024
c124b79
Change version
tatiana Dec 18, 2024
ee8e928
Try to fix version issue in CI when running hatch type-check
tatiana Dec 18, 2024
bbcce4a
Fix unit test
tatiana Dec 18, 2024
c484e50
Fix unit test
tatiana Dec 18, 2024
c7d282f
Expose more metrics in Scarf
tatiana Dec 19, 2024
e460db2
Fix spelling
tatiana Dec 19, 2024
0300809
Update cosmos/constants.py
tatiana Dec 19, 2024
e8c1a26
Add docs about telemetry
tatiana Dec 19, 2024
44387b3
Update cosmos/__init__.py
tatiana Dec 19, 2024
3ba1580
Add debug messages
tatiana Dec 19, 2024
79650e4
Add more logs
tatiana Dec 19, 2024
4de8c62
Fix import
tatiana Dec 19, 2024
5b0207a
Change version for testing
tatiana Dec 19, 2024
a84dc7c
Add more logs to troubleshoot
tatiana Dec 19, 2024
585f209
Bump the version to see changes in scarf
tatiana Dec 19, 2024
aac353e
Stop using context manager to measure time of emitting telemetry
tatiana Dec 19, 2024
a851920
troubleshot
tatiana Dec 19, 2024
adc54d1
Troubleshoot
tatiana Dec 19, 2024
1de1083
Upgrade telemetry
tatiana Dec 19, 2024
15c1688
Improve test coverage
tatiana Dec 19, 2024
8919ef0
Fix unit test
tatiana Dec 19, 2024
aaa19da
Change log levels to analyse behaviour in Astro CLI
tatiana Dec 20, 2024
fc7b502
Change logger
tatiana Dec 20, 2024
a1d63ee
Add more logs
tatiana Dec 20, 2024
7ef47f5
improve logs
tatiana Dec 20, 2024
2f3ddb7
Log more info
tatiana Dec 20, 2024
940a1a2
add more logs
tatiana Dec 20, 2024
7661ab3
Add logs
tatiana Dec 20, 2024
7ae6152
Force to fetch the actual dag, add more debug logs
tatiana Dec 20, 2024
5caf6a8
Force to fetch the actual dag, add more debug logs
tatiana Dec 20, 2024
430aec3
Force to fetch the actual dag, add more debug logs
tatiana Dec 20, 2024
bc98126
Force to fetch the actual dag, add more debug logs
tatiana Dec 20, 2024
41a9293
debugging
tatiana Dec 20, 2024
47fb46c
Attempt to work around RuntimeError: UNEXPECTED COMMIT - THIS WILL BR…
tatiana Dec 20, 2024
6dac09a
Remove DagBag from listener
tatiana Dec 20, 2024
f06f86a
Change logic to iterate over dag tasks
tatiana Dec 20, 2024
5b4fda2
Change logic to iterate over dag tasks
tatiana Dec 20, 2024
93da409
Improve identifying COsmos task class
tatiana Dec 20, 2024
ac93714
Clean up working telemetry
tatiana Dec 20, 2024
07752ac
Update privacy notice
tatiana Dec 20, 2024
7dc9f96
Fix privacy notice
tatiana Dec 20, 2024
f2aa9c7
Update cosmos/__init__.py
tatiana Dec 20, 2024
574c3ba
Update test_performance.py
tatiana Dec 20, 2024
c57f884
🎨 [pre-commit.ci] Auto format from pre-commit.com hooks
pre-commit-ci[bot] Dec 20, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Fix unit test
  • Loading branch information
tatiana committed Dec 19, 2024
commit bbcce4ac4999dea6e6556cfeb79fea73ddf200a1
2 changes: 1 addition & 1 deletion tests/test_telemetry.py
Original file line number Diff line number Diff line change
Expand Up @@ -81,7 +81,7 @@ def test_emit_usage_metrics_succeeds(caplog):
"event_type": "dag_run",
"status": "success",
"dag_hash": "d151d1fa2f03270ea116cc7494f2c591",
"task_count": 3,
"cosmos_task_count": 3,
}
is_success = telemetry.emit_usage_metrics(sample_metrics)
assert is_success
Expand Down