-
Notifications
You must be signed in to change notification settings - Fork 181
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: Add ability to broadcast daft metrics over HTTP #3778
Conversation
CodSpeed Performance ReportMerging #3778 will degrade performances by 18.04%Comparing Summary
Benchmarks breakdown
|
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #3778 +/- ##
==========================================
+ Coverage 75.59% 77.85% +2.26%
==========================================
Files 739 740 +1
Lines 97206 93803 -3403
==========================================
- Hits 73484 73034 -450
+ Misses 23722 20769 -2953
|
daft/dataframe/dataframe.py
Outdated
def broadcast_metrics(f): | ||
@wraps(f) | ||
def wrapped(self, *args, **kwargs): | ||
self._explain_broadcast() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I still think this is wrong because most of the functions f
will add to the plan before execution. E.g. write_parquet will add a write parquet node to the plan.
2778d2f
to
ca81621
Compare
Closing in favour of #3790. |
Overview
This PR adds the ability of broadcasting the serialized Mermaid query plan over port 3238.
Receiving services can listen to this TCP broadcast over 3238, read the data, and process/present it.
This will be useful for the dashboard, which will (through another proxy service) be listening over 3238 for the plan, so that it can display it to the end-user.
Notes
This is a refreshed version of this closed PR.
Port choice
The default port to broadcast on was chosen to be 3238. The reason here being because that is
daft
spelled over the telephone's alphanumeric keypad.