feat: apply post processing to chart data #15843
Conversation
Codecov Report
@@ Coverage Diff @@
## master #15843 +/- ##
==========================================
- Coverage 77.13% 76.88% -0.26%
==========================================
Files 984 985 +1
Lines 51706 51840 +134
Branches 6995 7037 +42
==========================================
- Hits 39882 39855 -27
- Misses 11600 11760 +160
- Partials 224 225 +1
Flags with carried forward coverage won't be shown. Click here to find out more.
Continue to review full report at Codecov.
|
|
Hi @betodealmeida, Thanks for this improvement. But, please hold on to this PR. Can we do this by constructing a different query object on the front end? |
@zhaoyongjie I don't think we can. We would have to express the aggregation functions for the pivoting using the query context. For other text-based visualization like the t-test table it would also by really hard to express the statistical significance logic in a backend-agnostic query. What are your concerns here? |
Sorry for reply is a little late. The newest v1 API has
|
|
Merging this even if it's not the ideal solution, since it's blocking some important work. Let's talk about this on the next meetup. |
|
@betodealmeida using this option on a pivot table v2 that has 'number' fields (in the group by) and using the CSV format, gives me the following error:
When selecting json format, it works fine. Removing the number fields seems to fix the issue. Is this a known problem/limitation? |
|
@betodealmeida I see it was fixed in #16259. |
* feat: apply post processing to chart data * Fix tests and lint * Fix lint * trigger tests
* feat: apply post processing to chart data * Fix tests and lint * Fix lint * trigger tests
* feat: apply post processing to chart data * Fix tests and lint * Fix lint * trigger tests
* feat: apply post processing to chart data * Fix tests and lint * Fix lint * trigger tests
SUMMARY
This PR introduces a new
ChartDataResultType,ChartDataResultType.POST_PROCESSED. It allows us to request the data for a given chart post-processed in the same way the visualization does in Javascript.Eg, for this pivot table:
Normally if we request the data for the chart we get the pre-processed (unpivoted) data.
Doing a
GETon "http://localhost:9000/api/v1/chart/72/data/?format=json" returns:{ "result": [ { "cache_key": "d432a37512f63a88562744f4da62021e", "cached_dttm": null, "cache_timeout": 86400, "annotation_data": {}, "error": null, "is_cached": false, "query": "SELECT gender AS gender,\n state AS state,\n sum(num) AS \"Births\"\nFROM birth_names\nWHERE ds >= TO_TIMESTAMP('1921-07-22 00:00:00.000000', 'YYYY-MM-DD HH24:MI:SS.US')\n AND ds < TO_TIMESTAMP('2021-07-22 11:56:04.000000', 'YYYY-MM-DD HH24:MI:SS.US')\nGROUP BY gender,\n state\nLIMIT 100", "status": "success", "stacktrace": null, "rowcount": 22, "colnames": [ "gender", "state", "Births" ], "coltypes": [ 1, 1, 0 ], "data": [ { "gender": "boy", "state": "PA", "Births": 2390275 }, { "gender": "girl", "state": "PA", "Births": 1615383 }, { "gender": "boy", "state": "OH", "Births": 2376385 }, { "gender": "girl", "state": "MI", "Births": 1326229 }, { "gender": "girl", "state": "FL", "Births": 1312593 }, { "gender": "boy", "state": "CA", "Births": 5430796 }, { "gender": "boy", "state": "other", "Births": 22044909 }, { "gender": "girl", "state": "OH", "Births": 1622814 }, { "gender": "girl", "state": "TX", "Births": 2313186 }, { "gender": "boy", "state": "NY", "Births": 3543961 }, { "gender": "boy", "state": "IL", "Births": 2357411 }, { "gender": "girl", "state": "CA", "Births": 3567754 }, { "gender": "girl", "state": "NY", "Births": 2280733 }, { "gender": "girl", "state": "IL", "Births": 1614427 }, { "gender": "girl", "state": "other", "Births": 15058341 }, { "gender": "boy", "state": "TX", "Births": 3311985 }, { "gender": "boy", "state": "FL", "Births": 1968060 }, { "gender": "boy", "state": "MA", "Births": 1285126 }, { "gender": "boy", "state": "MI", "Births": 1938321 }, { "gender": "boy", "state": "NJ", "Births": 1486126 }, { "gender": "girl", "state": "NJ", "Births": 992702 }, { "gender": "girl", "state": "MA", "Births": 842146 } ], "applied_filters": [], "rejected_filters": [] } ] }Note that the results above have 22 rows and 3 columns: "gender", "state", and "Birth" (the metric).
With this PR we can do a
GETrequest to "http://localhost:9000/api/v1/chart/72/data/?format=json&type=post_processed" to get:{ "result": [ { "cache_key": "d432a37512f63a88562744f4da62021e", "cached_dttm": null, "cache_timeout": 86400, "annotation_data": {}, "error": null, "is_cached": false, "query": "SELECT gender AS gender,\n state AS state,\n sum(num) AS \"Births\"\nFROM birth_names\nWHERE ds >= TO_TIMESTAMP('1921-07-22 00:00:00.000000', 'YYYY-MM-DD HH24:MI:SS.US')\n AND ds < TO_TIMESTAMP('2021-07-22 12:03:21.000000', 'YYYY-MM-DD HH24:MI:SS.US')\nGROUP BY gender,\n state\nLIMIT 100", "status": "success", "stacktrace": null, "rowcount": 3, "colnames": [ "Births CA", "Births FL", "Births IL", "Births MA", "Births MI", "Births NJ", "Births NY", "Births OH", "Births PA", "Births TX", "Births other", "Births All" ], "coltypes": [ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 ], "data": [ { "Births CA": 5430796, "Births FL": 1968060, "Births IL": 2357411, "Births MA": 1285126, "Births MI": 1938321, "Births NJ": 1486126, "Births NY": 3543961, "Births OH": 2376385, "Births PA": 2390275, "Births TX": 3311985, "Births other": 22044909, "Births All": 22044909, "gender": "boy" }, { "Births CA": 3567754, "Births FL": 1312593, "Births IL": 1614427, "Births MA": 842146, "Births MI": 1326229, "Births NJ": 992702, "Births NY": 2280733, "Births OH": 1622814, "Births PA": 1615383, "Births TX": 2313186, "Births other": 15058341, "Births All": 15058341, "gender": "girl" }, { "Births CA": 5430796, "Births FL": 1968060, "Births IL": 2357411, "Births MA": 1285126, "Births MI": 1938321, "Births NJ": 1486126, "Births NY": 3543961, "Births OH": 2376385, "Births PA": 2390275, "Births TX": 3311985, "Births other": 22044909, "Births All": 22044909, "gender": "All" } ], "applied_filters": [], "rejected_filters": [] } ] }Note that we now get the correct results, with 3 rows and many columns from the pivot.
This functionality was added to support a very small subset of visualization types:
Because of the small number of visualization types, I opted to do this by reimplementing the chart logic in Python. The alternative would be to run the visualization under Selenium and scrape the data, which is inefficient and still requires keeping the scraping logic up-to-date with the plugins.
BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
TESTING INSTRUCTIONS
Create and save a pivot chart (not v2). Check the data at "http://localhost:9000/api/v1/chart/{chartid}/data/?format=json&type=post_processed" and "http://localhost:9000/api/v1/chart/{chartid}/data/?format=json&type=full".
ADDITIONAL INFORMATION