Enable Estimated Stats on Query Plan for Instrumentation Monitoring #18682

fgwang7w · 2022-11-15T06:55:29Z

Test plan - (Please fill in how you tested your changes)
Enforce a testcase to verify the plan entirety with stats info using graphviz format and json format

== RELEASE NOTES ==

General Changes
* Add estimated stats on plan output for EXPLAIN format in GRAPHVIZ and JSON

Before Change:

                                                                                                                    Query Plan
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 {
   "id" : "5",
   "name" : "Output",
   "identifier" : "[key1, value1]",
   "details" : "",
   "children" : [ {
     "id" : "80",
     "name" : "RemoteStreamingExchange",
     "identifier" : "[GATHER]",
     "details" : "",
     "children" : [ {
       "id" : "0",
       "name" : "TableScan",
       "identifier" : "[TableHandle {connectorId='hive', connectorHandle='HiveTableHandle{schemaName=default, tableName=test_grouped_join1, analyzePartitionValues=Optional.empty}', layout='Optional[default.test_grouped_join1{buckets=13}]'}]",
       "details" : "LAYOUT: default.test_grouped_join1{buckets=13}\nvalue1 := value1:varchar(79):1:REGULAR (1:37)\nkey1 := key1:bigint:0:REGULAR (1:37)\n",
       "children" : [ ],
       "remoteSources" : [ ]
     } ],
     "remoteSources" : [ ]
   } ],
   "remoteSources" : [ ]
 }
(1 row)

Query 20221212_231733_00000_2kdqx, FINISHED, 1 node
Splits: 1 total, 1 done (100.00%)
0:01 [0 rows, 0B] [0 rows/s, 0B/s]

After Change:

                                                                                                                    Query Plan
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 {
   "id" : "5",
   "name" : "Output",
   "identifier" : "[key1, value1]",
   "details" : "",
   "children" : [ {
     "id" : "80",
     "name" : "RemoteStreamingExchange",
     "identifier" : "[GATHER]",
     "details" : "",
     "children" : [ {
       "id" : "0",
       "name" : "TableScan",
       "identifier" : "[TableHandle {connectorId='hive', connectorHandle='HiveTableHandle{schemaName=default, tableName=test_grouped_join1, analyzePartitionValues=Optional.empty}', layout='Optional[default.test_grouped_join1{buckets=13}]'}]",
       "details" : "LAYOUT: default.test_grouped_join1{buckets=13}\nvalue1 := value1:varchar(79):1:REGULAR (1:37)\nkey1 := key1:bigint:0:REGULAR (1:37)\n",
       "children" : [ ],
       "remoteSources" : [ ],
       "estimates" : [ {
         "outputRowCount" : 15000.0,
         "totalSize" : 1057364.0,
         "confident" : true,
         "variableStatistics" : {
           "value1<varchar(79)>" : {
             "lowValue" : "-Infinity",
             "highValue" : "Infinity",
             "nullsFraction" : 0.0,
             "averageRowSize" : 48.49093333333333,
             "distinctValuesCount" : 15000.0
           },
           "key1<bigint>" : {
             "lowValue" : 1.0,
             "highValue" : 60000.0,
             "nullsFraction" : 0.0,
             "averageRowSize" : "NaN",
             "distinctValuesCount" : 15000.0
           }
         }
       } ]
     } ],
     "remoteSources" : [ ],
     "estimates" : [ {
       "outputRowCount" : 15000.0,
       "totalSize" : 1057364.0,
       "confident" : true,
       "variableStatistics" : {
         "value1<varchar(79)>" : {
           "lowValue" : "-Infinity",
           "highValue" : "Infinity",
           "nullsFraction" : 0.0,
           "averageRowSize" : 48.49093333333333,
           "distinctValuesCount" : 15000.0
         },
         "key1<bigint>" : {
           "lowValue" : 1.0,
           "highValue" : 60000.0,
           "nullsFraction" : 0.0,
           "averageRowSize" : "NaN",
           "distinctValuesCount" : 15000.0
         }
       }
     } ]
   } ],
   "remoteSources" : [ ],
   "estimates" : [ {
     "outputRowCount" : 15000.0,
     "totalSize" : 1057364.0,
     "confident" : true,
     "variableStatistics" : {
       "value1<varchar(79)>" : {
         "lowValue" : "-Infinity",
         "highValue" : "Infinity",
         "nullsFraction" : 0.0,
         "averageRowSize" : 48.49093333333333,
         "distinctValuesCount" : 15000.0
       },
       "key1<bigint>" : {
         "lowValue" : 1.0,
         "highValue" : 60000.0,
         "nullsFraction" : 0.0,
         "averageRowSize" : "NaN",
         "distinctValuesCount" : 15000.0
       }
     }
   } ]
 }
(1 row)

fgwang7w · 2022-11-16T17:23:00Z

@aaneja could you please help give a first pass on this PR? Thanks!

v-jizhang · 2022-11-16T21:22:32Z

@bot kick off tests

aaneja

Some minor comments, but LGTM otherwise

presto-tests/src/main/java/com/facebook/presto/tests/AbstractTestQueries.java

presto-main/src/main/java/com/facebook/presto/sql/planner/planPrinter/JsonRenderer.java

presto-tests/src/test/java/com/facebook/presto/execution/TestEventListener.java

presto-main/src/main/java/com/facebook/presto/util/GraphvizPrinter.java

presto-tests/src/test/java/com/facebook/presto/execution/TestEventListener.java

presto-main/src/main/java/com/facebook/presto/sql/planner/planPrinter/JsonRenderer.java

presto-main/src/main/java/com/facebook/presto/sql/planner/planPrinter/PlanPrinter.java

aaneja

LGTM, minor comments

yingsu00

@fgwang7w George, Could you paste some sample of the plan output for both before and after?

presto-main/src/main/java/com/facebook/presto/sql/analyzer/QueryExplainer.java

presto-main/src/main/java/com/facebook/presto/sql/planner/planPrinter/PlanPrinter.java

presto-main/src/main/java/com/facebook/presto/cost/PlanNodeStatsEstimate.java

presto-main/src/main/java/com/facebook/presto/sql/Serialization.java

presto-main/src/test/java/com/facebook/presto/sql/planner/planPrinter/TestJsonRenderer.java

fgwang7w · 2022-12-12T23:23:16Z

@yingsu00 thank you for the review. I have updated the before/after change. All comments are addressed. Could you please help take a look again?

presto-main/src/main/java/com/facebook/presto/sql/planner/planPrinter/PlanPrinter.java

presto-main/src/main/java/com/facebook/presto/cost/PlanNodeStatsEstimate.java

presto-main/src/main/java/com/facebook/presto/sql/planner/planPrinter/PlanPrinter.java

presto-main/src/main/java/com/facebook/presto/util/GraphvizPrinter.java

presto-main/src/test/java/com/facebook/presto/sql/planner/planPrinter/TestJsonRenderer.java

presto-main/src/main/java/com/facebook/presto/sql/Serialization.java

presto-main/src/main/java/com/facebook/presto/sql/planner/BasePlanFragmenter.java

presto-main/src/main/java/com/facebook/presto/sql/planner/LocalExecutionPlanner.java

presto-main/src/test/java/com/facebook/presto/util/TestGraphvizPrinter.java

fgwang7w · 2022-12-18T20:56:40Z

Thank you @yingsu00 for helping review the PR. All comments are addressed. could you please help give another pass? Thanks!

fgwang7w · 2022-12-20T19:34:46Z

@prestodb/committers ping!

yingsu00

@fgwang7w Could you please show me how you resolved the TypeManager initialization issue in presto-main/src/main/java/com/facebook/presto/sql/Serialization.java?

presto-main/src/main/java/com/facebook/presto/sql/Serialization.java

yingsu00 · 2022-12-29T00:01:02Z

presto-main/src/main/java/com/facebook/presto/sql/Serialization.java

@@ -107,7 +107,15 @@ public void serialize(VariableReferenceExpression value, JsonGenerator jsonGener
    public static class VariableReferenceExpressionDeserializer
            extends KeyDeserializer
    {
-        private final TypeManager typeManager;
+        // when it is on the test code path, jackson deserializer instantiates the default constructor with no arg,


Is this paragraph still valid?

The fix to avoid using createTestFunctionAndTypeManager in Serialization main code path is to create a json mapper provider using the SerDe function for VariableReferenceExpression and use this custom json codec for serialization. This does not need any special no-args handling on the testing code path, e.g. TestExplainVerification, the planCodec is bind with JsonObjectMapperProvider that uses VariableReferenceExpressionDeserializer.

fgwang7w · 2022-12-29T19:42:39Z

tests all green. Many thanks for reviewing and approving this PR. @yingsu00 @aaneja

fgwang7w force-pushed the graphvizstats branch from 5fa2734 to d713726 Compare November 15, 2022 07:37

fgwang7w changed the title ~~Enable estimated stats on plan output for format in graphviz and json~~ Enable estimated stats on query plan for instrumentation monitoring Nov 16, 2022

fgwang7w changed the title ~~Enable estimated stats on query plan for instrumentation monitoring~~ Enable Estimated Stats on Query Plan for Instrumentation Monitoring Nov 16, 2022

fgwang7w force-pushed the graphvizstats branch from d713726 to 53bb9fc Compare November 16, 2022 17:22

fgwang7w marked this pull request as ready for review November 16, 2022 17:22

fgwang7w requested a review from a team as a code owner November 16, 2022 17:22

fgwang7w requested a review from presto-oss November 16, 2022 17:22

fgwang7w force-pushed the graphvizstats branch from 53bb9fc to c12d0f3 Compare November 16, 2022 18:45

fgwang7w force-pushed the graphvizstats branch from c12d0f3 to ab6bdcb Compare November 17, 2022 08:35

aaneja requested changes Nov 21, 2022

View reviewed changes

fgwang7w force-pushed the graphvizstats branch 5 times, most recently from 93ac48d to 7db18ba Compare December 6, 2022 01:52

fgwang7w requested a review from aaneja December 6, 2022 01:59

fgwang7w force-pushed the graphvizstats branch 2 times, most recently from 769bfad to bf82e3b Compare December 6, 2022 04:50

yingsu00 self-requested a review December 9, 2022 04:54

aaneja reviewed Dec 9, 2022

View reviewed changes

presto-main/src/main/java/com/facebook/presto/sql/planner/planPrinter/JsonRenderer.java Show resolved Hide resolved

aaneja reviewed Dec 9, 2022

View reviewed changes

presto-main/src/main/java/com/facebook/presto/sql/planner/planPrinter/PlanPrinter.java Outdated Show resolved Hide resolved

aaneja reviewed Dec 9, 2022

View reviewed changes

fgwang7w force-pushed the graphvizstats branch 2 times, most recently from e6eea28 to 9ac09ab Compare December 12, 2022 03:11

yingsu00 reviewed Dec 12, 2022

View reviewed changes

yingsu00 self-assigned this Dec 12, 2022

fgwang7w force-pushed the graphvizstats branch from 9ac09ab to 30b4d4a Compare December 12, 2022 22:57

yingsu00 reviewed Dec 13, 2022

View reviewed changes

fgwang7w force-pushed the graphvizstats branch 3 times, most recently from 885e8e4 to 3f4328c Compare December 14, 2022 02:07

fgwang7w added the in-final-review label Dec 14, 2022

fgwang7w force-pushed the graphvizstats branch from 3f4328c to 061ea67 Compare December 14, 2022 19:06

yingsu00 reviewed Dec 17, 2022

View reviewed changes

fgwang7w force-pushed the graphvizstats branch 2 times, most recently from c05c968 to d03908d Compare December 18, 2022 05:39

fgwang7w force-pushed the graphvizstats branch from d03908d to b06bdf1 Compare December 18, 2022 20:58

aaneja approved these changes Dec 22, 2022

View reviewed changes

fgwang7w force-pushed the graphvizstats branch from b06bdf1 to 11ed45e Compare December 23, 2022 07:15

fgwang7w closed this Dec 23, 2022

fgwang7w reopened this Dec 23, 2022

fgwang7w force-pushed the graphvizstats branch 3 times, most recently from 56c65e1 to df3d362 Compare December 27, 2022 05:49

yingsu00 reviewed Dec 29, 2022

View reviewed changes

fgwang7w force-pushed the graphvizstats branch from df3d362 to c488562 Compare December 29, 2022 06:14

fgwang7w added 3 commits December 28, 2022 22:30

Add estimates to graphviz of the plan

e5d3a52

Add estimates to json format of the plan

f300623

Add Graphviz query plan monitoring for instrumentation

da88672

fgwang7w force-pushed the graphvizstats branch from c488562 to da88672 Compare December 29, 2022 06:47

yingsu00 approved these changes Dec 29, 2022

View reviewed changes

yingsu00 merged commit 94c1d72 into prestodb:master Dec 30, 2022

fgwang7w deleted the graphvizstats branch December 30, 2022 05:30

wanglinsong mentioned this pull request Jan 12, 2023

Add release notes for 0.279 #18920

Merged

30 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable Estimated Stats on Query Plan for Instrumentation Monitoring #18682

Enable Estimated Stats on Query Plan for Instrumentation Monitoring #18682

fgwang7w commented Nov 15, 2022 •

edited

Loading

fgwang7w commented Nov 16, 2022

v-jizhang commented Nov 16, 2022

aaneja left a comment

aaneja left a comment

yingsu00 left a comment

fgwang7w commented Dec 12, 2022

fgwang7w commented Dec 18, 2022

fgwang7w commented Dec 20, 2022

yingsu00 left a comment

yingsu00 Dec 29, 2022

fgwang7w Dec 29, 2022

fgwang7w commented Dec 29, 2022

Enable Estimated Stats on Query Plan for Instrumentation Monitoring #18682

Enable Estimated Stats on Query Plan for Instrumentation Monitoring #18682

Conversation

fgwang7w commented Nov 15, 2022 • edited Loading

fgwang7w commented Nov 16, 2022

v-jizhang commented Nov 16, 2022

aaneja left a comment

Choose a reason for hiding this comment

aaneja left a comment

Choose a reason for hiding this comment

yingsu00 left a comment

Choose a reason for hiding this comment

fgwang7w commented Dec 12, 2022

fgwang7w commented Dec 18, 2022

fgwang7w commented Dec 20, 2022

yingsu00 left a comment

Choose a reason for hiding this comment

yingsu00 Dec 29, 2022

Choose a reason for hiding this comment

fgwang7w Dec 29, 2022

Choose a reason for hiding this comment

fgwang7w commented Dec 29, 2022

fgwang7w commented Nov 15, 2022 •

edited

Loading