[native] Table writer 2: Add Presto write protocols #18377

gggrace14 · 2022-09-21T01:13:37Z

Add PrestoNoCommitWriteProtocol and PrestoTaskCommitWriteProtocol,
which extends the write protocols of Hive connector and make commit()
return the specific format of outputs expected by Presto from
the table writer. Register the two write protocols during server start.
The right protocol will be picked up by the table writer,
given the CommitStrategy used by the TableWriter.

mbasmanova · 2022-11-02T08:05:20Z

@gggrace14 There are CI failures. Are these related?

mbasmanova

@gggrace14 Looks good overall. Some comments below.

mbasmanova · 2022-11-02T08:07:32Z

presto-native-execution/presto_cpp/main/types/PrestoToVeloxQueryPlan.cpp

+    case protocol::TableType::TEMPORARY:
+      return connector::hive::LocationHandle::TableType::kTemporary;
+    default:
+      throw std::invalid_argument("Unknown table type");


Use VELOX_USER_CHECK and include tableType in the error message (see toJsonString helper function) to simplify troubleshooting.

Revised here and other places. Found in this file we use VELOX_UNSUPPORTED for this case

mbasmanova · 2022-11-02T08:07:40Z

presto-native-execution/presto_cpp/main/types/PrestoToVeloxQueryPlan.cpp

+      return connector::hive::LocationHandle::WriteMode::
+          kDirectToTargetExistingDirectory;
+    default:
+      throw std::invalid_argument("Unknown write mode");


mbasmanova · 2022-11-02T08:09:05Z

presto-native-execution/presto_cpp/main/types/PrestoToVeloxQueryPlan.cpp

@@ -1662,6 +1717,7 @@ VeloxQueryPlanConverter::toVeloxQueryPlan(
      node->columnNames,
      insertTableHandle,
      outputType,
+      connector::WriteProtocol::CommitStrategy::kNoCommit,


Why is no-commit protocol hard-coded here?

Here is where a system sets the commit strategy it's going to use. CommitStrategy::kNoCommit is set for Presto.

Also discussed with MJ previously, and for PrestoSpark, he will change here to add a condition to set kTaskCommit for PrestoSpark and kNoCommit for Presto.

mbasmanova · 2022-11-02T08:09:52Z

presto-native-execution/presto_cpp/presto_protocol/WriteProtocol.h

+class PrestoNoCommitWriteProtocol
+    : public velox::connector::hive::HiveNoCommitWriteProtocol {
+ public:
+  ~PrestoNoCommitWriteProtocol() override {}


= default or remove this and let compiler auto-generate

mbasmanova · 2022-11-02T08:10:01Z

presto-native-execution/presto_cpp/presto_protocol/WriteProtocol.h

+        std::make_shared<PrestoNoCommitWriteProtocol>());
+  }
+
+  // private:


remove commented out code

mbasmanova · 2022-11-02T08:18:15Z

presto-native-execution/presto_cpp/presto_protocol/WriteProtocol.cpp

+    rowWrittenVector = std::dynamic_pointer_cast<FlatVector<int64_t>>(
+        BaseVector::create(BIGINT(), 1, pool));
+    rowWrittenVector->set(0, commitInfo.numWrittenRows());
+    columns.emplace_back(rowWrittenVector);


nit: consider shortening

columns.emplace_back(BaseVector::createConstant(commitInfo.numWrittenRows(), 1, pool))

Good to know

mbasmanova · 2022-11-02T08:18:52Z

presto-native-execution/presto_cpp/presto_protocol/WriteProtocol.cpp

+
+  vector_size_t numOutputRows = 1;
+  FlatVectorPtr<int64_t> rowWrittenVector;
+  FlatVectorPtr<StringView> fragmentsVector;


fragmentsVector and commitContextVector are used only in the 'else' branch. Consider moving them there.

mbasmanova · 2022-11-02T08:19:25Z

presto-native-execution/presto_cpp/presto_protocol/WriteProtocol.cpp

+        commitInfo.connectorCommitInfo());
+    numOutputRows = hiveCommitInfo.writerParameters().size() + 1;
+
+    // Set rows column


add period at the end of the sentence; ditto other places

mbasmanova · 2022-11-02T08:20:24Z

presto-native-execution/presto_cpp/presto_protocol/WriteProtocol.cpp

+    numOutputRows = hiveCommitInfo.writerParameters().size() + 1;
+
+    // Set rows column
+    rowWrittenVector = std::dynamic_pointer_cast<FlatVector<int64_t>>(


use BaseVector::create<FlatVector<int64_t>>(...)

ditto other places

mbasmanova · 2022-11-02T08:21:28Z

presto-native-execution/presto_cpp/presto_protocol/WriteProtocol.cpp

+  return std::make_shared<RowVector>(
+      pool,
+      commitInfo.outputType(),
+      BufferPtr(nullptr),


nit: just "nullptr"

Good to know

mbasmanova

@gggrace14 Looks good to me. Please, make sure all CI is green before landing.

mbasmanova · 2022-11-03T17:21:49Z

presto-native-execution/presto_cpp/presto_protocol/WriteProtocol.cpp

+         ("targetFileName", hiveWriterParameters->targetFileName())
+         ("fileSize", 0)))
+     ("rowCount", numWrittenRows)
+     ("inMemoryDataSizeInBytes", 0)


https://fburl.com/18vim7k4.

Is this accessible outside of Meta?

mbasmanova · 2022-11-03T17:22:29Z

presto-native-execution/presto_cpp/presto_protocol/WriteProtocol.h

+  }
+};
+
+class PrestoHiveTaskCommitWriteProtocol


Since we are inside presto::protocol namespace, "Presto" prefix can be removed.

Add PrestoNoCommitWriteProtocol and PrestoTaskCommitWriteProtocol, which extends the write protocols of Hive connector and make commit() return the specific format of outputs expected by Presto from the table writer. Register the two write protocols during server start. The right protocol will be picked up by the table writer, given the CommitStrategy used by the TableWriter.

gggrace14 force-pushed the insert branch 4 times, most recently from d39e15a to 47a898e Compare September 26, 2022 17:16

gggrace14 force-pushed the insert branch from 47a898e to e0be818 Compare September 29, 2022 21:19

gggrace14 force-pushed the insert branch 3 times, most recently from 596d4f9 to 78198b6 Compare October 7, 2022 20:33

gggrace14 marked this pull request as ready for review October 7, 2022 20:43

gggrace14 requested a review from a team as a code owner October 7, 2022 20:43

gggrace14 requested review from spershin and mbasmanova October 7, 2022 20:44

gggrace14 force-pushed the insert branch 2 times, most recently from 964c0eb to c84c999 Compare October 14, 2022 23:34

gggrace14 changed the title ~~[native] Support INSERT INTO unpartitioned table~~ [native] Add Presto write protocols Oct 14, 2022

gggrace14 changed the title ~~[native] Add Presto write protocols~~ [native] Table write 2: Add Presto write protocols Oct 14, 2022

gggrace14 force-pushed the insert branch 5 times, most recently from 9e68793 to c069b87 Compare October 21, 2022 08:22

gggrace14 changed the title ~~[native] Table write 2: Add Presto write protocols~~ [native] Table writer: Add Presto write protocols Oct 21, 2022

gggrace14 force-pushed the insert branch from c069b87 to 3f2675c Compare October 21, 2022 08:27

gggrace14 requested review from majetideepak and aditi-pandit October 21, 2022 16:43

gggrace14 force-pushed the insert branch from 3f2675c to b8ef630 Compare October 25, 2022 23:51

gggrace14 changed the title ~~[native] Table writer: Add Presto write protocols~~ [native] Table writer 2: Add Presto write protocols Oct 25, 2022

gggrace14 force-pushed the insert branch 3 times, most recently from 6c33e23 to 80b2b95 Compare October 27, 2022 22:25

gggrace14 force-pushed the insert branch 3 times, most recently from ca52b64 to 39ca0c5 Compare November 2, 2022 04:55

mbasmanova reviewed Nov 2, 2022

View reviewed changes

gggrace14 force-pushed the insert branch 3 times, most recently from b8f071f to f0d3a97 Compare November 3, 2022 16:20

mbasmanova approved these changes Nov 3, 2022

View reviewed changes

gggrace14 mentioned this pull request Nov 4, 2022

[native] Table writer 1: Pass LocationHandle to HiveInsertTableHandle #18490

Merged

gggrace14 force-pushed the insert branch 7 times, most recently from cca4b30 to ab21950 Compare November 11, 2022 23:16

gggrace14 force-pushed the insert branch from ab21950 to 8f5f930 Compare November 12, 2022 04:45

gggrace14 merged commit c329c38 into prestodb:master Nov 12, 2022

gggrace14 deleted the insert branch November 12, 2022 07:14

wanglinsong mentioned this pull request Jan 12, 2023

Add release notes for 0.279 #18920

Merged

30 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[native] Table writer 2: Add Presto write protocols #18377

[native] Table writer 2: Add Presto write protocols #18377

gggrace14 commented Sep 21, 2022 •

edited

Loading

mbasmanova commented Nov 2, 2022

mbasmanova left a comment

mbasmanova Nov 2, 2022

gggrace14 Nov 2, 2022

mbasmanova Nov 2, 2022

mbasmanova Nov 2, 2022

gggrace14 Nov 2, 2022

mbasmanova Nov 2, 2022

mbasmanova Nov 2, 2022

mbasmanova Nov 2, 2022

gggrace14 Nov 3, 2022

mbasmanova Nov 2, 2022

mbasmanova Nov 2, 2022

mbasmanova Nov 2, 2022

mbasmanova Nov 2, 2022

gggrace14 Nov 3, 2022

mbasmanova left a comment

mbasmanova Nov 3, 2022

mbasmanova Nov 3, 2022

gggrace14 Nov 3, 2022

[native] Table writer 2: Add Presto write protocols #18377

[native] Table writer 2: Add Presto write protocols #18377

Conversation

gggrace14 commented Sep 21, 2022 • edited Loading

mbasmanova commented Nov 2, 2022

mbasmanova left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mbasmanova left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gggrace14 commented Sep 21, 2022 •

edited

Loading