[native] Add support for REST based remote function #23568

Joe-Abraham · 2024-09-02T06:07:11Z

Description

Part of Implement REST API for Remote Functions in Presto (Based on RFC-0007) #23682

Dependency

Joe-Abraham · 2024-11-28T03:52:11Z

@aditi-pandit Can you please review the changes

pramodsatya

Thanks @Joe-Abraham, could you add e2e tests as well?

pramodsatya · 2025-01-31T17:22:11Z

presto-native-execution/presto_cpp/main/types/CMakeLists.txt

@@ -18,12 +18,20 @@ add_library(
  presto_types OBJECT
  PrestoToVeloxQueryPlan.cpp PrestoToVeloxExpr.cpp VeloxPlanValidator.cpp
  PrestoToVeloxSplit.cpp PrestoToVeloxConnector.cpp)
+


nit: revert newline.

reverted the line

pramodsatya · 2025-01-31T17:25:26Z

presto-native-execution/presto_cpp/main/types/PrestoToVeloxExpr.cpp

@@ -412,6 +421,18 @@ std::optional<TypedExprPtr> VeloxExprConverter::tryConvertLike(
      returnType, args, getFunctionName(signature));
 }

+#ifdef PRESTO_ENABLE_REMOTE_FUNCTIONS
+PageFormat fromSerdeString(const std::string_view& serdeName) {


Do we need this as a separate function? There is only one caller and this check can be moved to the caller.

moved it to the function

pramodsatya · 2025-01-31T17:27:23Z

presto-native-execution/presto_cpp/main/types/PrestoToVeloxExpr.cpp

@@ -412,6 +421,18 @@ std::optional<TypedExprPtr> VeloxExprConverter::tryConvertLike(
      returnType, args, getFunctionName(signature));
 }

+#ifdef PRESTO_ENABLE_REMOTE_FUNCTIONS
+PageFormat fromSerdeString(const std::string_view& serdeName) {
+  if (serdeName == "presto_page") {


Avoid magic literal and use constant variable for "presto_page".

The implementation is done similar to the below code.

presto/presto-native-execution/presto_cpp/main/RemoteFunctionRegisterer.cpp

Line 39 in a7a7f80

if (serdeName == "presto_page") {

pramodsatya · 2025-01-31T17:30:43Z

presto-native-execution/presto_cpp/main/types/PrestoToVeloxExpr.cpp

    return std::make_shared<CallTypedExpr>(
        returnType, args, getFunctionName(sqlFunctionHandle->functionId));
  }
+#ifdef PRESTO_ENABLE_REMOTE_FUNCTIONS


Can we move this logic to a separate function, say registerVeloxRemoteFunction?

Thanks for the feedback, I have made the improvements

pramodsatya · 2025-01-31T17:33:51Z

presto-native-execution/presto_cpp/presto_protocol/core/presto_protocol_core.h

 };
 class ParseError : public Exception {
 public:
-  explicit ParseError(const std::string& message) : Exception(message){};
+  explicit ParseError(const std::string& message) : Exception(message) {};


Nit: Revert these changes. I believe these format-fixes were automatically added on Macos? They would likely cause a failure in format-check CI which runs on linux.

these are auto-generated, I have reverted these changes.

pramodsatya · 2025-01-31T17:35:18Z

presto-native-execution/presto_cpp/presto_protocol/core/presto_protocol_core.yml

@@ -175,7 +175,7 @@ AbstractClasses:
    subclasses:
      - { name: BuiltInFunctionHandle,      key: $static }
      - { name: SqlFunctionHandle,          key: json_file }
-


Nit: revert this newline removal.

reverted back the changes, Is there a need of two empty line?

pramodsatya · 2025-01-31T17:37:21Z

presto-native-execution/presto_cpp/main/types/tests/CMakeLists.txt

@@ -74,6 +74,16 @@ target_link_libraries(
  ${GFLAGS_LIBRARIES}
  pthread)

+if(PRESTO_ENABLE_REMOTE_FUNCTIONS)


Why is this change needed? No tests have been added for remote functions so could this be reverted?

Added these as part of build failure in MacOS, removed it.

pdabre12

Thanks @Joe-Abraham,
Is it possible to add these test cases(#23777) as part of this PR?

pdabre12 · 2025-02-10T22:50:00Z

presto-native-execution/presto_cpp/main/types/PrestoToVeloxExpr.cpp

+  velox::exec::FunctionSignatureBuilder signatureBuilder;
+
+  for (const auto& typeVar : prestoSignature.typeVariableConstraints) {
+    signatureBuilder.typeVariable(typeVar.name);
+  }
+
+  for (const auto& longVar : prestoSignature.longVariableConstraints) {
+    signatureBuilder.integerVariable(longVar.name);
+  }
+
+  signatureBuilder.returnType(prestoSignature.returnType);
+
+  for (const auto& argType : prestoSignature.argumentTypes) {
+    signatureBuilder.argumentType(argType);
+  }
+
+  if (prestoSignature.variableArity) {
+    signatureBuilder.variableArity();
+  }
+
+  auto signature = signatureBuilder.build();
+  std::vector<velox::exec::FunctionSignaturePtr> veloxSignatures = {signature};


Maybe extract this into a separate function?

pramodsatya

Thanks @Joe-Abraham.

pramodsatya · 2025-02-13T13:46:35Z

presto-native-execution/presto_cpp/main/common/Configs.cpp

@@ -342,6 +342,10 @@ std::string SystemConfig::remoteFunctionServerSerde() const {
  return optionalProperty(kRemoteFunctionServerSerde).value();
 }

+std::string SystemConfig::remoteFunctionRestUrl() const {


nit: rename to remoteFunctionServerRestURL() for consistency

pramodsatya · 2025-02-13T13:48:35Z

presto-native-execution/presto_cpp/main/types/CMakeLists.txt

@@ -21,8 +21,16 @@ add_library(
 add_dependencies(presto_types presto_operators presto_type_converter velox_type
                 velox_type_fbhive)

-target_link_libraries(presto_types presto_type_converter velox_type_fbhive
-                      velox_hive_partition_function velox_tpch_gen velox_functions_json)
+target_link_libraries(


nit: please revert this formatting change.

pramodsatya · 2025-02-13T13:51:26Z

presto-native-execution/presto_cpp/main/types/PrestoToVeloxExpr.cpp

@@ -127,6 +135,61 @@ std::string getFunctionName(const protocol::SqlFunctionId& functionId) {

 } // namespace

+#ifdef PRESTO_ENABLE_REMOTE_FUNCTIONS
+TypedExprPtr VeloxExprConverter::registerRestRemoteFunction(


Should this be a member function or can it be moved to an anonymous namespace?

pramodsatya · 2025-02-13T13:51:43Z

presto-native-execution/presto_cpp/main/types/PrestoToVeloxExpr.cpp

@@ -504,10 +567,22 @@ TypedExprPtr VeloxExprConverter::toVeloxExpr(
              pexpr.functionHandle)) {
    auto args = toVeloxExpr(pexpr.arguments);
    auto returnType = typeParser_->parse(pexpr.returnType);
+


nit: revert newline.

Co-authored-by: Wills Feng <wills.feng@ibm.com>

Joe-Abraham · 2025-02-14T07:03:48Z

Thank you for the feedback @pdabre12 @pramodsatya, I have added basic test cases to this PR and e2e test case in #23777

pdabre12 · 2025-02-18T21:16:46Z

@Joe-Abraham Please take a look at the failing CI.

Joe-Abraham force-pushed the remote branch 6 times, most recently from 808f416 to 1c7ea23 Compare September 10, 2024 09:49

Joe-Abraham force-pushed the remote branch 7 times, most recently from 86c1f8b to 3345b51 Compare September 17, 2024 14:30

Joe-Abraham force-pushed the remote branch 3 times, most recently from 0d73f09 to f530b51 Compare September 26, 2024 08:08

Joe-Abraham force-pushed the remote branch 10 times, most recently from 32a8715 to 5726f00 Compare October 16, 2024 06:45

Joe-Abraham force-pushed the remote branch 4 times, most recently from a8afff9 to 26f9fe6 Compare October 29, 2024 06:06

Joe-Abraham force-pushed the remote branch from 83ecc48 to beccbc6 Compare November 27, 2024 08:45

Joe-Abraham mentioned this pull request Nov 27, 2024

Add e2e testcases for REST based remote functions #23777

Open

Joe-Abraham marked this pull request as ready for review November 28, 2024 03:51

Joe-Abraham requested review from a team as code owners November 28, 2024 03:51

Joe-Abraham requested a review from presto-oss November 28, 2024 03:51

prestodb-ci added the from:IBM PR from IBM label Nov 28, 2024

prestodb-ci requested review from a team, ScrapCodes and aaneja and removed request for a team November 28, 2024 03:51

Joe-Abraham force-pushed the remote branch from beccbc6 to 4356902 Compare December 9, 2024 05:12

Joe-Abraham force-pushed the remote branch 2 times, most recently from ef746fc to 098b074 Compare January 13, 2025 06:15

pramodsatya reviewed Jan 31, 2025

View reviewed changes

Joe-Abraham force-pushed the remote branch 3 times, most recently from 6af6ddb to 9929892 Compare February 3, 2025 07:53

Joe-Abraham requested a review from pramodsatya February 3, 2025 08:33

pdabre12 reviewed Feb 10, 2025

View reviewed changes

pramodsatya reviewed Feb 13, 2025

View reviewed changes

Joe-Abraham force-pushed the remote branch 2 times, most recently from f11e126 to 7fcb6b0 Compare February 14, 2025 06:47

Add support for REST based remote functions.

78f2afa

Co-authored-by: Wills Feng <wills.feng@ibm.com>

Joe-Abraham force-pushed the remote branch from 7fcb6b0 to 78f2afa Compare February 14, 2025 07:02

Joe-Abraham requested review from pdabre12 and pramodsatya February 14, 2025 16:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[native] Add support for REST based remote function #23568

[native] Add support for REST based remote function #23568

Joe-Abraham commented Sep 2, 2024 •

edited

Loading

Joe-Abraham commented Nov 28, 2024

pramodsatya left a comment

pramodsatya Jan 31, 2025

Joe-Abraham Feb 3, 2025

pramodsatya Jan 31, 2025

Joe-Abraham Feb 3, 2025

pramodsatya Jan 31, 2025

Joe-Abraham Feb 3, 2025

pramodsatya Jan 31, 2025

Joe-Abraham Feb 3, 2025

pramodsatya Jan 31, 2025

Joe-Abraham Feb 3, 2025

pramodsatya Jan 31, 2025

Joe-Abraham Feb 3, 2025

pramodsatya Jan 31, 2025

Joe-Abraham Feb 3, 2025

pdabre12 left a comment

pdabre12 Feb 10, 2025

pramodsatya left a comment

pramodsatya Feb 13, 2025

pramodsatya Feb 13, 2025

pramodsatya Feb 13, 2025

pramodsatya Feb 13, 2025

Joe-Abraham commented Feb 14, 2025

pdabre12 commented Feb 18, 2025

[native] Add support for REST based remote function #23568

Are you sure you want to change the base?

[native] Add support for REST based remote function #23568

Conversation

Joe-Abraham commented Sep 2, 2024 • edited Loading

Description

Dependency

Joe-Abraham commented Nov 28, 2024

pramodsatya left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pdabre12 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pramodsatya left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Joe-Abraham commented Feb 14, 2025

pdabre12 commented Feb 18, 2025

Joe-Abraham commented Sep 2, 2024 •

edited

Loading