[New Featrue] Support Vectorization Execution Engine Interface For Doris #6329

HappenLee · 2021-07-27T07:11:23Z

FE vecrizied plan code
Function register vec function
Diff function nullable type
New thirdpart code and new thirt struct

Proposed changes

Describe the big picture of your changes here to communicate to the maintainers why we should accept this pull request. If it fixes a bug or resolves a feature request, be sure to link to that issue.

Types of changes

What types of changes does your code introduce to Doris?
Put an x in the boxes that apply

Bugfix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
Documentation Update (if none of the other choices apply)
Code refactor (Modify the code structure, format the code, etc...)
Optimization. Including functional usability improvements and performance improvements.
Dependency. Such as changes related to third-party components.
Other.

Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code.

I have created an issue on (Fix #ISSUE) and described the bug/feature there in detail
Compiling and unit tests pass locally with my changes
I have added tests that prove my fix is effective or that my feature works
If these changes need document changes, I have updated the document
Any dependent changes have been merged

wangbo · 2021-07-27T08:28:51Z

Diff function nullable type

New thirdpart code and new thirt struct

What does point 3 and point 4 specifically refer to?

be/src/exec/exec_node.h

be/src/exec/schema_scanner.cpp

stdpain · 2021-07-30T09:50:08Z

be/CMakeLists.txt

@@ -704,3 +704,8 @@ install(FILES
    ${BASE_DIR}/../conf/odbcinst.ini
    DESTINATION ${OUTPUT_DIR}/conf)

+
+get_property(dirs DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR} PROPERTY INCLUDE_DIRECTORIES)


may be we should remove cmake log ?

yes, I will remove it

stdpain · 2021-07-30T09:51:45Z

be/src/exprs/aggregate_functions.cpp

@@ -1925,8 +1939,7 @@ DoubleVal AggregateFunctions::knuth_var_pop_finalize(FunctionContext* ctx,
 }

 DecimalV2Val AggregateFunctions::decimalv2_knuth_var_pop_finalize(FunctionContext* ctx,
-                                                                  const StringVal& state_sv) {
-    DCHECK(!state_sv.is_null);
+                                                  const StringVal& state_sv) {


need a code format here?

stdpain · 2021-07-30T09:54:04Z

build.sh

@@ -84,7 +92,7 @@ fi

 eval set -- "$OPTS"

-PARALLEL=$[$(nproc)/4+1]
+PARALLEL=$[$(nproc)+1]


set PARALLEL= $[$(nproc)/4+1]

morningman · 2021-08-05T07:21:38Z

fe/fe-core/src/main/java/org/apache/doris/analysis/CastExpr.java

@@ -17,6 +17,9 @@

 package org.apache.doris.analysis;

+import java.util.Arrays;
+import java.util.Map;


import order

morningman · 2021-08-05T07:23:14Z

fe/fe-core/src/main/java/org/apache/doris/analysis/CompoundPredicate.java

@@ -17,6 +17,9 @@

 package org.apache.doris.analysis;

+import com.google.common.collect.Lists;


import order

morningman · 2021-08-05T07:25:02Z

fe/fe-core/src/main/java/org/apache/doris/catalog/AggregateFunction.java

@@ -20,6 +20,7 @@
 import static org.apache.doris.common.io.IOUtils.readOptionStringOrNull;
 import static org.apache.doris.common.io.IOUtils.writeOptionString;

+import avro.shaded.com.google.common.collect.ImmutableSet;


wrong import

morningman · 2021-08-05T07:28:26Z

fe/fe-core/src/main/java/org/apache/doris/catalog/FunctionSet.java

@@ -1020,12 +1121,21 @@ private void initAggregateBuiltins() {
        // count(*)
        addBuiltin(AggregateFunction.createBuiltin(FunctionSet.COUNT,
            new ArrayList<Type>(), Type.BIGINT, Type.BIGINT,
-            prefix + "9init_zeroIN9doris_udf9BigIntValEEEvPNS2_15FunctionContextEPT_",
+            prefix + "18init_zero_not_nullIN9doris_udf9BigIntValEEEvPNS2_15FunctionContextEPT_",


yes, to keep count slot not null.

morningman · 2021-08-05T07:28:54Z

fe/fe-core/src/main/java/org/apache/doris/catalog/FunctionSet.java

@@ -1037,12 +1147,21 @@ private void initAggregateBuiltins() {
            // Count
            addBuiltin(AggregateFunction.createBuiltin(FunctionSet.COUNT,
                    Lists.newArrayList(t), Type.BIGINT, Type.BIGINT,
-                    prefix + "9init_zeroIN9doris_udf9BigIntValEEEvPNS2_15FunctionContextEPT_",
+                    prefix + "18init_zero_not_nullIN9doris_udf9BigIntValEEEvPNS2_15FunctionContextEPT_",


morningman · 2021-08-05T07:30:24Z

fe/fe-core/src/main/java/org/apache/doris/catalog/FunctionSet.java

@@ -1479,7 +1845,7 @@ private void initAggregateBuiltins() {
                prefix + "13rank_finalizeEPN9doris_udf15FunctionContextERNS1_9StringValE"));
        addBuiltin(AggregateFunction.createAnalyticBuiltin( "row_number",
                new ArrayList<Type>(), Type.BIGINT, Type.BIGINT,
-                prefix + "9init_zeroIN9doris_udf9BigIntValEEEvPNS2_15FunctionContextEPT_",
+                prefix + "18init_zero_not_nullIN9doris_udf9BigIntValEEEvPNS2_15FunctionContextEPT_",


morningman · 2021-08-05T07:31:29Z

fe/fe-core/src/main/java/org/apache/doris/common/FeMetaVersion.java

@@ -216,6 +216,8 @@
    public static final int VERSION_101 = 101;
    // add data encrypt
    public static final int VERSION_102 = 102;
+    // add vectorized to function
+    public static final int VERSION_103 = 103;


morningman · 2021-08-05T07:42:02Z

fe/fe-core/src/main/java/org/apache/doris/catalog/Function.java

@@ -104,40 +117,49 @@
    private HdfsURI location;
    private TFunctionBinaryType binaryType;

+    protected NullableMode nullableMode = NullableMode.DEPEND_ON_ARGUMENT;
+
+    private boolean vectorized;


Suggested change

private boolean vectorized;

private boolean vectorized = false;

morningman · 2021-08-05T07:42:11Z

fe/fe-core/src/main/java/org/apache/doris/catalog/Function.java

@@ -640,6 +664,9 @@ public void readFields(DataInput input) throws IOException {
        if (hasChecksum) {
            checksum = Text.readString(input);
        }
+        if (Catalog.getCurrentCatalogJournalVersion() >= FeMetaVersion.VERSION_101) {


remove meta version

morningman

LGTM

francisoliverlee · 2021-08-11T08:44:28Z

@HappenLee does those 2 PR all of implementation of ver-engine?

morningman · 2021-08-11T08:47:12Z

@HappenLee does those 2 PR all of implementation of ver-engine?

Not yet, more PR will coming.

HappenLee · 2021-08-11T08:47:14Z

@francisoliverlee yes，but it only contain the some interface of vec-engine.

hf200012 · 2021-08-11T08:48:36Z

Great, I look forward to it

francisoliverlee · 2021-08-11T11:48:59Z

@HappenLee does those 2 PR all of implementation of ver-engine?

Not yet, more PR will coming.

that's nice, waiting...o_o

…ris (apache#6329) 1. FE vectorized plan code 2. Function register vec function 3. Diff function nullable type 4. New thirdparty code and new thrift struct

…he#6341 and add back in apache#6329 by mistake

…and add back in #6329 by mistake (#9233)

…he#6341 and add back in apache#6329 by mistake (apache#9233)

wangbo reviewed Jul 27, 2021

View reviewed changes

be/src/exec/exec_node.h Show resolved Hide resolved

wangbo reviewed Jul 27, 2021

View reviewed changes

be/src/exec/schema_scanner.cpp Show resolved Hide resolved

morningman added kind/feature Categorizes issue or PR as related to a new feature. area/vectorization labels Jul 27, 2021

stdpain reviewed Jul 30, 2021

View reviewed changes

morningman reviewed Aug 5, 2021

View reviewed changes

HappenLee force-pushed the vectorized branch 2 times, most recently from a037693 to 250ad89 Compare August 7, 2021 11:27

[New Featrue] Support Vectorization Execution Engine Interface For Doris

3de4df8

HappenLee force-pushed the vectorized branch from 250ad89 to 3de4df8 Compare August 7, 2021 11:31

fix error result of agg function

36279c0

HappenLee force-pushed the vectorized branch from 930a931 to 36279c0 Compare August 9, 2021 10:53

morningman approved these changes Aug 11, 2021

View reviewed changes

morningman merged commit 9216735 into apache:master Aug 11, 2021

qidaye mentioned this pull request Aug 11, 2021

[Bug] fix compile error in AliasFunction #6425

Merged

yangzhg added a commit to yangzhg/doris that referenced this pull request Apr 26, 2022

remove gensrc/proto/palo_internal_service.proto, this removed in apac…

8316827

…he#6341 and add back in apache#6329 by mistake

yangzhg mentioned this pull request Apr 26, 2022

[refactor] remove gensrc/proto/palo_internal_service.proto, this removed in #6341 and add back in #6329 by mistake #9233

Merged

yangzhg added a commit to yangzhg/doris that referenced this pull request Apr 26, 2022

remove gensrc/proto/palo_internal_service.proto, this removed in apac…

90c4160

…he#6341 and add back in apache#6329 by mistake

yiguolei pushed a commit that referenced this pull request Apr 27, 2022

remove gensrc/proto/palo_internal_service.proto, this removed in #6341 …

c1ae1a0

…and add back in #6329 by mistake (#9233)

morningman pushed a commit that referenced this pull request Apr 28, 2022

remove gensrc/proto/palo_internal_service.proto, this removed in #6341 …

db4932b

…and add back in #6329 by mistake (#9233)

Kikyou1997 pushed a commit to Kikyou1997/incubator-doris that referenced this pull request May 9, 2022

remove gensrc/proto/palo_internal_service.proto, this removed in apac…

f724112

…he#6341 and add back in apache#6329 by mistake (apache#9233)

starocean999 pushed a commit to starocean999/incubator-doris that referenced this pull request May 19, 2022

remove gensrc/proto/palo_internal_service.proto, this removed in apac…

c791cc1

…he#6341 and add back in apache#6329 by mistake (apache#9233)

englefly pushed a commit to englefly/incubator-doris that referenced this pull request May 23, 2022

remove gensrc/proto/palo_internal_service.proto, this removed in apac…

a872430

…he#6341 and add back in apache#6329 by mistake (apache#9233)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[New Featrue] Support Vectorization Execution Engine Interface For Doris #6329

[New Featrue] Support Vectorization Execution Engine Interface For Doris #6329

HappenLee commented Jul 27, 2021

wangbo commented Jul 27, 2021 •

edited

Loading

stdpain Jul 30, 2021

HappenLee Jul 30, 2021

stdpain Jul 30, 2021

stdpain Jul 30, 2021

morningman Aug 5, 2021

morningman Aug 5, 2021

morningman Aug 5, 2021

morningman Aug 5, 2021

HappenLee Aug 7, 2021

morningman Aug 5, 2021

morningman Aug 5, 2021

morningman Aug 5, 2021

morningman Aug 5, 2021

morningman Aug 5, 2021

morningman left a comment

francisoliverlee commented Aug 11, 2021

morningman commented Aug 11, 2021

HappenLee commented Aug 11, 2021

hf200012 commented Aug 11, 2021

francisoliverlee commented Aug 11, 2021

		@@ -17,6 +17,9 @@

		package org.apache.doris.analysis;

		import com.google.common.collect.Lists;

	private boolean vectorized;
	private boolean vectorized = false;

[New Featrue] Support Vectorization Execution Engine Interface For Doris #6329

[New Featrue] Support Vectorization Execution Engine Interface For Doris #6329

Conversation

HappenLee commented Jul 27, 2021

Proposed changes

Types of changes

Checklist

wangbo commented Jul 27, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

morningman left a comment

Choose a reason for hiding this comment

francisoliverlee commented Aug 11, 2021

morningman commented Aug 11, 2021

HappenLee commented Aug 11, 2021

hf200012 commented Aug 11, 2021

francisoliverlee commented Aug 11, 2021

wangbo commented Jul 27, 2021 •

edited

Loading