[Enhencement](Export) support export with outfile syntax #18325

BePPPower · 2023-04-03T02:20:27Z

Proposed changes

Issue Number: close #xxx

Problem summary

Export syntax provides asynchronous export function, but Export does not achieve vectorization.
Outfile syntax provides synchronous export function`.
So we can reimplement the export syntax with oufile syntax.

Checklist(Required)

Does it affect the original behavior
Has unit tests been added
Has document been added or modified
Does it need to update dependencies
Is this PR support rollback (If NO, please explain WHY)

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

morningman · 2023-04-06T14:57:41Z

fe/fe-core/src/main/java/org/apache/doris/task/ExportExportingTask.java

+        connectContext.setEnv(Env.getCurrentEnv());
+        connectContext.setDatabase(job.getTableName().getDb());
+        // TODO(ftw): 这里的权限可能要改一下
+        connectContext.setQualifiedUser(UserIdentity.ROOT.getQualifiedUser());


Need to save user info in export job

fe/fe-core/src/main/java/org/apache/doris/analysis/ExportStmt.java

morningman · 2023-04-13T00:53:10Z

fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java

@@ -547,6 +546,7 @@ private Env(boolean isCheckpointCatalog) {
        this.routineLoadManager = new RoutineLoadManager();
        this.sqlBlockRuleMgr = new SqlBlockRuleMgr();
        this.exportMgr = new ExportMgr();
+        this.exportMgr.start();


Move this to startMasterOnlyDaemonThreads()

morningman · 2023-04-13T00:56:27Z

fe/fe-core/src/main/java/org/apache/doris/load/ExportMgr.java

    public ExportMgr() {
+        int poolSize = Config.export_running_job_num_limit == 0 ? 5 : Config.export_running_job_num_limit;
+        exportingExecutor = new MasterTaskExecutor("export-exporting-job", poolSize, true);
+        exportingExecutor.start();


Override the start() of MasterDaemon and move this exportingExecutor.start() to start()

morningman · 2023-04-13T06:35:56Z

fe/fe-core/src/main/java/org/apache/doris/load/ExportMgr.java

+        List<ExportJob> jobs = getExportJobs(JobState.PENDING);
+        // Because exportJob may be replayed from log
+        // we also need handle EXPORTING state exportJob.
+        jobs.addAll(getExportJobs(JobState.EXPORTING));


I think we don't need to handle exporting state jobs here.
All jobs replayed from log should be reset to PENDING.

morningman · 2023-04-13T06:39:14Z

fe/fe-core/src/main/java/org/apache/doris/load/ExportMgr.java

+        // we also need handle EXPORTING state exportJob.
+        jobs.addAll(getExportJobs(JobState.EXPORTING));
+        int runningJobNumLimit = Config.export_running_job_num_limit;
+        if (runningJobNumLimit > 0 && !jobs.isEmpty()) {


No need to calc the remaining job num.
The number of thread in executor pool is limit to 5, and the submitted job will be queued when all threads are taken.
So here, we can simply do:

change the job's state to EXPORTING, so that it won't be scheduled again.

submit the job to the executor.

Exporting state means this exportJob is exporting. If we change the job's state to EXPORTING here, the job may not be assigned to the thread in the executor pool. So I think the change of job's state should in ExportExportingTask.exec()

morningman · 2023-04-13T06:40:32Z

fe/fe-core/src/main/java/org/apache/doris/task/ExportExportingTask.java

+            return;
+        }
+
+        if (job.updateState(ExportJob.JobState.EXPORTING)) {


No need to persist EXPORTING state.
The state should already be changed before submitting to the executor thread pool

morningman · 2023-04-13T06:41:22Z

fe/fe-core/src/main/java/org/apache/doris/task/ExportExportingTask.java

-                    newFiles.add(destPath);
+    }
+
+    private Status makeSnapshots() {


Currently, we can comment out this makeSnapshots() method because it is not used.
It can be reopened after we implement exporting tablet one by one

morningman · 2023-04-13T12:21:55Z

fe/fe-core/src/main/java/org/apache/doris/load/ExportMgr.java

+
+                }
+            } catch (Exception e) {
+                LOG.warn("run export exporting job error", e);


Suggested change

LOG.warn("run export exporting job error", e);

LOG.warn("run export exporting job {}.", job, e);

morningman · 2023-04-13T12:25:26Z

fe/fe-core/src/main/java/org/apache/doris/load/ExportJob.java

@@ -714,6 +500,52 @@ public synchronized boolean isFinalState() {
        return this.state == ExportJob.JobState.CANCELLED || this.state == ExportJob.JobState.FINISHED;
    }

+    public Status makeSnapshots() {


Suggested change

public Status makeSnapshots() {

private Status makeSnapshots() {

github-actions · 2023-04-14T08:18:52Z

clang-tidy review says "All clean, LGTM! 👍"

BePPPower · 2023-04-14T09:22:18Z

run buildall

github-actions · 2023-04-14T09:24:14Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2023-04-14T13:43:24Z

clang-tidy review says "All clean, LGTM! 👍"

morningman

LGTM

github-actions · 2023-04-16T14:52:16Z

PR approved by at least one committer and no changes requested.

github-actions · 2023-04-16T14:52:18Z

PR approved by anyone and no changes requested.

morningman · 2023-04-16T14:53:42Z

run clickbench

morningman · 2023-04-16T14:55:08Z

run beut

hello-stephen · 2023-04-16T16:00:04Z

TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 37.01 seconds
stream load tsv: 430 seconds loaded 74807831229 Bytes, about 165 MB/s
stream load json: 22 seconds loaded 2358488459 Bytes, about 102 MB/s
stream load orc: 60 seconds loaded 1101869774 Bytes, about 17 MB/s
stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230419154003_clickbench_pr_131231.html

morningman · 2023-04-16T16:03:46Z

run feut

BePPPower · 2023-04-19T06:37:15Z

run buildall

BePPPower · 2023-04-19T06:47:26Z

run buildall

github-actions · 2023-04-19T07:03:20Z

clang-tidy review says "All clean, LGTM! 👍"

BePPPower · 2023-04-19T07:24:32Z

run buildall

github-actions · 2023-04-19T07:38:23Z

clang-tidy review says "All clean, LGTM! 👍"

BePPPower · 2023-04-19T09:26:41Z

run feut

BePPPower · 2023-04-19T13:39:54Z

run buildall

github-actions · 2023-04-19T13:50:05Z

clang-tidy review says "All clean, LGTM! 👍"

BePPPower · 2023-04-20T02:58:19Z

run p0

BePPPower · 2023-04-20T06:36:17Z

run p1

morningman

LGTM

`Export` syntax provides asynchronous export function, but `Export` does not achieve vectorization. `Outfile` syntax provides synchronous export function`. So we can reimplement the export syntax with oufile syntax.

github-actions bot added area/load Issues or PRs related to all kinds of load area/planner Issues or PRs related to the query planner kind/docs Categorizes issue or PR as related to documentation. labels Apr 3, 2023

BePPPower force-pushed the exportJob branch from 190f5a3 to 5e068c2 Compare April 6, 2023 03:28

morningman reviewed Apr 6, 2023

View reviewed changes

morningman added the kind/meta-version-change Categorizes issue or PR as related to changing meta version label Apr 12, 2023

BePPPower force-pushed the exportJob branch from 96cd0b5 to 4ed01a8 Compare April 13, 2023 04:13

morningman reviewed Apr 13, 2023

View reviewed changes

github-actions bot added the area/vectorization label Apr 14, 2023

BePPPower marked this pull request as ready for review April 14, 2023 09:21

BePPPower changed the title ~~[Enhencement](Export) support export~~ [Enhencement](Export) support export with outfile syntax Apr 14, 2023

morningman previously approved these changes Apr 16, 2023

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 16, 2023

github-actions bot added the reviewed label Apr 16, 2023

BePPPower dismissed morningman’s stale review via 54aa3c1 April 19, 2023 06:29

BePPPower added 4 commits April 19, 2023 14:36

support export

77a2198

fix 1

cb49cd9

fix 2

7c3ee85

modify persis

d17433e

BePPPower added 4 commits April 19, 2023 14:36

add outfile info

4365f8c

fix

18fb251

fix 3

c7983ae

fix fe ut

6fa3cfb

BePPPower force-pushed the exportJob branch from 54aa3c1 to 6fa3cfb Compare April 19, 2023 06:36

fix show export

0a72fe4

fix checkstyle

1746d8a

fix fe ut

a78db95

morningman approved these changes Apr 20, 2023

View reviewed changes

morningman merged commit 8e2146f into apache:master Apr 20, 2023

	LOG.warn("run export exporting job error", e);
	LOG.warn("run export exporting job {}.", job, e);

	public Status makeSnapshots() {
	private Status makeSnapshots() {

[Enhencement](Export) support export with outfile syntax #18325

[Enhencement](Export) support export with outfile syntax #18325

Conversation

BePPPower commented Apr 3, 2023 • edited Loading

Proposed changes

Problem summary

Checklist(Required)

Further comments

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Apr 14, 2023

BePPPower commented Apr 14, 2023

github-actions bot commented Apr 14, 2023

github-actions bot commented Apr 14, 2023

morningman left a comment

Choose a reason for hiding this comment

github-actions bot commented Apr 16, 2023

github-actions bot commented Apr 16, 2023

morningman commented Apr 16, 2023

morningman commented Apr 16, 2023

hello-stephen commented Apr 16, 2023 • edited Loading

morningman commented Apr 16, 2023

BePPPower commented Apr 19, 2023

BePPPower commented Apr 19, 2023

github-actions bot commented Apr 19, 2023

BePPPower commented Apr 19, 2023

github-actions bot commented Apr 19, 2023

BePPPower commented Apr 19, 2023

BePPPower commented Apr 19, 2023

github-actions bot commented Apr 19, 2023

BePPPower commented Apr 20, 2023

BePPPower commented Apr 20, 2023

morningman left a comment

Choose a reason for hiding this comment

BePPPower commented Apr 3, 2023 •

edited

Loading

hello-stephen commented Apr 16, 2023 •

edited

Loading