Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Fix](Nereids)Check bounding of expression in analyze process #18581

Merged
merged 1 commit into from
Apr 20, 2023

Conversation

LiBinfeng-01
Copy link
Collaborator

Probleam:
Dead loop cause of keep pushing analyze tasks into job stack. When doing analyze process and generate new operators, the same analyze rule would be pushed again, so it cause dead loop. And analyze process generate new operators when trying to bound order by key and aggregate function.
Solve:
We need to make it throw exception before complex analyze and rewrite process, so checking whether all expressions being bound should be done twice. One is done after bounding all expression, another is done after all analyze process in case of generate new expressions and new operators.
Example:
Cases were put in file: regression-test/suites/nereids_p0/except/test_bound_exception.groovy

Proposed changes

Issue Number: close #xxx

Problem summary

Describe your changes.

Checklist(Required)

  • Does it affect the original behavior
  • Has unit tests been added
  • Has document been added or modified
  • Does it need to update dependencies
  • Is this PR support rollback (If NO, please explain WHY)

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@morrySnow
Copy link
Contributor

run buildall

1 similar comment
@LiBinfeng-01
Copy link
Collaborator Author

run buildall

@LiBinfeng-01
Copy link
Collaborator Author

run buildall

@LiBinfeng-01
Copy link
Collaborator Author

run buildall

2 similar comments
@LiBinfeng-01
Copy link
Collaborator Author

run buildall

@LiBinfeng-01
Copy link
Collaborator Author

run buildall

@LiBinfeng-01
Copy link
Collaborator Author

run clickbench

morrySnow
morrySnow previously approved these changes Apr 19, 2023
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 19, 2023
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

…ounding and before apply more complex analyze rules
@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Apr 19, 2023
@LiBinfeng-01
Copy link
Collaborator Author

run buildall

@hello-stephen
Copy link
Contributor

TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 34.16 seconds
stream load tsv: 425 seconds loaded 74807831229 Bytes, about 167 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230419153856_clickbench_pr_131210.html

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 20, 2023
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@morrySnow morrySnow merged commit 668c681 into apache:master Apr 20, 2023
xiaokang pushed a commit to xiaokang/doris that referenced this pull request Apr 21, 2023
…pache#18581)

Probleam:
Dead loop cause of keep pushing analyze tasks into job stack. When doing analyze process and generate new operators, the same analyze rule would be pushed again, so it cause dead loop. And analyze process generate new operators when trying to bound order by key and aggregate function.

Solve:
We need to make it throw exception before complex analyze and rewrite process, so checking whether all expressions being bound should be done twice. One is done after bounding all expression, another is done after all analyze process in case of generate new expressions and new operators.

Example:
Cases were put in file: regression-test/suites/nereids_p0/except/test_bound_exception.groovy
gnehil pushed a commit to gnehil/doris that referenced this pull request Apr 21, 2023
…pache#18581)

Probleam:
Dead loop cause of keep pushing analyze tasks into job stack. When doing analyze process and generate new operators, the same analyze rule would be pushed again, so it cause dead loop. And analyze process generate new operators when trying to bound order by key and aggregate function.

Solve:
We need to make it throw exception before complex analyze and rewrite process, so checking whether all expressions being bound should be done twice. One is done after bounding all expression, another is done after all analyze process in case of generate new expressions and new operators.

Example:
Cases were put in file: regression-test/suites/nereids_p0/except/test_bound_exception.groovy
jackwener pushed a commit to jackwener/doris that referenced this pull request Apr 21, 2023
…pache#18581)

Probleam:
Dead loop cause of keep pushing analyze tasks into job stack. When doing analyze process and generate new operators, the same analyze rule would be pushed again, so it cause dead loop. And analyze process generate new operators when trying to bound order by key and aggregate function.

Solve:
We need to make it throw exception before complex analyze and rewrite process, so checking whether all expressions being bound should be done twice. One is done after bounding all expression, another is done after all analyze process in case of generate new expressions and new operators.

Example:
Cases were put in file: regression-test/suites/nereids_p0/except/test_bound_exception.groovy
jackwener pushed a commit to jackwener/doris that referenced this pull request Apr 21, 2023
…pache#18581)

Probleam:
Dead loop cause of keep pushing analyze tasks into job stack. When doing analyze process and generate new operators, the same analyze rule would be pushed again, so it cause dead loop. And analyze process generate new operators when trying to bound order by key and aggregate function.

Solve:
We need to make it throw exception before complex analyze and rewrite process, so checking whether all expressions being bound should be done twice. One is done after bounding all expression, another is done after all analyze process in case of generate new expressions and new operators.

Example:
Cases were put in file: regression-test/suites/nereids_p0/except/test_bound_exception.groovy
Reminiscent pushed a commit to Reminiscent/doris that referenced this pull request May 15, 2023
…pache#18581)

Probleam:
Dead loop cause of keep pushing analyze tasks into job stack. When doing analyze process and generate new operators, the same analyze rule would be pushed again, so it cause dead loop. And analyze process generate new operators when trying to bound order by key and aggregate function.

Solve:
We need to make it throw exception before complex analyze and rewrite process, so checking whether all expressions being bound should be done twice. One is done after bounding all expression, another is done after all analyze process in case of generate new expressions and new operators.

Example:
Cases were put in file: regression-test/suites/nereids_p0/except/test_bound_exception.groovy
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. area/nereids kind/test reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants