Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Enable ROCm to use tunable GEMM #12853

Merged
merged 4 commits into from
Sep 28, 2022
Merged

Enable ROCm to use tunable GEMM #12853

merged 4 commits into from
Sep 28, 2022

Conversation

cloudhan
Copy link
Contributor

@cloudhan cloudhan commented Sep 5, 2022 •
edited
Loading

Related PRs #12855 #12856 #12857

Description: Enable ROCm to use tunable GEMM for better performance.

Motivation and Context

  • Why is this change required? What problem does it solve?
    This drastically improve some GEMM performance, aka, the overall performance for bert inference.

@cloudhan
Copy link
Contributor Author

cloudhan commented Sep 5, 2022

For recording purpose, the perf difference with initial try

Latency(ms)     Latency_P50     Latency_P75     Latency_P90     Latency_P95     Latency_P99     Throughput(QPS) model   graph_optimization_level        intra_op_num_threads    batch_size      sequence_length test_cases      test_timesuse_gpu
113.03  113.01  113.15  113.26  113.38  113.53  9059.37 fbv_bert_fp16_rocm_no_attention_fusion.onnx     ENABLE_ALL      24      1024    128     10      10      True
94.89   94.88   94.92   94.96   94.98   95.02   10791.95        fbv_bert_fp16_rocm_no_attention_fusion.onnx     ENABLE_ALL      24      1024    128     10      10      True

@cloudhan cloudhan force-pushed the guangyunhan/ort-use-tunable-gemm branch from 773ea60 to 0148fbb Compare September 5, 2022 08:18
@cloudhan cloudhan changed the base branch from main to guangyunhan/tunableop-move-to-ep September 5, 2022 08:36
@cloudhan cloudhan marked this pull request as ready for review September 5, 2022 08:37
@cloudhan cloudhan mentioned this pull request Sep 5, 2022
Closed
@cloudhan cloudhan force-pushed the guangyunhan/tunableop-move-to-ep branch from 8e71431 to 14839aa Compare September 7, 2022 03:42
@cloudhan cloudhan force-pushed the guangyunhan/ort-use-tunable-gemm branch from 0148fbb to e856516 Compare September 7, 2022 11:31
@cloudhan cloudhan force-pushed the guangyunhan/tunableop-move-to-ep branch from 14839aa to 4988216 Compare September 8, 2022 04:12
@cloudhan cloudhan force-pushed the guangyunhan/ort-use-tunable-gemm branch from e856516 to f79b05b Compare September 8, 2022 04:13
@cloudhan cloudhan force-pushed the guangyunhan/tunableop-move-to-ep branch from 4988216 to 23a3f90 Compare September 21, 2022 11:33
Base automatically changed from guangyunhan/tunableop-move-to-ep to main September 23, 2022 03:10
@cloudhan cloudhan force-pushed the guangyunhan/ort-use-tunable-gemm branch from f79b05b to 833a4ea Compare September 27, 2022 05:22
@microsoft microsoft deleted a comment from lgtm-com bot Sep 27, 2022
@microsoft microsoft deleted a comment from lgtm-com bot Sep 27, 2022
@cloudhan cloudhan mentioned this pull request Sep 27, 2022
Merged
@cloudhan cloudhan merged commit 32c2c4b into main Sep 28, 2022
@cloudhan cloudhan deleted the guangyunhan/ort-use-tunable-gemm branch September 28, 2022 08:21
@cloudhan
Copy link
Contributor Author

This PR is split into 2, the following #13116 the enabling and testing for it.

cloudhan added a commit that referenced this pull request Sep 29, 2022
@cloudhan
Revert "Change ROCm to use tunable GEMM (#12853)"
be1ae4b 
This reverts commit 32c2c4b.
@cloudhan cloudhan mentioned this pull request Sep 29, 2022
Merged
cloudhan added a commit that referenced this pull request Sep 30, 2022
@cloudhan
Revert "Enable ROCm to use tunable GEMM" (#13160)
c93cb8f 
Reverts #12853 due to CI pipeline problem.
linnealovespie pushed a commit that referenced this pull request Sep 30, 2022
@cloudhan @linnealovespie
Change ROCm to use tunable GEMM (#12853)
8e5ed42 
Change ROCm to use tunable GEMM. It is not enabled in this PR. This will drastically improve GEMM performance in some shapes and dtypes configuration. This will benefit the overall performance for BERT inference and hopefully, training, when enabled.
linnealovespie pushed a commit that referenced this pull request Sep 30, 2022
@cloudhan @linnealovespie
Revert "Enable ROCm to use tunable GEMM" (#13160)
1f18f65 
Reverts #12853 due to CI pipeline problem.
@cloudhan cloudhan mentioned this pull request Oct 4, 2022
Merged
cloudhan added a commit that referenced this pull request Oct 5, 2022
@cloudhan
Update ROCm CI to use HIP LANGUAGE (#13214)
72076b1 
Update for ROCm CI before reland tunable GEMM #12853. This PR also update
composable kernel to use CMakes's HIP language support so that we can
mix C/C++ compiler with HIP compiler instead of locking to hip-clang
yuslepukhin pushed a commit that referenced this pull request Oct 5, 2022
@cloudhan @yuslepukhin
Update ROCm CI to use HIP LANGUAGE (#13214)
7a73a61 
Update for ROCm CI before reland tunable GEMM #12853. This PR also update
composable kernel to use CMakes's HIP language support so that we can
mix C/C++ compiler with HIP compiler instead of locking to hip-clang
cloudhan added a commit that referenced this pull request Oct 7, 2022
@cloudhan
Reland: Change ROCm to use tunable GEMM (#12853)
ad7899d 
Change ROCm to use tunable GEMM. It is not enabled in this PR. This will drastically improve GEMM performance in some shapes and dtypes configuration. This will benefit the overall performance for BERT inference and hopefully, training, when enabled.# This is a combination of 2 commits.
@cloudhan cloudhan mentioned this pull request Oct 7, 2022
Merged
cloudhan added a commit that referenced this pull request Oct 12, 2022
@cloudhan
Reland: Change ROCm to use tunable GEMM (#12853)
31345e3 
Change ROCm to use tunable GEMM. It is not enabled in this PR. This will drastically improve GEMM performance in some shapes and dtypes configuration. This will benefit the overall performance for BERT inference and hopefully, training, when enabled.# This is a combination of 2 commits.
zhangyaobit pushed a commit that referenced this pull request Oct 14, 2022
@cloudhan
Reland: Change ROCm to use tunable GEMM (#13231)
790e363 
Reland: Change ROCm to use tunable GEMM (#12853)
zhangyaobit pushed a commit that referenced this pull request Oct 20, 2022
@cloudhan
Enable/Disbale tunable GEMM by using tunable switch in provider optio…
fc12abf 
…ns and env var (#13116)

Related PRs #12853

This allows the user enable/disbale tunable GEMM on demand.
preetha-intel pushed a commit to intel/onnxruntime that referenced this pull request Nov 11, 2022
@cloudhan @preetha-intel
Change ROCm to use tunable GEMM (microsoft#12853)
e19fddd 
Change ROCm to use tunable GEMM. It is not enabled in this PR. This will drastically improve GEMM performance in some shapes and dtypes configuration. This will benefit the overall performance for BERT inference and hopefully, training, when enabled.
preetha-intel pushed a commit to intel/onnxruntime that referenced this pull request Nov 29, 2022
@cloudhan @preetha-intel
Change ROCm to use tunable GEMM (microsoft#12853)
2668d4c 
Change ROCm to use tunable GEMM. It is not enabled in this PR. This will drastically improve GEMM performance in some shapes and dtypes configuration. This will benefit the overall performance for BERT inference and hopefully, training, when enabled.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@zhangyaobit zhangyaobit zhangyaobit approved these changes

@PeixuanZuo PeixuanZuo Awaiting requested review from PeixuanZuo

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@cloudhan @zhangyaobit