add config option for ipex (https://github.com/min-jean-cho/serve/blob/ipex_enable/examples/IPEX/README.md) #1319

min-jean-cho · 2021-11-13T01:57:21Z

This PR will replace https://github.com/pytorch/serve/pull/1312/files

sagemaker-neo-ci-bot · 2021-11-13T02:10:02Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-win
Commit ID: 44ca0db
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2021-11-13T02:18:01Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-cpu
Commit ID: 44ca0db
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2021-11-13T02:21:54Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-gpu
Commit ID: 44ca0db
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

ashokei · 2021-11-15T23:10:44Z

thanks @min-jean-cho , example config could be.

cpu_launcher_enable=true
cpu_launcher_args=--ncore_per_instance 4 --ninstance 1
ipex_enable=true

msaroufim · 2021-11-15T23:24:28Z

Next steps

FB team to share BERT and resnet 50 benchmarks and basic repro instructions with a sample config.properties before and after PR (without launcher) - if conclusive go ahead with base handler change and if not conclusive put in examples/ instead
Intel: start a launcher PR, LPOT etc..

msaroufim · 2021-11-16T00:00:57Z

@min-jean-cho to pass CI

cd frontend 
./gradlew formatJava

jgong5 · 2021-11-16T02:44:05Z

ts/torch_handler/base_handler.py

-        return torch.as_tensor(data, device=self.device)
+        t = torch.as_tensor(data, device=self.device)
+        if ipex_enabled and t.dim() == 4:
+            t = t.to(memory_format=torch.channels_last)


Suggest not do this here, reasons:

Input will be converted to channels last automatically inside the op as long as model.to(memory_format=torch.channels_last) (https://github.com/pytorch/serve/pull/1319/files#diff-d30e1f5ef9fd05e2ab9f652c116461632586cc132dde5d23ac4bc5cd19c799ccR85)

The channels last conversion only benefits CNN models while we can't tell if the model is CNN just by its input shapes. Moreover, CNN models can also take 3D or 5D inputs. Checking 4D sounds too specific to me.

I definitely agree we can't assume CNN based on the input dimension being 4D (e.g., CNN for videos can take 5D inputs).

msaroufim · 2021-11-14T18:06:22Z

frontend/server/src/main/java/org/pytorch/serve/util/ConfigManager.java

@@ -67,6 +67,9 @@
    private static final String TS_NETTY_CLIENT_THREADS = "netty_client_threads";
    private static final String TS_JOB_QUEUE_SIZE = "job_queue_size";
    private static final String TS_NUMBER_OF_GPU = "number_of_gpu";
+    private static final String TS_CPU_LAUNCHER_ENABLE = "cpu_launcher_enable";


Can we add a comment on top of the IPEX args like //ipex arguments

Of course, will do so.

ts/torch_handler/base_handler.py

msaroufim · 2021-11-14T18:07:16Z

ts/torch_handler/base_handler.py

@@ -73,6 +81,9 @@ def initialize(self, context):
            self.model = self._load_torchscript_model(model_pt_path)

        self.model.eval()
+	if ipex_enabled:
+            self.model = self.model.to(memory_format=torch.channels_last)


Does this apply only to vision models?

Yes, but benign to non-CNN models.

msaroufim · 2021-11-14T18:07:44Z

ts/torch_handler/base_handler.py

@@ -73,6 +81,9 @@ def initialize(self, context):
            self.model = self._load_torchscript_model(model_pt_path)

        self.model.eval()
+	if ipex_enabled:
+            self.model = self.model.to(memory_format=torch.channels_last)
+            self.model = ipex.optimize(self.model, dtype=torch.float32, level='O1')


is there a document that describes what the ipex levels mean?

Also what happens here if a user passes in a model that's already in float16 or bffloat16?

is there a document that describes what the ipex levels mean?

Yes, there will be a user guide as part of IPEX 1.10 release. But here, I would suggest we use the default "level" recommended by IPEX, not hard-code 'O1' here.

Also what happens here if a user passes in a model that's already in float16 or bffloat16?

@min-jean-cho please remove the dtype arg. @msaroufim The default behavior without specifying dtype is do nothing about the data type conversion of the model. If the user specifies a low-precision data type (e.g., bfloat16 or float16, note that float16 is not supported now) different from the low-precision data type the model has, IPEX will do nothing about the data type conversion and issue a warning message. The behavior will be clarified in the user doc too.

@min-jean-cho Suggest to change this line to self.model = ipex.optimize(self.model)

@jgong5 it would also be great to clarify on different optimization levels in the doc as well.

@HamidShojanazeri Yes, this is in the plan. Some initial draft below, welcome comments:
level (string): "O0" or "O1". No optimizations are applied with
"O0". The optimizer function just returns the original model and
optimizer. With "O1", the following optimizations are applied:
conv+bn folding, weights prepack, dropout removal (inferenc model),
master weight split and fused optimizer update step (training model).
The optimization options can be further overridden by setting the
following options explicitly. The default value is "O1".

self.model = ipex.optimize(self.model, dtype=torch.float32, level='O1') this also only works for fp32; So if ipex_enabled: is insufficient condition to check

@min-jean-cho may I know the specific scenario you are concerned with if you change the code to self.model = ipex.optimize(self.model)?

@jgong5 You're right. I'll integrate your two suggestions regarding dtype arg and self.model = ipex.optiize(self.model) - that would work.

@jgong5 You're right. I'll integrate your two suggestions regarding dtype arg and self.model = ipex.optiize(self.model) - that would work.

wouldn't we still need the level?

We don't need to explicitly set the level as level = 'O1' since it is the default value and we are optimizing the model with ipex enabled

frontend/server/src/main/java/org/pytorch/serve/util/ConfigManager.java

sagemaker-neo-ci-bot · 2021-11-17T20:37:00Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-gpu
Commit ID: e000094
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2021-11-17T20:44:55Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-win
Commit ID: e000094
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2021-11-17T20:45:50Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-gpu
Commit ID: 62d6d9f
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2021-11-17T20:50:43Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-win
Commit ID: 62d6d9f
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2021-11-17T20:50:57Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-gpu
Commit ID: 94414d9
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2021-11-17T20:51:19Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-cpu
Commit ID: e000094
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2021-11-17T20:58:30Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-gpu
Commit ID: 5b50072
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2021-11-17T20:59:15Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-win
Commit ID: 94414d9
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2021-11-17T21:03:35Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-cpu
Commit ID: 62d6d9f
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2021-11-17T21:06:45Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-win
Commit ID: 5b50072
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2021-11-17T21:09:16Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-cpu
Commit ID: 94414d9
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2021-11-17T21:17:17Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-cpu
Commit ID: 5b50072
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

HamidShojanazeri · 2021-11-22T19:07:19Z

ts/torch_handler/base_handler.py

@@ -73,6 +81,9 @@ def initialize(self, context):
            self.model = self._load_torchscript_model(model_pt_path)

        self.model.eval()
+	if ipex_enabled:
+            self.model = self.model.to(memory_format=torch.channels_last)
+            self.model = ipex.optimize(self.model)


@min-jean-cho does it choose optimization level 01 by default?

Yes, ipex will choose level 'O1' by default.

sagemaker-neo-ci-bot · 2021-11-22T19:43:42Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-gpu
Commit ID: c92905e
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2021-11-22T19:49:21Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-win
Commit ID: c92905e
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2021-11-22T19:56:52Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-cpu
Commit ID: c92905e
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2021-11-22T20:22:34Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-gpu
Commit ID: 3022ab6
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2021-11-22T20:34:17Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-win
Commit ID: 3022ab6
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2021-11-22T20:41:46Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-cpu
Commit ID: 3022ab6
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2021-11-22T22:37:12Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-win
Commit ID: 4dfc64d
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2021-11-22T22:46:42Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-cpu
Commit ID: 4dfc64d
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2021-11-22T22:50:35Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-gpu
Commit ID: 4dfc64d
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

min-jean-cho · 2021-11-22T23:49:40Z

rn50_pytorch_fp32_bs1:
inference time: 0.012870311737060547
preprocess time: 0.009124755859375
postprocess time: 5.078315734863281e-05

rn50_ipex_fp32_bs1:
inference time: 0.0069921016693115234
preprocess time: 0.009052753448486328
postprocess time: 8.249282836914062e-05

rn50_ipex_int8_bs1:
inference time: 0.0020923614501953125
preprocess time: 0.008910655975341797
postprocess time: 4.839897155761719e-05

msaroufim · 2021-11-23T01:46:40Z

I just tried running a BERT benchmark with and without IPEX and the improvements are clear with a 30% improvement. On an m6i.4x with 4 workers without the launcher. I'm running a couple more but I think it'd be safe to go ahead and get this merged in.
W/o IPEX: 126.33 ms
W/ IPEX: 95ms

Longer form benchmarks here
https://gist.github.com/msaroufim/3611f0565f72706557789a8a244a8271
https://gist.github.com/nskool/6f2166116a791c3e02b052d1842b1da4 (edited)

sagemaker-neo-ci-bot · 2021-11-23T01:47:56Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-gpu
Commit ID: 4dfc64d
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2021-11-23T02:02:59Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-win
Commit ID: 4dfc64d
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

msaroufim · 2021-11-19T20:17:24Z

ts/torch_handler/base_handler.py

+        import intel_extension_for_pytorch as ipex
+        ipex_enabled = True
+    except:
+        pass


if users are saying they want to use ipex but it's not available then it makes more sense to print("pip install ipex") instead of pass otherwise users will still get a failure later

agree, we can log a message saying ipex was not installed hence not used.

min-jean-cho · 2021-11-23T02:34:33Z

<IPEX release/1.1 + int8 patch, + with launcher JIT, inference only time (s) >
m6i.24xlarge

rn50_pytorch_fp32_bs1:0.013046979904174805
rn50_ipex_fp32_bs1: 0.0050411224365234375
rn50_ipex_int8_bs1: 0.002028942108154297
rn50_pytorch_fp32_bs64: 0.39560604095458984
rn50_ipex_fp32_bs64: 0.1628246307373047
rn50_ipex_int8_bs64: 0.05131959915161133
bert_pytorch_fp32_bs1: 0.03835415840148926
bert_ipex_fp32_bs1: 0.03265023231506348
bert_ipex_int8_bs1: 0.01756596565246582
bert_pytorch_fp32_bs64: 2.070930242538452
bert_ipex_fp32_bs64: 1.7104699611663818
bert_ipex_int8_bs64: 0.9398586750030518

sagemaker-neo-ci-bot · 2021-11-23T02:53:53Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-gpu
Commit ID: 1905e47
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2021-11-23T03:07:15Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-win
Commit ID: 1905e47
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2021-11-23T03:16:32Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-cpu
Commit ID: 1905e47
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2021-11-23T04:00:21Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-win
Commit ID: 7cecdfa
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2021-11-23T04:13:09Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-cpu
Commit ID: 7cecdfa
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2021-11-23T04:19:03Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-gpu
Commit ID: 7cecdfa
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2021-11-23T17:20:24Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-gpu
Commit ID: 7cecdfa
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2021-11-23T17:44:37Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-win
Commit ID: 7cecdfa
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

jgong5 reviewed Nov 16, 2021

View reviewed changes

msaroufim requested changes Nov 16, 2021

View reviewed changes

HamidShojanazeri reviewed Nov 22, 2021

View reviewed changes

fixed formatting

4dfc64d

min-jean-cho force-pushed the master branch from 3022ab6 to 4dfc64d Compare November 22, 2021 22:24

min-jean-cho closed this Nov 22, 2021

msaroufim reopened this Nov 23, 2021

msaroufim requested review from HamidShojanazeri, lxning and msaroufim November 23, 2021 02:15

msaroufim approved these changes Nov 23, 2021

View reviewed changes

added warning

1905e47

fixed pylint formatting

7cecdfa

msaroufim mentioned this pull request Nov 23, 2021

PyTorch profiler integration #1297

Merged

10 tasks

HamidShojanazeri approved these changes Nov 23, 2021

View reviewed changes

msaroufim merged commit c779018 into pytorch:master Nov 23, 2021

min-jean-cho changed the title ~~add config option for ipex~~ add config option for ipex (https://github.com/min-jean-cho/serve/blob/ipex_enable/examples/IPEX/README.md) Nov 24, 2021

This was referenced Nov 29, 2021

add config option for launcher #1349

Closed

add config option for launcher #1354

Merged

msaroufim mentioned this pull request May 14, 2022

Modularize base_handler.py into handler.utils #1631

Closed

add config option for ipex (https://github.com/min-jean-cho/serve/blob/ipex_enable/examples/IPEX/README.md) #1319

add config option for ipex (https://github.com/min-jean-cho/serve/blob/ipex_enable/examples/IPEX/README.md) #1319

Conversation

min-jean-cho commented Nov 13, 2021 • edited Loading

sagemaker-neo-ci-bot commented Nov 13, 2021

AWS CodeBuild CI Report

sagemaker-neo-ci-bot commented Nov 13, 2021

AWS CodeBuild CI Report

sagemaker-neo-ci-bot commented Nov 13, 2021

AWS CodeBuild CI Report

ashokei commented Nov 15, 2021

msaroufim commented Nov 15, 2021 • edited Loading

msaroufim commented Nov 16, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jgong5 Nov 16, 2021 • edited Loading

Choose a reason for hiding this comment

HamidShojanazeri Nov 16, 2021 • edited Loading

Choose a reason for hiding this comment

jgong5 Nov 17, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sagemaker-neo-ci-bot commented Nov 17, 2021

AWS CodeBuild CI Report

sagemaker-neo-ci-bot commented Nov 17, 2021

AWS CodeBuild CI Report

sagemaker-neo-ci-bot commented Nov 17, 2021

AWS CodeBuild CI Report

sagemaker-neo-ci-bot commented Nov 17, 2021

AWS CodeBuild CI Report

sagemaker-neo-ci-bot commented Nov 17, 2021

AWS CodeBuild CI Report

sagemaker-neo-ci-bot commented Nov 17, 2021

AWS CodeBuild CI Report

sagemaker-neo-ci-bot commented Nov 17, 2021

AWS CodeBuild CI Report

sagemaker-neo-ci-bot commented Nov 17, 2021

AWS CodeBuild CI Report

sagemaker-neo-ci-bot commented Nov 17, 2021

AWS CodeBuild CI Report

sagemaker-neo-ci-bot commented Nov 17, 2021

AWS CodeBuild CI Report

sagemaker-neo-ci-bot commented Nov 17, 2021

AWS CodeBuild CI Report

sagemaker-neo-ci-bot commented Nov 17, 2021

AWS CodeBuild CI Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sagemaker-neo-ci-bot commented Nov 22, 2021

AWS CodeBuild CI Report

sagemaker-neo-ci-bot commented Nov 22, 2021

AWS CodeBuild CI Report

sagemaker-neo-ci-bot commented Nov 22, 2021

AWS CodeBuild CI Report

sagemaker-neo-ci-bot commented Nov 22, 2021

AWS CodeBuild CI Report

sagemaker-neo-ci-bot commented Nov 22, 2021

AWS CodeBuild CI Report

sagemaker-neo-ci-bot commented Nov 22, 2021

AWS CodeBuild CI Report

sagemaker-neo-ci-bot commented Nov 22, 2021

AWS CodeBuild CI Report

sagemaker-neo-ci-bot commented Nov 22, 2021

AWS CodeBuild CI Report

sagemaker-neo-ci-bot commented Nov 22, 2021

AWS CodeBuild CI Report

min-jean-cho commented Nov 22, 2021

msaroufim commented Nov 23, 2021

sagemaker-neo-ci-bot commented Nov 23, 2021

AWS CodeBuild CI Report

sagemaker-neo-ci-bot commented Nov 23, 2021

min-jean-cho commented Nov 13, 2021 •

edited

Loading

msaroufim commented Nov 15, 2021 •

edited

Loading

jgong5 Nov 16, 2021 •

edited

Loading

HamidShojanazeri Nov 16, 2021 •

edited

Loading

jgong5 Nov 17, 2021 •

edited

Loading