python session.run() fallback to CPU/CUDA provider for EP failures. #1960
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…untime into jywu_py_fallback
HectorSVC
reviewed
Oct 1, 2019
| try: | ||
| return self._sess.run(output_names, input_feed, run_options) | ||
| except C.EPFail as err: | ||
| if self._enable_fallback: |
There was a problem hiding this comment.
if self._enable_fallback: [](start = 12, length = 25)
Is it possible that for CPUProvider only, if failed, it will try CPU one more time?
There was a problem hiding this comment.
no, the intent is to retry only for EPFail status, which should only get returned from EP's Compile() or EP's compute_func
HectorSVC
approved these changes
Oct 1, 2019
pranavsharma
approved these changes
Oct 2, 2019
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
python api session.run() fallback mechanism.
Problem Statement: Execution Provider internal failures cause session.run() to fail.
An execution provider's GetCapability() implementation is supposed to accurately return what subgraphs on an ONNX model the EP can execute. However, for 3rd party EP's, sometimes the response is not accurate, leading to Onnxruntime assigning a subgraph to an EP, but the EP fails during runtime.
To satisfy ORT's requirement to be able to execute all valid ONNX models, we introduce a fallback/retry mechanism for the session.Run() api, which is enabled by default.
If Run() fails due to internal EP failure, it will recreate a new session and retry using default execution providers ['CPUExecutionProvider'] or ['CUDAExecutionProvider, 'CPUExecutionProvider'] (if gpu capable)
For example, if session.run() is invoked and the subgraph is assigned to TensorrtExecutionProvider but fails during execution, the session will be recreated and retried using CUDAExecutionProvider.
Similarly, if NGRAPHExecutionProvider fails, a new session is recreated and the session.run() retried using CPUExecutionProvider.
The session is recreated at most once. The retry only happens when EPFail exception is thrown by session.Run(). Thus, other exceptions thrown by ORT will not trigger the retry.
Once the session is recreated and set to default providers, subsequent run() invocations will use the default provider settings. (to prevent the same failure from happening again)
For this PR, NGRAPHExecution provider was updated to return the proper EPFail status code.
Other EP's will be updated in subsequent PRs.
Tested against previously failing onnx backend tests (e.g. test_hardmax_negative_axis) which fails using NGraph. with this PR, it fallsback and retries on CPU and succeeds.