fix: CUDA error 710 bugfix #1424
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
gs-olive
commented
Oct 28, 2022
|
@bowang007 Make sure to review this |
|
From my perspective see nothing wrong with sampling between |
- Issue arising when compiling BERT models with 3+ inputs - Added temporary fix by decreasing the range of allowed values to the random number generator for creating input tensors to [0,2), instead of [0,5) - Used random float inputs in the range [0, 2) instead of int, then casted to desired type. The ultimate effect of this change with regard to bug pytorch#1418, is random floats are selected in the range [0, 2), then casted to Int, effectively making the range of allowed ints {0, 1}, as required by the model - More robust fix to follow
8f41d83 to
595b9f4
Compare
gs-olive
commented
Nov 3, 2022
|
|
||
| // Make the value range for input tensor a uniform (float) distribution | ||
| // over [LoValIncl, HiValExcl), then cast to the desired dtype | ||
| auto in = ((HiValExcl - LoValIncl) * at::rand(shape, {at::kCUDA}) + LoValIncl).to(type); |
There was a problem hiding this comment.
Used float inputs in the range
There was a problem hiding this comment.
seems like a little bit hard-coded for this model only, but will be resolved once the input range is open to users by this RFC #1425.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Resolves a CUDA 710 error Issue arising when compiling BERT models with 3+ inputs. The issue arises due to the role of the third tensor in inference computations. Specifically, as specified in the BERT model code linked here, the third argument,$[0,1]$ . This means that when values outside of this set are used, the input is invalid.
token_type_idsis of typetorch.LongTensor, but can only take indices inThis becomes problematic when the inputs are, for example, indices in a dictionary or embedding - which seems to be the case here. Specifically,$[0,4]$ .
aten::embeddingis used with Tensors which are the product oftoken_type_ids. The issue traces to one line in theshape_analysiscode previewed below, which initializes a random tensor with values in the rangeThis tensor is run through the
forwardfunction of the module to determine the shapes of outputs and causes the model compilation-time error, as featured here in the shape analysis code.I have added a temporary fix by decreasing the range of allowed values to the random number generator for creating input tensors to 0-1, instead of 0-4, and am working on a more robust fix.
Fixes #1418
Type of change
Please delete options that are not relevant and/or add your own.
Checklist: