[QUANTIZE] Improve explicitness of rules during annotation/realization #3828
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ZihengJiang
commented
Aug 24, 2019
|
|
||
| def _find_scale_by_kl(arr, | ||
| quantized_dtype='int8', | ||
| num_bins=8001, |
There was a problem hiding this comment.
How do we choose the parameter? @vinx13
There was a problem hiding this comment.
this parameter is a tradeoff between precision of the computed KLD and speed of calibration, the default one is good in my experiments
ZihengJiang
commented
Aug 27, 2019
| # TODO: need to fix accuracy | ||
| # Config('mobilenetv2_1.0', nbit_input=8, dtype_input='int8', nbit_output=16, dtype_output='int16', global_scale=4.0), | ||
| # resnet18_v1 best configuration | ||
| Config('resnet18_v1', nbit_input=8, dtype_input='int8', nbit_output=16, dtype_output='int16', global_scale=8.0, expected_acc=0.675), |
There was a problem hiding this comment.
It seems this method brings some accuracy drop, I will hold this PR until find workaround
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In the previous version, we have put too many duties to realization: transform the simulated graph; decide data type casting after
simulated_quantize; foraddoperation, it also will decide the output scale and unify its operands. Things are quite complicated in this situation with many implicit rules. This PR would like to move those extra functions out of the realization procedure.Insert
cast_hintexplicitly during annotation. Thedtypefiled inQRealizeExprhas been removed.Currently,
cast_hintdoes two things: 1.It has been inserted duringPartition, and simulated_quantize to INPUT will be inserted before it during annotation. This is for storing low-precision output of residual block. 2. It has been inserted duringAnnotation, and will be transformed tocastduring realization. Before realization, it just pass the input through likeidentity, so will has no effect to the output before realization.Modify annotate/realize rule for
addition.Previously, we will quantize operands of addition separately during annotation, then unify their scale during realization. This way has a lot burden since there exist many combinations of operands' kind. Currently, we will quantize operands into the same
dom_scaleby two sq but with the samedom_scaleparameter. So that we only need to do identity transform during realization.This PR also did some refactor work for the calibration part: thanks @vinx13 evaluation script for KL, the collect stats part has been moved into internal
collect_stats. A new configcalibration_modehas been added.Minor improvement: saturation for left_shift inside of the
QuantizeRealize