Fp8 integration #1086
Fp8 integration #1086
Conversation
|
The documentation is not available anymore as the PR was closed or merged. |
src/accelerate/accelerator.py
Outdated
| default to the value in the environment variable `ACCELERATE_MIXED_PRECISION`, which will use the default | ||
| value in the accelerate config of the current system or the flag passed with the `accelerate.launch` | ||
| command. 'fp16' requires pytorch 1.6 or higher. 'bf16' requires pytorch 1.10 or higher. | ||
| Whether or not to use mixed precision training (fp16 or bfloat16). Choose from 'no','fp16','bf16 or 'fp8'. |
There was a problem hiding this comment.
| Whether or not to use mixed precision training (fp16 or bfloat16). Choose from 'no','fp16','bf16 or 'fp8'. | |
| Whether or not to use mixed precision training (fp8, fp16, or bfloat16). Choose from 'no','fp16','bf16 or 'fp8'. |
There was a problem hiding this comment.
Based on the earlier comment, this could be "fp16, bfloat16, or fp8", or we remove the () and just have "Choose from"
There was a problem hiding this comment.
Removing the parenthesis entirely.
src/accelerate/accelerator.py
Outdated
| f"The current device has compute capability of {torch.cuda.get_device_capability()} which is " | ||
| "insufficient for FP8 mixed precision training (requires a GPU Hopper or higher, compute " | ||
| "capability of 9 or higher). Will using FP16 instead." |
There was a problem hiding this comment.
Should this only warn? Or should it not explicitly raise an error?
There was a problem hiding this comment.
It still uses transformer engine instead of the regular model, so useful for testing on A100s
| import transformer_engine.pytorch as te | ||
|
|
||
|
|
||
| def convert_model(model, to_transformer_engine=True, _convert_linear=True, _convert_ln=True): |
There was a problem hiding this comment.
Should this run an explicit try/catch for is_fp8_available and raise an error if not?
There was a problem hiding this comment.
It shouldn't be called if is_fp8_available is False.
Co-authored-by: Sourab Mangrulkar <13534540+pacman100@users.noreply.github.com>
This PR brings FP8 mixed precision training to Accelerate. Using this requires a GPU Hopper or higher (hard to find at the moment!) and the
transformers_enginelibrary.