Fix deterministic mode (and use it for tests); simplify checkpointing tests #203
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
1. Deterministic mode on GPUs requires the CUBLAS_WORKSPACE_CONFIG environment variable to be set _before_ the training script is launched. The launch script can properly set this variable 2. Renamed composer.utils.determinism to composer.utils.reproducability to match the pytorch naming in the docs 3. Switched the testrunner to use deterministic mode 4. Renamed the run directory environment variable to COMPOSER_RUN_DIRECTORY, so it would have a parallel name to COMPOSER_USE_DETERMINISTIC_MODE
Averylamp
suggested changes
Jan 6, 2022
…et that in the hparams. Further reduced number of checkpoint tests
Averylamp
approved these changes
Jan 6, 2022
coryMosaicML
pushed a commit
to coryMosaicML/composer
that referenced
this pull request
Feb 23, 2022
… tests (mosaicml#203) 1. Deterministic mode on GPUs requires the `CUBLAS_WORKSPACE_CONFIG`. Setting this variable when configuring deterministic mode. 2. Renamed composer.utils.determinism to composer.utils.reproducibility to match the pytorch naming in the docs 3. Configured tests to use deterministic mode by default. 4. Configured checkpoints to not compress by default, since compression is slow 5. Made the gpu and deepspeed markers mutually exclusive, since they require different process initialization
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
CUBLAS_WORKSPACE_CONFIG. Setting this variable when configuring deterministic mode.