FSDP Compatible, Sequential SparseGPT #1947
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
bfineran
approved these changes
Jan 10, 2024
bfineran
pushed a commit
that referenced
this pull request
Jan 11, 2024
* initial recipe re-loading * loading for input recipe * persist structure across recipe loads * clean up fn names * clean up duplicated code * delete extra file * unit tests * fix failing test * quantization edge cases * quant tests * fixes for stage name clashes * clean up documentation * setup StageRunner class * running one_shot from text_gen script * cleanup helper fns * precision support * formatting * WIP for alternating * fixing device issue * MVP for alternating flows * add apply flag during finalization as well * clarity comments * clean up docstrings * fix unit test * WIP FSDP support * fix for unwrapping * WIP for state reloading between stages * example fsdp config updates * add finetuning README * fix for 2nd oneshot stage * cleaning up stage logic * mvp for single GPU fsdp * WIP obcq FSDP * quality * sgpt wrapper * clean up on finalize, improve logging * cleaning up device args * merge alternating * WIP alternating * fsdp compatible, training loss issue * fix for loss bug * fixing checkpoint issue * fix for quantization saving * cleanup after merge * unit test fix * clean up logging * move FSDP to helper files * update docstrings, clean up * fix circular import * unmodify example * fix typo! * setup FSDP for when starting from oneshot * update setup and readme * fix CLI issue, update README * POC for sequential FSDP OBCQ (#1947) * fix GHA line lost in merge * fix calib loading * fix dependencies * reverting OBCQ merged changes for now * restore SparseCausalModel for now * add progress bar for calibration forward pass (#1950) ---------
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adding back the
sequential_updateflag toSparseGPTModifierandWandaModifier. Previously this flag affected whether or not we calibrate modules within a transformers block sequentially. Regardless of this flag we would always perform OBCQ sequentially across different transformer blocks.In the updated FSDP compatible implementation, transformers blocks are calibrated in parallel with no option for sequential calibration. The new
sequential_updateflag affects whether transformer blocks are processed sequentially, modules within blocks are always calibrated in parallel. Note that running sequential updates requires a lot more computation, as the whole model forward pass it runnum_calibration_samplestimes for each transformer block