Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add support for fp16 execution and memory tracking #40

Merged
merged 81 commits into from
Dec 16, 2021
Merged

Add support for fp16 execution and memory tracking #40

merged 81 commits into from
Dec 16, 2021

Conversation

santhnm2
Copy link
Collaborator

No description provided.

santhnm2 and others added 30 commits September 20, 2021 14:27
@santhnm2
Add fp16 support
099f0d4
@santhnm2
Add fp16 support for GPT2
b5d1da8
@santhnm2
Fix formatting
4bce097
@santhnm2
Fix tests with no GPU available
7a3e573
@santhnm2
Add flag to skip adding AllGather ops in transforms
82e1309
@santhnm2
Add fp16 support to simulator calibration
b621cb1
@siddharth-krishna
Create DataFrame of calibration data and dump to csv
eb777a6
@santhnm2
Directly store device parameters
60abe92
@santhnm2
GPT fp16 fixes
2b12914
@santhnm2
Merge remote-tracking branch 'origin/fp16' into allgather_flag
fb8c06e
@santhnm2
Change default to fp16, set use_gpu to true for scripts
2e0e34f
@santhnm2
MLP model fix
11aab23
@santhnm2
Remove explicit fp16 cast
253459b
@santhnm2
In place SGD updates
13bbf5c
@santhnm2
Add all_world_sizes flag to script
682c25b
@santhnm2
Fix simulation parameter order
8a80da0
@santhnm2
Add debug prints
d534b20
@santhnm2
Empty cache between grid search runs
b964d55
@santhnm2
Remove NaN check, add start/end batch size args to pure baselines script
9c110e2
@santhnm2
Add register entry for SGDOptimizer with 2 inputs
a39edc2
@santhnm2
Skip SGD outputs when updating live memory estimate
ec3bcc7
@santhnm2
Updated amulet file
441b875
@santhnm2
Merge branch 'allgather_flag' of github.com:microsoft/dist-ir into al…
194025f 
…lgather_flag
@siddharth-krishna
Upload GPT onnx file via amlt
f6d4ee5
@santhnm2
Lower MLP model sizes, fix GPT-2 fp16
e6a75a6
@santhnm2
Only generate one weight tensor for MLP
cef30f0
@santhnm2
Restore original mlp sizes, add debug prints
17a6a1b
@santhnm2
Update calibration data sizes
e47177a
@santhnm2
[WIP] Add flag for peak memory tracking
0c23b75
@santhnm2
Peak memory fixes, add flag for mlp.py
5ee470c
siddharth-krishna
siddharth-krishna reviewed Oct 28, 2021
View reviewed changes
Copy link
Collaborator

@siddharth-krishna siddharth-krishna left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review part 1: mostly on torch.py. It looks good, but I've suggested some cleanups and asked some questions. I'll try to do the rest before the weekend.

(Should we update the PR title to reflect the other changes, such as experiment scripts?)

@siddharth-krishna siddharth-krishna marked this pull request as ready for review October 29, 2021 10:47
siddharth-krishna
siddharth-krishna reviewed Oct 29, 2021
View reviewed changes
Copy link
Collaborator

@siddharth-krishna siddharth-krishna left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review part 2/2. Thank you, everything looks pretty good, just some minor comments again.

siddharth-krishna
siddharth-krishna approved these changes Nov 8, 2021
View reviewed changes
Copy link
Collaborator

@siddharth-krishna siddharth-krishna left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, thanks! If we fix the merge conflicts, we can merge this PR in.

@santhnm2 santhnm2 merged commit a881214 into main Dec 16, 2021
@santhnm2 santhnm2 deleted the memory_tracking branch December 16, 2021 22:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@siddharth-krishna siddharth-krishna siddharth-krishna approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@santhnm2 @siddharth-krishna