Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

truncated normal initializer #38

Closed
ruotianluo opened this issue Nov 19, 2018 · 2 comments
Closed

truncated normal initializer #38

ruotianluo opened this issue Nov 19, 2018 · 2 comments

Comments

@ruotianluo
Copy link

I have a reasonable truncated normal approximation. (Actually that is what tf does).
https://discuss.pytorch.org/t/implementing-truncated-normal-initializer/4778/16?u=ruotianluo
@thomwolf
Copy link
Member

thomwolf commented Nov 20, 2018 •
edited
Loading

We could try that. Not sure how important it is though. Did you try it?

@thomwolf
Copy link
Member

Ok I think we will stick to the normal_initializer for now. Thanks for indicating this option!

@maeotaku maeotaku mentioned this issue May 23, 2019
Closed
xloem pushed a commit to xloem/transformers that referenced this issue Apr 9, 2023
@KSGulin @bfineran
Upgrade to transformers release V4.23.1 (huggingface#62)
46f78c1 
* Update trainer and model flows to accommodate sparseml

Disable FP16 on QAT start (huggingface#12)

* Override LRScheduler when using LRModifiers

* Disable FP16 on QAT start

* keep wrapped scaler object for training after disabling

Using QATMatMul in DistilBERT model class (huggingface#41)

Removed double quantization of output of context layer. (huggingface#45)

Fix DataParallel validation forward signatures (huggingface#47)

* Fix: DataParallel validation forward signatures

* Update: generalize forward_fn selection

Best model after epoch (huggingface#46)

fix sclaer check for non fp16 mode in trainer (huggingface#38)

Mobilebert QAT (huggingface#55)

* Remove duplicate quantization of vocabulary.

enable a QATWrapper for non-parameterized matmuls in BERT self attention (huggingface#9)

* Utils and auxillary changes

update Zoo stub loading for SparseZoo 1.1 refactor (huggingface#54)

add flag to signal NM integration is active (huggingface#32)

Add recipe_name to file names

* Fix errors introduced in manual cherry-pick upgrade

Co-authored-by: Benjamin Fineran <bfineran@users.noreply.github.com>
jameshennessytempus pushed a commit to jameshennessytempus/transformers that referenced this issue Jun 1, 2023
@jamesthesnake
Merge pull request huggingface#38 from jamesthesnake/nah
b7ecfb6 
Nah
@lwmlyy lwmlyy mentioned this issue Aug 15, 2023
Merged
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@thomwolf @ruotianluo