Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix _scaled_dot_product_attention bug #941

Merged
merged 61 commits into from
Mar 20, 2024
Merged

fix _scaled_dot_product_attention bug #941

Changes from all commits
Commits
Show all changes
61 commits
Select commit Hold shift + click to select a range
804644c
init falcon
hypertseng Oct 20, 2023
45e0995
Merge remote-tracking branch 'upstream/master'
hypertseng Nov 5, 2023
c685c11
init on server
hypertseng Nov 8, 2023
d65b127
Merge branch 'mindspore-lab:master' into master
hypertseng Nov 14, 2023
1c0ac42
module test
hypertseng Nov 15, 2023
19dbc89
Merge branch 'mindspore-lab:master' into master
hypertseng Nov 15, 2023
4f3d94e
Merge branch 'master' of https://github.com/Zeng-Zixuan/mindnlp
hypertseng Nov 15, 2023
814c9c5
Merge branch 'mindspore-lab:master' into master
hypertseng Nov 18, 2023
bbb0cc2
falcon ut
hypertseng Nov 18, 2023
7b89732
Merge branch 'mindspore-lab:master' into master
hypertseng Nov 22, 2023
dd996f3
Merge branch 'master' of https://github.com/Zeng-Zixuan/mindnlp
hypertseng Nov 22, 2023
6889171
Merge branch 'mindspore-lab:master' into master
hypertseng Nov 25, 2023
342d152
Merge branch 'mindspore-lab:master' into master
hypertseng Nov 27, 2023
02331d9
Add falcon_config
hypertseng Nov 28, 2023
6b8c467
Add convert_custum_code_checkpoint
hypertseng Nov 28, 2023
c7b8ea2
revise loss bug
hypertseng Nov 28, 2023
b57dffc
Add falcon_test
hypertseng Nov 28, 2023
56b439a
Merge branch 'mindspore-lab:master' into master
hypertseng Dec 3, 2023
3078482
Merge branch 'mindspore-lab:master' into master
hypertseng Dec 4, 2023
4f83bad
Merge branch 'mindspore-lab:master' into master
hypertseng Dec 5, 2023
5745d10
add falcon model
Dec 5, 2023
0dcf872
Merge branch 'mindspore-lab:master' into master
hypertseng Dec 6, 2023
787b5b6
add falcon model
Dec 7, 2023
287a934
add _LazyModule
Dec 7, 2023
dc535e4
add blip_2 model
Dec 7, 2023
fd23480
Merge branch 'mindspore-lab:master' into master
hypertseng Dec 7, 2023
35a418f
Merge branch 'master' of https://github.com/Zeng-Zixuan/mindnlp
Dec 7, 2023
b7997e1
delete falcon test
Dec 7, 2023
9a227d5
pylint check
Dec 7, 2023
ebc9981
pylint check
Dec 7, 2023
364e279
remove init in transformers style
Dec 7, 2023
c44b973
pylint check
Dec 7, 2023
0dfc1b0
rename falcon model
Dec 8, 2023
c4eee72
Merge branch 'master' of https://github.com/Zeng-Zixuan/mindnlp
Dec 8, 2023
fc767bc
fix pylint
Dec 8, 2023
b07bb8d
delete LazyModule
Dec 8, 2023
935e597
add falcon config
Dec 8, 2023
145d32e
resolve cyclic error
Dec 8, 2023
085d1c7
resolve cyclic error
Dec 8, 2023
5ce76b6
resolve cyclic error
Dec 8, 2023
78573eb
resolve cyclic error
Dec 8, 2023
0666e31
Merge branch 'master' of https://github.com/Zeng-Zixuan/mindnlp
Dec 10, 2023
1d979d0
Merge branch 'master' of https://github.com/Zeng-Zixuan/mindnlp
Dec 12, 2023
2bc7706
Merge branch 'mindspore-lab:master' into master
hypertseng Dec 19, 2023
c60dc15
Merge branch 'mindspore-lab:master' into master
hypertseng Dec 28, 2023
10eff6d
add falcon config
Dec 28, 2023
f25a112
add falcon finetune
Dec 28, 2023
68d982a
add falcon finetune
Jan 9, 2024
b4ed82e
Merge branch 'mindspore-lab:master' into master
hypertseng Jan 11, 2024
3090013
add falcon finetune
Jan 11, 2024
cdc93b9
Merge branch 'mindspore-lab:master' into master
hypertseng Jan 12, 2024
3e2b961
add falcon finetune
Jan 12, 2024
a69db60
add falcon finetune
Jan 13, 2024
bc93072
modify finetune config
hypertseng Jan 15, 2024
d52ee42
add falcon_finetune readme
Jan 18, 2024
e6d0383
remove logger for test
Jan 18, 2024
6facade
remove comments
Jan 18, 2024
e07709e
Merge branch 'mindspore-lab:master' into master
hypertseng Jan 24, 2024
c869156
Merge branch 'mindspore-lab:master' into master
hypertseng Mar 17, 2024
a1fb86a
Merge branch 'mindspore-lab:master' into master
hypertseng Mar 20, 2024
a99417b
fix _scaled_dot_product_attention
Mar 20, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion mindnlp/_legacy/functional.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -273,7 +273,7 @@ def _in_projection_packed(q, k, v, w, b, k_is_v, q_is_k):

def _scaled_dot_product_attention(query, key, value, attn_mask, dropout_p, is_causal, is_training):
embed_size = query.shape[-1]
scaling_factor = sqrt(sqrt(Tensor(embed_size, mindspore.float32)))
scaling_factor = sqrt(Tensor(embed_size, dtype=query.dtype))
query = query / scaling_factor

if is_causal:
Expand Down
Loading