-
Notifications
You must be signed in to change notification settings - Fork 28.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
support fixed ntk rope in modeling_rope_utils.py #32424
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hey there 👋 Thank you for being proactive and opening a PR! Now that we've abstracted the inv_freq
computations, we are indeed open to adding more RoPE types!
The PR LGTM. Only one item missing: a numerical test for the new function. See e.g. this test :)
device (`torch.device`): | ||
The device to use for initialization of the inverse frequencies. | ||
seq_len (`int`, *optional*): | ||
The current sequence length, used to update the dynamic RoPE at inference time. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The current sequence length, used to update the dynamic RoPE at inference time. | |
The current sequence length. Unused for this type of RoPE. |
# seq_len: default to max_position_embeddings, e.g. at init time | ||
seq_len = seq_len if seq_len is not None and seq_len > max_position_embeddings else max_position_embeddings |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
# seq_len: default to max_position_embeddings, e.g. at init time | |
seq_len = seq_len if seq_len is not None and seq_len > max_position_embeddings else max_position_embeddings |
(unused, as it is not dynamic)
Hello, I have update test and remove |
@gante can you make a review? Thanks so much |
What does this PR do?
Related PR:
#32135
#31999
the Origin NTK scaling factor is fixed, but now modeling_rope_utils.py only supports dynamic ntk scaling.
Refer to
Colab code: https://colab.research.google.com/drive/1VI2nhlyKvd5cw4-zHvAIk00cAVj2lCCC#scrollTo=b80b3f37
some examples of opensource project:
Before submitting
Pull Request section?
to it if that's the case.
documentation guidelines, and
here are tips on formatting docstrings.
Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.