Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

support fixed ntk rope in modeling_rope_utils.py #32424

Closed
wants to merge 4 commits into from
Closed

support fixed ntk rope in modeling_rope_utils.py #32424

wants to merge 4 commits into from

Conversation

Fazziekey
Copy link

@Fazziekey Fazziekey commented Aug 5, 2024

What does this PR do?

Related PR:
#32135
#31999

the Origin NTK scaling factor is fixed, but now modeling_rope_utils.py only supports dynamic ntk scaling.

Refer to
Colab code: https://colab.research.google.com/drive/1VI2nhlyKvd5cw4-zHvAIk00cAVj2lCCC#scrollTo=b80b3f37

import transformers

old_init = transformers.models.llama.modeling_llama.LlamaRotaryEmbedding.__init__
def ntk_scaled_init(self, dim, max_position_embeddings=2048, base=10000, device=None):

    #The method is just these three lines
    max_position_embeddings = 16384
    a = 8 #Alpha value
    base = base * a ** (dim / (dim-2)) #Base change formula

    old_init(self, dim, max_position_embeddings, base, device)

some examples of opensource project:

Before submitting

  • This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
  • Did you read the contributor guideline,
    Pull Request section?
  • Was this discussed/approved via a Github issue or the forum? Please add a link
    to it if that's the case.
  • Did you make sure to update the documentation with your changes? Here are the
    documentation guidelines, and
    here are tips on formatting docstrings.
  • Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@Fazziekey
Copy link
Author

Fazziekey commented Aug 5, 2024

@Fazziekey @gante

Copy link
Member

@gante gante left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey there 👋 Thank you for being proactive and opening a PR! Now that we've abstracted the inv_freq computations, we are indeed open to adding more RoPE types!

The PR LGTM. Only one item missing: a numerical test for the new function. See e.g. this test :)

device (`torch.device`):
The device to use for initialization of the inverse frequencies.
seq_len (`int`, *optional*):
The current sequence length, used to update the dynamic RoPE at inference time.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
The current sequence length, used to update the dynamic RoPE at inference time.
The current sequence length. Unused for this type of RoPE.

Comment on lines 202 to 203
# seq_len: default to max_position_embeddings, e.g. at init time
seq_len = seq_len if seq_len is not None and seq_len > max_position_embeddings else max_position_embeddings
Copy link
Member

@gante gante Aug 7, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# seq_len: default to max_position_embeddings, e.g. at init time
seq_len = seq_len if seq_len is not None and seq_len > max_position_embeddings else max_position_embeddings

(unused, as it is not dynamic)

@Fazziekey
Copy link
Author

Hey there 👋 Thank you for being proactive and opening a PR! Now that we've abstracted the inv_freq computations, we are indeed open to adding more RoPE types!

The PR LGTM. Only one item missing: a numerical test for the new function. See e.g. this test :)

Hello, I have update test and remove seq_len, can you make a review , thanks.

@Fazziekey
Copy link
Author

@gante can you make a review? Thanks so much

@Fazziekey Fazziekey closed this by deleting the head repository Nov 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants