Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IDEFICS: allow interpolation of vision's pos embeddings #26029

Merged
Merged
Changes from 1 commit
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
c16631d
add pos embed interpolation for vision encoder
leot13 Sep 7, 2023
242ca96
style
leot13 Sep 7, 2023
a89b7bf
update config with interpolate_pos_encoding arg
leot13 Sep 7, 2023
ca94372
fix imports formatting
leot13 Sep 7, 2023
bb67a9f
take off copied from on vision embeddings
leot13 Sep 7, 2023
498a881
add test for image embeddings interpolation
leot13 Sep 7, 2023
857b45c
add credit for interpolation code
leot13 Sep 7, 2023
8601455
Update src/transformers/models/idefics/configuration_idefics.py
leot13 Sep 8, 2023
52be895
Update src/transformers/models/idefics/vision.py
leot13 Sep 8, 2023
21d0bfa
fix condition to check nbr image patches match shape of pos embeddings
leot13 Sep 8, 2023
2840e4a
use kwargs in the forward methods for interpolation
leot13 Sep 8, 2023
4c9391f
fix tests
leot13 Sep 8, 2023
689e809
have interpolate_pos_encoding default to False instead of None
leot13 Sep 8, 2023
1a2d2e5
Update tests/models/idefics/test_modeling_idefics.py
leot13 Sep 11, 2023
3400654
Update tests/models/idefics/test_modeling_idefics.py
leot13 Sep 11, 2023
07b0bff
Update tests/models/idefics/test_modeling_idefics.py
leot13 Sep 11, 2023
79d4c43
Update src/transformers/models/idefics/configuration_idefics.py
leot13 Sep 11, 2023
bd8a67d
take off for loop meant to print k,v
leot13 Sep 11, 2023
2f1d449
add interpolate_pos_encoding arg in prepare_inputs_for_generation
leot13 Sep 12, 2023
ee2ce91
add test for interpolated generation
leot13 Sep 12, 2023
365c2a2
fix edge case num_patches == num_positions and height == width
leot13 Sep 12, 2023
e44649a
add test for edge case
leot13 Sep 12, 2023
9decae1
fix pos_embed in interpolate
leot13 Sep 12, 2023
776ce7a
allow interpolation in bf16 with upcasting
leot13 Sep 14, 2023
ee98321
Update src/transformers/models/idefics/vision.py
leot13 Sep 14, 2023
7483d5a
Update src/transformers/models/idefics/vision.py
leot13 Sep 14, 2023
82fda57
add multiple images tests for interpolation and generation
leot13 Sep 14, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
take off for loop meant to print k,v
  • Loading branch information
leot13 committed Sep 11, 2023
commit bd8a67d2b670ecffe8d748cc98065bb0c143698b
1 change: 0 additions & 1 deletion tests/models/idefics/test_modeling_idefics.py
Original file line number Diff line number Diff line change
Expand Up @@ -302,7 +302,6 @@ def test_training(self):
model.to(torch_device)
model.train()
inputs = self._prepare_for_class(inputs_dict, model_class, return_labels=True)
for k, v in inputs.items():
loss = model(**inputs).loss
loss.backward()

Expand Down