Code

Add PaliGemma#30814

Code

Merged

molbap merged 120 commits intomainhuggingface/transformers:mainfrom add_palmahuggingface/transformers:add_palmaCopy head branch name to clipboardMay 14, 2024

+1,890-8

Commits

Commits on Mar 4, 2024

• add new model like

  molbap

  committed
  2fddcc9

  Copy full SHA for 2fddcc9

• Merge pull request #9 from huggingface/update

  Show description for e536f6a
  molbap

  authored
  e536f6a

  Copy full SHA for e536f6a

Commits on Mar 5, 2024

• add state dict slicing + new model config

  molbap

  committed
  6ca0bf7

  Copy full SHA for 6ca0bf7

Commits on Mar 6, 2024

• update palma config and weights, passes vision activations

  molbap

  committed
  21db4a7

  Copy full SHA for 21db4a7

• fix

  molbap

  committed
  7985fc4

  Copy full SHA for 7985fc4

Commits on Mar 7, 2024

• update

  molbap

  committed
  bcb341d

  Copy full SHA for bcb341d

Commits on Mar 8, 2024

• reorder loading/unpacking

  molbap

  committed
  38ad70e

  Copy full SHA for 38ad70e

Commits on Mar 11, 2024

• clean up

  molbap

  committed
  929746a

  Copy full SHA for 929746a

Commits on Mar 12, 2024

• add debug statements

  molbap

  committed
  ae55ad9

  Copy full SHA for ae55ad9

• change device

  molbap

  committed
  9d5f8fb

  Copy full SHA for 9d5f8fb

• Merge branch 'add_palma' of github.com:huggingface/new-model-addition-palma into add_palma

  molbap

  committed
  524a073

  Copy full SHA for 524a073

• fix

  molbap

  committed
  d171c4e

  Copy full SHA for d171c4e

Commits on Mar 13, 2024

• debugging

  molbap

  committed
  1638a73

  Copy full SHA for 1638a73

Commits on Mar 14, 2024

• fix noncausal mask

  molbap

  committed
  4aad850

  Copy full SHA for 4aad850

• fixup sdpa + causal mask

  molbap

  committed
  2972674

  Copy full SHA for 2972674

• fix activation function

  molbap

  committed
  71fa912

  Copy full SHA for 71fa912

• remove debug before changing modeling file

  molbap

  committed
  94e4806

  Copy full SHA for 94e4806

Commits on Mar 15, 2024

• add variants

  molbap

  committed
  7b6f0b3

  Copy full SHA for 7b6f0b3

• debug attention mask in generate

  molbap

  committed
  4e8e1c6

  Copy full SHA for 4e8e1c6

• revert to non-debug sdpa

  molbap

  committed
  ba8fb4e

  Copy full SHA for ba8fb4e

• revert gemma modifications

  molbap

  committed
  6c2348d

  Copy full SHA for 6c2348d

• add custom language modeling

  molbap

  committed
  906e87f

  Copy full SHA for 906e87f

• use Processor

  molbap

  committed
  f26361d

  Copy full SHA for f26361d

• add language modeling file to init

  molbap

  committed
  b3e4a03

  Copy full SHA for b3e4a03

• try thin wrapper around generate

  molbap

  committed
  500a360

  Copy full SHA for 500a360

Commits on Mar 18, 2024

• Update

  molbap

  committed
  96a82e2

  Copy full SHA for 96a82e2

Commits on Mar 19, 2024

• update mask

  molbap

  committed
  347df2c

  Copy full SHA for 347df2c

• breakpoints galore

  molbap

  committed
  d01b502

  Copy full SHA for d01b502

• remove conflict

  molbap

  committed
  bb8030c

  Copy full SHA for bb8030c

• switch to left-padding

  molbap

  committed
  a6056c6

  Copy full SHA for a6056c6

• add incomplete model doc

  molbap

  committed
  dacd297

  Copy full SHA for dacd297

• add paligemma global files

  molbap

  committed
  f133a1d

  Copy full SHA for f133a1d

• batch rename paligemma

  molbap

  committed
  37c1368

  Copy full SHA for 37c1368

• make generation match outputs and captioning

  molbap

  committed
  9935e2e

  Copy full SHA for 9935e2e

• style

  molbap

  committed
  e7fd049

  Copy full SHA for e7fd049

• style

  molbap

  committed
  edcf1b1

  Copy full SHA for edcf1b1

• remove copied from + doc

  molbap

  committed
  2a29a2e

  Copy full SHA for 2a29a2e

• remove more copied from

  molbap

  committed
  5607c9d

  Copy full SHA for 5607c9d

• remove copy from projector

  molbap

  committed
  8ce77f3

  Copy full SHA for 8ce77f3

• minor fix

  molbap

  committed
  3563926

  Copy full SHA for 3563926

• update config and style

  molbap

  committed
  f48f61d

  Copy full SHA for f48f61d

Commits on Mar 21, 2024

• add readme - dummy

  molbap

  committed
  b84d1c1

  Copy full SHA for b84d1c1

Commits on Mar 22, 2024

• CORRECT image captioning

  molbap

  committed
  deb35ba

  Copy full SHA for deb35ba

Commits on Mar 26, 2024

• moving to args

  molbap

  committed
  9275d53

  Copy full SHA for 9275d53

Commits on Mar 27, 2024

• add siglip proper + fix merging image + text features

  molbap

  committed
  5487734

  Copy full SHA for 5487734

• take update_causal_mask from upstream

  molbap

  committed
  7f9c479

  Copy full SHA for 7f9c479

• remove breakpoint

  molbap

  committed
  438b143

  Copy full SHA for 438b143

Commits on Mar 28, 2024

• leverage AutoModel

  molbap

  committed
  00d2922

  Copy full SHA for 00d2922

• fix input_ids slicing

  molbap

  committed
  72697b6

  Copy full SHA for 72697b6

• make siglip head conditional

  molbap

  committed
  17b30fd

  Copy full SHA for 17b30fd

• remove encoder_decoder value

  molbap

  committed
  26cb46e

  Copy full SHA for 26cb46e

• remove unneeded modeling file

  molbap

  committed
  f49f389

  Copy full SHA for f49f389

• add commented 4d attention mask

  molbap

  committed
  d1964ea

  Copy full SHA for d1964ea

Commits on Mar 29, 2024

• FIXED generation with 4D mask

  molbap

  committed
  6da4d2e

  Copy full SHA for 6da4d2e

Commits on Mar 30, 2024

• Merge branch 'main' of github.com:huggingface/new-model-addition

  ArthurZucker

  committed
  ef8c0fb

  Copy full SHA for ef8c0fb

Commits on Mar 31, 2024

• Update src/transformers/models/siglip/modeling_siglip.py

  Show description for 9f261d7
  

  molbap

  and

  ArthurZucker

  authored
  9f261d7

  Copy full SHA for 9f261d7

Commits on Apr 2, 2024

• fix left padding detection

  molbap

  committed
  11c3488

  Copy full SHA for 11c3488

• shuffle order of verifications

  molbap

  committed
  6d527e4

  Copy full SHA for 6d527e4

• Merge branch 'main' into add_palma

  molbap

  committed
  8b679b1

  Copy full SHA for 8b679b1

Commits on Apr 3, 2024

• fix missing labels for training

  molbap

  committed
  fdba83a

  Copy full SHA for fdba83a

• fix

  molbap

  committed
  eee3c1e

  Copy full SHA for eee3c1e

Commits on Apr 10, 2024

• vectorize merging of features, improve slicing

  molbap

  committed
  e4869dd

  Copy full SHA for e4869dd

• improve testing before conversion

  molbap

  committed
  b973365

  Copy full SHA for b973365

• handle merging in processor

  molbap

  committed
  8cab59c

  Copy full SHA for 8cab59c

• image token index depends on checkpoint

  molbap

  committed
  9186b79

  Copy full SHA for 9186b79

Commits on Apr 15, 2024

• add variants, save processor too

  molbap

  committed
  78398a1

  Copy full SHA for 78398a1

• save processors, base tokenizer off spm file

  molbap

  committed
  4ede86e

  Copy full SHA for 4ede86e

Commits on Apr 17, 2024

• Merge branch 'main' of github.com:huggingface/transformers into main

  ArthurZucker

  committed
  c899136

  Copy full SHA for c899136

Commits on Apr 24, 2024

• expand model embeddings due to additional image token

  molbap

  committed
  a912273

  Copy full SHA for a912273

• pass image processing args

  molbap

  committed
  2c8a508

  Copy full SHA for 2c8a508

• add convert rgb to siglip processor

  molbap

  committed
  7201d43

  Copy full SHA for 7201d43

Commits on May 3, 2024

• add \n token separately

  molbap

  committed
  c74f3a3

  Copy full SHA for c74f3a3

Commits on May 9, 2024

• Merge branch 'main' of github.com:huggingface/transformers

  ArthurZucker

  committed
  23b12a3

  Copy full SHA for 23b12a3

• Merge branch 'main' of github.com:huggingface/new-model-addition

  ArthurZucker

  committed
  4e4a957

  Copy full SHA for 4e4a957

• fix tokenizer and prompts

  molbap

  committed
  1f465c0

  Copy full SHA for 1f465c0

• fix docstrings

  molbap

  committed
  38a0401

  Copy full SHA for 38a0401

• change to camel

  molbap

  committed
  16baabc

  Copy full SHA for 16baabc

• fix casing

  molbap

  committed
  9119db4

  Copy full SHA for 9119db4

• debug pos_ids and sdpa

  molbap

  committed
  267a0da

  Copy full SHA for 267a0da

• Merge branch 'main' into add_palma

  molbap

  committed
  3e32118

  Copy full SHA for 3e32118

• pass and use cache_position

  molbap

  committed
  4ed0594

  Copy full SHA for 4ed0594

Commits on May 10, 2024

• add flag for newline tokenization

  molbap

  committed
  46c5c0b

  Copy full SHA for 46c5c0b

• Update src/transformers/models/paligemma/processing_paligemma.py

  Show description for 5b47a5b
  

  molbap

  and

  merveenoyan

  authored
  5b47a5b

  Copy full SHA for 5b47a5b

• simplify conversion script

  molbap

  committed
  b80c591

  Copy full SHA for b80c591

• add copied from

  molbap

  committed
  7d21409

  Copy full SHA for 7d21409

Commits on May 12, 2024

• add precision to conversion script

  molbap

  committed
  b1eda0e

  Copy full SHA for b1eda0e

Commits on May 13, 2024

• Update src/transformers/models/paligemma/modeling_paligemma.py

  Show description for 404abd8
  

  molbap

  and

  pcuenca

  authored
  404abd8

  Copy full SHA for 404abd8

• clean up

  molbap

  committed
  a7e2446

  Copy full SHA for a7e2446

• Merge branch 'add_palma' of github.com:huggingface/new-model-addition-palma into add_palma

  molbap

  committed
  2f6b17d

  Copy full SHA for 2f6b17d

• Shift attention mask from 1:

  Show description for 60ad9c5
  pcuenca

  committed
  60ad9c5

  Copy full SHA for 60ad9c5

• Merge pull request #6 from pcuenca/add_palma_shift_mask

  Show description for daccbe7
  molbap

  authored
  daccbe7

  Copy full SHA for daccbe7

• add docs, fix quality

  molbap

  committed
  91c7aab

  Copy full SHA for 91c7aab

• Merge branch 'add_palma' of github.com:huggingface/new-model-addition-palma into add_palma

  molbap

  committed
  372d566

  Copy full SHA for 372d566

• quality, tied weights inheritance, and logits/label alignment

  molbap

  committed
  63464ec

  Copy full SHA for 63464ec

• fix more tests

  molbap

  committed
  55187a1

  Copy full SHA for 55187a1

• pass attn_implementation to language model correctly

  molbap

  committed
  5ea6c9f

  Copy full SHA for 5ea6c9f

• add SiglipVisionTransformer to no split modules

  molbap

  committed
  ddb7ac7

  Copy full SHA for ddb7ac7

• skip paligemma test for sdpa dispatch to flash

  molbap

  committed
  7f32718

  Copy full SHA for 7f32718

• skip incompatible tests

  molbap

  committed
  ac9fd9a

  Copy full SHA for ac9fd9a

• quality

  molbap

  committed
  cceb3d0

  Copy full SHA for cceb3d0

• [broken archive maps]

  molbap

  committed
  a264824

  Copy full SHA for a264824

Commits on May 14, 2024

• Apply suggestions

  Show description for 9310873
  

  molbap

  and

  ArthurZucker

  authored
  9310873

  Copy full SHA for 9310873

• Update src/transformers/utils/dummy_pt_objects.py

  Show description for 0711b12
  

  molbap

  and

  ArthurZucker

  authored
  0711b12

  Copy full SHA for 0711b12

• simplify conversion script

  molbap

  committed
  e7ec216

  Copy full SHA for e7ec216

• add suggestions

  molbap

  committed
  8b0724d

  Copy full SHA for 8b0724d

• add suggestions

  molbap

  committed
  7bcea3e

  Copy full SHA for 7bcea3e

• add copied from

  molbap

  committed
  498bbde

  Copy full SHA for 498bbde

• fix

  molbap

  committed
  a8bd223

  Copy full SHA for a8bd223

• move labels out

  molbap

  committed
  04d962f

  Copy full SHA for 04d962f

• revert

  molbap

  committed
  e7caa8a

  Copy full SHA for e7caa8a

• fix

  molbap

  committed
  ac5ed67

  Copy full SHA for ac5ed67

• remove placeholder labels if None

  molbap

  committed
  72f6fdc

  Copy full SHA for 72f6fdc

• use cache_position

  molbap

  committed
  c824771

  Copy full SHA for c824771

• fix quality + docstrings

  molbap

  committed
  7a8e62e

  Copy full SHA for 7a8e62e

• fix quality

  molbap

  committed
  4913c07

  Copy full SHA for 4913c07

• Merge branch 'main' of github.com:huggingface/transformers into main

  molbap

  committed
  0c8f2c9

  Copy full SHA for 0c8f2c9

• Merge branch 'main' into add_palma

  molbap

  committed
  54fd284

  Copy full SHA for 54fd284

• fix paligemma 4d gemma mask incompatibility

  molbap

  committed
  99c3ac5

  Copy full SHA for 99c3ac5

• fix config docstring

  molbap

  committed
  75c36c2

  Copy full SHA for 75c36c2

• fix query and attn_mask dtype

  molbap

  committed
  9b49838

  Copy full SHA for 9b49838