Code

Add Zamba2#34517

Code

Draft

pglorio wants to merge 76 commits intohuggingface:mainhuggingface/transformers:mainfrom Zyphra:zamba2Zyphra/transformers_zamba:zamba2Copy head branch name to clipboard

+4,437

Commits

Commits on Oct 24, 2024

• First commit

  pglorio

  committed
  acd25b7

  Copy full SHA for acd25b7

Commits on Oct 28, 2024

• Finish model implementation

  pglorio

  committed
  70639b8

  Copy full SHA for 70639b8

Commits on Oct 29, 2024

• First commit

  pglorio

  committed
  d111b98

  Copy full SHA for d111b98

• Finish model implementation

  pglorio

  committed
  8f36dba

  Copy full SHA for 8f36dba

• Merge branch 'zamba2' of https://github.com/Zyphra/transformers_zamba into zamba2

  pglorio

  committed
  f0c547c

  Copy full SHA for f0c547c

Commits on Oct 30, 2024

• Register zamba2

  pglorio

  committed
  700fbf0

  Copy full SHA for 700fbf0

Commits on Nov 4, 2024

• generated modeling and configuration

  pglorio

  committed
  70a6021

  Copy full SHA for 70a6021

Commits on Nov 5, 2024

• Merge pull request #2 from Zyphra/main

  Show description for 88c4b26
  pglorio

  authored
  88c4b26

  Copy full SHA for 88c4b26

• generated modeling and configuration

  pglorio

  committed
  685906a

  Copy full SHA for 685906a

• added hybrid cache

  pglorio

  committed
  4da8d5f

  Copy full SHA for 4da8d5f

• fix attention_mask in mamba

  pglorio

  committed
  6b5a9be

  Copy full SHA for 6b5a9be

• dropped unused loras

  pglorio

  committed
  248350d

  Copy full SHA for 248350d

• fix flash2

  pglorio

  committed
  d1d2c66

  Copy full SHA for d1d2c66

• Merge pull request #3 from Zyphra/main

  Show description for eb6063e
  pglorio

  authored
  eb6063e

  Copy full SHA for eb6063e

Commits on Nov 6, 2024

• config docstrings

  

  root

  committed
  5f5d01e

  Copy full SHA for 5f5d01e

Commits on Nov 7, 2024

• fix config and fwd pass

  pglorio

  committed
  c1b7647

  Copy full SHA for c1b7647

• make fixup fixes

  pglorio

  committed
  979b99b

  Copy full SHA for 979b99b

Commits on Nov 9, 2024

• text_modeling_zamba2

  pglorio

  committed
  9d9b2eb

  Copy full SHA for 9d9b2eb

• Merge pull request #4 from Zyphra/main

  Show description for 3a457f5
  pglorio

  authored
  3a457f5

  Copy full SHA for 3a457f5

• small fixes

  pglorio

  committed
  549d4cb

  Copy full SHA for 549d4cb

Commits on Nov 11, 2024

• make fixup fixes

  pglorio

  committed
  987bba9

  Copy full SHA for 987bba9

• Merge pull request #5 from Zyphra/main

  Show description for ffc2a58
  pglorio

  authored
  ffc2a58

  Copy full SHA for ffc2a58

• Fix modular model converter

  pglorio

  committed
  9adf85e

  Copy full SHA for 9adf85e

Commits on Nov 19, 2024

• added inheritances in modular, renamed zamba cache

  pglorio

  committed
  904da4e

  Copy full SHA for 904da4e

• Merge pull request #6 from Zyphra/main

  Show description for 4725983
  pglorio

  authored
  4725983

  Copy full SHA for 4725983

• modular rebase

  pglorio

  committed
  0be27d7

  Copy full SHA for 0be27d7

• Rebase

  pglorio

  committed
  cc0c549

  Copy full SHA for cc0c549

Commits on Nov 20, 2024

• new modular conversion

  pglorio

  committed
  ac77a09

  Copy full SHA for ac77a09

• fix generated modeling file

  pglorio

  committed
  e59980e

  Copy full SHA for e59980e

• fixed import for Zamba2RMSNormGated

  pglorio

  committed
  73a647a

  Copy full SHA for 73a647a

Commits on Nov 21, 2024

• modular file cleanup

  pglorio

  committed
  c2b72a5

  Copy full SHA for c2b72a5

• rebase

  pglorio

  committed
  0eb39a5

  Copy full SHA for 0eb39a5

• make fixup and model tests

  pglorio

  committed
  10a0b1e

  Copy full SHA for 10a0b1e

Commits on Nov 23, 2024

• dropped inheritance for Zamba2PreTrainedModel

  pglorio

  committed
  0270667

  Copy full SHA for 0270667

• make fixup and unit tests

  pglorio

  committed
  189c8c5

  Copy full SHA for 189c8c5

Commits on Dec 5, 2024

• Add inheritance of rope from GemmaRotaryEmbedding

  pglorio

  committed
  fa5f79e

  Copy full SHA for fa5f79e

• moved rope to model init

  pglorio

  committed
  8079ae0

  Copy full SHA for 8079ae0

• drop del self.self_attn and del self.feed_forward

  pglorio

  committed
  d6206eb

  Copy full SHA for d6206eb

• Rebase onto upstream

  pglorio

  committed
  f832699

  Copy full SHA for f832699

• fix tests

  pglorio

  committed
  cf613b7

  Copy full SHA for cf613b7

Commits on Dec 7, 2024

• renamed lora -> adapter

  pglorio

  committed
  337faed

  Copy full SHA for 337faed

• rewrote adapter implementation

  pglorio

  committed
  f1b31a1

  Copy full SHA for f1b31a1

• rebase

  pglorio

  committed
  8925c15

  Copy full SHA for 8925c15

• fixed tests

  pglorio

  committed
  11fdd47

  Copy full SHA for 11fdd47

Commits on Dec 18, 2024

• Merge branch 'main' into zamba2

  pglorio

  committed
  02dd042

  Copy full SHA for 02dd042

Commits on Dec 19, 2024

• Fix torch_forward in mamba2 layer

  pglorio

  committed
  5d0a5d4

  Copy full SHA for 5d0a5d4

• Fix torch_forward in mamba2 layer

  pglorio

  committed
  ef055c9

  Copy full SHA for ef055c9

• Fix torch_forward in mamba2 layer

  pglorio

  committed
  b993a78

  Copy full SHA for b993a78

• Dropped adapter in-place sum

  pglorio

  committed
  bf93251

  Copy full SHA for bf93251

• removed rope from attention init

  pglorio

  committed
  99708af

  Copy full SHA for 99708af

• updated rope

  pglorio

  committed
  d9b4a50

  Copy full SHA for d9b4a50

• created get_layers method

  pglorio

  committed
  095d853

  Copy full SHA for 095d853

Commits on Dec 20, 2024

• rebase

  pglorio

  committed
  10ebad5

  Copy full SHA for 10ebad5

• make fixup fix

  pglorio

  committed
  99e343e

  Copy full SHA for 99e343e

• make fixup fixes

  pglorio

  committed
  4e40975

  Copy full SHA for 4e40975

• make fixup fixes

  pglorio

  committed
  61bb32f

  Copy full SHA for 61bb32f

Commits on Jan 7, 2025

• fix merge conflicts

  pglorio

  committed
  bb9b24b

  Copy full SHA for bb9b24b

Commits on Jan 13, 2025

• update to new attention standard

  pglorio

  committed
  cb90bb4

  Copy full SHA for cb90bb4

• fixes for merge

  pglorio

  committed
  8ed701e

  Copy full SHA for 8ed701e

• update to new attention standard

  pglorio

  committed
  1dbc8c7

  Copy full SHA for 1dbc8c7

• make fixup fixes

  pglorio

  committed
  f24e452

  Copy full SHA for f24e452

Commits on Jan 16, 2025

• rebase

  pglorio

  committed
  676f862

  Copy full SHA for 676f862

• minor fixes

  pglorio

  committed
  2b29338

  Copy full SHA for 2b29338

• cache_position

  pglorio

  committed
  b212cb2

  Copy full SHA for b212cb2

• removed cache_position postion_ids use_cache

  pglorio

  committed
  1e3b51e

  Copy full SHA for 1e3b51e

• remove config from modular

  pglorio

  committed
  5ace701

  Copy full SHA for 5ace701

• removed config from modular (2)

  pglorio

  committed
  535b631

  Copy full SHA for 535b631

• rebase

  pglorio

  committed
  5a16aa9

  Copy full SHA for 5a16aa9

• import apply_rotary_pos_emb from llama

  pglorio

  committed
  1c92266

  Copy full SHA for 1c92266

• fixed rope_kwargs

  pglorio

  committed
  99bde93

  Copy full SHA for 99bde93

• Instantiate cache in Zamba2Model

  pglorio

  committed
  baf2ed3

  Copy full SHA for baf2ed3

Commits on Jan 17, 2025

• fix cache

  pglorio

  committed
  9afb57e

  Copy full SHA for 9afb57e

• fix @slow decorator

  pglorio

  committed
  d1687f9

  Copy full SHA for d1687f9

Commits on Jan 20, 2025

• rebase

  pglorio

  committed
  4299889

  Copy full SHA for 4299889

Commits on Jan 21, 2025

• rebase

  pglorio

  committed
  a0545bf

  Copy full SHA for a0545bf

• small fix in modular file

  pglorio

  committed
  903f6dc

  Copy full SHA for 903f6dc