Code

Finetune LORA#2632

Code

Merged

ggerganov merged 247 commits intoggerganov:masterggerganov/llama.cpp:masterfrom xaedes:finetune-loraxaedes/llama.cpp:finetune-loraCopy head branch name to clipboardSep 28, 2023

+6,941-2,353

Commits

Commits on Jul 28, 2023

• fix track_max_mem in forward_batch_wo_cache_flash_attn_train

  xaedes

  committed
  5d124d0

  Copy full SHA for 5d124d0

• remove unnecessary Adam(W) optimizer tensors.

  Show description for d39c8e6
  xaedes

  committed
  d39c8e6

  Copy full SHA for d39c8e6

• add gradient clipping to AdamW

  xaedes

  committed
  d395b19

  Copy full SHA for d395b19

• Fix reset of unused g->nodes and g->grads to NULL

  xaedes

  committed
  d7003a9

  Copy full SHA for d7003a9

• implement gradient checkpointing for training

  Show description for 6e3f95b
  xaedes

  committed
  6e3f95b

  Copy full SHA for 6e3f95b

• remove unused compute buffer 3

  xaedes

  committed
  e05e441

  Copy full SHA for e05e441

• add and use function ggml_build_backward_expand to avoid stack overflows with large maximum number of nodes

  Show description for ed4319e
  xaedes

  committed
  ed4319e

  Copy full SHA for ed4319e

• change AdamW decay parameter to work like the torch AdamW decay parameter

  Show description for a80f184
  xaedes

  committed
  a80f184

  Copy full SHA for a80f184

• change default AdamW weight decay parameter used in training to 0.1 as used in nanoGPT

  xaedes

  committed
  f175ead

  Copy full SHA for f175ead

• change default AdamW weight decay parameter defined in ggml to 0.0, making Adam default instead of AdamW

  Show description for 97964a4
  xaedes

  committed
  97964a4

  Copy full SHA for 97964a4

• bug fixes for cross entropy loss

  Show description for 2c6985f
  xaedes

  committed
  2c6985f

  Copy full SHA for 2c6985f

• fix test-grad0 for cross_entropy_loss

  Show description for 2d1e6e0
  xaedes

  committed
  2d1e6e0

  Copy full SHA for 2d1e6e0

• fix test-grad0 for soft_max

  Show description for 864e7e3
  xaedes

  committed
  864e7e3

  Copy full SHA for 864e7e3

• improve finite differences of test-grad0 by using double instead of float

  xaedes

  committed
  87febee

  Copy full SHA for 87febee

• change cross_entropy_loss to output average over all rows

  Show description for 51dc770
  xaedes

  committed
  51dc770

  Copy full SHA for 51dc770

• improve gradient checkpointing

  Show description for 3744a9b
  xaedes

  committed
  3744a9b

  Copy full SHA for 3744a9b

• disable gradient checkpointing debug output

  xaedes

  committed
  fc379a2

  Copy full SHA for fc379a2

• llama : fix rope usage in train-text-from-scratch after ChatGLM change

  xaedes

  committed
  d0fbb7d

  Copy full SHA for d0fbb7d

• add more training parameters:

  Show description for c6a18e1
  xaedes

  committed
  c6a18e1

  Copy full SHA for c6a18e1

• replace memcpy with reshape operation so that the graph is not cut at the input

  Show description for ce937bc
  xaedes

  committed
  ce937bc

  Copy full SHA for ce937bc

• remove unused function argument from get_example_targets_batch

  xaedes

  committed
  ff759d9

  Copy full SHA for ff759d9

• measure and print total training time

  xaedes

  committed
  e843d6e

  Copy full SHA for e843d6e

• add optimization callback to ggml_opt_resume_g

  Show description for bfc3119
  xaedes

  committed
  bfc3119

  Copy full SHA for bfc3119

• use optimization callback in training

  Show description for d7aa4d9
  xaedes

  committed
  d7aa4d9

  Copy full SHA for d7aa4d9

• add minimum number of tensor dimensions to apply weight decay (default 2)

  Show description for e6ff072
  xaedes

  committed
  e6ff072

  Copy full SHA for e6ff072

• rename training parameter cos-decay-alpha to cos-decay-min and clarify that adam-min-alpha also applies to warmup

  xaedes

  committed
  58024d3

  Copy full SHA for 58024d3

• fix increase of model.train_samples and model.train_tokens

  Show description for 17a0898
  xaedes

  committed
  17a0898

  Copy full SHA for 17a0898

• change sampling parameters for prediction after training to defaults of common.h

  Show description for 24a4b09
  xaedes

  committed
  24a4b09

  Copy full SHA for 24a4b09

• tighten abs error bounds for cross_entropy_loss in test-grad0

  xaedes

  committed
  1065c3b

  Copy full SHA for 1065c3b

• add conditional compilation of using F16 exp in flash attention

  Show description for dbbc263
  xaedes

  committed
  dbbc263

  Copy full SHA for dbbc263

• tighten abs error bounds for flash_attn in test-grad0

  xaedes

  committed
  47055c9

  Copy full SHA for 47055c9

• tighten abs error bounds for sqrt in test-grad0

  xaedes

  committed
  0f6a8ab

  Copy full SHA for 0f6a8ab

• remove out-commented vectorized code of opt_adam

  Show description for 87035b9
  xaedes

  committed
  87035b9

  Copy full SHA for 87035b9

• ggml : update ggml_rms_norm_back with configurable eps

  xaedes

  committed
  ecdc161

  Copy full SHA for ecdc161

• llama training : fix ggml_rms_norm_back calls to pass configurable eps

  xaedes

  committed
  c1a5e11

  Copy full SHA for c1a5e11

• remove trailing whitespace

  xaedes

  committed
  22cb368

  Copy full SHA for 22cb368

Commits on Aug 6, 2023

• Merge branch 'master' into pr-train-mem-usage-improvements

  xaedes

  committed
  d43af4b

  Copy full SHA for d43af4b

• add train function using automatic gradient checkpointing backward pass and allocator

  xaedes

  committed
  2bf422e

  Copy full SHA for 2bf422e

Commits on Aug 14, 2023

• in train function replace add_inplace by regular add

  Show description for fc826c8
  xaedes

  committed
  fc826c8

  Copy full SHA for fc826c8

• don't use allocate hash_map on context

  Show description for d437415
  xaedes

  committed
  d437415

  Copy full SHA for d437415

• correctly clone reshape and permute operations by also cloning tensor->nb values

  xaedes

  committed
  cfddc36

  Copy full SHA for cfddc36

• fix variable name and add missing type cast

  xaedes

  committed
  0dd496c

  Copy full SHA for 0dd496c

• terminate recursive tensor cloning when reaching tensor without src tensors

  xaedes

  committed
  52c92c0

  Copy full SHA for 52c92c0

• correctly clone view tensors by setting data pointers

  Show description for 345f516
  xaedes

  committed
  345f516

  Copy full SHA for 345f516

• fix variable names

  xaedes

  committed
  5a11b75

  Copy full SHA for 5a11b75

• swap arguments to commutative ops to be the same as in forward_batch_wo_cache_flash_attn

  xaedes

  committed
  b2f1310

  Copy full SHA for b2f1310

• add input tensors as checkpoints

  Show description for 5884b43
  xaedes

  committed
  5884b43

  Copy full SHA for 5884b43

• fix variable name and add missing boolean negation

  xaedes

  committed
  9716eb8

  Copy full SHA for 9716eb8

• make sure some tensors are not reallocated by inserting new temporary nodes depending on them:

  Show description for 38f4438
  xaedes

  committed
  38f4438

  Copy full SHA for 38f4438

• fix ASSERT to work with zero layers

  xaedes

  committed
  d6c5b03

  Copy full SHA for d6c5b03

• add training options whether to use allocator and/or unified training function

  xaedes

  committed
  4ed096c

  Copy full SHA for 4ed096c

• integrate unified training function which may use memory allocator

  Show description for 865c4cd
  xaedes

  committed
  865c4cd

  Copy full SHA for 865c4cd

• format name of cloned tensors with " (clone)" suffix

  xaedes

  committed
  3e99a8d

  Copy full SHA for 3e99a8d

• set names for tensors in unified train function for easier debugging

  xaedes

  committed
  75baed2

  Copy full SHA for 75baed2

• allocate graph on context using ggml_new_graph

  xaedes

  committed
  fe788a1

  Copy full SHA for fe788a1

• remove handwritten training functions

  xaedes

  committed
  c954f41

  Copy full SHA for c954f41

• remove unused training parameters "use_scratch" and "use_unified"

  xaedes

  committed
  271e4d6

  Copy full SHA for 271e4d6

• remove trailing whitespace

  xaedes

  committed
  6f161c7

  Copy full SHA for 6f161c7

• remove unused train params: mem_compute1_gb & mem_compute2_gb

  Show description for 3794dce
  xaedes

  committed
  3794dce

  Copy full SHA for 3794dce

• remove unused forward_batch function

  xaedes

  committed
  6e280b2

  Copy full SHA for 6e280b2

• add debug asserts in ggml_allocr_alloc to some common pitfalls when using this function directly

  xaedes

  committed
  faf3e21

  Copy full SHA for faf3e21

• only use ggml_allocr_alloc when tensor has NULL data and is no view

  xaedes

  committed
  098654c

  Copy full SHA for 098654c

• fix test when to create temporary backward graph

  Show description for 3e6468b
  xaedes

  committed
  3e6468b

  Copy full SHA for 3e6468b

• fix memory "leak" in optimizers

  Show description for 5622846
  xaedes

  committed
  5622846

  Copy full SHA for 5622846

• reverse order of for loop in ggml_build_backward_expand to save memory when using gradient checkpointing and allocator

  Show description for 3b5515b
  xaedes

  committed
  3b5515b

  Copy full SHA for 3b5515b

Commits on Aug 15, 2023

• add API functions to access llama model tensors

  xaedes

  committed
  316b070

  Copy full SHA for 316b070

• add stub example for finetuning, based on train-text-from-scratch

  xaedes

  committed
  5e059ac

  Copy full SHA for 5e059ac

• move and remove code

  xaedes

  committed
  9eb1ef8

  Copy full SHA for 9eb1ef8

Commits on Aug 16, 2023

• add API functions to access remaining model parameters:

  Show description for c0a372f
  xaedes

  committed
  c0a372f

  Copy full SHA for c0a372f

• first draft for LORA finetune training

  xaedes

  committed
  28ee0c8

  Copy full SHA for 28ee0c8

• remove const model and layer arguments in API functions for accessing model tensors

  xaedes

  committed
  50b1e66

  Copy full SHA for 50b1e66

• bug fixes to make finetune compile

  Show description for be7e564
  xaedes

  committed
  be7e564

  Copy full SHA for be7e564

• add debug prints for training memory improvements

  xaedes

  committed
  6202753

  Copy full SHA for 6202753

• fix names of lora tensors

  xaedes

  committed
  0ab2507

  Copy full SHA for 0ab2507

• avoid stack overflow resulting from big ggml_cgraph

  Show description for 39a2d15
  xaedes

  committed
  39a2d15

  Copy full SHA for 39a2d15

• replace llama API functions to get model tensors by one function to get model tensor by name

  Show description for 1151653
  xaedes

  committed
  1151653

  Copy full SHA for 1151653

• remove unused call to not existing llama_get_layer_from_model

  xaedes

  committed
  79ad888

  Copy full SHA for 79ad888

• implement ggml_compute_forward_out_prod_q_f32

  xaedes

  committed
  83cb9ed

  Copy full SHA for 83cb9ed

• remove trailing whitespace

  xaedes

  committed
  83a4ad7

  Copy full SHA for 83a4ad7

• add lora finetune support on quantized base model tensors

  xaedes

  committed
  f80e245

  Copy full SHA for f80e245

• add ggml_add_cast API function

  Show description for 9198b24
  xaedes

  committed
  9198b24

  Copy full SHA for 9198b24

• use ggml_add_cast in finetuning

  Show description for 714fec0
  xaedes

  committed
  714fec0

  Copy full SHA for 714fec0

Commits on Aug 17, 2023

• bug fix: actually use result type passed to ggml_add_cast

  xaedes

  committed
  0bb897c

  Copy full SHA for 0bb897c

Commits on Aug 18, 2023

• make sure base model tensors data cannot be used in viewable operations

  Show description for 44526cb
  xaedes

  committed
  44526cb

  Copy full SHA for 44526cb

• fix bug in ggml_out_prod which resulted in wrong n_dims of result tensors

  xaedes

  committed
  a252111

  Copy full SHA for a252111

• avoid keeping in memory ALL of the gradients

  Show description for f358204
  xaedes

  committed
  f358204

  Copy full SHA for f358204

• remove trailing whitespace

  xaedes

  committed
  011f47f

  Copy full SHA for 011f47f

• remove debug prints and function to compute tensor data hash

  xaedes

  committed
  a0c2752

  Copy full SHA for a0c2752

• improve optimization iteration prints

  xaedes

  committed
  113c90f

  Copy full SHA for 113c90f

• adjust maximal values to support finetuning 3B models

  xaedes

  committed
  7a63d42

  Copy full SHA for 7a63d42

• change default finetune params lora_r and lora_alpha to match the n_rank parameters of 4

  xaedes

  committed
  63cb374

  Copy full SHA for 63cb374

• bug fix: make sure finetune input gradient is allocated at begin and kept until end

  xaedes

  committed
  6c98640

  Copy full SHA for 6c98640

• remove unnecessary src tensor from ggml_get_rows_back

  Show description for 65b0561
  xaedes

  committed
  65b0561

  Copy full SHA for 65b0561

• remove unnecessary src tensor from ggml_repeat & ggml_repeat_back

  Show description for 3e47890
  xaedes

  committed
  3e47890

  Copy full SHA for 3e47890

• resolve todo

  Show description for 37dfb54
  xaedes

  committed
  37dfb54

  Copy full SHA for 37dfb54

Commits on Aug 20, 2023

• mixing multiple LORA adapters is now possible

  Show description for d61ed6b
  xaedes

  committed
  d61ed6b

  Copy full SHA for d61ed6b

• add option to save finetune output every N iterations

  xaedes

  committed
  27c24ff

  Copy full SHA for 27c24ff

Commits on Aug 21, 2023

• also save latest finetune output with ITERATION="LATEST" and print where files are saved

  Show description for 8b4106a
  xaedes

  committed
  8b4106a

  Copy full SHA for 8b4106a

Commits on Aug 23, 2023

• update checkpoint train stats before saving via "--save-every"

  xaedes

  committed
  77a3092

  Copy full SHA for 77a3092

• add command line option --rank-wo N for rank of wo tensor

  xaedes

  committed
  1a5f0a3

  Copy full SHA for 1a5f0a3

• update finetune README

  xaedes

  committed
  7df517c

  Copy full SHA for 7df517c

Commits on Aug 28, 2023

• Merge branch 'master' into finetune-lora

  Show description for b04263c
  xaedes

  committed
  b04263c

  Copy full SHA for b04263c

• fix dump_non_result_info_yaml to output multiple lora adapters

  xaedes

  committed
  aecc3b3

  Copy full SHA for aecc3b3

• bug fix: replace GGML_TYPE_SIZE[t] by ggml_type_size(t)

  xaedes

  committed
  aa8016e

  Copy full SHA for aa8016e

• replace llama_n_mult by llama_n_ff

  xaedes

  committed
  daedc6f

  Copy full SHA for daedc6f

• finetune bug fixes to compile with merged in code from master

  xaedes

  committed
  5ce92ae

  Copy full SHA for 5ce92ae

• remove prediction related code to reduce duplicated code with main

  Show description for 271c030
  xaedes

  committed
  271c030

  Copy full SHA for 271c030

• reduce large memory overhead in train-text-from-scratch

  Show description for 9a28bce
  xaedes

  committed
  9a28bce

  Copy full SHA for 9a28bce

• add comment explaining why finetune checkpoints are allocated in one block

  xaedes

  committed
  49af7fb

  Copy full SHA for 49af7fb

• make default value of float member a float literal

  xaedes

  committed
  007280c

  Copy full SHA for 007280c

• handle rms_norm and rope parameters the same as in train-text-from-scratch

  xaedes

  committed
  1faee64

  Copy full SHA for 1faee64

• remove unused code

  xaedes

  committed
  a3b4529

  Copy full SHA for a3b4529

• remove vocab related code as it is unnecessary

  xaedes

  committed
  ca97583

  Copy full SHA for ca97583

• add LLM_KV_TRAINING_TYPE to train-text-from-scratch checkpoints

  Show description for e030f7b
  xaedes

  committed
  e030f7b

  Copy full SHA for e030f7b

• add gguf constants and load/save functions from train-text-from-scratch

  xaedes

  committed
  ecb1b20

  Copy full SHA for ecb1b20

Commits on Aug 29, 2023

• add load & save lora finetune checkpoints via gguf

  xaedes

  committed
  0564f4e

  Copy full SHA for 0564f4e

• add python script to convert old finetune checkpoint files to gguf

  xaedes

  committed
  6134ad4

  Copy full SHA for 6134ad4

• remove old checkpoint save & load code

  xaedes

  committed
  1425968

  Copy full SHA for 1425968

• remove code to print data checksums which was used to verify correctness of new gguf code

  xaedes

  committed
  ebff3a1

  Copy full SHA for ebff3a1

• omit tokenization when training is disabled, only save llama lora adapter

  Show description for 5813ac8
  xaedes

  committed
  5813ac8

  Copy full SHA for 5813ac8

• remove trailing whitespace

  xaedes

  committed
  a6165da

  Copy full SHA for a6165da

• update README.md

  xaedes

  committed
  e28cf7e

  Copy full SHA for e28cf7e

• implement ggml_compute_forward_repeat_f16

  xaedes

  committed
  794bb7e

  Copy full SHA for 794bb7e

• avoid stack overflow of large cgraphs in test-grad0

  xaedes

  committed
  5f0a4e9

  Copy full SHA for 5f0a4e9

• add ggml API functions ggml_unravel_index, ggml_get_i32_nd and its analogs for set and for f32

  Show description for 82c5247
  xaedes

  committed
  82c5247

  Copy full SHA for 82c5247

• increase test-grad0 context mem size to accommodate for bigger cgraph

  xaedes

  committed
  5fcfa7e

  Copy full SHA for 5fcfa7e

• add sanity check to ggml_compute_backward, asserting the correct shape of gradients

  xaedes

  committed
  b1aa26f

  Copy full SHA for b1aa26f

• fix ggml_acc_or_set to return tensor of correct shape

  xaedes

  committed
  a76e66a

  Copy full SHA for a76e66a

• remove unused 'inplace' argument from ggml_compute_backward function

  Show description for dd4e4bc
  xaedes

  committed
  dd4e4bc

  Copy full SHA for dd4e4bc

• add missing argument 'int i0' to ggml_get_i32_nd & ggml_set_i32_nd header declarations

  xaedes

  committed
  8a96d4c

  Copy full SHA for 8a96d4c

• Merge branch 'master' into finetune-lora

  xaedes

  committed
  281245a

  Copy full SHA for 281245a

• fix error message in ggml_allocr_alloc to display actual max_avail

  xaedes

  committed
  5854f51

  Copy full SHA for 5854f51

• fix check_gradient

  Show description for bf70e27
  xaedes

  committed
  bf70e27

  Copy full SHA for bf70e27

Commits on Aug 30, 2023

• Merge branch 'master' into finetune-lora

  xaedes

  committed
  b1709f2

  Copy full SHA for b1709f2

• use tensor->view_src instead of ggml_is_view and get_view_source

  xaedes

  committed
  2392b67

  Copy full SHA for 2392b67

• move gradient checkpointing code into ggml, new API function:

  Show description for d487e05
  xaedes

  committed
  d487e05

  Copy full SHA for d487e05

• replace custom data getters and setters by ggml functions

  xaedes

  committed
  e6b7158

  Copy full SHA for e6b7158

• train-text-from-scratch can train (full finetune) gguf models

  Show description for fc456ed
  xaedes

  committed
  fc456ed

  Copy full SHA for fc456ed

• remove trailing whitespace

  xaedes

  committed
  f3590ad

  Copy full SHA for f3590ad

• add option to save train-text-from-scratch output every N iterations

  xaedes

  committed
  b26bd4c

  Copy full SHA for b26bd4c

• update README.md

  xaedes

  committed
  4e986ac

  Copy full SHA for 4e986ac

• fix warnings

  xaedes

  committed
  0c57f9f

  Copy full SHA for 0c57f9f

• fix warnings

  xaedes

  committed
  4fd51c4

  Copy full SHA for 4fd51c4

Commits on Aug 31, 2023

• remove finetune option to disable allocator

  Show description for e0da168
  xaedes

  committed
  e0da168

  Copy full SHA for e0da168

• add tensor checkpoints only when gradient checkpointing is enabled

  xaedes

  committed
  4914f85

  Copy full SHA for 4914f85

Commits on Sep 1, 2023

• initialize opt ggml context if none was provided

  xaedes

  committed
  d554a70

  Copy full SHA for d554a70

• add ggml-alloc API function 'ggml_allocr_max_size' to get max size of alloc

  Show description for 7e01d11
  xaedes

  committed
  7e01d11

  Copy full SHA for 7e01d11

• finetune: automatically allocate all memory and changes to command line options

  Show description for 5bba329
  xaedes

  committed
  5bba329

  Copy full SHA for 5bba329

• add finetune to Makefile

  xaedes

  committed
  6cbf55a

  Copy full SHA for 6cbf55a

• update README.md

  xaedes

  committed
  7acb124

  Copy full SHA for 7acb124

• Merge branch 'master' into finetune-lora

  Show description for 6809eb7
  xaedes

  committed
  6809eb7

  Copy full SHA for 6809eb7

• print time per iteration and estimate remaining time

  xaedes

  committed
  c32ad44

  Copy full SHA for c32ad44

Commits on Sep 2, 2023

• increase measured alloc size by tensor_alignment

  Show description for 6ee12b1
  xaedes

  committed
  6ee12b1

  Copy full SHA for 6ee12b1

• fix README.md

  xaedes

  committed
  cfe217f

  Copy full SHA for cfe217f

• add some more allocator debug prints

  xaedes

  committed
  ded6382

  Copy full SHA for ded6382

• bug fix, probably solves the 'ggml_allocr_alloc: not enough space in the buffer' issue

  xaedes

  committed
  8d982c8

  Copy full SHA for 8d982c8

• revert last commit

  Show description for 1ce7023
  xaedes

  committed
  1ce7023

  Copy full SHA for 1ce7023

• remove unnecessary "0x" before "%p" output

  xaedes

  committed
  2d2bdc0

  Copy full SHA for 2d2bdc0

• move measurement memory segment to upper region of the address space

  xaedes

  committed
  80ac697

  Copy full SHA for 80ac697

Commits on Sep 3, 2023

• update README.md

  xaedes

  committed
  406e075

  Copy full SHA for 406e075

• fix printf format warnings

  xaedes

  committed
  e07f5c5

  Copy full SHA for e07f5c5

• add missing gguf_free in load_checkpoint_lora_file

  xaedes

  committed
  bdb7092

  Copy full SHA for bdb7092

• load default rms_norm and rope parameters from base model

  xaedes

  committed
  50589ed

  Copy full SHA for 50589ed

Commits on Sep 4, 2023

• Merge branch 'master' into finetune-lora

  Show description for 9ea2f7f
  xaedes

  committed
  9ea2f7f

  Copy full SHA for 9ea2f7f

• Merge branch 'master' into finetune-lora

  Show description for d3afd71
  xaedes

  committed
  d3afd71

  Copy full SHA for d3afd71

• add gradient accumulation

  Show description for c1c3b0e
  xaedes

  committed
  c1c3b0e

  Copy full SHA for c1c3b0e

Commits on Sep 5, 2023

• fix tracking of train_samples and train_tokens

  xaedes

  committed
  d07b6aa

  Copy full SHA for d07b6aa

• build : fix compile warnings

  ggerganov

  committed
  786e786

  Copy full SHA for 786e786

• ggml : fix L-BFGS linesearch loop

  ggerganov

  committed
  d375b8f

  Copy full SHA for d375b8f

• Merge branch 'master' into finetune-lora

  xaedes

  authored
  867e7c2

  Copy full SHA for 867e7c2

Commits on Sep 6, 2023

• improve finetune time measurement

  Show description for 8c2d7e3
  xaedes

  committed
  8c2d7e3

  Copy full SHA for 8c2d7e3

• specify default lora rank with '--lora-r N'

  Show description for c08fcf5
  xaedes

  committed
  c08fcf5

  Copy full SHA for c08fcf5

• Merge branch 'master' into finetune-lora

  Show description for 0393116
  xaedes

  committed
  0393116

  Copy full SHA for 0393116

• fix gradient accumulation bug where the same batch was used for each microstep

  xaedes

  committed
  de6170d

  Copy full SHA for de6170d

• fix gradient accumulation bug where the same batch was used for each microstep

  xaedes

  committed
  0c2c9c7

  Copy full SHA for 0c2c9c7

Commits on Sep 9, 2023

• support grouped-query-attention in ggml_flash_attn and ggml_flash_attn_back

  Show description for d7aade7
  xaedes

  committed
  d7aade7

  Copy full SHA for d7aade7

• add llama API functions to get grouped-query-attention n_head parameter 'n_head_kv'.

  xaedes

  committed
  833a56c

  Copy full SHA for 833a56c

• fix finetune to support grouped-query-attention (using flash-attention)

  Show description for 35260f7
  xaedes

  committed
  35260f7

  Copy full SHA for 35260f7

• support broadcastable a in out_prod(a, b) and backward pass of broadcasting mul_mat(a, b)

  xaedes

  committed
  aea8b6b

  Copy full SHA for aea8b6b

• test broadcasting mul_mat backward pass

  xaedes

  committed
  dd32786

  Copy full SHA for dd32786

• decouple random number generator of each operation test

  Show description for 9738526
  xaedes

  committed
  9738526

  Copy full SHA for 9738526

• add comment briefly describing what ggml_repeat_back does

  xaedes

  committed
  d3aaf08

  Copy full SHA for d3aaf08

• simplify broadcasting mul_mat backward using ggml_repeat_back

  xaedes

  committed
  d3f1b43

  Copy full SHA for d3f1b43

• add cgraph evaluation order member and corresponding enum type

  Show description for 917d287
  xaedes

  committed
  917d287

  Copy full SHA for 917d287

• measure max compute size for each cgraph eval order and use best order

  Show description for ace9088
  xaedes

  committed
  ace9088

  Copy full SHA for ace9088

• Merge branch 'master' into finetune-lora

  Show description for 54b21a3
  xaedes

  committed
  54b21a3

  Copy full SHA for 54b21a3

• remove unused command line options

  xaedes

  committed
  1cef459

  Copy full SHA for 1cef459

Commits on Sep 13, 2023

• add sample start patterns and options to force new or by default resume last shuffling

  xaedes

  committed
  0e32932

  Copy full SHA for 0e32932

• update shuffle rng state on reshuffle

  xaedes

  committed
  7898652

  Copy full SHA for 7898652

• exclude known zero values from computations in flash_attn_f32 & flash_attn_back_f32

  xaedes

  committed
  ec57689

  Copy full SHA for ec57689

• remove probably unnecessary exception type flags from stringstream

  xaedes

  committed
  7f378a7

  Copy full SHA for 7f378a7

Commits on Sep 14, 2023

• pass correct max number of tokens to llama_tokenize

  xaedes

  committed
  f627e2f

  Copy full SHA for f627e2f

• account for possible leading whitespace that will be added by tokenizer

  Show description for 2c59f7b
  xaedes

  committed
  2c59f7b

  Copy full SHA for 2c59f7b

• use unrolled vec_mad in out_prod

  Show description for 20cf1a4
  xaedes

  committed
  20cf1a4

  Copy full SHA for 20cf1a4

• set lora_alpha to value of lora_r if it is not set via command line

  Show description for 3a9c1d7
  xaedes

  committed
  3a9c1d7

  Copy full SHA for 3a9c1d7

• reshuffle original sample order instead of the previous shuffled order

  Show description for 0971fee
  xaedes

  committed
  0971fee

  Copy full SHA for 0971fee

• block tiling for out-prod inspired by mul-mat

  Show description for d88dae2
  xaedes

  committed
  d88dae2

  Copy full SHA for d88dae2

• exclude some more known zero values from computations in flash_attn_f32 & flash_attn_back_f32

  xaedes

  committed
  76804fa

  Copy full SHA for 76804fa

Commits on Sep 15, 2023

• add static keywords

  xaedes

  committed
  4f2ce91

  Copy full SHA for 4f2ce91

• remove outcommented old code

  xaedes

  committed
  cc60b3f

  Copy full SHA for cc60b3f

• update train-text-from-scratch with tokenization, sample selection and shuffling from finetune

  xaedes

  committed
  ab56b63

  Copy full SHA for ab56b63

Commits on Sep 16, 2023

• remove lbfgs related train parameters

  xaedes

  committed
  00b656f

  Copy full SHA for 00b656f

• move common train functions into common/train.[h|cpp]

  xaedes

  committed
  9f4b1bf

  Copy full SHA for 9f4b1bf

• move train state into struct train_state

  xaedes

  committed
  a8c8907

  Copy full SHA for a8c8907

• move train data saving code into callback to unify code of opt_callback

  Show description for ee27333
  xaedes

  committed
  ee27333

  Copy full SHA for ee27333

• move common train params into common/train

  xaedes

  committed
  e9758ae

  Copy full SHA for e9758ae

• move common opt_callback into common/train

  xaedes

  committed
  bef1e97

  Copy full SHA for bef1e97

• fix consume_common_train_arg

  xaedes

  committed
  7aa9ea7

  Copy full SHA for 7aa9ea7

• save and load head_count_kv in lora checkpoints

  xaedes

  committed
  48d3509

  Copy full SHA for 48d3509

• increase train_samples by used_samples instead of number of batches

  Show description for 571dc94
  xaedes

  committed
  571dc94

  Copy full SHA for 571dc94

• Merge branch 'master' into finetune-lora

  Show description for d3e06d3
  xaedes

  committed
  d3e06d3

  Copy full SHA for d3e06d3

• fix usage of llama_tokenize

  xaedes

  committed
  7930caf

  Copy full SHA for 7930caf

• remove static from process_escape since we need it exposed in header

  xaedes

  committed
  8d82d4c

  Copy full SHA for 8d82d4c

• fix code formating of long function declarations

  xaedes

  committed
  9139fec

  Copy full SHA for 9139fec

• fix condition in load_train_state_gguf

  xaedes

  committed
  1d33ec5

  Copy full SHA for 1d33ec5

• use die("msg") instead of replace GGML_ASSERT(!"msg") or throw std::runtime_error("msg")

  xaedes

  committed
  1d09965

  Copy full SHA for 1d09965

• fix saving and loading of training type

  xaedes

  committed
  9db2664

  Copy full SHA for 9db2664

• remove terminating '\0' from tokenization

  Show description for dd3e763
  xaedes

  committed
  dd3e763

  Copy full SHA for dd3e763

• fix compile warnings

  xaedes

  committed
  83061fb

  Copy full SHA for 83061fb

• fix compile warnings

  xaedes

  committed
  8721785

  Copy full SHA for 8721785

Commits on Sep 17, 2023

• use new/delete for train_state instead of malloc/free

  Show description for ddf5ac2
  xaedes

  committed
  ddf5ac2

  Copy full SHA for ddf5ac2

• assert that sample_count > 0, avoiding division by zero

  xaedes

  committed
  151bfe9

  Copy full SHA for 151bfe9

• fix frand to return value in interval [0,1)

  xaedes

  committed
  bf2ad65

  Copy full SHA for bf2ad65

• add train option "--sample-random-offsets"

  Show description for d1bb6fb
  xaedes

  committed
  d1bb6fb

  Copy full SHA for d1bb6fb

• deduplicate code into function

  xaedes

  committed
  56a03fa

  Copy full SHA for 56a03fa

• remove n_rot hparam, as it must always be hparam.n_embd_head()

  xaedes

  committed
  1dbd6bc

  Copy full SHA for 1dbd6bc

• align code

  xaedes

  committed
  5ed3098

  Copy full SHA for 5ed3098

• assert correct base model tensor shapes

  xaedes

  committed
  b0ee563

  Copy full SHA for b0ee563

• move some params from lora hparams into model hparams and load model params from gguf

  Show description for 934ad8d
  xaedes

  committed
  934ad8d

  Copy full SHA for 934ad8d

• remove now unnecessary llama API functions to get model params that where added by this PR

  xaedes

  committed
  dd94ce4

  Copy full SHA for dd94ce4

• train-text-from-scratch: automatically allocate model tensors, remove option '--mem-model N'

  xaedes

  committed
  9e10fa9

  Copy full SHA for 9e10fa9

• train-text-from-scratch: automatically allocate opt context

  xaedes

  committed
  db38d2b

  Copy full SHA for db38d2b

• train-text-from-scratch: automatically allocate input tensors

  xaedes

  committed
  f9b5d9b

  Copy full SHA for f9b5d9b

• train-text-from-scratch: automatically allocate compute memory

  xaedes

  committed
  c993246

  Copy full SHA for c993246

• remove unused options and equalize train-text-from-scratch with finetune

  xaedes

  committed
  3b9d974

  Copy full SHA for 3b9d974

• initialize opt->loss_after with zero

  xaedes

  committed
  5ce74ee

  Copy full SHA for 5ce74ee

Commits on Sep 22, 2023

• add export-lora program

  xaedes

  committed
  0ede0f4

  Copy full SHA for 0ede0f4

• remove trailing whitespace

  xaedes

  committed
  b91e3dd

  Copy full SHA for b91e3dd

• add export-lora build in Makefile

  xaedes

  committed
  d38260b

  Copy full SHA for d38260b

• remove unused struct tensor_info from export-lora

  xaedes

  committed
  904c19b

  Copy full SHA for 904c19b

• add export-lora build dependency to llama

  Show description for 758c46c
  xaedes

  committed
  758c46c

  Copy full SHA for 758c46c

• update finetune README.md

  xaedes

  committed
  9145c87

  Copy full SHA for 9145c87

• cancel optimization when specified number of epochs is completed

  xaedes

  committed
  da05205

  Copy full SHA for da05205

Commits on Sep 24, 2023

• improve handling of export-lora arguments

  Show description for 2912f17
  xaedes

  committed
  2912f17

  Copy full SHA for 2912f17

• Fix export-lora.cpp "not enough space in the context's memory pool" (#1)

  Show description for ad64e33
  

  meatbag-18a

  and

  xaedes

  authored
  ad64e33

  Copy full SHA for ad64e33

• improve handling of not yet supported tensor types

  xaedes

  committed
  1660658

  Copy full SHA for 1660658

Commits on Sep 28, 2023

• Merge branch 'master' into HEAD

  ggerganov

  committed
  5461129

  Copy full SHA for 5461129