Code

[Feature Branch] KV Cache Interface#1083

Code

Merged

bfineran merged 109 commits intomainneuralmagic/deepsparse:mainfrom feature/damian/fb_kv_cacheneuralmagic/deepsparse:feature/damian/fb_kv_cacheCopy head branch name to clipboardJul 12, 2023

+1,488-93

Commits

Commits on Jun 5, 2023

• initial commit

  dbogunowicz

  committed
  48ac0ac

  Copy full SHA for 48ac0ac

• Update src/deepsparse/license.py

  dbogunowicz

  authored
  cf7f2b9

  Copy full SHA for cf7f2b9

Commits on Jun 6, 2023

• Merge branch 'main' into feature/damian/do_not_save_to_tmp

  dbogunowicz

  authored
  832630a

  Copy full SHA for 832630a

Commits on Jun 7, 2023

• Merge branch 'main' into feature/damian/do_not_save_to_tmp

  dbogunowicz

  authored
  9958c83

  Copy full SHA for 9958c83

• limit to 150mb

  dbogunowicz

  committed
  e6d2b03

  Copy full SHA for e6d2b03

• ready to review

  dbogunowicz

  committed
  7f9935b

  Copy full SHA for 7f9935b

Commits on Jun 8, 2023

• initial commit

  

  dbogunowicz

  authored and

  markurtz

  committed
  b1cf01b

  Copy full SHA for b1cf01b

• [Codegen][ORT][Static Seq Length] TextGenerationPipeline (#946)

  Show description for 0a3f48d
  

  dbogunowicz

  authored and

  markurtz

  committed
  0a3f48d

  Copy full SHA for 0a3f48d

• [CodeGen][Documentation] (#956)

  Show description for add4625
  

  2 peopleauthored and

  markurtz

  committed
  add4625

  Copy full SHA for add4625

• reimplementation for generative pipelines

  markurtz

  committed
  22d2746

  Copy full SHA for 22d2746

• restore text generation from examples

  

  dbogunowicz

  authored and

  markurtz

  committed
  7f1651d

  Copy full SHA for 7f1651d

• [CodeGen] ONNX model loading to support >2Gb models / two engines (#991)

  

  dbogunowicz

  authored and

  markurtz

  committed
  b85746d

  Copy full SHA for b85746d

• refactor sucessfull

  

  dbogunowicz

  authored and

  markurtz

  committed
  aadc608

  Copy full SHA for aadc608

• Pipeline fully refactored, time to test engine support. Note: Sliding window not yet implemented!

  

  dbogunowicz

  authored and

  markurtz

  committed
  58bc2b0

  Copy full SHA for 58bc2b0

• First iteration with Sage

  

  dbogunowicz

  authored and

  markurtz

  committed
  d538444

  Copy full SHA for d538444

• Apply suggestions from code review

  

  dbogunowicz

  authored and

  markurtz

  committed
  e19676b

  Copy full SHA for e19676b

• ORT agrees with the Engine. But they both give not entirely correct result. Hey, this is good news still

  

  dbogunowicz

  authored and

  markurtz

  committed
  7908b74

  Copy full SHA for 7908b74

• dynamic ORT vs static DS

  

  dbogunowicz

  authored and

  markurtz

  committed
  4bc3472

  Copy full SHA for 4bc3472

• pipeline handles OPT multitoken pass

  

  dbogunowicz

  authored and

  markurtz

  committed
  c07f7ed

  Copy full SHA for c07f7ed

• fixes to get static pipeline a little further along

  

  Benjamin

  authored and

  markurtz

  committed
  fb77838

  Copy full SHA for fb77838

• adjust shapes and slicing to enable static autoregressive pass - ISSUE: tokens past the base seq len are repeated

  

  Benjamin

  authored and

  markurtz

  committed
  2097463

  Copy full SHA for 2097463

• migrate from cache_length to positions input

  

  Benjamin

  authored and

  markurtz

  committed
  5eb10a9

  Copy full SHA for 5eb10a9

• got if working for multitoken + single token scenario

  

  dbogunowicz

  authored and

  markurtz

  committed
  9213f29

  Copy full SHA for 9213f29

• cleanup the pipeline

  

  dbogunowicz

  authored and

  markurtz

  committed
  d9af004

  Copy full SHA for d9af004

• further cleanup post merge

  

  dbogunowicz

  authored and

  markurtz

  committed
  476f25d

  Copy full SHA for 476f25d

• Pipeline working for single-token inference only

  

  dbogunowicz

  authored and

  markurtz

  committed
  fab44e4

  Copy full SHA for fab44e4

• do not load the onnx model with external files twice

  

  dbogunowicz

  authored and

  markurtz

  committed
  d454e2f

  Copy full SHA for d454e2f

• pipeline never redundantly saves the external data + more robust tokenizer

  

  dbogunowicz

  authored and

  markurtz

  committed
  1613e25

  Copy full SHA for 1613e25

• Stop saving tmp files, otherwise the engine looks for external files in the wrong place

  

  dbogunowicz

  authored and

  markurtz

  committed
  b61055c

  Copy full SHA for b61055c

• Left pad support

  

  Benjamin

  authored and

  markurtz

  committed
  6ee25fc

  Copy full SHA for 6ee25fc

• cleanup

  

  dbogunowicz

  authored and

  markurtz

  committed
  5d3004b

  Copy full SHA for 5d3004b

• cleanup2

  

  dbogunowicz

  authored and

  markurtz

  committed
  ace6fa5

  Copy full SHA for ace6fa5

• Add in pipeline timing

  markurtz

  committed
  388586d

  Copy full SHA for 388586d

• add in force tokens logic

  markurtz

  committed
  afd0139

  Copy full SHA for afd0139

• remove input validation for text generation pipelines

  markurtz

  committed
  30eeda7

  Copy full SHA for 30eeda7

• remove multitoken support for now

  markurtz

  committed
  5882b56

  Copy full SHA for 5882b56

• remove kv cache engine and other fixes

  markurtz

  committed
  4bbe33d

  Copy full SHA for 4bbe33d

• nest input shape override

  markurtz

  committed
  afa5746

  Copy full SHA for afa5746

• comment out input shape override

  markurtz

  committed
  e2bb78c

  Copy full SHA for e2bb78c

• add non batch override for ORT

  markurtz

  committed
  2299009

  Copy full SHA for 2299009

Commits on Jun 9, 2023

• clean up generation pipeline

  markurtz

  committed
  2935b77

  Copy full SHA for 2935b77

Commits on Jun 11, 2023

• Merge branch 'main' into feature/damian/do_not_save_to_tmp

  dbogunowicz

  authored
  b89b156

  Copy full SHA for b89b156

Commits on Jun 13, 2023

• initial commit

  dbogunowicz

  committed
  dc3d61b

  Copy full SHA for dc3d61b

• Update src/deepsparse/license.py

  dbogunowicz

  committed
  a294265

  Copy full SHA for a294265

• limit to 150mb

  dbogunowicz

  committed
  af97f2b

  Copy full SHA for af97f2b

• ready to review

  dbogunowicz

  committed
  c117788

  Copy full SHA for c117788

• fix the erronous Makefile

  dbogunowicz

  committed
  4ad5f49

  Copy full SHA for 4ad5f49

• Merge branch 'feature/damian/do_not_save_to_tmp' of https://github.com/neuralmagic/deepsparse into feature/damian/do_not_save_to_tmp

  dbogunowicz

  committed
  9e816bb

  Copy full SHA for 9e816bb

• perhaps fixed GHA

  dbogunowicz

  committed
  f97467f

  Copy full SHA for f97467f

• take into consideration that GHA creates four files

  dbogunowicz

  committed
  6be8d87

  Copy full SHA for 6be8d87

• initial commit

  dbogunowicz

  committed
  e2f088d

  Copy full SHA for e2f088d

• Merge remote-tracking branch 'origin/feature/damian/do_not_save_to_tmp' into feature/damian/codegen_pipeline_clean

  dbogunowicz

  committed
  9fc6c64

  Copy full SHA for 9fc6c64

• tested with actual model

  dbogunowicz

  committed
  a610faf

  Copy full SHA for a610faf

• remove val_inp argument

  dbogunowicz

  committed
  347d1fb

  Copy full SHA for 347d1fb

• Update README.md

  dbogunowicz

  authored
  e11027c

  Copy full SHA for e11027c

• Apply suggestions from code review

  dbogunowicz

  authored
  a950910

  Copy full SHA for a950910

• Update README.md

  dbogunowicz

  authored
  c1d02dc

  Copy full SHA for c1d02dc

• Merge branch 'main' into feature/damian/codegen_pipeline_clean

  dbogunowicz

  authored
  711cdfb

  Copy full SHA for 711cdfb

Commits on Jun 14, 2023

• Merge branch 'main' into feature/damian/codegen_pipeline_clean

  dbogunowicz

  authored
  e602662

  Copy full SHA for e602662

Commits on Jun 16, 2023

• [BugFix] Update deepsparse dockerfile (#1069)

  Show description for 2085c37
  

  rahul-tuli

  authored and

  dbogunowicz

  committed
  2085c37

  Copy full SHA for 2085c37

• initial implementation

  dbogunowicz

  committed
  2f7bc95

  Copy full SHA for 2f7bc95

• working implementation for pipeline input

  dbogunowicz

  committed
  e18fab7

  Copy full SHA for e18fab7

• [Fix] Fix CLI benchmark errors (#1071)

  Show description for 0358d87
  dbogunowicz

  committed
  0358d87

  Copy full SHA for 0358d87

• Merge branch 'main' into feature/damian/codegen_pipeline_clean

  dbogunowicz

  authored
  06b5246

  Copy full SHA for 06b5246

• Merge branch 'feature/damian/codegen_pipeline_clean' into feature/damian/text_gen_perplexity

  dbogunowicz

  authored
  2cab681

  Copy full SHA for 2cab681

• Clean a typo in the pipeline code

  dbogunowicz

  authored
  63b116b

  Copy full SHA for 63b116b

Commits on Jun 21, 2023

• initial commit

  dbogunowicz

  committed
  cde08b9

  Copy full SHA for cde08b9

Commits on Jun 22, 2023

• Merge branch 'main' into feature/damian/fb_kv_cache

  dbogunowicz

  authored
  99d125c

  Copy full SHA for 99d125c

Commits on Jun 26, 2023

• Merge branch 'main' into feature/damian/fb_kv_cache

  dbogunowicz

  authored
  67ffe47

  Copy full SHA for 67ffe47

Commits on Jun 27, 2023

• Merge branch 'main' into feature/damian/fb_kv_cache

  dbogunowicz

  authored
  9937686

  Copy full SHA for 9937686

Commits on Jun 28, 2023

• [KV Cache Interface] DecoderKVCache (#1084)

  Show description for 0d6a423
  dbogunowicz

  authored
  0d6a423

  Copy full SHA for 0d6a423

• [WiP] [KV Cache Interface] Text Generation & Decoder Engine Implementation (#1089)

  Show description for 0809aea
  

  4 peopleauthored
  0809aea

  Copy full SHA for 0809aea

Commits on Jun 29, 2023

• working implementation, time to cleanup

  dbogunowicz

  committed
  7001a6e

  Copy full SHA for 7001a6e

• now kv cache decoder holds information about the num of tokens preprocessed. also encountered first bug when running with the engine

  dbogunowicz

  committed
  c1bf5b7

  Copy full SHA for c1bf5b7

• cleanup the old files

  dbogunowicz

  committed
  79251e6

  Copy full SHA for 79251e6

• Update src/deepsparse/transformers/engines/nl_decoder_engine.py

  dbogunowicz

  authored
  9efbdb6

  Copy full SHA for 9efbdb6

• ready for review

  dbogunowicz

  committed
  da5e93e

  Copy full SHA for da5e93e

• ready for testing

  dbogunowicz

  committed
  a680dac

  Copy full SHA for a680dac

• managed to get first logits right

  dbogunowicz

  committed
  7099994

  Copy full SHA for 7099994

• Delete example

  dbogunowicz

  authored
  1d4d96d

  Copy full SHA for 1d4d96d

• cleanup before sharing with Ben and Sage

  dbogunowicz

  committed
  08e5421

  Copy full SHA for 08e5421

• Merge branch 'feature/damian/pipeline_engine_support' of https://github.com/neuralmagic/deepsparse into feature/damian/pipeline_engine_support

  dbogunowicz

  committed
  bfaa072

  Copy full SHA for bfaa072

• Update src/deepsparse/transformers/engines/nl_decoder_engine.py

  dbogunowicz

  authored
  fbeeb4a

  Copy full SHA for fbeeb4a

Commits on Jul 3, 2023

• assert proper padding on pipeline init

  dbogunowicz

  committed
  f83dcab

  Copy full SHA for f83dcab

• now also supporting kv cache perplexity. time for cleanup

  dbogunowicz

  committed
  e659c33

  Copy full SHA for e659c33

• ready for review

  dbogunowicz

  committed
  cf74ad7

  Copy full SHA for cf74ad7

• correctly print engine info

  dbogunowicz

  committed
  853f876

  Copy full SHA for 853f876

• work with left padding of the tokenizer

  dbogunowicz

  committed
  e8da07e

  Copy full SHA for e8da07e

• quality

  dbogunowicz

  committed
  58b12c8

  Copy full SHA for 58b12c8

Commits on Jul 5, 2023

• fix the multitoken inference

  dbogunowicz

  committed
  eecd232

  Copy full SHA for eecd232

• Perplexity Eval for Text Generation Models (#1073)

  Show description for 10c804a
  

  5 peopleauthored
  10c804a

  Copy full SHA for 10c804a

• Merge branch 'main' into feature/damian/fb_kv_cache

  dbogunowicz

  authored
  7bd23d6

  Copy full SHA for 7bd23d6

Commits on Jul 7, 2023

• [Text Generation] Run deepsparse engine without the LIB.kv_cache object (#1108)

  dbogunowicz

  authored
  10ba82e

  Copy full SHA for 10ba82e

• added few improvements that turned out to be useful post manual testing

  dbogunowicz

  committed
  e81c327

  Copy full SHA for e81c327

• Update src/deepsparse/transformers/engines/nl_decoder_engine.py

  dbogunowicz

  authored
  b737f77

  Copy full SHA for b737f77

• fixed the logic to assert correct multibatch inference

  dbogunowicz

  committed
  042cb79

  Copy full SHA for 042cb79

• Merge branch 'feature/damian/fb_kv_cache' of https://github.com/neuralmagic/deepsparse into feature/damian/fb_kv_cache

  dbogunowicz

  committed
  bf4eac3

  Copy full SHA for bf4eac3

• fix integration tests

  dbogunowicz

  committed
  c8a1f93

  Copy full SHA for c8a1f93

Commits on Jul 10, 2023

• initial implementation

  dbogunowicz

  committed
  d2d3dc1

  Copy full SHA for d2d3dc1

• perplexity working, so as batched inference for different sized inputs

  dbogunowicz

  committed
  6ce1ca4

  Copy full SHA for 6ce1ca4

• Merge branch 'main' into feature/damian/fb_kv_cache

  dbogunowicz

  authored
  47dc986

  Copy full SHA for 47dc986

• fix the integration test

  dbogunowicz

  committed
  ef77d91

  Copy full SHA for ef77d91

• Merge branch 'feature/damian/fb_kv_cache' of https://github.com/neuralmagic/deepsparse into feature/damian/fb_kv_cache

  dbogunowicz

  committed
  f0d74b0

  Copy full SHA for f0d74b0

• better solution for fixing the issues caused by this PR in GHA

  dbogunowicz

  committed
  186c80c

  Copy full SHA for 186c80c

• revert changes to yolo pipeline

  dbogunowicz

  committed
  09993e7

  Copy full SHA for 09993e7

Commits on Jul 11, 2023

• Merge branch 'main' into feature/damian/fb_kv_cache

  dbogunowicz

  authored
  ba8c126

  Copy full SHA for ba8c126

• Update src/deepsparse/transformers/engines/nl_decoder_engine.py

  Show description for 37e8a02
  

  dbogunowicz

  and

  rahul-tuli

  authored
  37e8a02

  Copy full SHA for 37e8a02

• response to Rahuls comments

  dbogunowicz

  committed
  0d308b9

  Copy full SHA for 0d308b9

Commits on Jul 12, 2023

• Merge remote-tracking branch 'origin/main' into feature/damian/fb_kv_cache

  dbogunowicz

  committed
  41e9306

  Copy full SHA for 41e9306