Code

Improve tokenizer performance#1841

Code

Closed

ggmichaelgo wants to merge 39 commits intomainShopify/liquid:mainfrom ianks-lazy-tokenizerShopify/liquid:ianks-lazy-tokenizerCopy head branch name to clipboard

+195-10

Commits

Commits on Oct 30, 2024

• fix parsing quirky incomplete expressions

  ggmichaelgo

  committed
  12b5a35

  Copy full SHA for 12b5a35

• refactor lexer unit test

  ggmichaelgo

  committed
  5969907

  Copy full SHA for 5969907

• lazy tokenizer

  ggmichaelgo

  committed
  b545e3a

  Copy full SHA for b545e3a

• micro optimization

  ggmichaelgo

  committed
  128935d

  Copy full SHA for 128935d

• more micro optimization

  ggmichaelgo

  committed
  ddb969a

  Copy full SHA for ddb969a

• more micro optimization

  ggmichaelgo

  committed
  6ad7508

  Copy full SHA for 6ad7508

• more micro optimization

  ggmichaelgo

  committed
  1b3927e

  Copy full SHA for 1b3927e

• more micro optimization

  ggmichaelgo

  committed
  e3c2fd4

  Copy full SHA for e3c2fd4

• more micro optimization

  ggmichaelgo

  committed
  b81d1c8

  Copy full SHA for b81d1c8

• more micro optimization

  ggmichaelgo

  committed
  c2baa4c

  Copy full SHA for c2baa4c

• more micro optimization

  ggmichaelgo

  committed
  e06ac1b

  Copy full SHA for e06ac1b

• more micro optimization

  ggmichaelgo

  committed
  9e0c5f4

  Copy full SHA for 9e0c5f4

• more micro optimization

  ggmichaelgo

  committed
  48d6a03

  Copy full SHA for 48d6a03

• more micro optimization

  ggmichaelgo

  committed
  649b64f

  Copy full SHA for 649b64f

• more micro optimization

  ggmichaelgo

  committed
  0346337

  Copy full SHA for 0346337

• more micro optimization

  ggmichaelgo

  committed
  91e1563

  Copy full SHA for 91e1563

• more micro optimization

  ggmichaelgo

  committed
  54975dd

  Copy full SHA for 54975dd

• more micro optimization

  ggmichaelgo

  committed
  a406603

  Copy full SHA for a406603

• Avoid method dispatch

  

  ianks

  authored and

  ggmichaelgo

  committed
  aa45356

  Copy full SHA for aa45356

• more opt

  

  ianks

  authored and

  ggmichaelgo

  committed
  f42c6b0

  Copy full SHA for f42c6b0

• more opt

  

  ianks

  authored and

  ggmichaelgo

  committed
  d76663f

  Copy full SHA for d76663f

• more opt

  

  ianks

  authored and

  ggmichaelgo

  committed
  be8e329

  Copy full SHA for be8e329

• fix typo

  ggmichaelgo

  committed
  b7b6400

  Copy full SHA for b7b6400

• remove unnecessary comparison jump table entries

  ggmichaelgo

  committed
  fecbc62

  Copy full SHA for fecbc62

• refactor tokenizer to simulate original regex properly

  ggmichaelgo

  committed
  936f73d

  Copy full SHA for 936f73d

• more less strict strscan version requirement

  ggmichaelgo

  committed
  15d1179

  Copy full SHA for 15d1179

• add more quirky lexer parsing unit test

  ggmichaelgo

  committed
  2a829b8

  Copy full SHA for 2a829b8

• backward compatible tokenizer

  ggmichaelgo

  committed
  78ef372

  Copy full SHA for 78ef372

Commits on Oct 31, 2024

• pre-calculate tokens

  ggmichaelgo

  committed
  b4907f6

  Copy full SHA for b4907f6

• Update strscan comment

  ianks

  authored
  a0c64e4

  Copy full SHA for a0c64e4

Commits on Nov 1, 2024

• mini opt

  ggmichaelgo

  committed
  2e4d02f

  Copy full SHA for 2e4d02f

• mini opt

  ggmichaelgo

  committed
  7d9ffdc

  Copy full SHA for 7d9ffdc

• undo mini opt

  ggmichaelgo

  committed
  d2b68b8

  Copy full SHA for d2b68b8

• mini opt

  ggmichaelgo

  committed
  f49ceed

  Copy full SHA for f49ceed

• use String#split for liquid tag tokenization

  ggmichaelgo

  committed
  7ac53a3

  Copy full SHA for 7ac53a3

• mini opt

  ggmichaelgo

  committed
  579b899

  Copy full SHA for 579b899

• appease rubocop

  ggmichaelgo

  committed
  f770758

  Copy full SHA for f770758

• mini opt

  ggmichaelgo

  committed
  fa51b8b

  Copy full SHA for fa51b8b

• don't create string scanner for liquid tag

  ggmichaelgo

  committed
  fe2ba1d

  Copy full SHA for fe2ba1d