Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor(es/lexer): Add fast lexer implementation #10145

Merged
merged 100 commits into from
Mar 6, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
100 commits
Select commit Hold shift + click to select a range
53eae71
Init
kdy1 Mar 4, 2025
866e243
init
kdy1 Mar 4, 2025
884b4fe
token.rs
kdy1 Mar 4, 2025
7799127
Use Atom for tokens
kdy1 Mar 4, 2025
2e3d63e
mod token;
kdy1 Mar 4, 2025
54bd46d
Dep
kdy1 Mar 4, 2025
25dfddc
lexer
kdy1 Mar 4, 2025
af38716
parser strcuture
kdy1 Mar 4, 2025
4d7303b
more
kdy1 Mar 4, 2025
b4d8d3e
lockfile
kdy1 Mar 4, 2025
4bdc070
skeleton
kdy1 Mar 5, 2025
e191bca
more fix
kdy1 Mar 5, 2025
6cdf377
more fix
kdy1 Mar 5, 2025
b887046
Dep
kdy1 Mar 5, 2025
7e19c4c
lockfile
kdy1 Mar 5, 2025
5965670
fix imports
kdy1 Mar 5, 2025
c0d034e
fix imports
kdy1 Mar 5, 2025
28c95d9
fix imports
kdy1 Mar 5, 2025
85bf228
fix imports
kdy1 Mar 5, 2025
4f939f6
fix imports
kdy1 Mar 5, 2025
3b68875
fix imports
kdy1 Mar 5, 2025
7dd30d5
pub crate
kdy1 Mar 5, 2025
f7a276e
pub crate
kdy1 Mar 5, 2025
ea9297b
`;`
kdy1 Mar 5, 2025
2e00c36
remove wrong imports
kdy1 Mar 5, 2025
efae2ca
New Token: `?.`
kdy1 Mar 5, 2025
e8bf342
benchmark
kdy1 Mar 5, 2025
7a1bdd9
Disable parser
kdy1 Mar 5, 2025
77286a5
lockfile
kdy1 Mar 5, 2025
e5b1356
optimize
kdy1 Mar 5, 2025
8acbdf4
fix clippy
kdy1 Mar 5, 2025
532e1b0
String => Cow
kdy1 Mar 6, 2025
620205d
lint
kdy1 Mar 6, 2025
90199fc
cursorrules
kdy1 Mar 6, 2025
225716d
cursorrules
kdy1 Mar 6, 2025
bf65054
cursorrules
kdy1 Mar 6, 2025
edcd01d
cursorrules
kdy1 Mar 6, 2025
a511156
Optimize `keyword_to_token_type`
kdy1 Mar 6, 2025
4f57b40
feature nightly
kdy1 Mar 6, 2025
e0d0618
util: likely + unlikely
kdy1 Mar 6, 2025
a76e5c6
Optimize cursor.rs
kdy1 Mar 6, 2025
82e8f3b
lint
kdy1 Mar 6, 2025
081fb3f
SIMD
kdy1 Mar 6, 2025
9f819ba
fix build
kdy1 Mar 6, 2025
bdead70
Lookup Table
kdy1 Mar 6, 2025
2f5f7f4
Reduce allocation (correct)
kdy1 Mar 6, 2025
359f2ac
lint
kdy1 Mar 6, 2025
20ec91e
Reduce allocation (future)
kdy1 Mar 6, 2025
65fe1c7
FNV optimization
kdy1 Mar 6, 2025
ee5cd35
More optimization
kdy1 Mar 6, 2025
2b4fed6
pub(crate)
kdy1 Mar 6, 2025
2f073a9
fix bug
kdy1 Mar 6, 2025
12fa017
fix (AI)
kdy1 Mar 6, 2025
ea367a6
panic with loc
kdy1 Mar 6, 2025
06b114a
Fix reset_to
kdy1 Mar 6, 2025
9ecca0f
dollar lbrace
kdy1 Mar 6, 2025
6bf64f9
tests.rs
kdy1 Mar 6, 2025
8de3c31
lexer test
kdy1 Mar 6, 2025
6416ff6
tetss
kdy1 Mar 6, 2025
a1d4664
tetss
kdy1 Mar 6, 2025
21716c0
More tests
kdy1 Mar 6, 2025
6b577c5
more tests
kdy1 Mar 6, 2025
61829b1
lints
kdy1 Mar 6, 2025
b9858b1
Add a test
kdy1 Mar 6, 2025
02f605f
Dep
kdy1 Mar 6, 2025
422984e
lockfile
kdy1 Mar 6, 2025
5fdc424
Fix by using phf
kdy1 Mar 6, 2025
912a101
test
kdy1 Mar 6, 2025
52d6e80
Update test
kdy1 Mar 6, 2025
c54d721
Improve test system
kdy1 Mar 6, 2025
2e8e8bf
Fix test def
kdy1 Mar 6, 2025
d684d0b
#[track_caller]
kdy1 Mar 6, 2025
11fbf49
fix identifier
kdy1 Mar 6, 2025
64d6362
verify_token
kdy1 Mar 6, 2025
5193e88
verify_token
kdy1 Mar 6, 2025
3cb3a81
cursorrules
kdy1 Mar 6, 2025
ce3b3d4
English
kdy1 Mar 6, 2025
459190c
Fix read_template call
kdy1 Mar 6, 2025
6a56dee
Rename
kdy1 Mar 6, 2025
19a54ee
Improve verify_tokens
kdy1 Mar 6, 2025
ba36aeb
${
kdy1 Mar 6, 2025
a83edf2
in_template_expr
kdy1 Mar 6, 2025
f2bf090
in_template = false
kdy1 Mar 6, 2025
d72ed58
fix template literal lexing
kdy1 Mar 6, 2025
0fd96b2
Dep on wide
kdy1 Mar 6, 2025
0cc2758
Dep on wide
kdy1 Mar 6, 2025
e7eebda
lockfile
kdy1 Mar 6, 2025
0bc57c0
REmove SIMD
kdy1 Mar 6, 2025
617d9b8
assert
kdy1 Mar 6, 2025
7fdeced
find_byte: SIMD
kdy1 Mar 6, 2025
5454003
CMT
kdy1 Mar 6, 2025
27bbd45
allow()
kdy1 Mar 6, 2025
78c7215
Reaplce bench name
kdy1 Mar 6, 2025
270d9cb
find_string_end: SIMD
kdy1 Mar 6, 2025
02cb682
advance_while: SIMD
kdy1 Mar 6, 2025
a81a959
clippy
kdy1 Mar 6, 2025
81a56b9
Revert "advance_while: SIMD"
kdy1 Mar 6, 2025
136a0ae
skip_whitespace: SIMD
kdy1 Mar 6, 2025
af3567b
u32
kdy1 Mar 6, 2025
6d80b83
Rename
kdy1 Mar 6, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .cursorrules
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
1. You should write performant code. Always prefer performance over other things.
2. Always write comments in English.
3. Do not use unstable, nightly only features.
4. When creating Atom instances, it's better to use Cow<str> or &str instead of String. Note that `&str` is better than `Cow<str>` here.
45 changes: 45 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -127,6 +127,7 @@ resolver = "2"
wasm-bindgen-futures = "0.4.41"
wasmer = { version = "=5.0.5-rc1", default-features = false }
wasmer-wasix = { version = "0.35.0", default-features = false }
wide = "0.7.32"

[profile.release]
lto = true
Expand Down
41 changes: 41 additions & 0 deletions crates/swc_ecma_fast_parser/Cargo.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
[package]
authors = ["강동윤 <kdy1997.dev@gmail.com>"]
description = "Feature-complete es2019 parser."
documentation = "https://rustdoc.swc.rs/swc_ecma_fast_parser/"
edition = { workspace = true }
include = ["Cargo.toml", "src/**/*.rs", "examples/**/*.rs"]
license = { workspace = true }
name = "swc_ecma_fast_parser"
publish = false
repository = { workspace = true }
version = "1.0.0"

[features]
nightly = []

[dependencies]
swc_atoms = { version = "5.0.0", path = "../swc_atoms" }
swc_common = { version = "8.0.0", path = "../swc_common" }
swc_ecma_ast = { version = "8.0.0", path = "../swc_ecma_ast" }

num-bigint = { workspace = true }
phf = { workspace = true, features = ["macros"] }
wide = { workspace = true }

[dev-dependencies]
criterion = { workspace = true }
pretty_assertions = { workspace = true }
serde_json = { workspace = true }
walkdir = { workspace = true }

codspeed-criterion-compat = { workspace = true }
swc_ecma_ast = { version = "8.0.0", path = "../swc_ecma_ast", features = [
"serde-impl",
] }
swc_ecma_visit = { version = "8.0.0", path = "../swc_ecma_visit" }
swc_malloc = { version = "1.2.2", path = "../swc_malloc" }
testing = { version = "8.0.0", path = "../testing" }

[[bench]]
harness = false
name = "lexer"
100 changes: 100 additions & 0 deletions crates/swc_ecma_fast_parser/benches/lexer.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,100 @@
extern crate swc_malloc;

use codspeed_criterion_compat::{black_box, criterion_group, criterion_main, Bencher, Criterion};
use swc_common::FileName;
use swc_ecma_fast_parser::{token::TokenType, JscTarget, Lexer, Syntax};

fn bench_module(b: &mut Bencher, syntax: Syntax, src: &'static str) {
let _ = ::testing::run_test(false, |cm, _| {
let fm = cm.new_source_file(FileName::Anon.into(), src.into());

b.iter(|| {
let mut count = 0;
let mut lexer = Lexer::new(&fm.src, JscTarget::EsNext, syntax, None);

loop {
if lexer.current.token_type == TokenType::EOF {
break;
}
count += 1;
let token = lexer.next_token();

black_box(token).unwrap_or_else(|err| {
let loc = cm.lookup_char_pos(err.span.lo);
panic!("{err:?}: {loc:?}");
});
}

assert_ne!(count, 0);
});
Ok(())
});
}

fn bench_files(c: &mut Criterion) {
c.bench_function("es/fast-lexer/angular", |b| {
bench_module(
b,
Default::default(),
include_str!("../../swc_ecma_parser/benches/files/angular-1.2.5.js"),
)
});

c.bench_function("es/fast-lexer/backbone", |b| {
bench_module(
b,
Default::default(),
include_str!("../../swc_ecma_parser/benches/files/backbone-1.1.0.js"),
)
});

c.bench_function("es/fast-lexer/jquery", |b| {
bench_module(
b,
Default::default(),
include_str!("../../swc_ecma_parser/benches/files/jquery-1.9.1.js"),
)
});

c.bench_function("es/fast-lexer/jquery mobile", |b| {
bench_module(
b,
Default::default(),
include_str!("../../swc_ecma_parser/benches/files/jquery.mobile-1.4.2.js"),
)
});
c.bench_function("es/fast-lexer/mootools", |b| {
bench_module(
b,
Default::default(),
include_str!("../../swc_ecma_parser/benches/files/mootools-1.4.5.js"),
)
});

c.bench_function("es/fast-lexer/underscore", |b| {
bench_module(
b,
Default::default(),
include_str!("../../swc_ecma_parser/benches/files/underscore-1.5.2.js"),
)
});

c.bench_function("es/fast-lexer/three", |b| {
bench_module(
b,
Default::default(),
include_str!("../../swc_ecma_parser/benches/files/three-0.138.3.js"),
)
});

c.bench_function("es/fast-lexer/yui", |b| {
bench_module(
b,
Default::default(),
include_str!("../../swc_ecma_parser/benches/files/yui-3.12.0.js"),
)
});
}

criterion_group!(benches, bench_files);
criterion_main!(benches);
Loading
Loading