Refactor Lexer class and update documentation #511

Job	Run time
Check / Check with clippy and rustfmt	41s
Build / build	47s
Check / Check with Biome	9s
test_deno	5s
test_linux_browser (chrome)	26s
benchmarks	1m 25s
coverage	28s
test_linux_browser (firefox)	37s
test_macos_browser (chrome)	59s
test_nodejs (18.x)	28s
test_macos_browser (firefox)	59s
test_windows_browser (chrome)	1m 20s
test_nodejs (20.x)	27s
test_windows_browser (firefox)	1m 18s
test_windows_browser (edge)	1m 29s
	11m 38s

Provide feedback