Fix null byte `\x00` issue in byte level fsm resulting in `KeyError` in `BetterFSM::FSMInfo` #930

Merged

rlouf merged 8 commits intodottxt-ai:mainfrom lapp0:solve-833

Jun 5, 2024

+307-81

Commits on Jun 4, 2024

Fix null byte \x00 issue by switching to numba.types.unicode_type

M0gician
authored and
lapp0
committed
ensure byte fsm unicode_type compatibility by prefixing hex-bytes with \x00
lapp0
committed
index token -> transition key sequence for efficient _walk_fsm
lapp0
committed
pass token_transition_sequence to walk_fsm in parsing.py
lapp0
committed
add assertions for numba behavior re: UnicodeCharSeq / unicode_type
lapp0
committed
Add docstrings to get_token/vocab_transition_keys, improve naming
lapp0
committed
remove cleanup step redundant with cleanup_lark_import
lapp0
committed
Improve naming of token and vocab transition keys in regex.py
lapp0
committed