Ready-made tokenizer library for working with GPT and tiktoken

Fast and versatile tokenizer for language models, compatible with SentencePiece, Tokenizers, Tiktoken and more. Supports BPE, Unigram and WordPiece tokenization in JavaScript, Python and Rust.

C API for tiktoken-rs

Text tokenization service (Rust, Axum) (2025)

[Rust] Unofficial implementation of "SuperBPE: Space Travel for Language Models" in Rust

Implementation of BPE Tokenizer in Rust

This crate is a rust porting of Andrej Karpathy implementation of Byte Pair Encoding (BPE) algorithm available here https://github.com/karpathy/minbpe

minbpe-rs库的中文注解

Minimal implementation of BPE (Byte Pair Encoding) algorithm in Rust

Simple rust port of bpe