Show HN: Quicktok, an exact BPE tokenizer 7x faster than tiktoken

Category: library

Tags: tokenizer, bpe, c-plus-plus, python, llm, performance, openai

Score: 8.0/10 (Innovation: 7, Technical: 9, Documentation: 9, Utility: 7)

Quicktok is an exact BPE tokenizer implemented in C++ that achieves 3.5–11x speedup over OpenAI's tiktoken through novel data structures like 2-byte tries and SIMD-optimized pretokenizers. It provides drop-in Python and C++ APIs, byte-identical outputs for major model encodings, and ships zero-dependency with bundled vocabularies, making it a compelling upgrade for any LLM pipeline.

Target audience: backend devs, ml engineers, data engineers, python developers

Repository: https://github.com/dmatth1/quicktok · C++ · MIT · 6 stars

View on Hacker News