tiktoken: BPE tokenizer for OpenAI models
Fast BPE tokenizer for OpenAI language models.
Learn more about tiktoken
tiktoken is a tokenization library that implements byte pair encoding (BPE), a compression algorithm that converts text into sequences of numeric tokens. The library is written in Rust with Python bindings, providing both standard encodings for OpenAI models and an extensible architecture for custom tokenizers. It performs lossless, reversible tokenization that works on arbitrary text and compresses input by mapping text to subword units, with tokens typically representing about 4 bytes of text on average. The tool is commonly used in applications that need to count tokens for API billing, prepare text for language models, or implement custom tokenization schemes.
Rust-Backed Performance
Written in Rust with Python bindings rather than pure Python, delivering significantly faster tokenization than transformers library implementations. Handles large-scale text processing with minimal overhead for production workloads.
Pre-Built Model Encodings
Includes native encodings for OpenAI models (o200k_base, cl100k_base, gpt-4o) with exact token counts for API billing. Educational submodule provides BPE visualization tools for understanding tokenization mechanics.
Plugin-Based Extensibility
Supports custom tokenizer encodings through a plugin architecture. Add proprietary model tokenizers or modified encoding schemes without forking the core library, enabling experimentation with novel tokenization approaches.
import tiktoken
encoding = tiktoken.get_encoding("cl100k_base")
text = "Hello, how are you doing today?"
tokens = encoding.encode(text)
print(f"Token count: {len(tokens)}")
print(f"Tokens: {tokens}")Release notes do not specify breaking changes, new requirements, or feature details for this version.
- –Review the commit history or changelog file directly to identify actual changes before upgrading.
- –Test thoroughly in a staging environment as the scope of modifications is undocumented.
Release notes do not specify breaking changes, new requirements, or feature additions for this version.
- –Review the commit history or changelog file directly to identify changes before upgrading production systems.
- –Test thoroughly in staging as the scope of modifications and potential compatibility impacts are undocumented.
Release notes do not specify breaking changes, requirements, or new features for this version.
- –Review the commit history or changelog file directly to identify changes before upgrading.
- –Test thoroughly in a non-production environment as impact and compatibility are undocumented.
Related Repositories
Discover similar tools and frameworks used by developers
llama_index
Connect LLMs to external data via RAG workflows.
Mask2Former
Unified transformer architecture for multi-task image segmentation.
bolt.new
LLM-powered browser IDE with integrated WebContainers runtime.
LLaMA-Factory
Parameter-efficient fine-tuning framework for 100+ LLMs.
llama
PyTorch inference for Meta's Llama language models.