tiktoken: BPE tokenizer for OpenAI models
Fast BPE tokenizer for OpenAI language models.
Learn more about tiktoken
tiktoken is a tokenization library that implements byte pair encoding (BPE), a compression algorithm that converts text into sequences of numeric tokens. The library is written in Rust with Python bindings, providing both standard encodings for OpenAI models and an extensible architecture for custom tokenizers. It performs lossless, reversible tokenization that works on arbitrary text and compresses input by mapping text to subword units, with tokens typically representing about 4 bytes of text on average. The tool is commonly used in applications that need to count tokens for API billing, prepare text for language models, or implement custom tokenization schemes.
Rust-Backed Performance
Written in Rust with Python bindings rather than pure Python, delivering significantly faster tokenization than transformers library implementations. Handles large-scale text processing with minimal overhead for production workloads.
Pre-Built Model Encodings
Includes native encodings for OpenAI models (o200k_base, cl100k_base, gpt-4o) with exact token counts for API billing. Educational submodule provides BPE visualization tools for understanding tokenization mechanics.
Plugin-Based Extensibility
Supports custom tokenizer encodings through a plugin architecture. Add proprietary model tokenizers or modified encoding schemes without forking the core library, enabling experimentation with novel tokenization approaches.
import tiktoken
encoding = tiktoken.get_encoding("cl100k_base")
text = "Hello, how are you doing today?"
tokens = encoding.encode(text)
print(f"Token count: {len(tokens)}")
print(f"Tokens: {tokens}")See how people are using tiktoken
Top in AI & ML
Related Repositories
Discover similar tools and frameworks used by developers
whisper.cpp
Lightweight C++ port of OpenAI Whisper for cross-platform speech recognition.
Continue
Multi-LLM coding agent with interactive and automated modes.
OpenPose
Multi-person 135-keypoint anatomical detection in C++.
ByteTrack
Multi-object tracker associating low-confidence detections across frames.
Ultralytics YOLO
PyTorch library for YOLO-based real-time computer vision.
Related Reading
Guides and comparisons from the Greptile content library
Best AI Coding Tools for Solo Developers (2026 Review)
The best AI coding tools for solo developers in 2026, compared by where they fit in the software development lifecycle: writing code, prototyping, reviewing, testing, and background automation. Plus which tools fit which languages.
What is Source Code Management (SCM): Everything Engineering Leaders Need to Know
Source code management is more than branching and merging. Here is how SCM works, the top tools, the best practices, and why good code review is the foundation of source control in an agentic era.
Guide to Self-Hosted AI Code Review for GitLab and GitHub Enterprise [2026]
How self-hosted AI code review works on GitLab Self-Managed and GitHub Enterprise Server, how it compares to SaaS review tools, what to look for in a self-hosted tool, and what a self-hosted rollout actually takes.
Best Developer Productivity Tools in 2026
The best developer productivity tools in 2026, compared by the workflow problem they solve: code review, autocomplete, AI-native editing, terminal agents, code search, observability, and security.