xFormers: Optimized Transformer building blocks library
Memory-efficient PyTorch components for transformer architectures.
Learn more about xFormers
import xformers.ops as xops\noutput = xops.memory_efficient_attention(query, key, value)
Custom CUDA kernels
xFormers includes its own optimized CUDA implementations for attention mechanisms and fused operations rather than relying solely on PyTorch primitives. This allows for memory-efficient exact attention computation and other specialized operations not available in mainstream frameworks.
Composable architecture
Components are designed as independent, domain-agnostic building blocks that can be combined without boilerplate code. This modular approach enables researchers to construct custom transformer variants while maintaining clean, readable code.
Research-oriented components
The library prioritizes access to emerging techniques and experimental operations before they reach mainstream libraries. It provides implementations of sparse attention, block-sparse attention, and other attention variants alongside standard components.
import xformers.ops as xops
attention_out = xops.memory_efficient_attention(
query, key, value
)Stable wheels for PyTorch 2.10+
- –Migrated xFormers to the PyTorch stable API/ABI, which means that binary builds targeting PyTorch 2.10+ will be compatible with any later version
- –Removed optimized fast-path of SwiGLU (which was only available for A100 GPUs)
- –Removed most legacy components
Wheels for PyTorch 2.9.1
Fixed wheel upload to pypi
- –Fixed wheel upload to pypi
Top in AI & ML
Related Repositories
Discover similar tools and frameworks used by developers
Crush
LLM-powered coding agent with LSP and MCP integration.
Ultralytics YOLO
PyTorch library for YOLO-based real-time computer vision.
CUTLASS
CUDA C++ templates and Python DSLs for high-performance matrix multiplication on GPUs.
InvokeAI
Node-based workflow interface for local Stable Diffusion deployment.
FastMCP
Build Model Context Protocol servers with decorators.
Related Reading
Guides and comparisons from the Greptile content library
Best AI Code Review Tools for GitLab
Enterprise-grade AI code review tools for GitLab teams. In-depth comparison of accuracy, security, and integration features to accelerate your merge request workflows.
Top AI Code Review Tools for Small Dev Teams 2026
Affordable, lightweight AI code review tools that help small teams catch bugs faster. Practical comparisons of accuracy, ease of use, and value for teams without enterprise budgets.
What is agentic coding? Benefits, risks, and best tools to consider
What engineering leaders need to know about agentic coding: how it differs from AI coding and vibe coding, how it works, the benefits and risks, and the tools worth evaluating.
10 Powerful Code Quality Tools That Catch Bugs Before Deployment
We tested 10 code quality tools that catch bugs before production. Detailed comparison with real-world examples, pricing, and ROI analysis for dev teams.