Crawl4AI: Open-source web crawler for LLM applications
Async browser automation extracting web content for LLMs.
Learn more about crawl4ai
Crawl4AI is an open-source Python library designed to crawl and extract web content optimized for consumption by large language models. It operates through asynchronous browser automation to render JavaScript-heavy pages, capturing dynamic content that traditional HTTP-based scrapers cannot access. The crawler processes web pages to extract clean, structured content while removing navigation elements, advertisements, and other noise that would interfere with LLM processing. It implements configurable extraction strategies to transform raw HTML into markdown or structured data formats suitable for embedding in vector databases or direct LLM prompts.
LLM-Ready Markdown Output
Extracts web content into structured Markdown with preserved semantic elements like headings, tables, and code blocks. Designed specifically for RAG systems and language model ingestion rather than general HTML parsing.
Async Browser Pooling
Manages concurrent crawl requests through a pool of reusable browser instances with asynchronous execution. Reduces startup overhead and enables parallel processing compared to sequential single-browser approaches.
Programmable Extraction Hooks
Inject custom JavaScript, define site-specific behaviors, and chain LLM-based extraction strategies through a hook system. Enables adaptive crawling logic and intelligent content filtering without forking the codebase.
import asyncio
from crawl4ai import AsyncWebCrawler
async def crawl_page():
async with AsyncWebCrawler() as crawler:
result = await crawler.arun(url="https://example.com")
print(result.markdown)
asyncio.run(crawl_page())Adds webhook support to Docker job queue API endpoints, enabling real-time notifications with automatic retry instead of polling.
- –Configure webhooks for /crawl/job and /llm/job endpoints with custom headers and full payload delivery.
- –Set global webhook URLs in config.yml; delivery includes exponential backoff retry on failure.
Requires Python 3.10+; deprecates proxy parameter in favor of proxy_config structure and adds cssselect dependency.
- –Upgrade to Python 3.10 or later and migrate proxy usage to the new proxy_config structure before deploying.
- –Use Docker hooks at 8 pipeline points for auth or performance; fixes JWT validation and URL query parameter handling.
Release notes do not specify breaking changes, new requirements, or feature details; consult CHANGELOG.md for actual changes.
- –Install via PyPI with `pip install crawl4ai==0.7.4` or pull Docker image `unclecode/crawl4ai:0.7.4`.
- –Review the project CHANGELOG.md on GitHub to identify breaking changes, deprecations, or new capabilities before upgrading.
See how people are using crawl4ai
Related Repositories
Discover similar tools and frameworks used by developers
llama.cpp
Quantized LLM inference with hardware-accelerated CPU/GPU backends.
YOLOX
PyTorch anchor-free object detector with scalable model variants.
deepface
Python library wrapping multiple face recognition deep learning models.
lerobot
PyTorch library for robot imitation learning and sim-to-real transfer.
mlx
Lazy-evaluated NumPy-like arrays optimized for Apple silicon.