🚀🤖 Crawl4AI: Open-source LLM Friendly Web Crawler & Scraper.
🚀 Crawl4AI Cloud API — Closed Beta (Launching Soon)
Reliable, large-scale web extraction, now built to be drastically more cost-effective than any of the existing solutions.
👉 Apply here for early access
We’ll be onboarding in phases and working closely with early users.
Limited slots.
Crawl4AI turns the web into clean, LLM ready Markdown for RAG, agents, and data pipelines. Fast, controllable, battle tested by a 50k+ star community.
✨ Check out latest update v0.9
✨ New in v0.9: Major secure-by-default release of the Docker API server. Auth is on by default, the server binds loopback unless given a token, and the request body is now an untrusted trust boundary. Breaking changes for the self-hosted server only; the pip library is unchanged. If you self-host the Docker API, read the migration guide before upgrading. Release notes →
✨ Recent v0.8.7: Security-hardening release. Fixes critical Docker API vulnerabilities (RCE, SSRF, auth bypass, file write, XSS, hardcoded JWT secret), adds DomainMapper, and ships scraping, deep-crawl, and LLM fixes. Release notes →
✨ Recent v0.8.6: Security hotfix that replaced litellm with unclecode-litellm due to a PyPI supply chain compromise.
✨ Previous v0.8.0: Crash Recovery & Prefetch Mode! Deep crawl crash recovery with resume_state and on_state_change callbacks for long-running crawls. New prefetch=True mode for 5-10x faster URL discovery. Release notes →
✨ Previous v0.7.8: Stability & Bug Fix Release! 11 bug fixes addressing Docker API issues, LLM extraction improvements, URL handling fixes, and dependency updates. Release notes →
I grew up on an Amstrad, thanks to my dad, and never stopped building. In grad school I specialized in NLP and built crawlers for research. That’s where I learned how much extraction matters.
In 2023, I needed web-to-Markdown. The “open source” option wanted an account, API token, and $16, and still under-delivered. I went turbo anger mode, built Crawl4AI in days, and it went viral. Now it’s the most-starred crawler on GitHub.
I made it open source for availability, anyone can use it without a gate. Now I’m building the platform for affordability, anyone can run serious crawls without breaking the bank. If that resonates, join in, send feedback, or just crawl something amazing.
- LLM ready output, smart Markdown with headings, tables, code, citation hints
- Fast in practice, async browser pool, caching, minimal hops
- Full control, sessions, proxies, cookies, user scripts, hooks
- Adaptive intelligence, learns site patterns, explores only what matters
- Deploy anywhere, zero keys, CLI and Docker, cloud friendly
🚀 Quick Start
- Install Crawl4AI:
# Install the package
pip install -U crawl4ai
# For pre release versions
pip install crawl4ai --pre
# Run post-installation setup
crawl4ai-setup
# Verify your installation
crawl4ai-doctor
If you encounter any browser-related issues, you can install them manually:
python -m playwright install --with-deps chromium
- Run a simple web crawl with Python:
import asyncio
from crawl4ai import *
async def main():
async with AsyncWebCrawler() as crawler:
result = await crawler.arun(
url="https://www.nbcnews.com/business",
)
print(result.markdown)
if __name__ == "__main__":
asyncio.run(main())
- Or use the new command-line interface:
# Basic crawl with markdown output
crwl https://www.nbcnews.com/business -o markdown
# Deep crawl with BFS strategy, max 10 pages
crwl https://docs.crawl4ai.com --deep-crawl bfs --max-pages 10
# Use LLM extraction with a specific question
crwl https://www.example.com/products -q "Extract all product prices"
💖 Support Crawl4AI
🎉 Sponsorship Program Now Open! After powering 51K+ developers and 1 year of growth, Crawl4AI is launching dedicated support for startups and enterprises. Be among the first 50 Founding Sponsors for permanent recognition in our Hall of Fame.
Crawl4AI is the #1 trending open-source web crawler on GitHub. Your support keeps it independent, innovative, and free for the community — while giving you direct access to premium benefits.
🤝 Sponsorship Tiers
- 🌱 Believer ($5/mo) — Join the movement for data democratization
- 🚀 Builder ($50/mo) — Priority support & early access to features
- 💼 Growing Team ($500/mo) — Bi-weekly syncs & optimization help
- 🏢 Data Infrastructure Partner ($2000/mo) — Full partnership with dedicated support
Custom arrangements available - see SPONSORS.md for details & contact
Why sponsor?
No rate-limited APIs. No lock-in. Build and own your data pipeline with direct guidance from the creator of Crawl4AI.
✨ Features
- 🧹 Clean Markdown: Generates clean, structured Markdown with accurate formatting.
- 🎯 Fit Markdown: Heuristic-based filtering to remove noise and irrelevant parts for AI-friendly processing.
- 🔗 Citations and References: Converts page links into a numbered reference list with clean citations.
- 🛠️ Custom Strategies: Users can create their own Markdown generation strategies tailored to specific needs.
- 📚 BM25 Algorithm: Employs BM25-based filtering for extracting core information and removing irrelevant content.
- 🤖 LLM-Driven Extraction: Supports all LLMs (open-source and proprietary) for structured data extraction.
- 🧱 Chunking Strategies: Implements chunking (topic-based, regex, sentence-level) for targeted content processing.
- 🌌 Cosine Similarity: Find relevant content chunks based on user queries for semantic extraction.
- 🔎 CSS-Based Extraction: Fast schema-based data extraction using XPath and CSS selectors.
- 🔧 Schema Definition: Define custom schemas for extracting structured JSON from repetitive patterns.
- 🖥️ Managed Browser: Use user-owned browsers with full control, avoiding bot detection.
- 🔄 Remote Browser Control: Connect to Chrome Developer Tools Protocol for remote, large-scale data extraction.
- 👤 Browser Profiler: Create and manage persistent profiles with saved authentication states, cookies, and settings.
- 🔒 Session Management: Preserve browser states and reuse them for multi-step crawling.
- 🧩 Proxy Support: Seamlessly connect to proxies with authentication for secure access.
- ⚙️ Full Browser Control: Modify headers, cookies, user agents, and more for tailored crawling setups.
- 🌍 Multi-Browser Support: Compatible with Chromium, Firefox, and WebKit.
- 📐 Dynamic Viewport Adjustment: Automatically adjusts the browser viewport to match page content, ensuring complete rendering and capturing of all elements.
- 🖼️ Media Support: Extract images, audio, videos, and responsive image formats like
srcsetandpicture. - 🚀