Why CRW? — Firecrawl & Crawl4AI Alternative

• Single binary, 6 MB RAM — no Redis, no Node.js, no containers. Firecrawl needs 5 containers and 4 GB+. Crawl4AI requires Python + Playwright
• 5.5x faster than Firecrawl — 833ms avg vs 4,600ms (see benchmarks). P50 at 446ms
• 73/100 search win rate — beats Firecrawl (25/100) and Tavily (2/100) in head-to-head benchmarks
• Free self-hosting — $0/1K scrapes vs Firecrawl's $0.83–5.33. No infra, no cold starts (85ms). No API key required for local mode
• Agent ready — add to any MCP client in one command. Embedded mode: no server needed
• Firecrawl-compatible API — drop-in replacement. Same /v1/scrape, /v1/crawl, /v1/map endpoints. HTML to markdown, structured data extraction, website crawler — all built-in
• Built for RAG pipelines — clean LLM-ready markdown output for vector databases and AI data ingestion
• Open source — AGPL-3.0, developed transparently. Join our community

| Metric | CRW (self-hosted) | fastcrw.com (cloud) | Firecrawl | Tavily | Crawl4AI | |---|---|---|---|---|---| | Coverage (1K URLs) | 92.0% | 92.0% | 77.2% | — | — | | Avg Scrape Latency | 833ms | 833ms | 4,600ms | — | — | | Avg Search Latency | 880ms | 880ms | 954ms | 2,000ms | — | | Search Win Rate | 73/100 | 73/100 | 25/100 | 2/100 | — | | Idle RAM | 6.6 MB | 0 (managed) | ~500 MB+ | — (cloud) | — | | Cold start | 85 ms | 0 (always-on) | 30–60 s | — | — | | Self-hosting | Single binary | — | Multi-container | No | Python + Playwright | | Cost / 1K scrapes | $0 (self-hosted) | From $13/mo | $0.83–5.33 | — | $0 | | License | AGPL-3.0 | Managed | AGPL-3.0 | Proprietary | Apache-2.0 |

Web Scraping & Crawling Features

Core

| Feature | Description | |---------|-------------| | Scrape | Convert any URL to markdown, HTML, JSON, or links | | Crawl | Async BFS website crawler with rate limiting | | Map | Discover all URLs on a site instantly | | Search | Web search + content scraping — bundled SearXNG sidecar, free Tavily alternative |

More

| Feature | Description | |---------|-------------| | LLM Extraction | Send a JSON schema, get validated structured data back | | LLM Summary | formats: ["summary"] on /v1/scrape — clean prose digest of any page. BYOK (Anthropic / OpenAI / Azure / DeepSeek / any OpenAI-compatible). | | LLM Search Answer | answer: true / summarizeResults: true on /v1/search — synthesized answer with citations or per-result summaries | | JS Rendering | Auto-detect SPAs, render via LightPanda or Chrome | | CLI | Scrape any URL from your terminal — no server needed | | MCP Server | Built-in stdio + HTTP transport for any AI agent |

Use Cases: RAG pipelines · AI agent web access · content monitoring · data extraction · HTML to markdown conversion · web archiving

🚀 Quick Start

# Install:
curl -fsSL https://raw.githubusercontent.com/us/crw/main/install.sh | CRW_BINARY=crw sh

# Interactive setup wizard (recommended):
crw setup

# Scrape:
crw example.com

# Add to Claude Code (local):
claude mcp add crw -- npx crw-mcp
# Add to Claude Code (cloud — includes web search, 500 free credits at fastcrw.com):
claude mcp add -e CRW_API_URL=https://fastcrw.com/api -e CRW_API_KEY=your-key crw -- npx crw-mcp

Or: pip install crw (Python SDK) · npx crw-mcp (zero install) · brew install us/crw/crw (Homebrew) · All install options →

Scrape

Convert any URL to clean markdown, HTML, or structured JSON.

from crw import CrwClient

client = CrwClient(api_url="https://fastcrw.com/api", api_key="YOUR_API_KEY")  # local: CrwClient()
result = client.scrape("https://example.com")
print(result["markdown"])

Local mode: CrwClient() with no arguments runs a self-contained scraping engine — no server, no API key, no setup. The SDK automatically downloads the crw-mcp binary on first use.

CLI:

crw example.com
crw example.com --format html
crw example.com --js --css 'article'

Self-hosted (crw-server running on :3000):

curl -X POST http://localhost:3000/v1/scrape \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com"}'

Cloud:

curl -X POST https://fastcrw.com/api/v1/scrape \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com"}'

With LLM summary (BYOK — bring your own provider key):

curl -X POST http://localhost:3000/v1/scrape \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://en.wikipedia.org/wiki/Tokio_(software)",
    "formats": ["markdown", "summary"],
    "summaryPrompt": "Answer in two sentences.",
    "maxContentChars": 50000,
    "llmProvider": "openai-compatible",
    "llmModel": "deepseek-chat",
    "baseUrl": "https://api.deepseek.com/v1",
    "llmApiKey": "YOUR_PROVIDER_KEY"
  }'

Output:

# Example Domain

This domain is for use in illustrative examples in documents.
You may use this domain in literature without prior coordination.

Renderer selection & response metadata

CRW picks between three rendering backends per request:

• http (1 credit) — plain HTTP fetch. Used for static pages.
• lightpanda (1 credit) — lightweight JS renderer for most SPAs.
• chrome (2 credits) — full Chromium for sites where LightPanda's hydration crashes (e.g. some Next.js App Router pages).

By default the engine auto-select