jevcrawl

The web crawler for AI agents that finds the answer, not just pages.

A small decision model picks every link and every passage, so your agent's LLM reads only what answers its question at a few cents per crawl.

Quick start · Benchmarks · Tools · How it works · Configuration

Why jevcrawl

Agents that browse a site have two bad options. They can read everything a crawler fetches, which buries the answer in navigation and boilerplate and burns tokens. Or they can let the LLM pick each link, which is slow and costs dollars per question. jevcrawl does neither:

• 🎯 Finds the answer page more often. 54% of tasks vs 33% when Claude picks the links, and 2x as often as Crawl4AI's goal-directed crawl.
• 💸 About 8x cheaper per task, 10x per correct answer. The crawl decisions cost about a cent.
• ⚡ 2x faster than an LLM-driven crawl. Four pages at a time, one decision request per page.
• 📜 Verbatim, with sources. Every passage is copied from the page with its URL, heading path and character offsets. The decision model only scores, so it cannot invent text.
• 🛡️ Safe against prompt injection by design. Page text is only ever scored, never followed.
• 🔌 Works everywhere MCP does: Claude Code, Claude Desktop, Cursor, or the CLI.

Quick start

You need a Jev API key from console.typesafe.ai and uv.

Claude Code

claude mcp add jevcrawl -e TYPESAFE_API_KEY=<your key> -- uvx jevcrawl

Claude Desktop, Cursor and other MCP clients (mcpServers in the client's config file)

{
  "mcpServers": {
    "jevcrawl": {
      "command": "uvx",
      "args": ["jevcrawl"],
      "env": { "TYPESAFE_API_KEY": "<your key>" }
    }
  }
}

Then ask your agent something like "Use jevcrawl to find which Python version added the walrus operator, starting from docs.python.org". Check the setup with uvx jevcrawl doctor. Agents that load skills can also use skills/jevcrawl/SKILL.md.

Pages that need JavaScript (optional). jevcrawl reads plain HTML. To also render pages that are empty without JavaScript, add the js extra and download Chromium once:

claude mcp add jevcrawl -e TYPESAFE_API_KEY=<your key> -- uvx --from "jevcrawl[js]" jevcrawl
uvx --from "jevcrawl[js]" playwright install chromium

(In mcpServers, use "args": ["--from", "jevcrawl[js]", "jevcrawl"].) Only pages detected as JavaScript shells are opened in the browser, at most five per call, and they are listed in stats.js_rendered_urls. Everything else is still fetched as plain HTML.

Benchmarks

Every system starts on a site's home page with the same 30-page budget and the same question. What each one collects goes to the same answering LLM, and the same judge grades the answer, so the crawler is the only thing that changes.

jevcrawl vs Crawl4AI

Crawl4AI is one of the most popular open-source crawlers for LLMs. Its BestFirstCrawlingStrategy with a keyword scorer is its way to crawl toward a question without an LLM, so we compared the two on whether they fetch the page that holds the answer.

Where the numbers come from

The benchmark code and how to reproduce every number: bench/.

Tools

{"blocks": [{"url": "https://docs.python.org/3/whatsnew/3.8.html",
             "heading_path": "What’s New In Python 3.8 > New Features > Assignment expressions",
             "text": "There is new syntax := that assigns values to variables as part of a larger expression. ...",
             "char_start": 307, "char_end": 580, "relevance": 0.97}],
 "uncertain": [],
 "pages": [{"url": "https://docs.python.org/3/", "title": "3.15.0 Documentation", "lang": "en",
            "text_sha256": "b624ce93..."}],
 "stats": {"pages_fetched": 6, "tokens_raw": 80034, "tokens_returned": 1873, "jev_calls": 11,
           "cache_hits": 0, "jev_input_tokens": 164139, "jev_cost_usd": 0.0069,
           "model": "jev-1.13.0", "degraded": false, "degraded_reason": null,
           "needs_js_urls": [], "stopped_because": "answered", "elapsed_seconds": 8.3}}

Trimmed to one block and one page. 80,034 tokens of pages became 1,873 tokens of answer, for $0.0069, in 8.3 seconds.

• blocks are the relevant passages, best first; uncertain are borderline ones worth reading before concluding something is missing.
• char_start/char_end index into the page text whose sha256 is in pages, so a quote can be checked later.
• stopped_because is answered, frontier_empty, max_pages, max_seconds, budget or auth_error.
• Expected failures come back as {"error": "<code>", "message": "<what to do>"}; the tools never raise into the agent's session.

The same operations work from the command line (uv tool install jevcrawl or pip install jevcrawl puts jevcrawl on your path):

jevcrawl crawl https://docs.python.org/3/ --goal "What does the walrus operator do?" [--json]
jevcrawl scrape https://example.com/help/returns [--goal "..."] [--json]
jevcrawl map https://example.com --goal "..." [--limit 20]
jevcrawl doctor

How it works