webclaw

Most web scraping tools give your agent one of two bad outputs:

• a blocked page, login wall, or empty app shell
• raw HTML full of nav, scripts, styling, ads, and duplicated boilerplate

webclaw.io is the hosted web extraction API for webclaw. This repo contains the open-source CLI, MCP server, extraction engine, and self-hostable server.

webclaw turns a URL into clean content your tools can actually use.

webclaw https://example.com --format markdown

# Example Domain

This domain is for use in illustrative examples in documents.

You may use this domain in literature without prior coordination or asking for permission.

Use it from the terminal, wire it into Claude/Cursor through MCP, call the hosted API from your app, or self-host the OSS server.

Install

Agent setup

The fastest way to connect webclaw to Claude Code, Claude Desktop, Cursor, Windsurf, OpenCode, Codex CLI, and other MCP-compatible tools:

npx create-webclaw

The installer detects supported clients and configures the MCP server for you.

Homebrew

brew tap 0xMassi/webclaw
brew install webclaw

Prebuilt binaries

Download macOS, Linux, and Windows binaries from GitHub Releases.

Windows (x64)

The prebuilt ZIP is the easiest installation and does not require Rust. On the v0.6.22 release page, download webclaw-v0.6.22-x86_64-pc-windows-msvc.zip. Extract it in File Explorer, open PowerShell in the extracted folder containing webclaw.exe, and run:

.\webclaw.exe --version
.\webclaw.exe https://example.com --format markdown

PowerShell requires the .\ prefix for programs in the current directory. Add that directory to your user PATH only if you want to run webclaw from other folders. If Windows reports a missing Visual C++ runtime DLL, install the Microsoft Visual C++ Redistributable for x64. The ZIP contains x64 binaries; Windows ARM64 is not covered by the native installation check.

To build from source on Windows, install Rust with the x86_64-pc-windows-msvc toolchain, Visual Studio Build Tools with Desktop development with C++ and the Windows SDK, Git, CMake 3.22 or newer, LLVM (including libclang.dll), and NASM. Open a fresh Developer PowerShell for Visual Studio after installation. Ensure CMake and NASM are on PATH and set LIBCLANG_PATH to your LLVM bin directory if bindgen cannot find it, for example:

$env:LIBCLANG_PATH = "C:\Program Files\LLVM\bin"
cargo install --git https://github.com/0xMassi/webclaw.git --tag v0.6.22 --locked webclaw-cli
webclaw --version

The Cargo package is webclaw-cli; the executable is webclaw. These packages are installed from Git, not crates.io. A first source build can take several minutes. The Windows CI workflow checks the Git installation and published ZIP on a hosted runner with preinstalled build tools; it does not model a clean PC. Linux containers, including Docker on macOS, do not validate Windows support.

Docker

docker run --rm ghcr.io/0xmassi/webclaw https://example.com

Cargo

cargo install --git https://github.com/0xMassi/webclaw.git --tag v0.6.22 --locked webclaw-cli
cargo install --git https://github.com/0xMassi/webclaw.git --tag v0.6.22 --locked webclaw-mcp

If building from source fails because native build tools are missing, install the platform prerequisites:

Python

The python/ directory builds a Python package that bundles the webclaw binary and calls it from Python: scrape, select, tables, find_text, similar, crawl, map and screenshot, with no API key and no Python dependencies.

cd python && uv build --wheel && pip install dist/webclaw_cli-*.whl

import webclaw_cli as wc

md = wc.scrape("https://example.com", "llm")
titles = wc.select("https://books.toscrape.com", css="h3 a::attr(title)")
for page in wc.crawl("https://quotes.toscrape.com", max_pages=10):
    print(page["url"])

See python/README.md for every function and option.

Quick Start

Scrape one page

webclaw https://stripe.com --format markdown

Return LLM-optimized text

webclaw https://docs.anthropic.com --format llm

Keep only the main content

webclaw https://example.com/blog/post --only-main-content

Include or exclude selectors

webclaw https://example.com \
  --include "article, main, .content" \
  --exclude "nav, footer, .sidebar, .ad"

Pick values with CSS selectors or XPath

webclaw https://books.toscrape.com --select "h3 a::attr(href)"
webclaw https://books.toscrape.com --select "article.product_pod" \
  --field "title=h3 a::attr(title)" --field price=.price_color

--select prints a JSON array of what the selector matches: each element's text, or an attribute or the element's HTML when the selector ends in ::attr(NAME) or ::html. Links in href and src come back absolute. With --field NAME=SELECTOR, each match becomes a JSON object whose fields run inside it, taking the first value or null. Pages that need JavaScript render first, as with extraction.

webclaw https://books.toscrape.com --xpath "//li[@class='next']/a/@href"
webclaw https://books.toscrape.com --xpath "//article[@class='product_pod']" \
  --field "title=.//h3/a/@title" --field "price=.//p[@class='price_color']"

--xpath takes an XPath 1.0 expression instead, over the same parsed page: elements and text nodes give their text, attributes their value, and count() or string() style expressions one value. With --xpath, each --field is an expression evaluated from the match, so it usually starts with .. The MCP select tool takes xpath in place of selector.

Data tables as JSON

webclaw "https://en.wikipedia.org/wiki/List_of_S%26P_500_companies" --tables

--tables prints the page's data tables as a JSON array, each with its caption (when it has one), headers and rows of cell text. A cell that spans several rows or columns repeats its text in each one, so every row has one value per column. Stacked header rows are joined per column with /, as in Population / % of world. The MCP select tool takes tables: true in place of a selector.

Header rows are the rows of <thead> and the leading rows made only of <th>; a leading row that is one cell across the whole table becomes the caption. These tables are skipped as page layout:

• tables marked role="presentation" or role="none";
• tables with fewer than two columns of text;
• tables without a header row that have a single body row, wrap another table, hold only links, or hold paragraphs, lists or other blocks in a row of more than one cell.

Text hidden with hidden, an inline display:none or <template> is left out. Text hidden by a stylesheet is kept, because pages are not styled. A cell keeps its first 64 KB of text, and a very large table is cut short with "truncated": true. Once the tables on a page reach 64 MB, the tables after that are left out.

Find elements by text, or everything like one example

webclaw https://books.toscrape.com --find-text "tipping the"
webclaw https://books.toscrape.com --find-regex '^£5\d\.\d\d$'
webclaw https://book