by 0xMassi
Fast, local-first web content extraction for LLMs. Scrape, crawl, extract structured data — all from Rust. CLI, REST API, and MCP server.
# Add to your Claude Code skills
git clone https://github.com/0xMassi/webclawGuides for using ai agents skills like webclaw.
Last scanned: 5/3/2026
{
"issues": [],
"status": "PASSED",
"scannedAt": "2026-05-03T06:26:10.723Z",
"semgrepRan": false,
"npmAuditRan": true,
"pipAuditRan": true
}Most web scraping tools give your agent one of two bad outputs:
webclaw.io is the hosted web extraction API for webclaw. This repo contains the open-source CLI, MCP server, extraction engine, and self-hostable server.
webclaw turns a URL into clean content your tools can actually use.
webclaw https://example.com --format markdown
# Example Domain
This domain is for use in illustrative examples in documents.
You may use this domain in literature without prior coordination or asking for permission.
Use it from the terminal, wire it into Claude/Cursor through MCP, call the hosted API from your app, or self-host the OSS server.
The fastest way to connect webclaw to Claude Code, Claude Desktop, Cursor, Windsurf, OpenCode, Codex CLI, and other MCP-compatible tools:
npx create-webclaw
The installer detects supported clients and configures the MCP server for you.
brew tap 0xMassi/webclaw
brew install webclaw
Download macOS, Linux, and Windows binaries from GitHub Releases.
docker run --rm ghcr.io/0xmassi/webclaw https://example.com
cargo install --git https://github.com/0xMassi/webclaw.git webclaw-cli
cargo install --git https://github.com/0xMassi/webclaw.git webclaw-mcp
If building from source fails because native build tools are missing, install the platform prerequisites:
| OS | Command |
|---|---|
| Debian / Ubuntu | sudo apt install -y pkg-config libssl-dev cmake clang git build-essential |
| Fedora / RHEL | sudo dnf install -y pkg-config openssl-devel cmake clang git make gcc |
| Arch | sudo pacman -S pkg-config openssl cmake clang git base-devel |
| macOS | xcode-select --install |
webclaw https://stripe.com --format markdown
webclaw https://docs.anthropic.com --format llm
webclaw https://example.com/blog/post --only-main-content
webclaw https://example.com \
--include "article, main, .content" \
--exclude "nav, footer, .sidebar, .ad"
webclaw https://docs.rust-lang.org --crawl --depth 2 --max-pages 50
webclaw https://github.com --brand
webclaw https://example.com/pricing --format json > pricing-old.json
webclaw https://example.com/pricing --diff-with pricing-old.json
webclaw ships with an MCP server for AI agents.
Zero-install — point any MCP client at the npx launcher:
{
"mcpServers": {
"webclaw": {
"command": "npx",
"args": ["-y", "@webclaw/mcp"]
}
}
}
Or run npx create-webclaw to auto-detect your AI tools and write their configs for you.
Then ask your agent things like:
Scrape these competitor pricing pages and summarize the differences.
Crawl this documentation site and prepare clean context for a RAG index.
Extract the brand colors, fonts, and logos from this company website.
Add webclaw to Claude Code, Cursor, Windsurf, and other MCP agents in one command:
npx skills add 0xMassi/webclaw-skill
Your agent gets scrape, crawl, map, extract, summarize, diff, brand, and search
as native tools. Most sites extract locally with no API key. Set WEBCLAW_API_KEY
to handle bot-protected and JavaScript-rendered pages.
Find it on skills.sh.
| Tool | What it does | Local |
|---|---|---|
scrape |
Extract one URL as markdown, text, JSON, LLM format, or HTML | Yes |
crawl |
Follow same-origin links and extract discovered pages | Yes |
map |
Discover URLs without extracting every page | Yes |
batch |
Scrape multiple URLs in parallel | Yes |
extract |
Convert page content into structured data | Yes, with local or configured LLM |
summarize |
Summarize a page | Yes, with local or configured LLM |
diff |
Compare page content snapshots | Yes |
brand |
Extract colors, fonts, logos, and metadata | Yes |
search |
Search the web and scrape results | Hosted API |
research |
Multi-source research workflow | Hosted API |
npm install @webclaw/sdk
pip install webclaw
go get github.com/0xMassi/webclaw-go
import { Webclaw } from "@webclaw/sdk";
const client = new Webclaw({ apiKey: process.env.WEBCLAW_API_KEY! });
const page = await client.scrape({
url: "https://example.com",
formats: ["markdown"],
only_main_content: true,
});
console.log(page.markdown);
from webclaw import Webclaw
client = Webclaw(api_key="wc_your_key")
page = client.scrape(
"https://example.com",
formats=["markdown"],
only_main_content=True,
)
print(page.markdown)
curl -X POST https://api.webclaw.io/v1/scrape \
-H "Authorization: Bearer $WEBCLAW_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://example.com",
"formats": ["markdown"],
"only_main_content": true
}'
| Format | Use it when you need |
|---|---|
markdown |
Clean page content with structure preserved |
llm |
Compact context for agents and RAG pipelines |
text |
Plain text with minimal formatting |
json |
Structured metadata, links, images, and extracted fields |
html |
Cleaned HTML for custom processing |
The CLI and MCP server work locally without an account for the core extraction path.
Use the hosted API at webclaw.io when you need:
export WEBCLAW_API_KEY=wc_your_key
webclaw https://example.com --cloud
| Use case | Example |
|---|---|
| AI agent web access | Give Claude, Cursor, or another MCP client clean page context |
| RAG ingestion | Crawl docs, help centers, blogs, and knowledge bases |
| Competitor monitoring | Track pricing pages, changelogs, docs, and product pages |
| Structured extraction | Turn messy pages into typed JSON for automations |
| Research workflows | Search, scrape, summarize, and cite multiple sources |
| Brand intelligence | Extract logos, colors, fonts, and social metadata |
webclaw/
crates/
webclaw-core HTML to markdown, text, JSON, and LLM-ready output
webclaw-fetch Fetching, crawling, batching, and mapping
webclaw-llm Local and hosted LLM provider support
webclaw-pdf PDF text extraction
webclaw-mcp MCP server for AI agents
webclaw-cli Command-line interface
webclaw-core is pure extraction logic: no network I
webclaw is an open-source ai agents skill for AI coding assistants such as Claude Code, Codex CLI, and ChatGPT, built by 0xMassi. Fast, local-first web content extraction for LLMs. Scrape, crawl, extract structured data — all from Rust. CLI, REST API, and MCP server. It has 2,123 GitHub stars.
Yes. webclaw passed SkillsLLM's automated security scan — a dependency vulnerability audit plus prompt-injection heuristics — with no high-severity issues. You can read the full report in the Security Report section on this page.
Clone the repository with "git clone https://github.com/0xMassi/webclaw" and add it to your Claude Code skills directory (see the Installation section above).
webclaw is primarily written in Rust. It is open-source under 0xMassi on GitHub, so you can review or fork the full source.
Yes. SkillsLLM lists many other AI Agents skills you can browse and compare side by side. Open the AI Agents category from the badge at the top of this page, or use the Related Skills and comparison links further down to weigh webclaw against similar tools.
No comments yet. Be the first to share your thoughts!