Maintainer of this skill?
by YOLOW92
Rust web scraper & crawler for AI agents: URLs, PDFs and Office files → clean Markdown/JSON. Firecrawl- and Jina Reader-compatible API, MCP server, Python/Node SDKs. Self-hosted single binary.
# Add to your Claude Code skills
git clone https://github.com/YOLOW92/pagesiftSee how pagesift compares with popular alternatives.
pagesift is an open-source ai agents skill for AI coding assistants such as Claude Code, Codex CLI, and ChatGPT, built by YOLOW92. Rust web scraper & crawler for AI agents: URLs, PDFs and Office files → clean Markdown/JSON. Firecrawl- and Jina Reader-compatible API, MCP server, Python/Node SDKs. Self-hosted single binary. It has 175 GitHub stars.
pagesift's catalog security scan is still queued. You can run an instant dependency and prompt-injection check now with the "Scan for vulnerabilities" button above.
Clone the repository with "git clone https://github.com/YOLOW92/pagesift" and add it to your Claude Code skills directory (see the Installation section above).
pagesift is primarily written in Rust. It is open-source under YOLOW92 on GitHub, so you can review or fork the full source.
Yes. SkillsLLM lists many other AI Agents skills you can browse and compare side by side. Open the AI Agents category from the badge at the top of this page, or use the Related Skills and comparison links further down to weigh pagesift against similar tools.
No comments yet. Be the first to share your thoughts!
Available after the catalog security scan passes.
⚠️ Third-Party Software Notice
This skill is third-party open-source software developed and hosted independently on GitHub. SkillsLLM is an informational directory and does not control or maintain the underlying repository.
Any security checks, ratings, or warnings displayed by SkillsLLM are automated and limited in scope. They do not constitute a security certification or guarantee that the software is safe, error-free, or free from malicious code, vulnerabilities, compromised dependencies, or prompt-injection risks.
Review the source code, permissions, dependencies, and configuration before installing or running any third-party skill. Use is at your own risk. To the maximum extent permitted by applicable law, SkillsLLM is not liable for losses arising from third-party software.
The deep catalog scan for this skill is still queued. Run an instant dependency check now instead.
Turn web pages, PDFs and Office files into clean Markdown for LLMs and AI agents.
One self-hosted Rust binary: CLI, Firecrawl-compatible HTTP API, Jina Reader endpoint, MCP server, Python and Node.js packages.
English · 简体中文 · 日本語 · Español
pagesift fetches a URL (rendering JavaScript only when the page needs it) or reads a local file, keeps the main content, and returns Markdown plus metadata. It scrapes single pages, batches and whole sites, and runs as a command, a library, an HTTP service or an MCP server.
Can replace or sit alongside: Firecrawl (same v1/v2 API, self-hosted), Jina Reader (/r/<url>),
Crawl4AI, trafilatura / Readability, pymupdf4llm / pypdf, Kreuzberg.
/r/ · MCP (stdio or HTTP) · Python · Node.js / TypeScript · Rust crateghcr.io/yolow92/pagesift| Install | First call | |
|---|---|---|
| CLI | cargo install pagesift --locked |
pagesift scrape https://example.com |
| Python | pip install pagesift |
pagesift.scrape("https://example.com").markdown |
| Node.js | npm install @pagesift/node |
(await scrape("https://example.com")).markdown |
| Server | docker run -p 3002:3002 ghcr.io/yolow92/pagesift:0.81.4 |
curl localhost:3002/r/https://example.com |
| MCP | — | claude mcp add pagesift -- pagesift mcp |
CLI
pagesift scrape "https://news.ycombinator.com" # Markdown to stdout
pagesift scrape report.pdf # local PDF, Office, EPUB, CSV or HTML file
pagesift crawl "https://docs.rs" --limit 500 --out ./data/ # one Markdown file per page
pagesift crawl "https://www.example.edu" --query "graduate admission" --adaptive
pagesift map "https://docs.rs" # list a site's URLs
pagesift doctor # check browsers and environment
Python
import pagesift
page = pagesift.scrape("https://example.com", formats=["markdown", "metadata"])
print(page.markdown)
print(page.metadata.title, page.metadata.published_date)
results = pagesift.scrape_many(["https://site1.com", "https://site2.com"], concurrency=8)
Node.js / TypeScript
import { scrape, scrapeMany } from '@pagesift/node'
const page = await scrape('https://example.com', { formats: ['markdown', 'links'] })
console.log(page.markdown)
for await (const p of scrapeMany(['https://a.com', 'https://b.com'], { concurrency: 4 })) {
console.log(p.url, p.elapsedMs)
}
HTTP server (Firecrawl and Jina Reader compatible)
docker run -d -p 3002:3002 -v pagesift-data:/data \
-e PAGESIFT_API_KEYS=sk-test ghcr.io/yolow92/pagesift:0.81.4
from firecrawl import FirecrawlApp
app = FirecrawlApp(api_url="http://localhost:3002", api_key="sk-test") # existing Firecrawl code, new URL
print(app.scrape_url("https://example.com", params={"formats": ["markdown"]})["markdown"])
curl -s http://localhost:3002/r/https://example.com -H 'Authorization: Bearer sk-test' # Jina Reader style
curl -s http://localhost:3002/v2/parse -H 'Authorization: Bearer sk-test' \
-F file=@report.pdf -F 'options={"formats":["markdown"]}' # upload a file
Endpoints: /v1 and /v2 scrape, batch/scrape, crawl, map; /v2/parse (file upload, up to 50 MB,
details); /r/<url> (plain Markdown or text/event-stream).
MCP server for AI agents
pagesift mcp # stdio, for one agent
pagesift mcp http --port 3003 # one shared local service (127.0.0.1 only)
claude mcp add --scope user --transport http pagesift http://127.0.0.1:3003/mcp
Tools: pagesift_scrape, pagesift_batch_scrape, pagesift_crawl, pagesift_crawl_status, pagesift_map.
Shared browsers and cache, an audit log and an issue-report tool: shared MCP service.
Good fit
Pick something else when
↑ higher is better · ↓ lower is better. Strongest result per domain; methods and sources in the full report.
| Task | Metric | PageSift | Best other tool | Gap |
|---|---|---|---|---|
| PDF → Markdown — olmOCR-bench 1,269 pages, per page (single thread) | ms/page ↓ | 2.2 | pypdf 50.0 | ×22.32 |
| HTML → Markdown — 600 generated documents, per page | ms/page ↓ | 0.24 | Turndown 1.9 | ×8.24 |
| Main content — Firecrawl 597 pages, single thread | p50 ms/page ↓ | 1.0 | crw 7.4 | ×7.23 |
| Publication dates — htmldate 1,000 pages, date search | ms/page ↓ | 0.31 | htmldate (fast) 2.0 | ×6.45 |
| Office / e-books — B4 81 files, in-process | files/s ↑ | 94.1 | Kreuzberg 19.5 | ×4.82 |
| RAG chunking — 5 corpora, 1.4 MB | total ms ↓ | 11 | chonkie recursive 35 | ×3.24 |
| RAG query filter — 472 questions, default settings | answer recall ↑ | 0.929 | Crawl4AI BM25 0.673 | +0.256 |
| Wild crawl — polite, 40 sites × 60 pages, coverage | coverage ↑ | 0.740 | Crawl4AI (HTTP) 0.529 | +0.211 |
Still behind: A2 replay, 269 pages, 1 worker 2.4 pages/s vs spider 156.6 pages/s; fast, 40 sites × 60 pages, time 16.6 seconds vs spider 1.4 seconds; B5 134 pages, completion 0.112 vs Docling (GPU) 1.000.
A2 throughput is measured against a replay server on a single loopback address, so all 269 pages count as one host. PageSift waits 1 s between requests to the same host by default; spider has no such limit. This row mostly reflects that politeness interval, not engine speed.
Scale (0.70, synthetic local origin, not real-internet speed): 200,000,000 requests on 192 vCPU with 0 failures at 947,611 pages/s and 888 MiB peak RSS; inside a 768 MiB cgroup it still sustained 817,346 pages/s with no OOM. Sources: 0.70 run, all benchmarks.
includeTags / excludeTags CSS filters; warnings such as partial_content, pages_missing, output_truncated instead of silently incomplete resultsengine: auto fetches over HTTP first and opens a headless browser only when that adds content;