# Add to your Claude Code skills
git clone https://github.com/spider-rs/spiderLast scanned: 4/23/2026
{
"issues": [],
"status": "PASSED",
"scannedAt": "2026-04-23T06:08:11.965Z",
"semgrepRan": false,
"npmAuditRan": true,
"pipAuditRan": true
}No comments yet. Be the first to share your thoughts!
Spider is an open-source web crawler and scraper for Rust. Point it at a URL and you get the pages back as a stream — no batching, no waiting for the crawl to finish.
[dependencies]
spider = "2"
use spider::{tokio, website::Website};
#[tokio::main]
async fn main() {
let mut website = Website::new("https://example.com");
let mut rx = website.subscribe(16);
tokio::spawn(async move {
while let Ok(page) = rx.recv().await {
println!("{} {}", page.status_code, page.get_url());
}
});
website.crawl().await;
website.unsubscribe();
}
The whole program. Pages stream as they arrive. The crawler stops on its own.
Crawling the modern web takes ongoing work — proxy pools, headless Chrome, Cloudflare updates, fingerprinting. We maintain all of that as a service so you don't have to.
Spider Cloud is a managed backend for the same crawler. The code stays the same; the operational load goes away.
use spider::configuration::{SpiderCloudConfig, SpiderCloudMode};
let cloud = SpiderCloudConfig::new("sk-...")
.with_mode(SpiderCloudMode::Smart);
let mut website = Website::new("https://example.com")
.with_spider_cloud_config(cloud)
.build()?;
Smart mode uses the unblocker only on pages that need it, so you don't pay for bypass on requests that work fine on their own.
Free tier on signup at spider.cloud — no card required.
| You want… | Run |
|---|---|
| Rust library | cargo add spider |
| Command-line tool | cargo install spider_cli |
| Node.js package | npm i @spider-rs/spider-rs |
| Python package | pip install spider_rs |
| MCP server (Claude, Cursor, …) | cargo install spider_mcp |
| Managed crawling | spider.cloud |
let mut website = Website::new("https://example.com")
.with_limit(50) // concurrent requests
.with_depth(10) // how deep to follow links
.with_delay(500) // polite pause between hits (ms)
.with_respect_robots_txt(true)
.with_subdomains(true)
.with_user_agent(Some("MyBot/1.0"))
.with_stealth(true)
.build()
.unwrap();
Defaults are reasonable — you only set what you care about. Full reference in the Configuration docs.
Need JavaScript rendering? Turn on features = ["chrome"] and call crawl_smart() — Spider tries HTTP first and only spins up Chrome on pages that need it.
→ 50+ runnable examples to start from.
PRs welcome. See CONTRIBUTING.md.
cargo test -p spider # unit tests
RUN_LIVE_TESTS=1 cargo test # live network tests
MIT.