spider

Spider is an open-source web crawler and scraper for Rust. Point it at a URL and you get the pages back as a stream — no batching, no waiting for the crawl to finish.

[dependencies]
spider = "2"

use spider::{tokio, website::Website};

#[tokio::main]
async fn main() {
    let mut website = Website::new("https://example.com");
    let mut rx = website.subscribe(16);

    tokio::spawn(async move {
        while let Ok(page) = rx.recv().await {
            println!("{}  {}", page.status_code, page.get_url());
        }
    });

    website.crawl().await;
    website.unsubscribe();
}

The whole program. Pages stream as they arrive. The crawler stops on its own.

What's inside

• ⚡ Built for speed — handles large crawls without falling over
• 🌊 Streams pages as they're fetched
• 🧠 Renders JavaScript only when a page needs it
• 🛡️ Proxies, retries, and stealth built in
• 🧩 Scales from one script to a fleet of workers — same API
• 🦀 Pure Rust, embeddable anywhere

Don't want to run the plumbing?

Crawling the modern web takes ongoing work — proxy pools, headless Chrome, Cloudflare updates, fingerprinting. We maintain all of that as a service so you don't have to.

Spider Cloud is a managed backend for the same crawler. The code stays the same; the operational load goes away.

use spider::configuration::{SpiderCloudConfig, SpiderCloudMode};

let cloud = SpiderCloudConfig::new("sk-...")
    .with_mode(SpiderCloudMode::Smart);

let mut website = Website::new("https://example.com")
    .with_spider_cloud_config(cloud)
    .build()?;

Smart mode uses the unblocker only on pages that need it, so you don't pay for bypass on requests that work fine on their own.

Free tier on signup at spider.cloud — no card required.

Install

| You want… | Run | |---|---| | Rust library | cargo add spider | | Command-line tool | cargo install spider_cli | | Node.js package | npm i @spider-rs/spider-rs | | Python package | pip install spider_rs | | MCP server (Claude, Cursor, …) | cargo install spider_mcp | | Managed crawling | spider.cloud |

A little configuration

let mut website = Website::new("https://example.com")
    .with_limit(50)                    // concurrent requests
    .with_depth(10)                    // how deep to follow links
    .with_delay(500)                   // polite pause between hits (ms)
    .with_respect_robots_txt(true)
    .with_subdomains(true)
    .with_user_agent(Some("MyBot/1.0"))
    .with_stealth(true)
    .build()
    .unwrap();

Defaults are reasonable — you only set what you care about. Full reference in the Configuration docs.

Need JavaScript rendering? Turn on features = ["chrome"] and call crawl_smart() — Spider tries HTTP first and only spins up Chrome on pages that need it.

A few things people do with it

• Ingest the open web into vector stores for LLM pipelines
• Watch sites for SEO or price changes
• Export pages as Markdown, JSON, or WARC archives
• Drive headless Chrome for AI browsing agents
• Build small search indexes on a single machine

→ 50+ runnable examples to start from.

Going deeper

• 📚 Guides — recipes and integrations
• 📖 API docs — every option and method
• 💬 Discord — questions and ideas
• 🐛 Issues — bugs and feature requests

Contributing

PRs welcome. See CONTRIBUTING.md.

cargo test -p spider                  # unit tests
RUN_LIVE_TESTS=1 cargo test           # live network tests

License

MIT.