BrowserPilot

Tell a browser what you want in plain English. It scrapes any website — even the ones that block everyone else.

Why BrowserPilot?

Most scraping tools break the moment a site has Cloudflare, DataDome, or Akamai. BrowserPilot doesn't.

What You Can Do

# Single page — just describe what you want
"Go to Amazon and extract all laptop prices under $1000 as JSON"

# Bulk scrape — hit hundreds of pages across protected sites
curl -X POST http://localhost:8000/bulk -H "Content-Type: application/json" -d '{
  "urls": ["https://nike.com", "https://wayfair.com", "https://footlocker.com"],
  "prompt": "Extract product data",
  "format": "json",
  "max_workers": 3
}'

# Watch it work — live browser stream in your browser
# Open http://localhost:8000 and watch the AI navigate in real-time

Output formats: JSON, CSV, PDF, HTML, Markdown, plain text — just ask.

See It Work

Reddit's new React frontend — navigates feeds, clicks posts, scrolls comments. No selectors, no DOM parsing.

StatsMuse — scraped 191 rows of La Liga stats in seconds. JS-rendered data, no API needed.

10 protected sites in 60 seconds — DataDome, Akamai, Cloudflare, PerimeterX. Zero blocks.

Stealth That Actually Works

We don't just claim stealth — we prove it. BrowserPilot passes every major bot detection benchmark:

Don't trust screenshots — reproduce it yourself:

make benchmark          # or: python -m backend.benchmark

Launches Ghost Mode against these detectors live and saves a screenshot + page text for each to outputs/benchmark/, plus an automated pass/fail summary. Use --only sannysoft to target one, or --dry-run to list them without launching a browser.

Tested Against Real Anti-Bot Systems

These are the systems that block 99% of automation tools. BrowserPilot loaded 11 out of 14:

How the stealth works

• Patchright — Playwright fork that never calls Runtime.enable (defeats CDP detection)
• Full Chromium + xvfb — real browser window, real GPU, real WebGL fingerprints
• Fingerprint rotation — each session gets a unique viewport, UA, DPR, locale, timezone
• Human behavior — Bezier mouse curves, variable typing speed, natural scroll patterns
• Geo-matching — proxy country auto-maps to correct timezone + locale
• WebRTC blocked — local IP never leaks

No noise injection. Anti-bots detect canvas/WebGL noise by rendering known values. Real fingerprints from real hardware, varied through configuration, is stronger.

Bulk Scraping at Production Scale

Not a demo — a production bulk engine that scrapes hundreds of pages concurrently without getting blocked.

# Start a bulk job
curl -X POST http://localhost:8000/bulk \
  -H "Content-Type: application/json" \
  -d '{
    "urls": ["https://site1.com/page1", "https://site2.com/page2", "..."],
    "prompt": "Extract product names and prices",
    "format": "json",
    "max_workers": 5,
    "block_resources": true
  }'

# Check progress
curl http://localhost:8000/bulk/{job_id}

# Resume after crash
curl -X POST http://localhost:8000/bulk/{job_id}/resume

Quick Start

Docker (recommended)

git clone https://github.com/ai-naymul/BrowserPilot.git
cd BrowserPilot
echo 'GOOGLE_API_KEY=your_key_here' > .env
docker-compose up -d

Open http://localhost:8000 — done.

Manual

# 1. Backend
git clone https://github.com/ai-naymul/BrowserPilot.git && cd BrowserPilot
pip install -r requirements.txt
patchright install chromium
echo 'GOOGLE_API_KEY=your_key_here' > .env

# 2. Build the dashboard the app serves (required — the API serves frontend/dist)
cd frontend && npm install && npm run build && cd ..

# 3. Run
python -m uvicorn backend.main:app --reload

Configuration

# Required
GOOGLE_API_KEY=your_gemini_api_key

# Optional — proxies for heavy scraping
SCRAPER_PROXIES=[{"server": "http://proxy:port", "username": "user", "password": "pass", "location": "US"}]

Use Cases

Price monitoring — Track competitor pricing across Amazon, Walmart, Best Buy. Get structured JSON, schedule with cron.

Lead generation — Extract company data from LinkedIn, G2, Crunchbase. BrowserPilot handles login walls and infinite scroll.

Real estate data — Pull listings from Zillow, Realtor.com, Redfin. Export as CSV for analysis.

Market research — Monitor product launches on Product Hunt, reviews on Trustpilot, job postings on Indeed.

Academic research — Collect data from government portals, research databases, news sites that block standard scrapers.

How It Works

You type: "Extract laptop prices from Best Buy"
    |
    v
AI Vision (Gemini 2.5 Flash) sees the page like you do
    |
    v
Decides: click search, type query, scroll, extract data
    |
    v
Ghost Mode stealth keeps it undetected
    |
    v
S