google-browser-scraper

Quickstart · Output · HTTP API · MCP · Docker · How it works · Options

Latest version benchmarks (v0.0.6):

• 100% success rate
• 0.8 MB per search request
• 9 seconds per search request per instance

Scrape Google search results (SERP) with a real Chrome browser and your own residential proxy, and get clean JSON back. Use it as a CLI, a Python library, a local HTTP API, or an MCP server that gives AI agents live Google search.

Plain HTTP is no longer a reliable way to scrape Google: it often expects a browser that runs JavaScript, and a fresh browser on a fresh proxy exit can be refused quickly. This tool takes a browser path instead. It warms a sticky proxy session with ordinary browsing, opens Google's results page for each query in that browser, checks whether Google served results or refused, and writes one JSON record per results page. Captchas, blocks and failures are reported as such, never as "no results".

Features

• Real browser scraping: real Chrome via Patchright, with a warm-up before every new identity
• Bring your own proxy: any residential proxy with sticky sessions, built-in NodeMaven support
• SERP API alternative: JSON with organic_results, ads, related_questions, related_searches, pagination
• Honest failures: captchas and blocks come back labeled, not as empty results
• Cost tracking: bytes per page and cost per 1000 pages in every run summary
• Four ways to run it: CLI, Python library, HTTP API, MCP server for AI agents
• Docker image for self-hosted deployment

Use cases

• SEO rank tracking and SERP monitoring for your own keywords
• Keyword and competitor research from live Google results
• Web search for AI agents and LLM apps through MCP
• A self-hosted alternative to paid Google search APIs when you already pay for proxies

Install

pip install google-browser-scraper
patchright install chromium

Quickstart

You need a residential proxy with sticky sessions. Put {session} where your provider expects a session id in the username:

google-browser-scraper search "best running shoes" \
  --proxy "http://USER-session-{session}:PASS@gate.example.com:7000" \
  -o results.jsonl

The first query takes a few minutes, because a new exit is warmed up first, and the warm-up costs noticeably more traffic than a search. Later queries on the same exit take seconds. A summary of pages served and traffic used, with the hosts that took most of it, is printed at the end of every run.

With NodeMaven residential proxies, the session id is handled for you and the credentials come from the environment:

pip install "google-browser-scraper[nodemaven]"
export NODEMAVEN_LOGIN=... NODEMAVEN_PASSWORD=...
google-browser-scraper search -f queries.txt --nodemaven --country us -o results.jsonl

To cap a run's traffic, --nodemaven-demo creates a sub-user with a traffic limit from your dashboard API key, shares it across workers, and deletes it when the run ends (--keep-subuser keeps it). The account needs sub-users enabled. A fresh warm-up alone uses about 60 MB, so keep the cap well above that:

export NODEMAVEN_API_KEY=...
google-browser-scraper search -f queries.txt --nodemaven-demo --traffic-limit-mb 300 -o results.jsonl

To try it without a proxy, use your own connection:

google-browser-scraper search "best running shoes" --no-proxy

Google then sees one address for every query, so it starts answering with captchas sooner than it would across a pool of exits. Those pages come back as captcha in page_verdict, not as empty results.

From Python:

from google_browser_scraper import ProxyTemplate, Scraper, Settings

proxy = ProxyTemplate("http://USER-session-{session}:PASS@gate.example.com:7000")
for page in Scraper(proxy, Settings(pages=2)).run(["best running shoes"]):
    for result in page.get("organic_results", []):
        print(result["position"], result["title"], result["link"])

What you get

One JSON object per results page:

{
  "search_metadata": {"status": "Success", "page_verdict": "ok"},
  "search_parameters": {"engine": "google", "q": "best running shoes", "page": 1},
  "organic_results": [
    {"position": 1, "title": "...", "link": "https://...", "snippet": "..."}
  ],
  "related_questions": [{"question": "..."}],
  "exit": {"session": "...", "queries_on_exit": 1, "bytes": "..."}
}

The schema uses the field names common to SERP APIs: organic_results, ads, related_questions, related_searches, pagination and search_metadata, so simple consumers can often switch with minimal mapping. Knowledge panels, shopping, images and other special blocks are not parsed.

Each record also has an exit block: which proxy session answered and how many bytes the page cost. search_metadata.timings says where the page's time went, in seconds per step (type, load, ready, read, parse, and links, which overlaps the next query and is not part of total_time_taken). A query that failed is a record too, with an error. --format csv writes the organic results as a table, and --price-per-gb adds the cost per 1000 pages to the summary.

Run it as an API

google-browser-scraper serve --nodemaven --country us --prewarm
curl "http://127.0.0.1:8000/search.json?q=best+running+shoes"

Same record format as above. Requests run one at a time on one held identity. --prewarm searches once at startup, moving to a new exit until Google serves one, so a request sent after it does not wait for a warm-up. /health answers 503 warming until then; a request sent earlier waits in line behind it. Set GBS_API_KEY to require a key; without one the server only listens on localhost.

Use it from an AI agent (MCP)

{
  "mcpServers": {
    "google-search": {
      "command": "google-browser-scraper",
      "args": ["mcp", "--nodemaven", "--country", "us", "--prewarm"],
      "env": {"NODEMAVEN_LOGIN": "...", "NODEMAVEN_PASSWORD": "..."}
    }
  }
}

It exposes one tool, google_search(query, pages).

Docker

A ready image is published to GitHub Container Registry with every release:

docker run --rm ghcr.io/nodemaven/google-browser-scraper doctor
docker run --rm -p 8000:8000 -e GBS_API_KEY=change-me \
  -e GBS_PROXY='http://USER-session-{session}:PASS@gate.example.com:7000' \
  ghcr.io/nodemaven/google-browser-scraper serve --host 0.0.0.0

To build it yourself: docker build -t google-browser-scraper .

How it works

1. New identity: a sticky proxy session and a fresh browser profile.
2. Warm-up: a few ordinary pages, staying on each for 20-45 seconds.
3. Search: each query opens Google's results page directly, which is faster than the search box and skips the autocomplete traffic of typing.
4. Check before parsing: the page is classified first, so a captcha or a block is reported as one and never parsed as an empty result.
5. Keep or drop: an exit Google answers is reused for the next queries; an exit it refuses is dropped and the query is retried on a new one. If exit after exit is refused, the run stops instead of spending more traffic.

Holding exits. A served exit is kept for up to 300 queries. Every new exit pays for a warm-up, so a short cap mostly buys more warm-up traffic. Every 50 results pages the browser is restarted on the same profile and exit, which keeps its memory from growing over a long hold. A browser error on a search is retried once on the same exit before the exit is dropped; a network error, running out of memory or a crashed browser drops it at once.

Copying cookies. The command line copies the cookies of the first exit Google serves and starts new exits from that copy, on new sticky sessions, instead of warming each one; all workers share the copy. Google's acceptance follows the cookies more than the address, so a copy usually starts out served and costs a few MB instead of a full warm-up. A copy waits 40-60 seconds after its browser starts before the first search.

Copying the cache. Pages g