name: x-tweet-fetcher description: > Fetch tweets, replies, timelines, search results, X Lists, and X Articles from X/Twitter without login or API keys. Single tweets: zero dependencies (FxTwitter). Timelines/search/replies: a Nitter instance (XTF_NITTER). Lists/Articles: a browser driver (Camofox or Playwright). Unified JSON schema across all backends; machine-readable error_code for agent branching. Field reports and agent-use questions: Agent Waystation #22 Teahouse: https://github.com/ythx-101/openclaw-qa/discussions/22

X Tweet Fetcher

Fetch tweets from X/Twitter without authentication. For agent-use stories, failures, and field reports, start from #22 Teahouse / 茶座.

Feature Overview

python3 scripts/fetch_tweet.py accepts the same flags (v1-compatible entry point).

Basic Usage (Zero Dependencies)

# JSON output (default)
xtf --url https://x.com/user/status/1234567890

# Human-readable
xtf --url https://x.com/user/status/1234567890 --text-only

# Output covers: text, author, stats (likes/retweets/views), media URLs,
# quoted tweets, and full article text for tweet-embedded articles.

Timeline / Search / Replies (Nitter)

export XTF_NITTER=http://127.0.0.1:8788   # your instance; comma-separate for failover
xtf --user elonmusk --limit 20
xtf --search "openclaw" --limit 10
xtf --url https://x.com/user/status/123 --replies

Backend selection: --backend auto (default, Nitter first then browser), --backend nitter, --backend browser.

Lists & Articles (Browser)

# Camofox running on localhost:9377 (default), or:
export XTF_BROWSER=playwright

xtf --list https://x.com/i/lists/1455045069516357634 --limit 30
xtf --article https://x.com/i/article/2011779830157557760

Note: full X Article text requires X login; without it you get title + public preview (is_partial: true).

Mentions Monitor (cron-friendly)

xtf --monitor @yourhandle
# exit 0 = no new mentions, 1 = new mentions found, 2 = setup error
# First run builds a baseline silently; later runs report only new URLs.

Error Handling for Agents

Every error result includes error (message) and error_code: invalid_input · not_found · rate_limited · upstream_down · backend_unavailable · all_backends_failed (with per-backend error_causes).

Python API

from xtf import Router
router = Router()
tweets = router.fetch_timeline("elonmusk", limit=20)  # list[Tweet]
print(tweets[0].to_dict())

Directory Structure

src/xtf/
├── backends/   # fxtwitter.py, nitter.py, browser.py
├── parsers/    # pure parsing functions, fixture-tested
├── router.py   # auto-fallback
├── monitor.py  # mentions monitor
└── cli.py      # the `xtf` command
scripts/fetch_tweet.py  # v1-compatible shim

Looking for Chinese-platform fetching (Weibo/Bilibili/WeChat) or tweet growth tracking? Those moved out of this repo in v2 — see MIGRATION.md.

x-tweet-fetcher

Fetch X/Twitter tweets, replies, timelines, lists, and articles — no login, no API keys.

Three backends · Auto fallback · Unified JSON schema · Built for AI agents

Quick Start · Backends · Capabilities · Python API · Self-hosted Nitter · Migrating from v1

😤 Problem

You: fetch that tweet / list / article for me
AI:  I can't access X/Twitter. Please copy-paste the content manually.

You: ...seriously?

X has no free API. Scraping gets you blocked. Browser automation is fragile in headless environments.

x-tweet-fetcher solves this with smart backend routing: FxTwitter for single tweets (zero deps), Nitter for timelines and search (direct HTTP), a browser driver for everything else — with automatic fallback between them.

🚀 Quick Start

git clone https://github.com/ythx-101/x-tweet-fetcher
cd x-tweet-fetcher && pip install .

# Single tweet — works instantly, zero configuration
xtf --url https://x.com/user/status/1234567890

# User timeline (needs a Nitter instance, see below)
export XTF_NITTER=http://127.0.0.1:8788
xtf --user elonmusk --limit 20

# Search
xtf --search "openclaw" --limit 10

# Human-readable output instead of JSON
xtf --user elonmusk --text-only

Prefer not to install? python3 scripts/fetch_tweet.py --url ... works straight from the clone (same flags).

🔀 Three Backends

xtf --user elonmusk                    # auto (default)
xtf --user elonmusk --backend nitter   # direct HTTP only
xtf --list 1455045069516357634         # lists always use the browser

Browser driver defaults to Camofox (localhost:9377). Playwright users:

pip install ".[playwright]"            # from the clone
export XTF_BROWSER=playwright          # or: --browser-driver playwright

📊 Capabilities

Exit codes (cron-friendly): 0 success / no new mentions · 1 error / new mentions found · 2 monitor setup error.

Errors are machine-readable. Every failure carries error (human message) plus error_code — one of invalid_input, not_found, rate_limited, upstream_down, backend_unavailable, all_backends_failed — so agents can branch on it. all_backends_failed additionally includes per-backend error_causes.

🐍 Python API

from xtf import Router, NotFound, RateLimited

router = Router()                                  # backend="auto"
tweet   = router.fetch_tweet("user", "1234567890") # dict, v1-compatible shape
tweets  = router.fetch_timeline("user", limit=20)  # list[Tweet]
replies = router.fetch_replies("user", "1234567890")
results = router.search("openclaw", limit=10)

for tw in tweets:
    print(tw.author, tw.likes, tw.text)
    print(tw.to_dict())                            # JSON-ready

All backends normalize into one Tweet / Reply / Profile / Article schema — your downstream prompt only ever needs to describe one shape.

⚙️ Configuration

Everything is an environment variable (CLI flags override):

NITTER_URL (the v1 name) is still honored as a fallback for XTF_NITTER.

🏗 Self-hosted Nitter

Public Nitter instances are unreliable and frequently dead. Self-hosting is strongly recommended for timeline/search/replies:

# See https://github.com/zedeus/nitter for full setup
docker run -d -p 8788:8080 --name nitter zedeus/nitter:latest
export XTF_NITTER=http://127.0.0.1:8788

Multiple instances failover automatically:

export XTF_NITTER=http://127.0.0.1:8788,https://your-backup-instance.example

If no instance is reachable, you get a clear error (error_code: "all_backends_failed", with each backend's reason — e.g. backend_unavailable — under error_causes) telling you exactly what to set. Never a silent empty result.

📁 Project Structure

src/xtf/
├── models.py        # Tweet / Reply / Profile / Article dataclasses
├── backends/
│   ├── fxtwitter.py # single tweets + profiles
│   ├── nitter.py    # direct HTTP, multi-instance failover
│   └── browser.py   # Camofox / Playwright snapshot fetching
├── parsers/         # pure functions, locked by fixture tests
├── router.py        # auto-fallback chain
├── monitor.py       # incremental mentions monitor
└── cli.py           # the `xtf` command
scripts/fetch_tweet.py   # v1-compatible entry point (thin shim)
tests/fixtures/          # captured page structures — regression protection

🔄 Migrating from v1

python3 scripts/fetch_tweet.py still works with all v1 flags and exit codes, and JSON fields are unchanged for every mode except --search, whose per-tweet schema is now unified with --user (fields renamed, url/has_media/media_urls dropped). See MIGRATION.md for the full list, including where the analytics/China/Obsidian scripts went (spoiler: their own repos — this project is now purely about fetching tweets; the old world lives at the v1-legacy tag).

🧪 Development

pip install -e ".[dev]"
pytest          # all parsers locked by fixture tests
ruff check src tests

When Nitter or X change their page structure, capture a fresh snapshot into tests/fixtures/ — the failing test will show exactly which parser and field broke.

🙏 Acknowledgments

• Nitter by zedeus — self-hosted Twitter frontend
• FxTwitter — public API for single tweet data
• Camofox — anti-fingerprint browser, default browser driver
• Playwright — alternative browser automation driver
• OpenClaw — AI agent framework this tool grew up in

📜 License

MIT

Three backends. Auto fallback. Built for AI agents.

GitHub · Issues · #22 Teahouse · Agent Waystation