PullMD

Anything in, clean Markdown out. Self-hosted conversion of web pages, documents, images, audio and YouTube videos into token-lean Markdown, for you and for your AI agents.

Quick start · Use with AI agents · API · Configuration · Changelog

Why PullMD

Agents are bad at reading the web. Raw HTML burns context on navigation, cookie banners and scripts, and built-in fetch tools often stumble on JavaScript-heavy pages, PDFs and comment threads. PullMD sits in between: one URL or file in, clean Markdown out, with source and metadata in YAML frontmatter instead of the body.

curl "https://pullmd.example.com/api?url=https://example.com/article"

What that saves, measured on three public pages (tokens counted with the cl100k_base tokenizer, 2026-10-16):

The curl column is what a naive fetch tool hands an agent. The last column is query-scoped extraction: only the sections relevant to a question.

Highlights

Reads almost anything

• Web pages through a fallback chain: Cloudflare's native Markdown, then Readability + Trafilatura (best result wins), then headless Chromium for JavaScript-heavy sites
• Reddit and Hacker News threads with their comment trees (depth and count configurable)
• Documents: PDF, Word, PowerPoint, Excel, EPUB, by URL or upload, plus an optional OCR tier for table-grade PDFs
• YouTube title, description and transcript with clickable timecodes, no API key needed
• Images and audio via any OpenAI-compatible or local model (captions, transcripts)

Built for agents

• MCP server for Claude Code and other MCP clients, with OAuth 2.1 for claude.ai and Claude Desktop
• Query-scoped extraction: ?query= returns only the sections relevant to a question, typically 70-95% fewer tokens on long pages
• Lean output: clean body, metadata in frontmatter, unpadded tables, configurable field allowlist
• Live share links: /s/:id re-fetches when the cached copy is older than an hour and serves the last good snapshot if the source dies

Comfortable to self-host

• One docker compose up, zero config, multi-arch images (amd64/arm64) on Docker Hub and GHCR
• Every sidecar is optional, and PullMD degrades gracefully without it
• Auth modes from open homelab to multi-user, SSRF protection, rate limits, a health endpoint for Uptime Kuma
• Per-site extraction recipes as declarative JSON

Four ways in: PWA · REST API · MCP server · Claude Code skill

Quick start

mkdir pullmd && cd pullmd
curl -O https://raw.githubusercontent.com/AeternaLabsHQ/pullmd/main/docker-compose.yml
docker compose up -d
# → http://localhost:3000

No .env needed: every variable has a sensible default. Add a .env next to the compose file to override anything (see Configuration). Then connect your agent: see AI-agent integration.

Upgrading? v3's only breaking change is the clean-body output format (one-line opt-out in MIGRATION.md). Running behind a reverse proxy? Since v3.13 you need PULLMD_TRUST_PROXY. Full history in the changelog.

docker-compose.yml (zero-config, abridged)

services:
  pullmd:
    image: aeternalabshq/pullmd:latest
    container_name: pullmd
    restart: unless-stopped
    ports:
      - "${PORT:-3000}:3000"
    environment:
      - PUBLIC_URL=${PUBLIC_URL:-http://localhost:${PORT:-3000}}
      - TRAFILATURA_URL=http://trafilatura:8001/extract
      - PLAYWRIGHT_URL=http://playwright:8002/render
      - MARKITDOWN_URL=http://markitdown:8003/convert
      - CACHE_DB=/data/cache.db
    volumes:
      - ./data:/data
    networks:
      - pullmd-internal
    depends_on:
      - trafilatura
      - playwright
      - markitdown

  trafilatura:
    image: aeternalabshq/pullmd-trafilatura:latest
    container_name: pullmd-trafilatura
    restart: unless-stopped
    networks:
      - pullmd-internal

  playwright:
    image: aeternalabshq/pullmd-playwright:latest
    container_name: pullmd-playwright
    restart: unless-stopped
    networks:
      - pullmd-internal

  markitdown:
    image: aeternalabshq/pullmd-markitdown:latest
    container_name: pullmd-markitdown
    restart: unless-stopped
    mem_limit: ${MARKITDOWN_MEM_LIMIT:-1g}
    networks:
      - pullmd-internal

networks:
  pullmd-internal:
    driver: bridge

Abridged for readability — the docker-compose.yml in the repo additionally passes every optional .env variable through to the containers (Reddit credentials, auth, media/OCR keys, YouTube options, output shaping). Use the curl -O command above rather than copying this block, or .env overrides beyond the basics won't reach the containers.

Note: the Playwright sidecar adds ~3.7 GB to your image cache (Chromium + Firefox + WebKit binaries from the official Playwright base image). It's optional — leave PLAYWRIGHT_URL unset and the playwright service block off, and PullMD silently degrades to static extraction with a fallback note in the metadata.

Note: the MarkItDown sidecar is optional. Leave MARKITDOWN_URL unset and remove the markitdown service block to disable document conversion. Web-page URLs always work without it.

Mirror on GHCR: ghcr.io/aeternalabshq/{pullmd,pullmd-trafilatura,pullmd-playwright,pullmd-markitdown}. Replace the image: lines if you prefer GitHub's registry.

Behind Traefik

For deployments behind Traefik with TLS, use docker-compose.traefik.yml instead. Same images, but with Traefik labels and the proxy external network. Set HOST_DOMAIN in .env:

curl -O https://raw.githubusercontent.com/AeternaLabsHQ/pullmd/main/docker-compose.traefik.yml
echo "HOST_DOMAIN=pullmd.example.com" > .env
docker compose -f docker-compose.traefik.yml up -d

This file sets PULLMD_TRUST_PROXY=1: Traefik is the only peer (no port is published), so PullMD takes the client address and the https scheme from Traefik's forwarding headers.

Behind another reverse proxy

PullMD ignores X-Forwarded-For, X-Forwarded-Proto and X-Forwarded-Host unless PULLMD_TRUST_PROXY says the connecting peer is a proxy. Behind nginx, Caddy, a tunnel or a load balancer, set:

PULLMD_TRUST_PROXY=1                          # one proxy hop, port not reachable directly
# or: PULLMD_TRUST_PROXY=172.18.0.0/16       # the proxy's address/network
PUBLIC_URL=https://pullmd.example.com

A hop count trusts whatever connects, so use it only when the container port is not published to anyone but the proxy; otherwise list the proxy's address. Without the setting, all clients share one rate-limit bucket (login, signup, share-link misses, OAuth, media budget) and PullMD logs a one-time warning. PUBLIC_URL keeps the session cookie Secure and is accepted as this site's host by the login form check. Do not send Referrer-Policy: no-referrer for PullMD's pages: browsers then post forms with Origin: null, which is refused. See MIGRATION.md.

Local development (no Docker)

git clone https://github.com/AeternaLabsHQ/pullmd.git
cd pullmd
npm install
npm start             # http://localhost:3000
npm test              # node --test

Configuration

All variables go in .env (copy from .env.example):

v3.0.0 output format change: the markdown body is clean by default - just # Title followed by content. The source URL, fetch date, and all extraction metadata remain in the YAML frontmatter unchanged - the body no longer duplicates them. Set PULLMD_SOURCE_HEADER=true to restore the old inline header. Use PULLMD_FRONTMATTER_FIELDS to pick which frontmatter fields are emitted (handy for trimming tokens in agent pipelines).