paper-fetch — a publisher-aware full-text PDF fetcher

Give it a DOI; it walks a route ladder to get the full-text PDF the cheapest, most legitimate way first — open access, then official publisher text-mining (TDM) APIs, then your own institutional library proxy, and finally it just prints your library's resolver link so you can finish by hand. Every route is one you're already allowed to use; there is no Sci-Hub fallback here.

It's the download end of a small paper pipeline — the piece that was missing longest, because it's the one that fights publishers:

# 30 seconds to first PDF (the OA route needs zero keys — just your email in config.yaml)
git clone https://github.com/drpwchen/paper-fetch && cd paper-fetch
pip install -r requirements.txt && cp config.example.yaml config.yaml   # set unpaywall_email
python paper_fetch.py 10.1186/s12984-023-01168-x out.pdf

This project ships no institution's access. You supply your own library's endpoints (in config.yaml) and your own account (in a local secret store). It automates your own authenticated session — it is not a paywall bypass and it does not share credentials. Most readers will point an AI agent at this repo to understand and adapt the method; the code — and AGENTS.md — are written for exactly that.

The route ladder (the idea)

The whole design is: try the cheapest, most-permitted route first, fall through on failure.

DOI
 ├─ 1. Open Access ─────── Unpaywall + Semantic Scholar openAccessPdf → every oa_location,
 │                          PMC→Europe PMC render, landing-page citation_pdf_url (repositories)
 ├─ 2. Publisher TDM API ─ Elsevier / Wiley / Springer official text-mining endpoints
 │                          (you register for your own key; entirely sanctioned)
 ├─ 3. Institutional proxy your library's off-campus remote-auth + EZproxy/NetScaler proxy
 │                          (your own login; for subscriptions you already have)
 └─ 4. Resolver link ────── print your library's SFX/OpenURL link → finish manually

Layer 1 works out of the box for anyone. Layers 2–3 are where the work is.

Two rules worth stealing:

• Validate %PDF magic bytes, never trust Content-Type. Paywalls and Cloudflare love to return 200 text/html that looks like a PDF response but isn't.
• Before blaming a route, check whether you're entitled to that article. A working route and a route you have no access through look identical. See docs/holdings.md — this is the single most expensive mistake in this problem space.

Publisher routes, worked out the hard way

The point of this repo. Each publisher exposes a different shape; these are the ones mapped and verified against live articles.

All four shapes live in one dispatch table — ROUTES in library_session.py, keyed by DOI prefix with a kind of tpl / meta / lww — and as of v1.0 every one of them ships working code:

10.1016 Elsevier goes through the TDM API in paper_fetch.py, never the proxy.

Adding a new publisher? Reach for the citation_pdf_url route first. Many sites with no DOI→PDF template still advertise the exact PDF URL in a <meta name="citation_pdf_url"> tag on the article page (it's what Google Scholar indexes). Resolve the DOI through the proxy, read the meta, fetch it with the article as Referer. No reverse engineering needed.

And if a route comes back cf_block, try headful navigation before declaring it dead. BMJ was documented here as a Cloudflare dead end that no stealth browser could clear. That was simply wrong: the WAF only blocks headless requests, and a real headful navigation passes on the first try — that's the whole _HEADFUL_META_PREFIXES variant. (Highwire sites like AJNR/JNM also loop on the generic doi-org resolver → give them an explicit host so the route uses that site's own /lookup/doi/.)

How this compares to other paper fetchers

Plenty of tools turn a DOI into a PDF. They differ on two axes: where the PDFs come from and what happens when a route fails.

The two rows nothing else has — entitlement ground truth and log-driven route health — exist because they answer the question every other tool leaves you guessing on: "is this route broken, or do I just not have access to this article?"

Install

python -m patchright install chromium      # only needed for the institutional proxy layer

(The clone / pip install / cp config.example.yaml config.yaml steps are in the quickstart above.) Runs on Windows, macOS and Linux.

Credentials never live in config.yaml — they go in a secret store, and the right one is picked for your platform with no configuration:

Put -w last in the security command so it prompts for the value — passing it inline leaves the password in your shell history and in ps output.

Override the choice with secrets.backend in config.yaml or the SECRETS_BACKEND environment variable (dpapi · keychain · env · none); an unknown name fails at startup rather than looking like a missing secret later. The env backend has nowhere to persist the proxy session cookie, so that layer logs in once per run.

The names to store: LIB_USER / LIB_PASS (your own library account), plus whichever publisher TDM keys you have — ELSEVIER_TDM_KEY, WILEY_TDM_TOKEN, SPRINGER_API_KEY.

Off-Windows caveat, accurate as of v1.2.0. paper_fetch.py still reads the publisher TDM keys through the Windows PowerShell store, so the TDM layer is Windows-only — and 10.1016 (Elsevier) is one of the busiest routes in the ladder. On macOS and Linux you get the full open-access ladder and the full institutional-proxy layer; TDM routes are skipped, with a hint that still names PowerShell. Porting them is the next piece of work.

How to register for the publisher TDM APIs (Elsevier / Wiley / Springer / Unpaywall) → docs/publisher-tdm-apis.md.

Use

python paper_fetch.py 10.1371/journal.pone.0000000 out.pdf   # OA / TDM — works out of the box
python paper_fetch.py --json 10.1016/xxx out.pdf             # agent mode: JSON envelope on stdout
python holdings.py 10.1097/xxxxx                             # do I even have access to this?
python library_session.py check                              # proxy layer (after config.yaml `auth:` is set)
python library_session.py fetch 10.1002/xxxxx out.pdf
python library_session.py stats                              # rate / block analysis
python library_sess