textbook-to-note

Turn your own PDF textbooks into an AI-searchable knowledge base and structured, fully-cited notes — figures included. A local-first pipeline that spends (almost) zero LLM tokens on the heavy lifting and reserves the frontier model for the one thing it's uniquely good at: synthesizing a note you can actually learn from.

繁體中文說明 → README.zh-TW.md

Why I built this

I've loved taking notes since med school, and over the years I've built up thousands of them — but I can no longer keep every one at the same quality by hand. In an age of information overload, trustworthy high-quality sources become the precious thing, and textbooks are among the best; yet my specialty alone has 40+ designated reference books, with a single concept scattered across chapters in many of them. Reading them all cover to cover just isn't realistic. LLMs are great at long context, but you can't dump hundreds of books on one at once — so the real unlock is pairing good search and a database with teaching the AI my own note-making process, so it produces grounded, structured notes I only have to absorb.

Left: one of the templates in templates/. Right: a real note in my vault, written against it — every claim traceable, every section where I expect it.

Why this is hard (and why naive approaches fail)

Feeding a raw PDF to a frontier model seems simple until you hit the real problems:

• Cost & latency — a 600-page book is 1-2M tokens. Re-reading it for every question is untenable.
• Silent data loss — scanned pages and broken font encodings produce garbage that the model quietly skips. You never learn your note is missing half the chapter.
• Scrambled text — most textbooks are two-column. Default PDF text extraction interleaves the columns, and a naive "is this garbled?" check passes the shuffled output because the characters themselves are fine.
• Figures vanish — the anatomy diagram, the classification table, the treatment algorithm: often the most valuable part of a chapter, and text extraction drops all of it.
• Ungrounded output — an AI writing notes from memory hallucinates. Every claim needs to trace back to a source.

This pipeline treats each of those as a distinct engineering problem with a deliberate solution, organized as five stages.

The five stages

1 · Convert — PDF/EPUB → clean markdown, 0 tokens

PyMuPDF text extraction at ~130 ms/page. Two non-obvious pieces:

• Silent-failure detection — a native text layer can lie (CID/Identity-H fonts, PUA codepoints). We score glyph-garble ratio, character density, and font risk to catch pages that "extracted fine" but didn't, and route only those to OCR.
• Column-aware reading order — line-level column clustering reconstructs true reading order on two-column layouts, with an exact-fallback path so single-column pages are byte-identical to the trivial extraction. (T2N_COLUMN_SORT=0 to disable.)
• Table pass, gated — pdfplumber's table detection is the slowest part of conversion. We gate it behind a cheap fitz pre-check (ruling-line signature incl. three-line tables, plus multilingual "Table"/表 keywords), so table-sparse books convert ~3.4× faster without missing tables. (T2N_TABLE_GATE=0 to disable.)
• Cross-page table merge — textbook tables often run past a page break. Opt in with T2N_TABLE_MERGE=1 to stitch a table that ends near a page bottom to a geometrically-matching table at the top of the next page (same column count / x-edges, no intervening heading), deduping a repeated header row and leaving a <!-- table continues from page N --> trace comment. Same exact-fallback discipline: default OFF ⇒ byte-identical output.
• Page-frame pseudo-table rejection — a page's decoration (a content frame plus the rule under the running header) gives pdfplumber enough intersecting edges to "find" a table covering the whole page body, with 1 column and every word on the page dumped into a single cell. That output is worse than a missing table: a real multi-column table arrives with its columns interleaved line by line, carrying its caption and every value, so it reads as clean citable data while the row↔column binding is destroyed. Candidates of one column whose largest cell exceeds 500 characters, or whose bbox alone covers half the page, are dropped and replaced by a <!-- ⚠️ page-frame pseudo-table rejected on page N --> comment — the page's own prose already carries the text. Measured at 9.9% of extracted tables across 128 books; on a hand-read sample of 34 pages from 10 books, 28 were genuine defects and the 6 legitimate boxed lists lost nothing (100% of their tokens are present in the page prose). Default ON — this corrects wrong output; T2N_TABLE_FRAME_REJECT=0 restores the old behaviour.
• Whole-book table-failure warning — table loss is bimodal: a book either extracts tables fine or silently loses every one. If pdfplumber parses 0 pages while fitz opens the file, or a book yields 0 tables despite ≥10 table captions, the conversion report and the markdown itself get a loud warning instead of nothing at all. Detection only — extraction is unchanged. Fires on 22 of 34 zero-table books in our corpus and on none of the 226 that extract tables. (T2N_BOOK_TABLE_CHECK=0 to disable.)

Scanned pages fall through an OCR ladder — Surya → PaddleOCR-VL → local vision model → frontier vision as the true last resort — chosen per-page, never committing a whole book to one method. See docs/ocr-ladder.md, including a hardware-tier model-selection table (no-GPU / Apple Silicon / NVIDIA 8GB / 16GB+) so you pick an engine and ollama model that actually fit your VRAM instead of OOMing mid-book.

2 · Chunk — into semantically searchable units

A table of contents is too coarse (one topic spans many chapters); manual tags don't scale and can't be granular enough. The answer is semantic embedding. But chunking is a design decision, not a fixed window: split too small and you lose context, too large and you dilute meaning. We chunk by heading structure and carry each chunk's parent-section context, so every retrieved unit is a self-contained concept with its provenance attached.

3 · Retrieve — find the right book among dozens

Cross-book search runs on the same engine as its sibling project vault-search — local LanceDB + bge-m3 embeddings, nothing leaves your machine. On top of plain similarity we add source weighting: boost your most-trusted references (exam-designated texts, official society textbooks) and down-weight by edition age, so on any topic the AI reaches for the source you trust first.

4 · Write — a note-writing algorithm, not a prompt

Note quality comes from the workflow, not the model:

• Blind draft first — the AI researches the topic fully from the textbook corpus before looking at any existing note, so an old note's structure and content can't bias the new draft. Merge comes last.
• Template-driven extraction — each topic type has a fixed template. This is load-bearing: it tells the AI exactly what to hunt for in the source, and it means every note has the same predictable shape so you read faster. Sections like a leading Summary and a Management-algorithm block exist specifically to aid comprehension, not just to hold data. The exact templates used daily are included under templates/, in both the original 繁體中文 and an English translation.
• Cite or it didn't happen — every claim carries book + chapter; anything the AI adds from its own knowledge is explicitly flagged as inferred.
• Non-destructive merge — replacing an existing note is gated on your vault being under version control; otherwise it writes a draft beside the original. It will not silently overwrite your hand-written notes.

See workflows/note-writing.md.

5 · Extract figures — the hard one

Every book lays figures out differently, so there's no single crop rule. We use a general geometric-matching method (a caption owns the nearest assignable raster) behind a deterministic QC gate: whitespace-fill, text-bleed, and OCR-long-line checks all run before any AI is allowed to judge the crop — and the gate hard-fails rather than guessing, so a wrong page yields a refusal, not a wrong figure. Everything runs locally, token-frugally.

When a specific book extracts wrong, you fix that book's logic once, and every later extraction from it is correct. This stage has been through many iterations and is still experimental — it doesn't yet handle every book, and improvement PRs are very welcome. See figures/CALIBRATION.md.

Bonus · Pluggable evidence enrichment

The note workflow has optional hook points to enrich a draft from external sources — a clinical-evidence API, a regulations/coverage database, a literature search. They live outside this repo to bound its scope; the workflow doc marks exactly where they slot in so you can wire in your own domain's sources.

The clinical-evidence hook I use daily is published separately as openevidence-tools — an OpenEvidence ask tool paired with a verifier that checks the returned citations. It takes htlin222/openevidence-mcp as its reference and inspiration; it is not a fork. Always run the verify step — cited sources can be wrong.

Designed to be deployed by an AI

You're probably here to have your AI set this up — that's the intended path:

Point Claude Code (or any capable coding agent) at this repo and say: "Read AGENTS.md and set this up for me."

AGENTS.md is written for t