lecture-to-notes
English · 繁體中文
Turn a lecture or conference recording into structured, slide-illustrated notes — and into a synced HTML viewer where the video, the timestamped transcript, and the curated summary sit on one page: the video highlights the matching note as it plays, and clicking any note timestamp seeks the video.
Every expensive stage runs locally: Whisper ASR on your GPU, frame extraction, OCR, and a local vision model for slide semantics. An LLM is used only at the end, to write prose from evidence the pipeline already assembled.
The design goal is not "summarize a video". It is traceability: every claim in the finished note should be attachable to a moment in the transcript and to the slide that was on screen at that moment. Most of the machinery here exists to make that link trustworthy rather than plausible.
Why this exists
Rehabilitation-medicine courses are notoriously hard to take notes on. Manual therapy, ultrasound scanning — the knowledge is in the motion, so at every course you see a forest of tripods: everyone records everything, planning to rewatch over dinner. Nobody actually rewatches hours of video.
With AI the first instinct was to teach it precise screenshotting, to squeeze the video back into a traditional text-plus-figures note. The turn was realizing a note doesn't have to be that shape at all: build a webpage that ties the video to the transcript, and you can jump straight to the moment you care about. What you actually want to know is how the maneuver is performed — and that only lives in motion. So the summary is for studying, the transcript is for verifying, and the video is one click away from both.
Bring whatever you captured
Real courses never produce one tidy recording. Some talks you filmed, some you
only audio-recorded, for some you photographed the slides with your phone, and
sometimes the organizer hands out a PDF deck afterwards. This pipeline takes
the folder as-is: route_inputs.py figures out what role each file plays,
and the alignment stage stitches every source onto one shared timeline —
so the finished note cites the right slide at the right moment even when that
slide never appeared inside the video.
What it does
flowchart TD
IN["material folder<br/>(video / audio / PDF deck / slide images)"]
IN --> R["route_inputs.py<br/><i>classifies the folder, prints the plan</i>"]
R --> T["transcribe_video.py<br/>faster-whisper, local GPU"]
R --> S{"slide source"}
S -->|video only| A["extract_slides.py<br/>frames + perceptual dedup"]
S -->|PDF deck| P["build_slides_from_pdf.py<br/>page render + embedded text"]
S -->|loose images| I["build_slides_from_images.py"]
A --> B["quick_ocr.py<br/>Stage B — cheap OCR triage"]
B --> C["dedup_semantic.py<br/>Stage C — canonical slides"]
C --> B2["ocr_surya.py<br/>Stage B2 — high-quality OCR"]
P --> D
I --> D
B2 --> D["vlm_signals.py<br/>Stage D — slide semantics (local VLM)"]
D --> E["ground_slides.py<br/>Stage E — tie slides to spoken words"]
T --> E
E --> F["Stage F — synthesis (LLM)<br/>tier pass, then write pass"]
F --> O["render_embeds.py → finalize_to_vault.py → audit_note.py"]
F --> W["export_web.py<br/><b>synced HTML viewer</b> — video + transcript + summary on one page"]
Stages A–E are plain Python and cost zero LLM tokens. They produce
slides_grounded.json: for each canonical slide, its OCR text, its semantic
signals, and the transcript segments spoken while it was on screen. That file is
the input to synthesis, and it is also readable on its own — if you never run
Stage F you still have a transcript, a deduplicated slide set, and the mapping
between them.
The HTML viewer: video, transcript and summary on one page
The signature output. export_web.py builds a single self-contained HTML
page per course where the video, the timestamped transcript, and the curated
summary notes are presented together and kept in sync both ways:
- Video → notes: as the video plays, the matching note bullet auto-highlights and scrolls into view.
- Notes → video: every bullet carries a
(Vn MM:SS)timestamp — click it and the video seeks to that moment. - Three read modes: summary only, transcript only, or both side by side — the summary layer (curated, workflow-ordered) and the transcript layer (verbatim, time-ordered) are separate chapters of the same timeline, so you can study top-down and verify bottom-up without leaving the page.
- Resizable panes, sidebar segment navigation, slide images inline where they were shown.
- Offline and shareable: one
.htmlplus one support folder (browser-ready clips, friendly-named slides, markdown + PDF copies). No server, no build step for the reader — send the folder, they double-click the page.--compressproduces a smaller H.264 set for handing around.
Real output from a cervical-ultrasound workshop (dates and speaker names blurred):
| Course hub — every segment with duration and a one-line hook | Summary view — per-video source table and key pearls |
|---|---|
 |
 |
The viewer UI lives in scripts/layout2/ (viewer.css, viewer.js);
export_web.py only generates the per-course timeline manifest and the synced
note HTML, so a UI tweak is an asset edit, not a generator change.
Markdown notes for your vault
The HTML viewer is the reading surface; plain markdown is the storage format, so everything also lands in your note vault:
finalize_to_vault.pyships the finished note plus its cited slide images into an Obsidian-style vault (attachment folder and inbox are configurable flags — any folder of markdown works).- The viewer's support folder contains a
markdown/copy of every transcript and summary, wikilink-rewritten so the folder itself opens directly as an Obsidian vault — same content as the webpage, readable and searchable in your normal note workflow, plus PDF copies for people who want neither.
So you get three durable forms of the same lecture: the synced webpage for studying, markdown for your knowledge base, PDF for handing to anyone.
Alignment: capture time is a hypothesis, cross-correlation is evidence
The part worth stealing, if you take nothing else.
When a talk is captured by more than one device — a room recording plus phone clips plus photos — you need a shared timeline. The obvious approach is to trust each file's capture timestamp. That approach is wrong often enough to matter: phone clocks drift, some files only have an mtime, and a recording that was stopped and restarted lies about its own start.
So this pipeline separates the two:
media_capture_index.pyreads capture times and emits them as claims, each with areliableflag. A source whose start came from mtime, or has no start at all, is marked unreliable and must not be aligned on.xcorr_media_offsets.pymeasures the actual offset by cross-correlating transcripts of the overlapping audio. That is evidence.- When claim and measurement disagree by more than 5 seconds, the pipeline sets
"conflict": trueand stops. It never auto-corrects.
The failure this prevents is a 44-minute misalignment that looks completely normal in the output, because every downstream stage faithfully processes the wrong pairing. A pipeline that silently reconciles conflicting evidence produces confident garbage; one that flags the conflict produces a question.
Requirements
Honest version:
- Tested on Windows 11 with an NVIDIA RTX 3070 Ti (8 GB). Nothing is Windows-specific by design, but Linux/macOS are untested.
- Python 3.12
- An NVIDIA GPU (8 GB+) is the fast path, not a requirement. Without one:
- CPU fallback is built in — transcription drops to int8 on CPU automatically and prints an honest ETA first (a 60-minute lecture takes hours instead of minutes). The OCR triage stage is CPU-friendly already.
- Offload transcription to a hosted Whisper —
scripts/groq_asr.pysends compressed audio to Groq'swhisper-large-v3-turbo(free tier works; the 25 MB request cap is handled by chunking) and returns the same segment format as the local path. Read its docstring first: hosted Whisper has no anti-collapse knobs for heavily code-switched audio, and anything you send leaves your machine — don't route confidential recordings there. - Apple Silicon should work in CPU mode (faster-whisper on CPU, ollama is native on macOS) — plausible but untested; reports welcome.
ffmpegandffprobeon PATH — not optional.- Optional: ollama with
minicpm-v:8bfor Stage D slide semantics; a separate venv with Surya for high-quality OCR;pandocfor the web/PDF export.
On an 8 GB card, GPU stages must be serialized — Whisper and the VLM cannot run
concurrently, and frame extraction must not run during transcription. The
measured sweet spot for transcription is --batch-size 3 --beam-size 10;
--beam-size 15 with sequential mode crashes.
Quickstart
git clone https://github.com/drpwchen/lecture-to-notes
cd lecture-to-notes
pip install -r requirements.txt
pip install -r requirements-optional.txt # recommended
cp config.example.yaml config.yaml # every value is