Transform YouTube videos into a compounding knowledge base with transcripts, vision analysis, and agentic search. Works as an MCP server for Claude, Copilot & more.
YouTube video knowledge engine — transcripts, vision, and persistent wiki.
mcptube-vision transforms YouTube videos into a persistent, structured knowledge base using both transcripts and visual frame analysis. Built on the Karpathy LLM Wiki pattern: knowledge compounds with every video you add.
Evolved from mcptube v0.1 — mcptube-vision replaces semantic chunk search with a persistent wiki that gets smarter with every video ingested.
🧠 How It Works
Traditional video tools re-discover knowledge from scratch on every query. mcptube-vision is different:
mcptube v0.1 mcptube-vision
┌─────────────────────┐ ┌─────────────────────────┐
│ Query → vector search│ │ Video ingested → LLM │
│ → raw chunks → LLM │ │ extracts knowledge → │
│ → answer (from scratch│ │ wiki pages created → │
│ every time) │ │ cross-references built │
└─────────────────────┘ │ │
│ Query → FTS5 + agent │
│ → reasons over compiled │
│ knowledge → answer │
└─────────────────────────┘
| | v0.1 (Video Search Engine) | vision (Video Knowledge Engine) |
|---|---|---|
| | Chunk transcript, embed in vector DB | LLM watches + reads, writes wiki pages |
| | Find similar chunks | Agent reasons over compiled knowledge |
| | Timestamp or keyword extraction | Scene-change detection + vision model |
| | Re-search all chunks each time | Connections already in the wiki |
| | Library of isolated videos | Compounding knowledge base |
First-time install walkthrough for Claude Code, Codex CLI, and ChatGPT.
On ingest
On query
Frames
Cross-video
Over time
🏗️ Technical Architecture
mcptube-vision is built around a core insight: video knowledge should compound, not be re-discovered. Every architectural decision flows from this principle.
The system overview shows three distinct subsystems connected by a unidirectional data flow. The Ingestion Pipeline (left) transforms a raw YouTube URL into structured knowledge through four stages: transcript extraction, scene-change frame detection, vision-model description, and LLM-powered knowledge extraction. Each stage enriches the signal — raw video becomes text, text becomes typed knowledge objects.
The Knowledge Store (center) is the persistent layer. The WikiEngine applies merge semantics — deciding whether to create new pages or append to existing ones — then writes JSON files to disk and updates the FTS5 search index in parallel. These two stores serve different access patterns: files for full-page reads and exports, FTS5 for sub-millisecond keyword retrieval.
The Retrieval layer (right) combines both stores. The Ask Agent first narrows via FTS5, then loads full pages from disk, and finally reasons over candidates with structural awareness from the wiki TOC. The CLI and MCP Server sit alongside as thin presentation layers — they never contain business logic.
The ingestion flow is a write-once pipeline — LLM-heavy at ingest time, but never repeated for the same video. This is the key cost tradeoff: invest tokens upfront to build compiled knowledge, so retrieval is cheap.
The sequence shows two critical branching points. First, after transcript extraction, the pipeline forks into vision processing (scene frames → LLM vision descriptions) and feeds both streams into the WikiExtractor. This dual-signal approach means the LLM sees both what was said and what was shown — critical for content like coding tutorials or slide-based lectures where the transcript alone misses visual information.
Second, the WikiEngine merge step is where knowledge compounding happens. Rather than blindly writing new pages, it checks for existing entities, topics, and concepts — appending new video contributions to existing pages and rewriting synthesis summaries. This is why ingesting video #10 makes the wiki smarter about videos #1–9 too: shared concepts get richer synthesis with each new source.
The final FTS5 index update runs synchronously after the file write, ensuring search consistency. There is no eventual-consistency window — once add_video returns, all new knowledge is immediately searchable.
The retrieval flow is deliberately two-stage to balance cost and intelligence. The first stage — FTS5 keyword search — runs entirely locally with zero LLM tokens, narrowing thousands of wiki pages to a ranked handful in milliseconds. Query sanitization strips special characters (e.g. ?, !) that would break FTS5 syntax, ensuring robustness for natural-language questions.
The second stage loads two types of context for the agent: the candidate pages (full detail — summaries, contributions, entity references) and the wiki TOC (a compact structural map of all knowledge). The TOC is critical — it gives the agent awareness of what it doesn't know. Without it, the agent would hallucinate answers from weak matches. With it, the agent can reason: "The wiki has pages on RLHF and scaling laws, but nothing on quantum computing — so I should say I don't have that information."
In CLI mode (BYOK), the agent is an LLM call that synthesizes the final answer with source citations. In MCP server mode (passthrough), this stage returns the raw candidates and TOC to the client — letting the client's own model (Copilot, Claude, Gemini) do the reasoning. This dual-mode design means the server never requires an API key when used via MCP.
Subsystem Breakdown
1. Ingestion Pipeline
YouTubeExtractor pulls transcript segments via youtube-transcript-api and video metadata via yt-dlp. Transcripts are chunked by natural segment boundaries, not fixed token windows — preserving semantic coherence.
SceneFrameExtractor uses ffmpeg's perceptual scene-change filter (select='gt(scene,{threshold})') rather than fixed-interval sampling. This is deliberate: fixed intervals waste tokens on static frames (slides held for 30s), while scene-change detection captures transitions — the moments of highest information density. The threshold (default 0.4) is configurable.
VisionDescriber sends detected frames to a vision-capable LLM (GPT-4o, Claude, Gemini — auto-detected via API key priority). Frame descriptions are plain prose, not structured JSON, to maximise the LLM's descriptive latitude.
Why this matters: A transcript of a coding tutorial misses the code on screen. Scene-change vision capture recovers that signal without the token cost of dense fixed-interval sampling.
The agent harness performance optimization system. Skills, instincts, memory, security, and research-first development for Claude Code, Codex, Opencode, Cursor and beyond.
Claude Code is an agentic coding tool that lives in your terminal, understands your codebase, and helps you code faster by executing routine tasks, explaining complex code, and handling git workflows - all through natural language commands.
