by alesha-pro
97 tested hypotheses on making huge MoE LLMs fast on consumer GPUs (4x RTX 3090 + experts in RAM): what worked, what failed and why. A playbook for people and a ready skill for coding agents.
From $49 · 30 days · terms & refunds
⚠️ Third-Party Software Notice
This skill is third-party open-source software developed and hosted independently on GitHub. SkillsLLM lists this project for discovery and does not control or maintain the underlying repository.
Any security checks, ratings, or warnings displayed by SkillsLLM are automated and limited in scope. They do not constitute a security certification or guarantee that the software is safe, error-free, or free from malicious code, vulnerabilities, compromised dependencies, or prompt-injection risks.
Review the source code, permissions, dependencies, and configuration before installing or running any third-party skill. Use is at your own risk. To the maximum extent permitted by applicable law, SkillsLLM is not liable for losses arising from third-party software.
Automated security scan — not a certification.
# Add to your Claude Code skills
git clone https://github.com/alesha-pro/inference-optimization-playbookGuides for using ai agents skills like inference-optimization-playbook.
A map of real experiments on speeding up LLM inference on consumer GPUs: what was tried, on which model, what the numbers were, what the decision was and why. All paths below are relative to the directory this file is in.
The 5 offloaded models in it behaved differently. A static hot set worked on 2 and was close to random on 2 others. The cheapest way to serve a miss flipped between models on the same hardware. So there is no fixed sequence to run. Find where the user's case is on the map, then use only the results that were obtained under the same conditions.
protocol/situation-card.md for the case in front of you: model, hardware, engine and placement, workload, current numbers. Unknown fields are the first things to find out, and the card says how. It ends with a tag line such as #offload #weak-skew #cpu-lane #multi-gpu #agent.playbook/00-start-here.md. Answer its 7 questions by regime, or enter by symptom. Each answer points to 1-2 chapters. Read the first lines of a chapter: every chapter says when to read it and when to skip it.ledger/hypotheses.md by your tags to see what was already tried under your conditions, with the verdict, the reason and a "revisit when". Then open the closest case from the traits table in cases/README.md.protocol/experiment-rules.md: ceiling first, one change, controls on both sides, thresholds written in advance, a stop rule.protocol/entry-template.md, failures included. Inside this repo follow AGENTS.md.Do not load every chapter. They are 3-13 KB each and most of them will not apply to one case.
grep "#weak-skew" ledger/hypotheses.md # results on models with weak routing skew
grep "#offload" ledger/hypotheses.md | grep "#tp" # tensor parallel with experts in RAM
grep "#drafter" ledger/hypotheses.md | grep "❌" # drafter ideas that failed
grep -n -i "prefetch\|numa" ledger/hypotheses.md # by keyword
Both are standard-library Python, no install.
| Tool | Question it answers | Input |
|---|---|---|
tools/route_sim.py |
Is routing skewed? Does a static hot set beat a random one at my VRAM budget? How far is a live cache from the best possible one? | a router trace in JSON Lines |
tools/fit_cost.py |
Which term of token time is larger, the constant part or the misses? What is the ceiling with no misses? | a CSV of 3 or more runs |
tools/README.md has the formats. Both have --demo.
| The user's problem | Read |
|---|---|
| Will offload be fast at all, how many experts fit, where does token time go, what is the ceiling | playbook/01-cost-model.md |
| Misses in decode: CPU or PCIe, memory tiers | playbook/02-serving-a-miss.md |
| Which experts to keep in VRAM, profiles, domains | playbook/03-expert-profile-and-placement.md |
| Cache policy, capacity, warming, predicting experts | playbook/04-live-expert-cache.md |
| 2-4 GPUs: layer split vs TP vs EP, exchange between cards, padding | playbook/05-multi-gpu.md |
| RAM channels, NUMA, thread count, pinning | playbook/06-cpu-and-ram.md |
| Slow prompt reading, chunk size, DMA, CUDA graphs | playbook/07-prefill.md |
| Drafters, MTP, acceptance, window policy, drafter training | playbook/08-speculative-decoding.md |
| Which quant, hybrids, KL noise floor | playbook/09-quants.md |
| Dropping, masking or merging experts | playbook/10-pruning-and-merging.md |
| Agent turns are slow, prompt cache misses, A → B → A | playbook/11-sessions-and-prompt-cache.md |
| Load time, THP stalls, a table too big for RAM | playbook/12-loading-and-big-tables.md |
| Vocabulary projection cost | playbook/13-output-head.md |
| Building a reference, replay, why a benchmark lied | playbook/14-benchmarking-and-traps.md |
| Did the change keep quality, should this build ship | playbook/15-quality-and-release-gates.md |
| A new model from zero | protocol/new-model-checklist.md |
| A term is unclear | glossary.md |
#weak-skew was not tested on weak skew.Decode of an offloaded MoE costs constant + misses per token × cost of one miss. So there are 4 levers: fewer misses at the same VRAM (profile, live cache), more experts in VRAM (fewer bits, smaller buffers), a cheaper miss (CPU lane, faster RAM, better CPU code), and a smaller constant (kernels, exchange, drafter). Fit the model first and work on the larger term. Prefill is a different regime: almost every expert is needed per chunk, so the lever is bytes moved over PCIe per token. Agent workloads are a third regime: most time is generation and short re-reads, so caches and snapshots matter more than cold prefill speed.
cases/README.md.When the map has no answer, say so. Then use the closest mechanism to design the cheapest test that would answer it, run it if you can, and record the result. A new ledger row with a clear verdict and tags is the most useful contribution.
Last scanned: 11/22/2026
{
"issues": [],
"status": "PASSED",
"scannedAt": "2026-11-22T12:18:39.467Z",
"npmAuditRan": true,
"pipAuditRan": true,
"promptInjectionRan": true
}See how inference-optimization-playbook compares with popular alternatives.
A map of 115 tested ideas for making huge LLMs fast on consumer GPUs. Each one has a verdict (worked, failed, or works only under conditions), the numbers, the reason, and the conditions it was tested under.
It comes from running DeepSeek V4.1 Flash, Qwen3.8-Flash-Next, GLM-5.3-Flash, MiniMax-M3.1-Flash and Tencent Hy4 Flash on 4x RTX 3090 with experts offloaded to RAM, and Inkling 2 Mini on a single RTX 3090 with every expert in VRAM. The big models do not fit in 96 GB of VRAM at the quants we wanted. Engines: Strata, Maya forks, an exllamav3 fork, llama.cpp, vLLM, most of them patched by hand.
| Model, setup | What changed | Before → after |
|---|---|---|
| DeepSeek V4.1 Flash, 4 cards | DMA from pinned RAM for every tensor format, layer skip on the prompt, auto chunk | prefill 32K: 430 → 2,199 tok/s |
| DeepSeek V4.1 Flash, CPU path on Zen 2 | 1 build flag (PDEP runs in microcode there) | 1 cold expert: 1.6 → 0.44 ms |
| Hy4 Flash, 1 card | 1 load of experts per block of up to 57,600 tokens, attention in tiles of 2,048 (full build) | prefill of 57,587 tokens: 415.6 → 1,510.2 tok/s |
| Hy4 Flash, 2 cards | cold experts computed on the CPU in parallel with hot ones on the GPU | decode of short code: 24.6 → 70.4 tok/s |
| Qwen3.8-Flash-Next, 1-2 cards | DFlash2 drafter | decode +44-46% |
| Qwen3.8-Flash-Next, Russian | a Cyrillic draft vocabulary, no training | draft acceptance 32% → 54% |
| MiniMax-M3.1-Flash | attention K/V gathered into VRAM once per chunk | prefill 64K ×3.3 |
| GLM-5.3-Flash at 2.05 bpw, 4 cards | tensor parallel instead of layer split | 48.3 → 66-68 tok/s |
| Inkling 2 Mini, 1 card | every expert in VRAM, a token as 1 CUDA graph, NVFP4 experts and attention on the tensor cores | decode 57 (llama.cpp) → 128.7 tok/s, 344K context |
And what did not work, which is most of the value:
Inkling 2 Mini shows the other regime: once everything fits one card, the cost is kernel launches and memory latency, not placement (case).
The 5 offloaded models behaved differently. The cheapest way to serve a miss flipped between models on the same hardware, tensor parallel lost on one and won on another, a long drafter window paid on skewed routing and not on flat routing. A fixed list of steps would be wrong for half of them.
So the playbook is built for orientation. You describe your case, find its place, and see only the results that were obtained under the same conditions.
It is written for 2 readers:
The numbers are from one rig. Yours will be different. The mechanisms are the same on any box with GPUs, a CPU and RAM, and chapters 01-15 each end with a checklist for moving the idea to another model.
| You want to | Open |
|---|---|
| Describe your case and get its tags | protocol/situation-card.md |
| Find your place by regime or by symptom | playbook/00-start-here.md |
| Check if an idea was already tried under your conditions | ledger/hypotheses.md |
| See how it went on a model like yours | cases/ |
| Run an experiment so the result means something | protocol/experiment-rules.md |
| Bring up a new model from zero | protocol/new-model-checklist.md |
| Look up a term | glossary.md |
playbook/ 16 chapters by mechanism
00 the map: entry by regime and by symptom, 15 findings
01 cost model: time = constant + misses × cost of a miss, ceilings
02 where to serve a miss: CPU lane vs PCIe copy
03 hot experts: profile, domain, placement
04 live expert cache
05 several GPUs: layer split, TP, EP, all-reduce
06 CPU and RAM: channels, NUMA, threads
07 prefill and re-reading the prompt
08 speculative decoding
09 quants: bits vs experts in VRAM
10 pruning and merging experts
11 sessions, KV and the prompt cache in agent work
12 loading, page cache, THP, big tables on NVMe
13 the output head
14 how to benchmark, and the traps
15 quality and release gates
ledger/ every hypothesis: verdict, numbers, why, "revisit when", condition tags
cases/ the rig, a traits table and 6 model case files
protocol/ situation card, experiment rules, new-model checklist, entry template
tools/ route_sim.py (skew and cache simulator), fit_cost.py (time model fit)
SKILL.md entry point for agents
AGENTS.md rules for editing this repo
The repo root is a skill directory: SKILL.md, plain markdown and 2 standard-library Python scripts.
Clone it where your agent looks for skills, for example:
git clone https://github.com/alesha-pro/inference-optimization-playbook ~/.claude/skills/inference-optimization-playbook
or into .agents/skills/ or .claude/skills/ of a project. Then ask the agent things like:
Without skills support, point the agent at the repo and tell it to read SKILL.md first.
The agent fills a situation card, gets a tag line such as #offload #weak-skew #cpu-lane #multi-gpu #agent, and filters the ledger with it:
grep "#weak-skew" ledger/hypotheses.md
python3 tools/route_sim.py --demo uniform --fraction 0.3
A benchmark harness, data on concurrent load, and hardware other than our rig. The tools here help to orient, they do not launch or time a server.
This is a living notebook. New models and new results are added as dated entries. When a later check changes a decision, the old result stays next to the new one with the date, so the reason stays visible.
Every number says where it came from: a real run, a simulator on real traces (sim), a calculation (calc), a microbenchmark (micro) or someone else's number (ext).
Tested something from here on other hardware and got a different result? Open an issue with the entry template filled in.
By @superalesha. MIT license.
inference-optimization-playbook is an open-source ai agents skill for AI coding assistants such as Claude Code, Codex CLI, and ChatGPT, built by alesha-pro. 97 tested hypotheses on making huge MoE LLMs fast on consumer GPUs (4x RTX 3090 + experts in RAM): what worked, what failed and why. A playbook for people and a ready skill for coding agents. It has 58 GitHub stars.
Yes. inference-optimization-playbook passed SkillsLLM's automated security scan — a dependency vulnerability audit plus prompt-injection heuristics — with no high-severity issues. You can read the full report in the Security Report section on this page.
Clone the repository with "git clone https://github.com/alesha-pro/inference-optimization-playbook" and add it to your Claude Code skills directory (see the Installation section above). inference-optimization-playbook ships a SKILL.md manifest, so compatible agents can discover and load it automatically.
inference-optimization-playbook is primarily written in Python. It is open-source under alesha-pro on GitHub, so you can review or fork the full source.
Yes. SkillsLLM lists many other AI Agents skills you can browse and compare side by side. Open the AI Agents category from the badge at the top of this page, or use the Related Skills and comparison links further down to weigh inference-optimization-playbook against similar tools.
No comments yet. Be the first to share your thoughts!