Agent skill for hill-climbing local LLM serving speed (decode, speculative decoding, prefill, KV, startup, TTFT) on any hardware and quantization
Available after the catalog security scan passes.
⚠️ Third-Party Software Notice
This skill is third-party open-source software developed and hosted independently on GitHub. SkillsLLM is an informational directory and does not control or maintain the underlying repository.
Any security checks, ratings, or warnings displayed by SkillsLLM are automated and limited in scope. They do not constitute a security certification or guarantee that the software is safe, error-free, or free from malicious code, vulnerabilities, compromised dependencies, or prompt-injection risks.
Review the source code, permissions, dependencies, and configuration before installing or running any third-party skill. Use is at your own risk. To the maximum extent permitted by applicable law, SkillsLLM is not liable for losses arising from third-party software.
The deep catalog scan for this skill is still queued. Run an instant dependency check now instead.
# Add to your Claude Code skills
git clone https://github.com/sybil-solutions/inference-optimisation-skillGuides for using ai agents skills like inference-optimisation-skill.
See how inference-optimisation-skill compares with popular alternatives.
An agent skill for hill-climbing the speed of a local LLM server on any hardware and any quantization. It covers:
It works under a memory cap, and every result has to survive a full-context stability check.
It encodes how we actually got these speedups. It is not a list of flags. The workflow:
It works with Claude Code and any agent that loads SKILL.md-style skills. Any model can also follow it from PROMPT.md.
Results from the hill-climbs this skill was distilled from. Every number was measured on the hardware named; see the linked sources for methods and raw data.
| Hardware | Model / quant / engine | Before → after |
|---|---|---|
| Intel Arc B390 iGPU (laptop, 64 GB unified LPDDR5X, ~115 GB/s) | Ornith-1.5-35B-A3B, EXL3 4 bpw, SGLang XPU + MTP | Decode, 1 user: 43.6 → 53 tok/s (median)Decode, 4 users: 84.7 → 103 tok/sPrefill 32K: 1,037 → 1,660 tok/sPrefill 256K: 297 → 628 tok/s (2.1×)KV cache: 264K tokens → 4 × 256KStartup: 113 → 43 sRegistry Lab gates: decode 28.6 → 41.0, prefill 255 → 667 |
| Intel Arc Pro B70 | Qwen3.8-27B, EXL3 4 bpw, vLLM + custom XPU kernels | Decode: 91.2 tok/s at 1 user, 365 at 16 (prose, thinking on)Same card with llama.cpp Q4_K_M: 25.0Prefill 4K/32K/128K/254K: 1,654/1,434/928/650 → 2,421/2,259/1,415/911Agent-session replay: 193 → 147 s |
| Intel Arc Pro B70 | Same model, SGLang + the same kernels | Decode, 1 user: 82.6 tok/s, vs vLLM 75.7 on the same card and day |
| RTX 3090 | Qwen3.8-27B EXL3 3 bpw + MTP | 96.2 tok/s prose / 141.1 code, 1 user |
| RTX 3090 | Qwen3.6-35B-A3B EXL3 3 bpw + MTP | 251.7 / 357.9 tok/s, 1 user |
| RTX 3090 | Flash-Next EXL3, 256K context | MTP: 45.5 → 65.6 tok/s |
| 4× H200 NVL | DeepSeek-V4.1-Flash, vLLM + speculative decoding | Crashed under load → 229.1 tok/s decode, 17,546 tok/s prefill; all six gates pass |
Sources:
xpu/README.md, xpu/docs/PROGRESS.md and xpu/docs/SGLANG.md for B70cuda/README.md for the RTX 3090That run is the one this skill grew out of. Its ceiling calculation said:
The skill makes the agent compute this up front, so it can tell you early when a target is above the physics.
┌────────────────────────────────────────────────────────────────┐
measure HW ─▶ │ THEORY.md: bytes/token ÷ bandwidth, FLOPs/token ÷ 50% peak, KV │
└────────────────────────────────────────────────────────────────┘
│
baseline ─▶ profile ─▶ pick lever (ceiling gap × likelihood) ─▶ configs/NEW.json
▲ │
│ queue.sh ctl NEW ctl (serialized, AC-only,
│ memory watchdog, ≥6 samples, medians)
│ │
PLAN.md + ledger.jsonl ◀── accept only if it beats BOTH neighbouring controls ──┘
│ and regresses nothing by >3%
▼
long-context needle gate ─▶ concurrent full-KV soak ─▶ plateau? ─▶ pinned clean build
─▶ acceptance gates ─▶ publish
The skill has three parts:
SKILL.md: the procedure.
references/: loaded only when needed.
ceilings.md: how to compute decode, prefill, TTFT, startup and KV maxima, with a worked example.methodology.md: measurement traps. Chunk-counted tokens, mis-sized prompts, acceptance counters, drift, unified-memory accounting.levers.md: every lever with measured evidence. It is tagged general vs stack-specific.gotchas.md: known traps in vLLM, SGLang, llama.cpp and ExLlama, and on Intel, NVIDIA, Apple and AMD hardware.publishing.md: pinning, clean builds, the six acceptance gates, proving the build matches the best, results cards.scripts/: a small, engine-agnostic harness. It is standard-library Python plus bash.
bench.py benchmarks any OpenAI-compatible server. It measures:
It flags looping output, failed requests and runs on battery. Token counts come from server usage stats, never from streamed chunks, and everything lands in ledger.jsonl.
ceiling.py computes the theoretical ceilings from a JSON of measured bandwidth and FLOPs plus component sizes.
queue.sh is a serialized, detachable run queue, for interleaving control/candidate/control runs.
mem.sh reports a container's real footprint. That includes driver residency on unified-memory GPUs, which cgroups don't see.
Claude Code (user-level):
git clone https://github.com/sybil-solutions/inference-optimisation-skill
mkdir -p ~/.claude/skills
cp -r inference-optimisation-skill/skills/inference-optimisation ~/.claude/skills/
Then ask, for example:
Other agents: point them at skills/inference-optimisation/SKILL.md, or paste PROMPT.md as the task prompt.
S=skills/inference-optimisation
python3 $S/scripts/ceiling.py $S/examples/ceiling-ornith-b390.json # ceilings
mkdir -p configs && cp $S/examples/config-existing-server.json configs/base.json # edit base_url/model
python3 $S/scripts/bench.py run configs/base.json --long 131072 # one run
bash $S/scripts/queue.sh base cand base # interleaved A/B
python3 $S/scripts/bench.py show # ledger table
PLAN.md is the list of things that did not work and why.PRs with new levers, gotchas or results are welcome. Each needs the hardware, model, quantization, engine version, before → after numbers, how they were measured, and a link to raw data.
MIT
inference-optimisation-skill is an open-source ai agents skill for AI coding assistants such as Claude Code, Codex CLI, and ChatGPT, built by sybil-solutions. Agent skill for hill-climbing local LLM serving speed (decode, speculative decoding, prefill, KV, startup, TTFT) on any hardware and quantization. It has 83 GitHub stars.
inference-optimisation-skill's catalog security scan is still queued. You can run an instant dependency and prompt-injection check now with the "Scan for vulnerabilities" button above.
Clone the repository with "git clone https://github.com/sybil-solutions/inference-optimisation-skill" and add it to your Claude Code skills directory (see the Installation section above).
inference-optimisation-skill is primarily written in Python. It is open-source under sybil-solutions on GitHub, so you can review or fork the full source.
Yes. SkillsLLM lists many other AI Agents skills you can browse and compare side by side. Open the AI Agents category from the badge at the top of this page, or use the Related Skills and comparison links further down to weigh inference-optimisation-skill against similar tools.
No comments yet. Be the first to share your thoughts!