llama-modchip
by synchopate
♟️ llama-modchip v2.1: Run 30B Sparse MoE LLMs on Pure CPU (0 GPUs) at 20–60.4 tok/s with a 256K Context Window via a 1.64MB Stockfish-PVS + RouteCrack L3 Drafter, 2MB HugePages & AVX-512 VNNI
71stars
11forks
Python
Added 11/3/2026
Available after the catalog security scan passes.
⚠️ Third-Party Software Notice
This skill is third-party open-source software developed and hosted independently on GitHub. SkillsLLM is an informational directory and does not control or maintain the underlying repository.
Any security checks, ratings, or warnings displayed by SkillsLLM are automated and limited in scope. They do not constitute a security certification or guarantee that the software is safe, error-free, or free from malicious code, vulnerabilities, compromised dependencies, or prompt-injection risks.
Review the source code, permissions, dependencies, and configuration before installing or running any third-party skill. Use is at your own risk. To the maximum extent permitted by applicable law, SkillsLLM is not liable for losses arising from third-party software.
The deep catalog scan for this skill is still queued. Run an instant dependency check now instead.
AI Agentsapple-siliconavx512claude-codecpu-inferenceggufhpcllama-cpplocal-llmmamba2mixture-of-expertsnemotronspeculative-decodingstockfishzen5
Maintainer of this skill?
Installation
# Add to your Claude Code skills
git clone https://github.com/synchopate/llama-modchipGetting Started
Guides for using ai agents skills like llama-modchip.
- Caveman: Cut Claude Token Use by 65%How agent-side prompt compression works, when to use it, and when not to.
- How to Test an AI Skill: A Practical Guide to EvalsEvaluate skills with realistic fixtures, a baseline, and repeatable checks.
- Getting Started with AI SkillsFirst-time install walkthrough for Claude Code, Codex CLI, and ChatGPT.
Compare with similar skills
See how llama-modchip compares with popular alternatives.
♟️ llama-modchip
Run 30B Sparse MoE LLMs on a pure CPU at 20–80 tok/s with a 256K context window in 14.35 GiB. Zero GPUs required.
-d29922?style=flat-square){kind=icon}
⚡ Quickstart · 📊 Benchmarks · 🧠 How It Works · 🏆 CPU Leaderboard · 📄 Technical Paper · ❓ FAQ
⚡ Quickstart
Clone the repository and run ./modchip chat. On first run, it automatically builds the patched llama.cpp engine (AVX-512 / AVX2 / Apple Metal), downloads Nemotron-3.5-30B-A3B (Q4_0), launches the background server + Web UI, and opens an interactive chat with live tok/s telemetry:
git clone https://github.com/synchopate/llama-modchip.git
cd llama-modchip
./modchip chat
All ./modchip Commands
| Command | What It Does |
|---|---|
./modchip chat |
💬 Interactive terminal chat with live tok/s, TTFT, and Stockfish-PVS acceptance footer |
./modchip web |
🌐 Opens the zero-dependency Dark-Mode Web Chat & Live Telemetry HUD (http://127.0.0.1:8787) |
./modchip bench |
📊 Runs a 30-second 2-stage speed test and prints a shareable CPU Benchmark Scorecard |
./modchip claude |
🤖 Launches Claude Code CLI (40 tools) wired to the local 256K CPU engine (18.9 ms warm restore) |
./modchip start [model.gguf] |
🚀 Starts the server + gateway in the background (supports Bring-Your-Own-GGUF for Qwen3-30B-A3B, Llama-3, etc.) |
./modchip status / ./modchip stop |
🔍 Inspects CPU readiness, 2MB HugePages, and /dev/shm/llama_modchip.bin, or stops services |
💡 Want to estimate your CPU's speed in 2 seconds without downloading anything?
curl -sL https://raw.githubusercontent.com/synchopate/llama-modchip/main/scripts/check_cpu_readiness.py | python3
📊 Benchmarks (Pure CPU — Zero GPUs)
Measured on an AMD Ryzen 9 9955HX (16C/32T Zen 5, 64 MB L3, 64 GB DDR5, 0 GPUs) running NVIDIA-Nemotron-3.5-30B-A3B-Q4_0:
Workload (22K – 256K Context) |
Standard llama.cpp Baseline |
llama-modchip v2.1 |
Improvement |
|---|---|---|---|
| Max Usable Context Window (64 GB RAM) | ~14,000 tokens (Timeout / OOM) | 262,144 tokens (256K) (only 432 MB KV cache) |
18.7x Larger |
| Cold Prompt Prefill Speed | < 20.0 tok/s | 165.2 – 241.0 tok/s (Peak: 306.8 tok/s) |
~12x Faster |
Warm Turn Latency (22.2K 40-Tool Prefix) |
130.8 s (Full re-eval) | 18.9 ms restore / 0.40 s TTFT |
~6,800x Faster |
| Decode — Technical Prose & New Code | ~4.5 tok/s | 18.8 – 20.9 tok/s (Peak 39.3 tok/s, 85.0% acc) |
4.5x Faster |
Decode — Agent <tool_call> & Mixed Refactor |
~4.5 tok/s | 28.6 – 34.5 tok/s ( Peak 48.5 tok/s, 86.1% acc) |
~7x Faster |
Decode — In-Context Refactoring (Stockfish M=7) |
~4.5 tok/s | 48.39 tok/s avg / 60.40–65.0 tok/s peak (mean len = 6.90) |
>11x Faster |
♟️ Why a 1.64 MB Stockfish Chess Engine Beats a 1B Neural Draft Model on CPU
On GPUs (3.35 TB/s HBM3), running a 1B neural draft model takes < 1 ms. On a CPU (71.7 GB/s Dual-Channel DDR5), 1B neural draft models fail catastrophically: streaming 600 MB of 1B draft weights for 5 steps reads 3.0 GB from RAM (40 ms) — slower than running the 30B Sparse MoE target model itself (1.8 GB active $\rightarrow 25\text{ ms}$), while completely flushing the 64 MB L3 cache!
Speculative Drafter on CPU (Nemotron-30B-A3B) |
Footprint | DDR5 Read / Draft | Draft Latency | L3 Cache Impact | Live 25K-Ctx Refactoring Speed |
|---|---|---|---|---|---|
1B Neural Draft Model (Q4_0) |
~600 MB |
3,000 MB |
~40.0 ms |
🔴 100% Flushed | ~11.5 tok/s (Slower than no speculation!) |
Vanilla llama.cpp ngram-simple |
< 1 MB |
0 MB |
0.005 ms |
🟢 0% |
~16.2 tok/s (41% wasted MoE reads + rollbacks) |
RouteCrack Static Cascade (M=5) |
~14 MB |
0 MB |
0.002 ms |
🟢 < 20% |
40.91 tok/s avg / 45.43 tok/s peak (len 5.47) |
♟️ RouteCrack + Stockfish-PVS v2 (M=7) |
1.64 MB TT |
0 MB (L2/L3) |
< 0.005 ms |
🟢 2.5% of 64MB L3 |
48.39 tok/s avg / 60.40 tok/s peak (96.4% acc, len 6.90!) |
🗜️ Block-64 2-Level Hierarchical Non-Linear MoE Quantization (17.59 GiB $\rightarrow$ 14.35 GiB, +28–33.5% Faster, 100/100 Quality Gate)
Standard 256-block K-quants and I-quants (Q3_K, IQ3_S) fail on Nemotron-3.5-30B-A3B because its MoE expert dimensions (1856 = 29 × 64, 2688 = 42 × 64) are not divisible by QK_K = 256, forcing llama-quantize to fall back to Q4_0. In v2.2.0, scripts/build_fractal_moe_gguf.sh introduces Block-64 2-Level Hierarchical Non-Linear Quantization (Fractal-64 / q2_0, 30 bytes per 64 weights = 3.75 bpw) and Asymmetric MXFP4 (4.25 bpw) with _MM_HINT_NTA cache-bypass streaming (full report & raw JSON logs in benchmarks/FRACTAL_MOE_QUALITY_GATE.md and benchmarks/results/):
Metric / Suite (benchmarks/eval_quality_and_tools.py) |
Baseline-Q4_0 (Official GGUF) |
Fractal-MXFP4 (from Q4_0) |
Fractal-64-BF16 (from BF16 58.84 GiB) |
Fractal-64-BF16 vs Baseline |
|---|---|---|---|---|
| GGUF File Size on Disk | 17.59 GiB (18,015 MiB) |
15.95 GiB (16,328 MiB) |
14.35 GiB (14,698 MiB) |
-3.24 GiB (-18.4%) |
MoE Expert Bitrate & SQNR (vs BF16) |
4.50 bpw (21.02 dB) |
4.25 bpw (18.51 dB) |
3.75 bpw (17.78 dB) |
+1.53 dB above Q3_K (16.25 dB) |
Quality, Tool-Calling, Code & 8.5K Needle (80 pts) |
80.0 / 80.0 |
80.0 / 80.0 |
80.0 / 80.0 |
100.0% parity (100/100 Total) |
Single-Turn Decode Throughput (tok/s) |
19.54 tok/s |
26.08 tok/s |
25.02 tok/s |
+28.0% (+33.5% on MXFP4) |
Idle Resident RAM (VmRSS, 256K Context) |
21.41 GiB |
17.46 GiB |
15.40 GiB |
-6.01 GiB (-28.1% RAM saved) |
Post-Benchmark Peak VmRSS (256K Context) |
21.71 GiB |
18.86 GiB |
17.27 GiB |
-4.44 GiB (-20.5% RAM saved) |
🧠 How It Works
We combine 5 core innovations (backed by 10 low-level C++/kernel patches documented in ARCHITECTURE.md and speculative/ROUTECRACK_TECHNIQUES.md):
- ♟️
RouteCrack + Stockfish-PVS v2(1.64 MBChess-Engine Search in L3 Cache)
RouteCrackacts as an $O(1)$ candidate move generator (pruning 131,072 vocabulary tokens to 4–8 candidates via an 8-level FNV-64 Hash Cascade, Online Session Trie, and 11,382-entry Macro Codebook).StockfishDrafter(rc_stockfish.h) acts as an orthogonal live-session tactician using Principal Variation Search (PVS) and 4-Way Bucketed Zobrist Transposition Tables (tt5/tt4/tt3), unlocking Singular Horizon Extensions up toM=7tokens/step (98.74%marginal precision at $D=5..7$). - 🗜️ Block-64 2-Level Hierarchical Non-Linear MoE Quantization (
Fractal-643.75 bpw&MXFP44.25 bpw)
Shrinks the 46 sparse MoE expert layers to3.75 bpw(30 bytesper 64 weights:2BFP16 super-scale +4Bshifted{4..19}4-bit sub-scales across8×8micro-blocks +24B3-bit indices mapped to{-14, -8, -4, -1, 1, 4, 8, 14}) while keeping the 6 GQA attention layers,ssm_out, and output head atQ8_0. In our AVX2 kernel (ggml_vec_dot_q2_0_q8_0_generic), sub-block rescaling is fused directly into_mm256_madd_epi16(p01, vmult01)(vpmaddwd) at zero extra SIMD instruction cost. - 📦
432 MBfor 256K Tokens (Hybrid Mamba-2 +_MM_HINT_NTANon-Temporal L3 Isolation)
InNemotron-3.5-30B-A3B, 46 blocks (23 Mamba-2 SSM + 23 Sparse MoE) maintain $O(1)$ recurrent state while only 6 layers use GQA attention (12KV heads total =432 MBat 256K vs7.1 GBon standard GQA models). Our_MM_HINT_NTApatch streams cold MoE weights (Q4_0,MXFP4,Fractal-64) and cold KV blocks without evicting hot state from the CPU's victim L3 cache. - ⚡ Crushing the 5-Million-Page TLB Wall (
2MB HugePages+AVX-512 VNNITile Repacking + Per-CCD Core Pinning)
Collapses 4.9 million 4KB pages into2MBTransparent HugePages (MADV_HUGEPAGE, expanding per-core L2 STLB coverage to6.14 GBand eliminating99.6%of page-table walks), repacksQ4_0weights into interleavedAVX-512 VNNI/AVX2SIMD tiles (+19%GEMV), and auto-tunes deterministic per-CCD physical core placement (386.0 tok/sprefill /80.14 tok/sdecode on 8-CCD Threadripper PRO 7985WX). - **🔧
"Priiloader"Auto-Profiler & Safety Locks for Any GGUF ([scripts/llama_modchip.py](scripts/llama_mod
Frequently Asked Questions
What is llama-modchip?
llama-modchip is an open-source ai agents skill for AI coding assistants such as Claude Code, Codex CLI, and ChatGPT, built by synchopate. ♟️ llama-modchip v2.1: Run 30B Sparse MoE LLMs on Pure CPU (0 GPUs) at 20–60.4 tok/s with a 256K Context Window via a 1.64MB Stockfish-PVS + RouteCrack L3 Drafter, 2MB HugePages & AVX-512 VNNI. It has 71 GitHub stars.
Is llama-modchip safe to use?
llama-modchip's catalog security scan is still queued. You can run an instant dependency and prompt-injection check now with the "Scan for vulnerabilities" button above.
How do I install llama-modchip?
Clone the repository with "git clone https://github.com/synchopate/llama-modchip" and add it to your Claude Code skills directory (see the Installation section above).
What programming language is llama-modchip written in?
llama-modchip is primarily written in Python. It is open-source under synchopate on GitHub, so you can review or fork the full source.
Are there alternatives to llama-modchip?
Yes. SkillsLLM lists many other AI Agents skills you can browse and compare side by side. Open the AI Agents category from the badge at the top of this page, or use the Related Skills and comparison links further down to weigh llama-modchip against similar tools.
Comments (0)
to leave a comment.
No comments yet. Be the first to share your thoughts!
Related Skills
Reverse engineer anything with agents, from app behavior down to native binaries.
420,582
105,013
TypeScript
AI Agentsagent-skillsai
The agent harness performance optimization system. Skills, instincts, memory, security, and research-first development for Claude Code, Codex, Opencode, Cursor and beyond.
295,087
43,680
JavaScript
AI Agentsai-agentsanthropic
Skills for Real Engineers. Straight from my .agents directory.
250,518
21,171
Shell
AI Agents
An agentic skills framework & software development methodology that works.
234,966
20,863
Shell
AI Agentsaibrainstorming