SkillRanker
The right skill for the next step, powered by Jev-compatible providers.
A standalone Rust CLI that puts Jev-compatible providers at the center of skill selection: Jev evaluates your agent's live context, compares the available skills, and estimates which ones fit the next step. SkillRanker supplies the session integration, local safeguards, and inspectable feedback around it.
Fresh ranking requires your own credential for the selected provider. TypeSafe.ai is the default; Cloudflare-hosted TypeSafe Jev is also supported.
sr capabilities --json # Check implemented commands and integrations in your build
sr demo --case useful # Inspect an offline fixture before connecting a session
sr rank --allow-network # Rank skills for the selected session
Contents
- TL;DR
- Quick Example
- Design Philosophy
- How It Compares
- Installation
- Quick Start
- Command Reference
- Configuration
- How Ranking Works
- Explain And Replay A Result
- Local Feedback And Calibration
- Evaluation, Sampling, And Risk Monitoring
- Control Requests And Interruptions
- Agent Hooks
- Inline TUI
- Architecture
- Privacy And Local State
- Performance
- Troubleshooting
- Limitations
- FAQ
- About Contributions
- License
TL;DR
The problem. A large skill library gives an agent plenty of procedures to choose from, but choosing is itself a task. Similar descriptions obscure useful distinctions. A skill that helped at the start of a conversation can be irrelevant three turns later. Loading a plausible but unsuitable skill consumes context and can redirect otherwise sensible work.
The solution. SkillRanker (sr) combines the recent conversation, current
request, workspace signals, and the selected harness's visible skill inventory.
The selected provider's Jev implementation is the key enabler of the system.
It first compares the candidates broadly, then reads richer excerpts from a
shortlist and evaluates whether each one fits. Both comparisons include a
real “none of these” option. The result is advisory: the agent follows the user's
instructions and decides what to consult.
For libraries with more than 254 eligible skills, Quill from FrankenSearch narrows the candidates locally before Jev evaluates them. Smaller rosters reach Jev in full. Explicit skill requests resolve locally before either stage.
SkillRanker does not include a local model. The ranking workflow requires your own account and credential for the selected provider. Local retrieval prepares the candidates; Jev supplies the evaluations that make the recommendations possible.
Why sr?
| Need | What SkillRanker provides |
|---|---|
| Evaluate meaning and task fit | The selected provider's typed Jev Choice and Noul evaluations power both ranking passes |
| Choose for the current step | Exact session identity, the newest prompt, recent tool evidence, and project signals |
| Suggest something the agent can load | Harness-aware visibility, override resolution, stable skill identities, and content revalidation |
| Respect an explicit request | Locally resolve a requested skill before probabilistic retrieval or ranking |
| Search a large library | Quill lexical prefiltering from FrankenSearch, admitting up to 254 skills plus a none option to each Choice |
| Separate similar skills | Detailed reranking with bounded descriptions and body excerpts |
| Recognize when no skill fits | Relevance gates, per-candidate fit checks, and sentinel-based abstention |
| Understand a missing suggestion | --why-not traces where a candidate was excluded, with thresholds and concrete recovery hints |
| Reproduce a surprising result | Opt-in case capture and offline replay compare compatible local policies without another Jev call |
| Get started without sharing a session | Offline fixture demos and a readiness report identify the next setup step |
| Keep the agent moving | A failed hook recommendation produces a quiet, non-blocking fallback |
| Control interruptions | Silent ordinary abstentions and scoped, expiring snoozes preserve explicit skill requests |
| Bound repeated expense | Optional shared HTTP-attempt allowances and a provider circuit breaker cover concurrent local sessions |
| Review what happens | Usefulness, interruptions, attempts, and cost share a report with explicit label coverage |
| Evaluate within a budget | Offline replay, explicit live-request caps, and reproducible samples with recorded selection probabilities |
| Assess recommendation harm | Controlled comparisons, uncertainty bounds, and optional monitoring across repeated evaluations |
| Control disclosure | Network opt-in, field-level disclosure receipts, a minimal context profile, and separate persistence controls |
The approach builds on the TypeSafe skill-suggestion recipe. SkillRanker adds session identity, harness visibility, bounded execution, and a local evaluation loop. The comprehensive plan explains the full design and acceptance criteria.
Quick Example
# See a labeled fixture result without a key, network, or private session.
sr demo --case useful
# Inspect local configuration and the available adapters.
sr doctor --json
sr doctor --config
sr capabilities --json
# Inspect visible, shadowed, and excluded skill records.
sr roster --json
# Preview the redacted wide-pass request without network or persistence effects.
sr rank --context scratch/context.json --dry-run
# Evaluate an explicitly selected conversation.
sr rank --context scratch/context.json --allow-network --json
# Find where an expected candidate was excluded; this adds no inference calls.
sr rank --context scratch/context.json --allow-network --why-not SKILL_ID --explain
# Preview the Claude hook settings change, then apply it.
sr install-hook claude
sr install-hook claude --apply
# Review observations without treating adoption as proof of usefulness.
sr stats --since 7d --by-skill
# Replay a labeled evaluation artifact without making network requests.
sr eval --dataset scratch/evaluation.json --explain
# Inspect description quality and suspected coverage gaps locally.
sr doctor --descriptions
sr gaps
Design Philosophy
- Choose for the next action. The current request matters, as do the recent failure, the task context, and the instructions already loaded.
- Resolve authority locally. The user decides what is required or excluded. The harness determines what can be loaded. A model answer cannot change either.
- Separate preference from applicability. Winning a comparison is not enough. A recommendation must survive fit, visibility, loaded-state, and none-option checks.
- Keep evidence inspectable. Preserve provider estimates and local arithmetic.
--explainexposes computations without inventing model-generated reasons. - Separate adoption from usefulness. Observing a load is useful telemetry. Learning a better policy requires independently judged examples and a holdout.
- Bound each evaluation. Input, discovery, subprocesses, networking, retries, persistence, and cleanup consume one ranking deadline. Batch evaluation also has explicit request and total-runtime limits.
- Stay standalone. Discovery, parsing, redaction, retrieval, and feedback live
in
sr. No skill-manager service or private database is required.
How It Compares
These are workflow choices, not benchmark rankings.
| Approach | Input to selection | Strength | Tradeoff |
|---|---|---|---|
| Manual selection | Your knowledge of the task and library | Direct control without a ranking service | Requires remembering each skill's coverage |
| Keyword search | A query over names and descriptions | Cheap local discovery | Synonyms and adjacent procedures can be hard to distinguish |
| Load every skill | The full library's instructions | Makes every procedure available immediately | Consumes context regardless of relevance |
| SkillRanker | Exact session, visible roster, and explicit constraints | Evaluates candidates and can abstain | Fresh Jev evaluations require authorized network access |
SkillRanker recommends procedures. It does not execute skills, grant permissions, or override the agent's governing instructions.
Installation
Install on Linux or macOS. Keep macOS state on a local APFS or HFS+ filesystem
with Unix permissions. Run sr capabilities --json after installation to check
command and integration availability in your build.
Installer
The installer selects a release for your platform, verifies its SHA256 checksum,
and installs sr into ~/.local/bin. When a release asset is unavailable, it
builds from the selected source revision using the pinned Rust toolchain:
curl -fsSL "https://raw.githubusercontent.com/Dicklesworthstone/skillranker/main/install.sh?$(date +%s)" | bash -s -- --verify
It requires Bash, Python 3, `cu