SkillEvaluator is NVIDIA’s open-source tool for testing AI agent skills, and it works in three tiers: a safety and format check, a search for duplicated guidance, and a live run that compares the same agent with and without the skill. The developer 旺旺 (@TiagerBao) pointed to it on X on 16 November, noting that it works with Claude Code and Codex and can run in CI. NVIDIA published its first benchmark of the tool on 19 August, and version 0.9.0 landed on PyPI on 13 November.
The short version
Skills are folders of instructions and files that an agent loads when a task calls for them, a format set out in the Agent Skills specification. What SkillEvaluator asks about one is simple: does it actually help? The code is on GitHub under the Apache 2.0 licence and installs with pip install skillevaluator. The post says the repository has passed 2,000 stars.
NVIDIA labels support as Experimental, which means community-supported, best effort and no SLA. GitHub dates the repository to 24 June 2026.
Three tiers, one question each
The three tiers don’t depend on each other, so you can run any one alone. They map onto the three layers the post describes.

| Tier | What it checks | Command | What you need |
|---|---|---|---|
| 1: Validation | Schema and frontmatter, quality score, prompt-injection and data-exfiltration scanning, secrets and personal data, licence, script linting | skillevaluator tier1 ./my-skill --no-llm | External scanners for full coverage; a model provider for the LLM checks |
| 2: Deduplication | Repeated guidance inside a skill, and overlap across a catalogue, using embedding similarity | skillevaluator tier2 ./my-skill | Embeddings and chat access; a catalogue file for cross-skill comparison |
| 3: Live evaluation | Whether the skill helps an agent finish tasks | skillevaluator tier3 ./my-skill | A model provider, a supported agent and a sandbox, Docker by default |
Tier 3 is where the post puts the emphasis. It runs on Harbor, an open-source framework for repeatable agent runs. Every evaluation case goes through the agent twice in separate sandboxes: once with the skill and once without, using the same prompt, model, task inputs and grading criteria. NVIDIA calls the score gap Skill Lift and reports it in points.
How to try it
The fastest route needs no API key, scanner, GPU or Docker:
pip install skillevaluator
skillevaluator quality-check ./my-skill
It prints a score out of 100, a grade and the findings behind them. The pass mark defaults to 70, and the README is clear that this covers static quality only; security and real impact sit in the other tiers.
- All three tiers: install
skillevaluator[all], setSKILL_EVAL_LLM_PROVIDER=nv_buildand an NVIDIA Build API key, then runskillevaluator validate ./my-skill. NVIDIA says eligible NVIDIA Developer Program members can use its hosted endpoints for free prototyping, within rate limits. - CI:
skillevaluator ci initopens a setup pull request in a GitHub repository. NVIDIA has a separate guide for GitLab. - From your agent:
npx skills add NVIDIA/SkillEvaluatoradds the evaluator as a skill in Claude Code, Codex or another agent, and you then ask it whether your skill is any good.
There’s also a fourth check, harness-check, which asks whether an agent will load your skill in the first place. It needs no model key or agent run, though it’s only in the GitHub source and not in PyPI 0.9.0. The README’s sample output, labelled illustrative, tests six agents: Codex cuts a description to 1,021 characters, Claude Code doesn’t scan an .agents/skills folder, and Copilot CLI refuses a description of 1,028 UTF-16 units against a limit of 1,024.
What NVIDIA’s benchmark found
The NVIDIA Technical Blog post of 19 August covers more than 300 verified skills across over 30 NVIDIA products. Each ran on two harnesses, meaning the agent apps that drive the model: Codex and Claude Code. The numbers come from a 12 August snapshot, with scores out of 100.
| Dimension | Without skill | With skill | Skill Lift |
|---|---|---|---|
| Correctness | 46 | 87 | +41 |
| Discoverability | 42 | 82 | +40 |
| Effectiveness | 39 | 78 | +39 |
| Efficiency | 43 | 78 | +35 |
| Security | 97 | 98 | +1 |

Leave security out and the average Skill Lift is 39 points, 42 on Claude Code and 36 on Codex; across all five dimensions it’s 34 and 29. NVIDIA argues the product matters more than the harness, since the two agents sit about 5 points apart on average while lift by product runs from roughly 2 to 46.
Token use is tracked apart from the efficiency score, and a skill can push it up. NVIDIA’s two single-attempt examples point in opposite directions. jetson-optimize-memory cut tokens from 617,306 to 142,540 (76.9%) and run time from 474.9 to 220.0 seconds. cuopt-install raised tokens from 25,227 to 55,582 (120.3%) and time from 34.0 to 41.1 seconds.

What the numbers cover
- NVIDIA ran its own skills. These are verified skills for NVIDIA products, and NVIDIA’s team ran the evaluations.
- Most ran once. By NVIDIA’s count 85% of skills had one attempt and 15% had two, and the post gives no confidence intervals.
- Baselines aren’t zero. Discoverability and Efficiency are scored against the skill itself, and NVIDIA says the without-skill scores of about 42 and 43 come from credit for productive tool use and for leaving the skill unloaded on unrelated tasks.
- Lift is in points. The scores aren’t pass probabilities, and lift isn’t a percentage change.
NVIDIA’s authors also posted a paper on 20 August that names the method ACES. It covers 947 paired cases from 58 of 64 production skills on four harnesses, with a mean composite lift of 0.2134 (95% confidence interval 0.1967 to 0.2301) and positive lift in 72.8% of paired cases. Across 145 skills, structural and LLM-judge scores correlated at a Spearman 0.14, and the paper says scan-only gates measure complementary facets of a skill.
Who else is using it
Two partner pilots appear in NVIDIA’s post. OpenClaw is piloting Tier 3 evaluations for official organisations on ClawHub, with with-skill and without-skill results shown in an Evals tab. Nous Research tested SkillEvaluator in Hermes Agent, where an optional advisory scan by SkillSpector runs in the skills install flow and checks for personal data, Unicode smuggling, script linting, licence and security issues before anything installs. NVIDIA says 29 tests pass and each scan takes about 1.4 to 1.5 seconds.
NVIDIA says it evaluates the catalogue continuously and publishes current scores in benchmarks.json in the nvidia/skills repository. Setup steps are in the SkillEvaluator documentation.
Plain-English glossary
| Term | What it means here |
|---|---|
| Skill | A folder of instructions and files an agent loads for a certain kind of task |
| Harness | The agent app that runs the model, such as Claude Code or Codex |
| Skill Lift | The with-skill score minus the without-skill score, in points |
| Sandbox | A sealed environment where an agent run happens, so runs don’t affect each other |
| Embedding | A numeric fingerprint of a piece of text, used to measure how similar two passages are |
Discover more from Sulat AI & Tech News
Subscribe to get the latest posts sent to your email.




