Sulat.com / Writing on AIWednesday, November 18, 2026
Agent Engineering/18 November 2026

NVIDIA’s SkillEvaluator tests agent skills with and without them, and NVIDIA’s own benchmark shows a 39-point lift

NVIDIA's open-source SkillEvaluator checks an agent skill for safety, duplicated guidance and real impact. NVIDIA's own benchmark of 300-plus skills shows an average lift of 39 points outside security.

A 3D NVIDIA logo linked to two rows of task cards, one row with skill folders and green ticks, one without

At a glance

  • NVIDIA's open-source SkillEvaluator checks an AI agent skill in three tiers: safety and format, duplicated guidance, and a live run with and without the skill.
  • The quickest check is skillevaluator quality-check, which needs no API key and scores a skill out of 100; version 0.9.0 reached PyPI on 13 November.
  • NVIDIA's benchmark of more than 300 skills on Codex and Claude Code shows correctness rising from 46 to 87 and effectiveness from 39 to 78.
  • Skill Lift ranges from about 2 to 46 points by product, and one skill raised token use by 120.3% while another cut it by 76.9%.
  • NVIDIA wrote and ran the evaluated skills, 85% of them ran one attempt, and the post reports no confidence intervals.

SkillEvaluator is NVIDIA’s open-source tool for testing AI agent skills, and it works in three tiers: a safety and format check, a search for duplicated guidance, and a live run that compares the same agent with and without the skill. The developer 旺旺 (@TiagerBao) pointed to it on X on 16 November, noting that it works with Claude Code and Codex and can run in CI. NVIDIA published its first benchmark of the tool on 19 August, and version 0.9.0 landed on PyPI on 13 November.

https://x.com/tiagerbao/status/2122312041088377144

The short version

Skills are folders of instructions and files that an agent loads when a task calls for them, a format set out in the Agent Skills specification. What SkillEvaluator asks about one is simple: does it actually help? The code is on GitHub under the Apache 2.0 licence and installs with pip install skillevaluator. The post says the repository has passed 2,000 stars.

NVIDIA labels support as Experimental, which means community-supported, best effort and no SLA. GitHub dates the repository to 24 June 2026.

Three tiers, one question each

The three tiers don’t depend on each other, so you can run any one alone. They map onto the three layers the post describes.

Stack of three checks: safe and well-formed, repeats itself, helps the agent
The checks run in order, and each one runs on its own.
TierWhat it checksCommandWhat you need
1: ValidationSchema and frontmatter, quality score, prompt-injection and data-exfiltration scanning, secrets and personal data, licence, script lintingskillevaluator tier1 ./my-skill --no-llmExternal scanners for full coverage; a model provider for the LLM checks
2: DeduplicationRepeated guidance inside a skill, and overlap across a catalogue, using embedding similarityskillevaluator tier2 ./my-skillEmbeddings and chat access; a catalogue file for cross-skill comparison
3: Live evaluationWhether the skill helps an agent finish tasksskillevaluator tier3 ./my-skillA model provider, a supported agent and a sandbox, Docker by default
Sources: SkillEvaluator README and NVIDIA Technical Blog.

Tier 3 is where the post puts the emphasis. It runs on Harbor, an open-source framework for repeatable agent runs. Every evaluation case goes through the agent twice in separate sandboxes: once with the skill and once without, using the same prompt, model, task inputs and grading criteria. NVIDIA calls the score gap Skill Lift and reports it in points.

How to try it

The fastest route needs no API key, scanner, GPU or Docker:

pip install skillevaluator
skillevaluator quality-check ./my-skill

It prints a score out of 100, a grade and the findings behind them. The pass mark defaults to 70, and the README is clear that this covers static quality only; security and real impact sit in the other tiers.

  • All three tiers: install skillevaluator[all], set SKILL_EVAL_LLM_PROVIDER=nv_build and an NVIDIA Build API key, then run skillevaluator validate ./my-skill. NVIDIA says eligible NVIDIA Developer Program members can use its hosted endpoints for free prototyping, within rate limits.
  • CI: skillevaluator ci init opens a setup pull request in a GitHub repository. NVIDIA has a separate guide for GitLab.
  • From your agent: npx skills add NVIDIA/SkillEvaluator adds the evaluator as a skill in Claude Code, Codex or another agent, and you then ask it whether your skill is any good.

There’s also a fourth check, harness-check, which asks whether an agent will load your skill in the first place. It needs no model key or agent run, though it’s only in the GitHub source and not in PyPI 0.9.0. The README’s sample output, labelled illustrative, tests six agents: Codex cuts a description to 1,021 characters, Claude Code doesn’t scan an .agents/skills folder, and Copilot CLI refuses a description of 1,028 UTF-16 units against a limit of 1,024.

What NVIDIA’s benchmark found

The NVIDIA Technical Blog post of 19 August covers more than 300 verified skills across over 30 NVIDIA products. Each ran on two harnesses, meaning the agent apps that drive the model: Codex and Claude Code. The numbers come from a 12 August snapshot, with scores out of 100.

DimensionWithout skillWith skillSkill Lift
Correctness4687+41
Discoverability4282+40
Effectiveness3978+39
Efficiency4378+35
Security9798+1
Average scores across both harnesses. Source: NVIDIA Technical Blog, 19 August 2026.
Bars: correctness 46 without a skill, 87 with; effectiveness 39 without, 78 with
NVIDIA’s skills lifted its own agent scores by about 40 points.

Leave security out and the average Skill Lift is 39 points, 42 on Claude Code and 36 on Codex; across all five dimensions it’s 34 and 29. NVIDIA argues the product matters more than the harness, since the two agents sit about 5 points apart on average while lift by product runs from roughly 2 to 46.

Token use is tracked apart from the efficiency score, and a skill can push it up. NVIDIA’s two single-attempt examples point in opposite directions. jetson-optimize-memory cut tokens from 617,306 to 142,540 (76.9%) and run time from 474.9 to 220.0 seconds. cuopt-install raised tokens from 25,227 to 55,582 (120.3%) and time from 34.0 to 41.1 seconds.

Bars: one skill cut tokens from 617,306 to 142,540; another raised them from 25,227 to 55,582
One skill cut tokens by 76.9%; another added 120.3%.

What the numbers cover

  • NVIDIA ran its own skills. These are verified skills for NVIDIA products, and NVIDIA’s team ran the evaluations.
  • Most ran once. By NVIDIA’s count 85% of skills had one attempt and 15% had two, and the post gives no confidence intervals.
  • Baselines aren’t zero. Discoverability and Efficiency are scored against the skill itself, and NVIDIA says the without-skill scores of about 42 and 43 come from credit for productive tool use and for leaving the skill unloaded on unrelated tasks.
  • Lift is in points. The scores aren’t pass probabilities, and lift isn’t a percentage change.

NVIDIA’s authors also posted a paper on 20 August that names the method ACES. It covers 947 paired cases from 58 of 64 production skills on four harnesses, with a mean composite lift of 0.2134 (95% confidence interval 0.1967 to 0.2301) and positive lift in 72.8% of paired cases. Across 145 skills, structural and LLM-judge scores correlated at a Spearman 0.14, and the paper says scan-only gates measure complementary facets of a skill.

Who else is using it

Two partner pilots appear in NVIDIA’s post. OpenClaw is piloting Tier 3 evaluations for official organisations on ClawHub, with with-skill and without-skill results shown in an Evals tab. Nous Research tested SkillEvaluator in Hermes Agent, where an optional advisory scan by SkillSpector runs in the skills install flow and checks for personal data, Unicode smuggling, script linting, licence and security issues before anything installs. NVIDIA says 29 tests pass and each scan takes about 1.4 to 1.5 seconds.

NVIDIA says it evaluates the catalogue continuously and publishes current scores in benchmarks.json in the nvidia/skills repository. Setup steps are in the SkillEvaluator documentation.

Plain-English glossary

TermWhat it means here
SkillA folder of instructions and files an agent loads for a certain kind of task
HarnessThe agent app that runs the model, such as Claude Code or Codex
Skill LiftThe with-skill score minus the without-skill score, in points
SandboxA sealed environment where an agent run happens, so runs don’t affect each other
EmbeddingA numeric fingerprint of a piece of text, used to measure how similar two passages are

Discover more from Sulat AI & Tech News

Subscribe to get the latest posts sent to your email.