The AWS Security Blog has published a two-part series on an “AI vulnerability harness”: a pipeline that takes raw scanner findings and narrows them to a smaller, evidence-backed set, plus a steering file that tells a model how to do that triage. The posts, dated 7 October, are by Nidhi Ramakant, Ievgeniia Ieromenko and Justin Kontny. The steering file is in an aws-samples GitHub repository.
Part 1 is about the architecture around the model. Vulnerability scanners, the authors write, produce findings faster than manual triage can process them, and many of those findings are unlikely to be exploited. The team built a three-layer filter that is meant to be tool-agnostic: “what to filter at each layer, what counts as evidence, where to stop trusting the model.”
Layer 1 is multi-scanner agreement: when two or more scanners converge on the same finding, confidence increases, and a finding from only one tool is flagged for extra scrutiny. Layer 2 is structural verification against the actual code, using an abstract syntax tree to check that the files, functions and data-flow path the scanner described exist; if they don’t, the pipeline rejects the finding. Layer 3 adds deployment context from infrastructure-as-code templates, and treats protective controls as a spectrum rather than a yes-or-no checkbox.
The authors say a critical finding behind strong controls may be less urgent than a medium finding that’s directly exposed, and that AI-generated claims should be verified against the code before a person reviews them. They also list limits: the harness doesn’t fix the code, doesn’t replace an engineer’s review of the highest-priority findings, won’t surface a vulnerability class none of the scanners look for, and, by default, doesn’t check whether a control defined in a template is still in place in the live environment. “Live verification,” they write, is a later, optional step, and only for findings that have already passed a confidence threshold.
Part 2 is about the instructions the model loads at the start of a session. Early, unsteered runs, the authors say, produced findings that sounded authoritative and then fell apart: a SQL injection in a function that didn’t exist, “HIGH” confidence with no structural backing, an AWS WAF in blocking mode sitting in the request path and being ignored. “The fix wasn’t a better model. It was better steering.”
“Never trust an LLM’s self-reported confidence about a vulnerability. Verify claims against the code structure… If a finding fails all structural checks, reject it regardless of how convincing the narrative sounds.”
— AWS Security Blog, in Configuring your AI vulnerability harness, Part 2
In their testing, about 30% of unsteered model findings referenced code structures that didn’t exist in the target repository — “not wrong interpretations, entirely fabricated paths.” With the verification instruction, they recorded none. The steering file also replaces the model’s own sense of confidence with a formula built from yes-or-no signals, weighting confirmed taint (user-controlled data reaching a sensitive operation) most heavily, at 0.30. Findings with confirmed taint and no sanitization were exploitable 89% of the time in their validation, they say, against 34% for findings the model called “HIGH confidence” without structural confirmation.
Infrastructure controls are applied as multipliers on specific vulnerability classes, with a floor of 0.15 so that stacked controls can’t be treated as if they reduced the risk to zero. Access-limiting controls, such as authentication, are treated separately from attack-blocking controls such as a WAF rule: authentication shrinks who can try an exploit, the authors write, but doesn’t change the impact if an authenticated user succeeds. Threat-intelligence signals from the CISA Known Exploited Vulnerabilities catalogue and EPSS can raise urgency, capped at a +0.50 boost, and findings are then sorted into four action tiers, from P0 (“fix immediately”) to P3. One instruction in the file is “Do not file P3 findings as security issues,” on the grounds that low-confidence tickets train teams to ignore the tooling.
They tested the file against a purpose-built vulnerable application with ten known issues, eight exploitable and two mitigated by infrastructure controls. Without the steering file, the model found all ten and correctly treated the two mitigated findings as lower priority, but assigned severity by intuition and produced no structural verification. With the file, it found nine of the ten, correctly downgraded the two mitigated findings to P3 with a documented rationale, showed its confidence calculation, and verified each claimed data flow. The miss was a hardcoded credential that had been embedded in unit-testing code and was defined but never called. An earlier version of the file applied authentication as a blanket multiplier that pushed every finding behind IAM down to P2; they changed that after validation, so a command injection behind IAM still comes out as P0.
The repository ships the same methodology as a Kiro steering file, a Kiro skill, and a CLAUDE.md for Claude Code, and the authors say the principles can be adapted for other assistants that load persistent instructions. They also list what the file doesn’t replace: a programmatic taint tracker, live checks of deployed infrastructure, current threat-intelligence feeds, and execution-based confirmation in a sandbox. “The steering file without them still produces measurably better results than an unsteered model,” they write, “but it represents the first layer of a mature harness, not the complete system.”
Discover more from Sulat AI & Tech News
Subscribe to get the latest posts sent to your email.
