Meta says an AI testing system that writes tests meant to fail now runs as part of its internal engineering infrastructure. The company calls the system Catching Just-in-Time Tests, or Catching JiTTests. It says that in the first half of 2026 the system helped catch 74 serious privacy bugs in code changes before they reached production.
Engineering at Meta, the company’s official engineering account, promoted the work on X in a post linking to a 29 October write-up on its engineering blog and to the team’s research paper. Read side by side, the two documents show how far the reported results have moved since January, when the paper described a small early trial. They also show how much of the newer evidence rests on Meta’s own counting.
🧠 Most AI bug-catching tools review your code the way a human reviewer would, only faster. What if the AI wrote a test that proved the bug instead?
— Engineering at Meta (@Meta_Engineers) November 9, 2026
That’s Catching Just-in-Time Tests (JiTTests) — a novel approach where LLMs generate executable tests designed to fail, surfacing behavioral changes in a code change before it lands.
After a year of research, it’s now part of Meta’s infrastructure.
✅ 589 engineers have engaged with it, reporting a 60% true-positive rate
📈 In the first half of 2026, it helped catch 74 privacy SEVs at the diff stage, lowering overall privacy SEV occurrence by 20%
🔗 Read how it works: engineering.fb.com/2026/10/29/sec…
📄 Read our FSE 2026 paper, “Just-in-Time Catching Test Generation at Meta”: arxiv.org/pdf/2601.22832
#AI #SoftwareEngineering #SoftwareTesting #DeveloperTools #Engineering
What the system does
Most AI code-review tools read a proposed change and leave comments, such as a warning that a line might be wrong. Meta’s system takes a different route. For a proposed code change, which Meta calls a diff, a large language model writes a test that passes on the old code and fails on the new code. A failing test of that kind is concrete evidence that the program now behaves differently.
That failure does not prove a bug, because many behaviour changes are deliberate. The paper calls such a failure a “weak catch”. It becomes a “strong catch” only when the change in behaviour turns out to be unintended. Most of the engineering effort described in both documents goes into telling the two apart before an engineer is asked to look.
According to the blog post, the production pipeline has five stages. It runs overnight on spare computing capacity and targets the previous day’s riskiest changes, using a ranking similar to Meta’s Diff Risk Score. Two test-writing workflows then run. One infers what the author meant the change to do, plants small deliberate faults, called mutants, and writes tests to expose them. The other treats the change itself as a possible fault.
Failing tests then go to a panel of AI judges. The paper names Llama 3.3 70B, Gemini 3 Pro and Claude Sonnet 4 among the models in that ensemble, and says each judge sees the test code, the failure output, the diff and the inferred intent. A separate rule-based filter, which Meta calls RubFake, applies fixed checks for patterns known to produce false alarms. Engineers sit at the end of the chain. Meta describes their role as “human-on-the-loop”: they set the criteria and review results in aggregate rather than inspecting every test.
From a 41-message trial to 589 engineers
The arXiv paper, posted on 30 January and listed for the FSE 2026 software engineering conference, describes the system’s early production trial. Its 15 authors are listed at Meta Platforms in London, Seattle and California. Among them is Mark Harman, the testing researcher whose earlier Sapienz work received an ISSTA 2026 Most Influential Paper award, according to Meta.
The trial numbers are small. From mid-September to the end of October 2025, the team contacted 41 engineers with short messages asking whether an observed change was intended. Eight confirmed real bugs, an engineer-experienced true-positive rate of 19.5 per cent. The paper says four of those eight would have caused severe failures in production, and that engineers needed only a few minutes to dismiss false alarms. One passage of the paper refers to “7 issues from 41 reach outs” while giving the same 19.5 per cent figure. Eight of 41 matches that rate, so the “7” appears to be a typo.
Across 22,126 generated tests, the paper reports that the change-aware workflows produced candidate catches about four times as often as Meta’s existing “hardening” test generators, and about 20 times as often as tests that failed by coincidence. A manual study of 600 diffs found that the AI judges cut the human review load by about 70 per cent.
The authors were cautious about their own results. “At this point in the evolution of our catch workflows, we have too few data points to make strong scientific claims of this nature,” they wrote.
“Catching JiTTests are no longer a research prototype; they’re part of Meta’s infrastructure.”
Engineering at Meta blog, 29 October 2026
The October blog post, by Sreeja Thummala, Sophie Zeng, Mark Harman, Weini Liu and Eric Liu, reports results at a much larger scale. It says 589 engineers, or 64 per cent of those Meta contacted, have engaged with the system, and that “these engineers have reported” a 60 per cent rate of true positives. It also says that in the first half of 2026 the system “played a key role in catching 74 serious (SEV) privacy bugs at the diff stage”, and that the overall occurrence of privacy SEVs fell by 20 per cent over the same period. SEV is the label Meta uses for these serious incidents.
| Measure | January paper (2025 trial) | October blog (2026 production) |
|---|---|---|
| Engineers contacted or engaged | 41 contacted | 589 engaged, 64% of those contacted |
| True-positive rate seen by engineers | 19.5% (8 of 41) | 60%, as reported by engineers |
| Serious bugs caught | 4 that would have caused severe failures | 74 privacy SEVs in the first half of 2026 |
| Wider effect claimed | About 70% less human review load | Privacy SEVs down 20% over the same period |
What the numbers leave open
The rise from 19.5 per cent to 60 per cent is the most striking change, and the blog post does not explain how the newer figure was measured. It does not say how many alerts sit behind it, whether it counts every alert or only those engineers answered, or how “true positive” was defined in production. The 64 per cent engagement figure implies Meta contacted roughly 920 engineers, but that is an inference from the post, which gives no total.
The privacy claim is also framed with care. Meta says the system played “a key role” in catching the 74 bugs and links that to a 20 per cent fall in privacy SEVs. It does not say how many of those bugs other checks would have caught, or what else changed in its privacy work during the same six months.
Limits of the evidence
Every figure in this story comes from Meta. The production results appear in a company blog post, not in peer-reviewed work, and no outside group has reproduced them. The system runs on Meta’s internal code and tools, and neither the blog post nor the paper mentions a public release. The blog post says the approach “requires more computational steps” than AI code review, but it gives no cost figures.
Replies to Meta’s post raised similar questions. One user wrote that “the hard part is discarding tests that fail because the change was intended”, which is the problem the AI judges and RubFake filter are meant to address. Another asked about token costs, which Meta has not published. A third questioned whether the approach is as “novel” as the post claims. The idea has a public history inside Meta: the company wrote in 2025 about ACH, an AI system that plants faults and writes tests to catch them, and about using it for compliance testing. The JiTTests paper uses ACH as one of its baselines.
What other teams can take from it
The tooling depends on Meta’s internal build, test and code-review systems, so outside teams cannot simply install it. The method is laid out in enough detail to copy in part. A model writes a test aimed at one specific change, automated judges and fixed rules discard failures that look intended, and the engineer gets one short question about whether a behaviour change was meant to happen.
The paper also finds that the wording of that question shapes how much a false alarm costs. Its authors write that a message such as “this expression, used to evaluate to true, but now evaluates to false; is that expected?” is typically easy for an engineer to judge. A question about the reordering of elements inside a large data structure can take orders of magnitude longer to dismiss. Meta’s rule-based filter therefore scores an alert higher when a false alarm would be quick to rule out.
Meta first presented the research publicly in February, when it described tests “designed to fail” for codebases of more than 100 million lines.
How do you catch critical bugs at the scale of 100M+ lines of code? 🛠️
— Engineering at Meta (@Meta_Engineers) February 11, 2026
In our new research paper, we explore Just-in-Time (JIT) catching test generation. Unlike traditional hardening tests that pass at generation, JIT tests are designed to fail, surfacing regressions before code lands.
This research demonstrates that JIT catching is a vital tool for maintaining stability in high-velocity, large-scale development environments. Read the full paper: go.meta.me/bf4344
Discover more from Sulat AI & Tech News
Subscribe to get the latest posts sent to your email.
