Sulat.com / Writing on AISunday, November 22, 2026
Agent Engineering/22 November 2026

Nvidia used Factory’s Droid harness to fix Nemotron 3 Ultra’s tool-use mistakes

Factory says Nvidia trained Nemotron 3 Ultra on 20,000 rows made with its Droid harness. The model's Terminal-Bench 2.0 score rose from 38.5 to 49 after a wider multi-harness training run.

Glossy 3D Nvidia and Factory logos joined by an arc of terminal windows and repaired tool-call tokens

At a glance

  • Factory says Nvidia ran its Droid coding-agent harness inside an air-gapped network to post-train Nemotron 3 Ultra, a model with 550 billion total and 55 billion active parameters.
  • Researchers read Droid's execution traces to find hallucinated tool calls, wrong argument formats and weak error recovery, then used the harness to reproduce those failures and write correct examples.
  • Nvidia trained on 20,000 rows using the harness and applied PivotRL, its reinforcement-learning method, to the tool-use errors.
  • Nemotron 3 Ultra's Terminal-Bench 2.0 score rose from 38.5 to 49 after a multi-harness run that included that data, measured by Nvidia on its own checkpoints.

Nvidia trained Nemotron 3 Ultra on 20,000 rows of data made with Factory’s Droid coding-agent harness, and the model’s Terminal-Bench 2.0 score rose from 38.5 to 49 after a wider training run that included that data. Factory laid out the work in a case study published on 18 November.

Nemotron 3 Ultra is Nvidia’s open model with 550 billion total and 55 billion active parameters, released on 4 June. Nvidia’s researchers wanted to see how it behaved on long agent tasks, not only whether it passed them, so they ran Droid inside Nvidia’s own air-gapped network.

Why Nvidia wanted the traces

A benchmark score says whether a task passed. It doesn’t say how the model got there. Factory’s case study spells out the gap: an agent can finish a task despite wrong tool calls, and a failure can come from poor planning, a misread instruction or an inability to recover from an error. Nvidia wanted trace-level visibility to pick the behaviours worth training on.

Droid as an enterprise harness really stood out because of the very strong instructions that were embedded into the harness. The system prompt has a very strong kind of steering for the model to plan…and draft its own tasks across its own trajectory of working through a problem.

Junkeun Yi, research scientist, agentic post-training for Nemotron, Nvidia

Droid’s planning tools made Nemotron draft tasks, update its plan as it worked and follow set formatting rules, which gave Nvidia a built-in test of instruction following.

The air gap mattered for a different reason. With the model and the harness sandboxed behind Nvidia’s firewalls, researchers could run experiments without worrying that one would “mess up something outside,” in the words of Jian Zhang, Nvidia’s director of applied research for Nemotron post-training. Factory says Droid was the only commercially supported enterprise platform deployed on-premises in that environment.

What the traces showed

Factory and Nvidia read Nemotron’s traces together and named specific failure modes: hallucinated tool calls, incorrect argument formats and poor error recovery. Factory also added hooks to Droid to capture the points where runs went wrong.

Five-step loop: run Droid, read traces, reproduce failures, train on 20,000 rows, send checkpoint back
Each training round ends with a fresh look at the traces.

Nvidia then used the harness to reproduce those failures and generate training examples of correct tool use. For Nemotron 3 Ultra it trained on 20,000 rows of data using the Factory harness, applying PivotRL to the hallucinated tool calls and other tool-use errors. Afterwards Nvidia sent the updated checkpoints back to Factory to check which behaviours had improved and which still needed work.

What PivotRL does

PivotRL is a reinforcement-learning method Nvidia researchers published in March. It starts from existing supervised fine-tuning trajectories and picks out “pivots”, intermediate turns where sampled actions give mixed results, some passing and some failing. It then runs short rollouts from those turns and rewards any action that does the same job as the demonstration, instead of demanding an exact text match. The paper reports competitive accuracy with end-to-end reinforcement learning on agentic coding tasks using four times fewer rollout turns. Yi is one of its three lead authors, and Nvidia had already used it to post-train Nemotron 3 Super.

The scores

Bars: Nemotron 3 Ultra 38.5 to 49 on Terminal-Bench 2.0; 3.5 Super VL 52.11 to 60.4 on 2.1
Both gains are Nvidia’s own results on its own checkpoints, on different Terminal-Bench versions.

Nemotron 3 Ultra’s Terminal-Bench 2.0 score went from 38.5 to 49. A preliminary checkpoint of Nemotron 3.5 Super VL went from 52.11 to 60.4 on Terminal-Bench 2.1 after the same wider pipeline. Factory puts the relative gains at 27.3% and 15.9%.

Both results are Nvidia’s, measured on Nvidia’s own checkpoints. They come from multi-harness training, not from Droid data alone, and Factory’s post says so: the improvements reflect Nvidia’s multi-harness work, “which incorporated data generated in the Factory harness.” Factory doesn’t publish a Droid-only score, and the case study doesn’t name the checkpoint the 38.5 started from.

One harness among several

Nvidia’s technical report for Nemotron 3 Ultra explains the design. It sorts training tasks into five groups: zero-to-one terminal and software engineering work, bug fixing in existing repos, office and general productivity, general knowledge, and search. Each group is trained under at least two of six harnesses: Stirrup, OpenHands, OpenCode, Terminus, Droid and Nvidia’s custom internal ones. The report says this stops the model learning one harness’s habits, so it holds up when the execution context changes.

Five task types each trained under at least two of six harnesses, with Droid highlighted as one
Droid is one of six harness options in Nemotron 3 Ultra’s training mix.

Yi said the behaviour analysis paid off beyond Droid: “Because of these in-depth behavior analyses, we were able to improve our model on not just Droid, but in various general agentic instruction following capabilities.”

Nvidia released the Nemotron 3 Ultra checkpoints and training data with the model in June. Factory’s post doesn’t say whether the 20,000 rows are among that data.

Other teams can copy the loop itself: run the agent in a harness that records every step, find the mistakes that keep repeating, turn them into training rows, and read the traces again once training is done.


Discover more from Sulat AI & Tech News

Subscribe to get the latest posts sent to your email.