Sulat.com / Writing on AIFriday, November 20, 2026
Agent Engineering/20 November 2026

OpenAI’s Performance Factory, explained: Codex agents that investigate ChatGPT slowdowns and propose the fix

OpenAI says its Performance Factory runs Codex agents in the cloud that investigate ChatGPT slowdowns, propose fixes and check recovery. Here is how it works, in plain words.

Glossy green 3D OpenAI logo beside a miniature factory conveyor with two white robot arms and a speed gauge in the green zone

At a glance

  • OpenAI says the Performance Factory is a cloud system where Codex agents investigate ChatGPT slowdowns, propose a fix for an engineer to review, and check that speed came back after deployment.
  • In just over three months, its first workflow, the Regression Workflow, tracked more than 1,700 investigations and found the root cause in over 600, which OpenAI puts at 17 automated investigations a day.
  • A separate Codex session tries to disprove each diagnosis, and in a manual review of 100 root cause suggestions OpenAI graded 70% accurate, with 30% needing more work or not fully establishing causality.
  • OpenAI says fixing a regression has been harder than finding its cause, so each proposed fix gets its own adversarial review, and an engineer reviews the pull request and decides whether it merges.
  • OpenAI says it's working on agents that spot slowdowns when no alert fires, and on shipping low-risk fixes without a human in the loop.

OpenAI says it has built the Performance Factory, a cloud system in which Codex agents investigate ChatGPT slowdowns, propose a fix for an engineer to review, and check afterwards that speed came back. In just over three months, OpenAI says, its first workflow tracked more than 1,700 investigations and found the root cause in over 600 of them, which it puts at 17 a day. Okay Zed and Cyrus Forbes, two OpenAI engineers, described the system in an engineering post dated 19 November. This piece walks through how it works in plain words, with each technical term explained where it first appears.

The short version

  • What it is: a system that turns a slowdown alert into a tracked task, hands each step to a Codex agent or an engineer, and keeps every finding in one place.
  • What the agents do: work out when ChatGPT got slower, which requests it hit and which change caused it, then open a pull request with a fix. Later, another agent checks whether speed returned to normal.
  • What people do: an engineer reviews the pull request and decides whether it ships, and steps in when a fix needs judgement.
  • Why OpenAI built it: it says the pace of agentic development has made traditional performance engineering unsustainable.
Regression Workflow numbers: 1,700+ investigations, 600+ root causes found, 17 automated investigations a day
In just over three months, OpenAI’s first workflow tracked more than 1,700 investigations.

The problem: ChatGPT gets slower and nobody knows why

A performance regression is a change that makes something slower than it used to be. OpenAI’s example is first token latency, the delay before the first piece of a ChatGPT reply appears. Teams set alerts on measurements like that, and an alert fires when a number crosses a threshold. It tells you that something got slower. It doesn’t tell you why.

Before Codex, OpenAI says, engineers answered that question by hand. They examined metrics (numbers tracked over time), read logs (a running diary of what a program did), compared traces (the path one request takes through many services), debugged live machines and read code. The work took a long time, and knowing where to look often needed deep knowledge of the service, so relevant signals were easy to miss. The post’s opening line is blunt: “The staggering pace of agentic development has made traditional approaches to performance engineering unsustainable.”

How OpenAI got from hand-digging to a factory

OpenAI describes six stages, and the early ones will look familiar to anyone who has used a coding assistant. Engineers first built more than 30 performance tools of their own, such as profilers that show where a program spends its processor time and its memory. Then they began asking Codex “Why did this get slower?” and working through the answer together, even in a system they didn’t know well.

Staircase of six steps from manual digging, through Codex as partner and skills, to automated investigations and the factory
Each step handed one more job from the engineer to Codex.

The third stage was a shared skill called perf-forensics. A skill is a saved set of instructions for an agent. This one tells Codex which tools exist, when to use each, and how to work through a slowdown from several angles: find when it began, identify the affected requests, and check whether a deployment or configuration change explains it. OpenAI tested and refined the skill with evals, repeatable tests built from past regressions. For the kinds of problems it focused on, such as ChatGPT latency and availability drops, Codex could take an alert and return a diagnosis in one run.

Then the engineers noticed they’d become a go-between. An alert fired, someone pasted it into Codex with the right skill, and someone pasted the findings back into Slack. OpenAI automated that by moving investigations to the cloud, so an alert could start a Codex session on its own. That helped, but the automation ended with the Slack reply. An engineer still had to develop a fix and check whether it worked after deployment, and as alerts piled up, just working out which ones mattered meant sifting through threads. The Performance Factory is the sixth stage, built to run the whole job from alert to recovery.

The four parts of the factory

OpenAI lists four components. The comparisons in the graphic below are my shorthand for readers new to the idea, not OpenAI’s wording.

Four boxes: task service, dispatch loop, Codex in the cloud and subscriptions, each with a plain-word comparison
The factory is a task board, a dispatcher, a cloud workshop and a set of wake-up triggers.

The task service is where engineers define a workflow by adding configuration files and Markdown prompts. The configuration lists the states a task can move through, the transitions allowed between them and which steps need an engineer, and separate prompts tell agents what to do at each step. OpenAI shows a code-change workflow as a folder with a config.toml file and two prompts, one for implementing and one for reviewing. Every piece of work becomes a task visible in a UI, where engineers can follow progress or share a task with a teammate who has more context.

Code
code-change/
├── config.toml
└── work/
    ├── implementing.md
    └── reviewing.md

The dispatch loop checks every so often which tasks are ready for their next step. For each one it starts a Codex turn with the instructions for the task’s current state, and the agent records its progress and moves the task along when it finishes the step. The agent works in Codex in the cloud, through an internal API that runs a turn on an environment inside OpenAI’s network. That environment has access to its code repositories and to the observability tools engineers use, meaning metrics, logs, traces, CPU profiles and memory snapshots. The agent runs under a service identity, so OpenAI can scope its permissions and see which actions came from its performance agents, and work runs in parallel without anyone’s laptop staying open.

The last part is subscriptions. Some work has to wait for a person or a deployment, and an agent shouldn’t sit running meanwhile. A waiting task can subscribe to events such as a review comment or a failed CI check (an automatic test run) on a GitHub pull request, which is a proposed code change a teammate reviews before it’s merged. When a matching event arrives, the task becomes ready again and the dispatch loop assigns an agent. For how Codex limits what its commands can touch on a laptop, see our earlier piece on the Codex sandbox.

The Regression Workflow, step by step

The first workflow built inside the factory is the Regression Workflow. A regression moves through four stages, which OpenAI names needs investigation, root caused, fix proposed and resolved, and the workflow follows the same path. When one of its configured performance alerts fires, it opens a task, and agents pick the task up and move it across the stages. The task stores its status and key findings outside any single agent’s context, so another agent or an engineer can take over at any point.

Flow from alert to recovery: system detects, agent finds cause and proposes fix, engineer reviews, agent checks; three outcomes
Agents do four of the five steps; an engineer decides what ships.

To find the root cause, the agent first works out which metrics got worse, when they started to change and which requests were affected. An alert fires when a threshold is crossed, but the slowdown may have started hours earlier, so the agent looks back through the data for the moment the metric first turned. It then compares groups of requests to see whether similar requests got slower or whether more of the traffic was already slow, because those two findings point to different causes. Only then does it examine deployments and feature flag changes (switches that turn a feature on or off without a new release), read the relevant code, and use logs or traces to check that a change actually reached the affected requests.

After an engineer deploys a fix, the workflow wakes an agent to compare performance with its earlier baseline. If it’s back to normal, the agent marks the task resolved. If it’s only partly better, the agent resolves the original regression and opens a new task for what remains, which may have a different cause. If nothing improved, it records what happened and sends the regression back for investigation.

A second agent tries to prove the first one wrong

Early on, OpenAI says, agents sometimes stopped at the first plausible explanation, and some of those diagnoses were wrong. The fix was an adversarial review: a separate Codex session whose job is to disprove the proposed diagnosis. It rechecks the assumptions and tests counterfactuals, which means asking what should be true if the diagnosis were right. If a rollout caused the slowdown, did similar requests that weren’t exposed stay healthy? If the suspected change was rolled back, did the affected metric recover?

A reviewer agent reopens a diagnosis because the file service had recovered while image requests still failed; 70 of 100 graded accurate
A second agent tries to disprove the first one’s answer.

OpenAI gives one example. An agent traced failing ChatGPT conversations to errors in image downloads and marked the cause as found. The reviewer noticed that the internal service responsible for locating files had already recovered while image requests were still failing, so the diagnosis didn’t explain the failures that continued. It reopened the investigation and recorded what still needed to be established. In a manual review of 100 root cause suggestions, OpenAI graded 70% accurate and 30% as needing refinement or not fully establishing causality, and it says it breaks larger jobs into focused Codex turns that investigate, challenge and propose as separate steps to raise those numbers.

Fixing is harder than finding

Once a diagnosis survives review, the workflow asks Codex to implement a fix and open a pull request, and an engineer decides whether to merge it. OpenAI says getting fixes right has been harder than finding causes. Agent-proposed changes were sometimes more complicated than necessary or introduced new problems, so each proposed fix now gets its own adversarial review to check that it addresses the root cause and to catch bugs. Some fixes also call for an architectural decision. Adding a cache, a store of recent results kept for speed, can make things faster and also serve stale data, and OpenAI says a tradeoff like that benefits from an engineer who knows the service.

The handoff matters as much as the fix. Each task carries an automatically generated report covering the regression, its impact, the root cause and the proposed fix, tuned to be easy to scan and built around metrics charts, CPU flame graphs (charts that show which parts of a program use the most processor time) and traces. Engineers can edit the pull request, write their own fix from the diagnosis, or leave comments, and a comment wakes an agent to address it. OpenAI is now comparing the fixes agents propose with the changes engineers merge, to find where proposals fall short and build evals from those cases. It’s also working on synthetic performance tests so agents can measure a fix before asking for review, and says it’s exploring whether agents can ship low-risk fixes without a human in the loop.

How OpenAI tests its own agents

OpenAI says updating a skill or upgrading the model can make the agent better at some cases and worse at others, so it tests changes before they go live, and it tests two things separately: how well the agent investigates and how useful its summaries are to engineers.

For investigations it backtests against past production incidents, turning them into a suite it can rerun after any change. Each case starts with a fresh Codex investigator and the original signal, stripped of details that would give away the answer. The agent can inspect code and query production telemetry within set limits, but it can’t use saved memories, read the incident write-up or see results from earlier runs, and OpenAI checks each transcript afterwards for anything that spoiled the test. It scores when the problem began, how big it was and which requests it hit, and it scores how the agent uses evidence to rank causes separately, since an agent can get the timing right and still blame the wrong deployment. Summaries are tested on snapshots of saved tasks, where a separate grader checks whether an engineer unfamiliar with the system can follow what happened, whether useful charts were kept and whether the proposed next step is backed by the evidence.

Two more lessons from running it

Protect engineers’ attention. Raw agent output is a wall of text, and agents can investigate more regressions than a team has time to read about. OpenAI’s first version gave engineers a UI full of findings and left them to decide which mattered. One regression can also show up as several incidents across alerts, monitors and channels, so the workflow now links related incidents, groups them by the metrics teams care about and ranks them by estimated impact.

Monitor the agents too. A broken profiler or a missing permission can stop an agent verifying a cause, and the final report may not show it. An agent now runs daily to read messages and logs from recent runs for tool failures and access errors. When the same error blocks several investigations it creates a task to fix it, an agent proposes the change when the cause is clear, and an engineer reviews it. After a fix merges, evals confirm that agents get past the step that used to block them.

What happens next

Today the Regression Workflow starts when an alert fires. OpenAI says it’s working on ways for the factory to find slowdowns and optimisations when no alert fires, by comparing the fine-grained data it already collects, including batches of CPU and memory profiles, heap dumps (snapshots of what’s in memory) and request traces across releases. An agent that notices a small CPU increase for one kind of request could open a task in the Regression Workflow to trace the cause, write a fix and open a pull request. Where OpenAI already knows what to look for, it’s encoding the pattern into a workflow of its own. Its service rightsizing skill, for example, helps agents spot autoscaling settings that keep more replicas running than a service needs.

OpenAI calls the Regression Workflow one instance of an agent factory pattern: tasks move through explicit states, agents advance them when work is ready, separate reviewers challenge the conclusions, and engineers step in when a decision needs judgement. Other workflows can reuse the same machinery with different states, instructions and events, and OpenAI says it hopes other service teams will learn from the process as they adopt agentic workflows in their own domains.

Who this affects

For people who use ChatGPT, OpenAI’s pitch is a line near the end of its post: “Next time you’re using ChatGPT and notice a slowdown, there’s a good chance the Performance Factory is already figuring out why.” It adds that the bottleneck today can be the wait for an engineer to review and merge the fix. For engineering teams, the post is a worked example of splitting a job into states, giving agents the steps that suit them and keeping a person on the decisions that carry tradeoffs. OpenAI closes with its own summary: “Codex already helps us build ChatGPT faster. Now it’s helping us keep ChatGPT fast.”

The terms in one place

TermIn plain words
RegressionA change that makes something slower than it used to be.
Root causeThe underlying change that explains a slowdown, not just where it shows up.
Metrics, logs, tracesNumbers over time, a running diary of events, and the path one request takes through services.
SkillA saved set of instructions an agent follows, such as perf-forensics.
EvalA repeatable test of how well an agent does a job, often built from past cases.
Adversarial reviewA second agent whose job is to disprove the first agent’s answer.
Pull requestA proposed code change that a teammate reviews before it’s merged.
Feature flagA switch that turns a feature on or off without a new release.
Flame graphA chart of which parts of a program use the most processor time.
Definitions are mine, written to match how OpenAI uses each term in its post.

Disclosure: I’m an OpenAI, Firecrawl and Devin ambassador.


Discover more from Sulat AI & Tech News

Subscribe to get the latest posts sent to your email.