Railway has published a detailed incident report into the security attack that began on 20 October. The company says a threat actor escaped isolation in its build system, later obtained a copy of Railway’s own monorepo, and on 21 October decrypted environment variables from at most 1.85% of workspaces, about 100,000, which it describes as an upper bound. Railway says the attacker was targeting AI provider credentials, and estimates unauthorised spend on stolen keys at about $740,000.
The 10 November report, by Jacob Cooper, Angelo Saraceno, Sarah Bedell and Luca Casonato, follows a preliminary write-up issued while recovery was still under way. Railway founder Jake, posting as JustJake, asked people to read the full post-mortem, and said attackers now have “alarmingly powerful tooling” and are hitting companies across the industry.
How the attacker got in
On 20 October, Railway says, the attacker exploited a previously unknown vulnerability in multi-tenant build infrastructure and partially escaped filesystem isolation. That let them read some files on the host virtual machine, including a credential the build servers used to authenticate to a subset of internal systems and to push images. An alert on anomalous registry API calls paged the on-call engineer later that day, and an incident response team began fixing the isolation escape, halting builds and watching the attacker.
The same build credential, intended only for work such as registry pushes, was also accepted as a trusted client by a self-hosted Temporal cluster that drove parts of the build process. Through that path the attacker read a limited subset of stored build histories, including environment variables and secrets passed to builds, and a recent build for Railway’s own support platform. Because the credential could fetch repository snapshots used for caching, they obtained a copy of Railway’s internal monorepo. The company says it has no indication that the attacker downloaded snapshots or repositories of Railway customers at that stage, and that up to that point the attacker appeared to be targeting Railway itself.
Railway revoked and replaced the build credential that evening and restricted what build servers can reach, then re-enabled customer builds after believing it had contained the escalation path. The attacker later tried to escape isolation again and failed. Railway says it did not yet know they had a copy of the codebase, and only realised that when probes hit endpoints that had been unused for weeks, months or years.
The second entry, and what was decrypted
On 21 October, after scanning the codebase, the attacker used an erroneously defined OIDC binding in one of Railway’s cloud environments to obtain a short-lived cloud credential. That was used to retrieve other internal credentials, including material that, with access to an internal network, allowed encrypted environment variables to be read from a production database, and an SSH key used by the CD system. The attacker reached a machine that ran orchestration tasks, found a longer-lived credential used to decrypt variables, and used it, with database access, to retrieve encrypted envelopes and then decrypt some of them.
Railway’s variables use envelope encryption: each environment has its own data-encryption key, itself encrypted by a root key held in an upstream cloud provider’s key management service. The company says one compromised key cannot expose the whole platform. Database logs, it says, show the attacker prioritising AI provider keys by variable name, and targeting customers with large numbers of variables. Railway disabled active cloud credentials, revoked static credentials, turned off key decryption and halted builds again. It says the attacker had no access to any customer data beyond a point that morning, though that fact was only established later the same day.
From available logs, Railway says at most 1.85% of workspaces, about 100,000, could have had any environment variables decrypted. The logs identify a subset of environments, not individual variables, so the figure is an upper bound. The attacker also read the stored histories of 353 builds, sealed variables included, and decrypted data included linked-account tokens (bring-your-own-key agent keys and GitHub OAuth tokens) for 1,852 users. Railway says it reset those tokens, notified each user, and identified no logins to Railway accounts using credentials taken in the incident.
What the attacker was after, and the cost to customers
Forensic work, Railway says, shows the attacker creating dummy accounts as early as 18 October, two days before the first page, with those early attempts unsuccessful. The company believes AI provider keys were the primary target and that they were passed on to “distributed transfer stations” for use in distillation. It also says an additional query matched cloud, database and payment-service credentials, but that forensics did not indicate customer databases were accessed.
During the incident Railway asked GitHub, GitLab, Anthropic, OpenAI, OpenRouter, SpaceXSI, Google and Stripe to revoke potentially compromised keys, and at first provided hashes of every key it held because it could not yet tell which were exposed. Unauthorised spend on stolen keys was estimated at about $740,000. Railway says it is working with providers on refunds and will backstop costs where a vendor will not. In the report it says AWS Bedrock and Google Gemini have initiated refunds under existing fraud policies; OpenAI and SpaceXSI are reviewing claims case by case through a shared bridge; it is in discussions with Anthropic and will cover Anthropic breached-key costs in the interim; and it will cover OpenRouter costs after OpenRouter enforced a no-refund policy. Customers are asked to report unauthorised charges through a form.
The company places the incident in a wider pattern of attacks on platforms that hold AI keys, and notes that OpenCode, Warp, Fireworks AI and several others disclosed similar thefts between 6 and 23 October. Wiz’s “Black October” research, it says, documents stolen AI credentials being checked and used by resale operations, and provides context, but “does not independently attribute the Railway incident.” Railway says the escalating response became “a contest of agent speed,” defending against “abliterated models and a harness similar to ARTEX, according to Wiz.”
Rebuild, and what customers felt
Existing deployments kept running, but builds were halted twice, sessions and tokens were invalidated, and new sign-ups were later paused until 2 November. Railway says it accelerated a new build system that had been planned for a gradual rollout the following week: each build now gets a dedicated ephemeral virtual machine, on top of the earlier isolation, with a credential that can only push a specific image. It also says it rebuilt CI, tightened secret decryption so only a secrets service can unwrap data-encryption keys, and rotated credentials, including a new root CA. A specialist forensics investigation is ongoing, and a complaint was filed with the FBI’s IC3.
Gergely Orosz, who writes The Pragmatic Engineer and says he has no Railway affiliation, called the report “extremely detailed” and said these attacks are “happening everywhere and FINALLY someone gives proper details.”
Discover more from Sulat AI & Tech News
Subscribe to get the latest posts sent to your email.
