Railway Build Breach Exposes AI Keys

Sometime between October 20 and October 22, Railway’s build system started compiling something its customers didn’t write. The attacker did not access Railway user accounts. They went after the machines that assemble and configure customer apps, where deploy-time credentials are injected into builds.

Railway’s preliminary report on the October security incident describes a “sustained and targeted campaign” that began in the build infrastructure and ended in the systems that store customer secrets. The stated target: AI provider API keys. At most 1.85% of workspaces had some environment variables potentially compromised, roughly 100,000 workspaces. That is 100,000 inference budgets someone else could spend.

No CVE is at the center of this incident. This is an infrastructure intrusion, not a flaw in a shipped product. Railway says the attacker did not access Railway user accounts, and that running deployments stayed online while new builds were paused. What the intruder reached was the machinery that builds and configures customer services, which is exactly where deploy-time credentials live.

What Railway Disclosed About the Build System

Railway’s timeline starts at 13:58 UTC on October 20, when monitoring paged the on-call engineer about higher-than-normal latency. During the next three hours, the team identified a hostile party attempting privilege escalation in the build infrastructure; by 16:51 it had halted build pipelines to protect users. The incident was declared on the status page at 17:03 UTC.

Railway’s report opens by dating awareness of the campaign to 19:53 UTC, the same timestamp its timeline assigns to evidence of escalation at larger scale across several parts of the system. The first detection was the monitoring page at 13:58.

The second day is the part worth studying closely. From 05:27 to 07:25, the attacker used compromised credentials to reach a CI environment and decrypted artifacts that held additional API access. From 07:25 to 09:18, Railway found database evidence of an unauthorized query against encrypted environment variables, and a subset of those variables were decrypted “using a combination of the compromised credentials.” Between 09:18 and 09:25, staff read logs from attacker payloads and saw the specific API key names being searched for. Vendors were asked to revoke keys at 09:25 UTC, minutes after that discovery.

The status page for incident QH8L8B3E fills in the operational shape. Builds were disabled, then re-enabled in tiers through October 22, with Hobby workspaces returning at 21:59 UTC and Free and Trial at 23:22 UTC. Railway also says it accelerated a move from third-generation multi-tenant builders to fourth-generation single-tenant environments that had been planned for the following week.

Read the sequence as a privilege problem rather than a storage problem. Nobody broke a cipher. The attacker used stolen build and CI credentials to reach the decryption workflow, so the question worth asking about any secrets system is who can request decryption, and from where.

The 1.85% Figure and the 100,000 Workspaces Behind It

A percentage that small is still 100,000 workspaces, and the containment came with a cost that is easy to miss. Because the full keys were envelope encrypted, Railway says, “this query alone cannot identify specifically which keys in the affected customers project/service were affected.” The company therefore notified external providers about everything it had stored for some services. Railway says it told vendors “up to ‘all of keys we stored, for all users on Railway,'” which means the number of customers who saw a revocation notice from GitHub, GitLab, Anthropic, OpenAI, OpenRouter or Stripe is larger than 100,000.

Railway also states that being notified by a vendor about a revocation “does not imply that their credentials were impacted by incident.” A GitHub token that stops working this week may simply have been in the blanket list. The October 21 security disclosure makes the same point from the other direction: impact may be limited to a subset of users, but all users were asked to rotate as a precaution.

The status page repeats the limit in plain language. Workspaces outside the higher-risk groups “are not confirmed safe from compromise.” That is why the practical advice is to rotate everything rather than to check whether your workspace landed inside the 1.85%. Railway sorted notifications into risk buckets using attacker behavior: workspaces with very large numbers of variables, and environments holding many AI vendor keys. Those buckets describe targeting, not proof of access.

As we covered in our analysis of that Vercel incident, the contested question was the same: how do you scope an exposure when secrets are stored centrally and read through a shared service? Railway’s answer still leaves roughly 100,000 workspaces holding credentials that may have been read.

Distillation Economics and Why Build Platforms Hold the Keys

Railway states the intent directly: “this attack was attempting to obtain AI provider API keys to perform distillation attacks.” Distillation here means querying a model at volume and using the responses to train or tune a cheaper model that reproduces part of the original’s behavior. The queries only pay off in bulk, so the attacker needs sustained, billable inference capacity that is not tied to their own payment method. A stolen key is exactly that: capacity without a card on file.

Environment variables are the obvious source. Deployment platforms encourage teams to keep provider keys there because the runtime needs them at process start, and one workspace can hold a dozen. A typical AI project carries a chat model key, an embedding key for search, a vision key for uploads, plus the database URL and registry token that Railway also names as typical contents. Railway’s forensics point to the two risk buckets it used for notifications, and both are proxies for concentration.

Security teams already have a label for the cost side of this. OWASP LLM10:2025 Unbounded Consumption covers excessive and uncontrolled inference, including the resource-management and unexpected-cost risks that follow. A stolen key moves that consumption onto the key owner’s invoice with no application-layer limit in the way. Dick O’Brien of Symantec’s Threat Hunter Team explained the underlying logic to The Hacker News this way: “API keys for AI models are valuable because anyone holding one can run inference at owner’s expense.” He also said his team has no evidence linking the activity it observed to this intrusion, so the quote explains the market, not the campaign.

Railway’s report adds a capability detail that defenders should sit with. Initial evidence, it says, suggests the actor “had access to abliterated models and substantial compute resources which enabled them to perform highly sophisticated exploits against our infrastructure.” Refusal-stripped open-weight models and rented GPUs are now part of the toolchain aimed at infrastructure, not only at content filters.

A Stored Gemini Key and the Credential Classes at Risk

The clearest cost signal comes from Railway’s own community forum rather than the official report. A customer wrote in the Station thread on the October 21 incident that a third party used their Gemini API key, stored as an environment variable on Railway. Google’s anomaly detection flagged the usage the same day: EUR 133.12 observed against a predicted baseline of EUR 0.02. Google declined the refund request, leaving EUR 118.35 in charges, and the customer counted about one day of developer time spent rotating every secret in the project. That is roughly 6,600 times the predicted spend, from a stored key.

A second poster in the same thread described a Gemini key billed USD 2.22 on October 1, then unused until charges of USD 100 on October 21 and USD 200 on October 22, after missing an email headed “Early signal anomaly: Unusual cost spike of $103 on Gemini API in project.” Two caveats belong next to those numbers. These are customer accounts on a public forum, not findings from Railway’s investigation, and Railway has not confirmed either amount. They are evidence of key misuse rather than proof of any specific decrypt path.

What they show is the shape of the loss at small scale: a bill, a revocation notice, and a day of engineering time. They also show detection working. Provider anomaly systems flagged the spike within a day. In the second case, the alert was missed before the next charge landed. That is a monitoring-routing problem, not a detection failure.

AI keys are not the only thing in that store. The October 21 disclosure names API keys, database credentials and registry credentials, and asks customers to rotate private registry credentials too. That maps to CWE-522, insufficiently protected credentials, which covers secrets stored where the surrounding system can read them without a second barrier. A vault for agent credentials, such as the proxy-and-placeholder pattern we examined in our review of OneCLI, removes plaintext from the process environment entirely.

Rotation Order, Spend Caps, and Usage Detection

Railway recommends that every customer who signed up before October 21 revoke and reapply sensitive environment variables stored on the platform, and its engineering blog now carries per-provider guidance for checking anomalous access on AWS, GCP and GitHub. Order matters more than the checklist. Rotating in the platform first and upstream second leaves a window where the old key is still valid at the provider and the new one is already documented.

A sequence that holds up under audit:

• Inventory before touching anything. Dump each service’s variables, scan the dump for live secrets, then delete the dump. GitGuardian’s write-up on this incident documents the approach with the Railway CLI and its scanner.
• Revoke at the provider, not only in the platform. Issue a new key, confirm the old one now fails authentication, and only then update the value in Railway.
• Redeploy dependent services. Railway’s status page says code from before the incident may fail to redeploy and advises disconnecting and reconnecting the GitHub integration if deployments fail.
• Rotate the adjacent credentials Railway names: database passwords, private registry tokens, and any deploy keys in the same project.
• Treat sealed variables as in scope. Sealed values cannot be read back through the UI, API or CLI, which limits casual exposure, but the disclosure does not say whether they were in the decrypted subset.
• Rebuild CI credentials. Railway invalidated API and project tokens created before the incident, and rolled GitHub App installations, so a stale token is a deployment failure waiting to happen.

Monitoring is the half most teams skip. Put a hard spend cap and a budget alert on every AI provider account, not just a dashboard, and route those alerts to an on-call address a human reads at 2 a.m. Break usage down per key or per project so a spike can be attributed to one service, and log which key each service presents so revocation has a known blast radius.

What to look for in provider logs, beyond raw spend: a key that suddenly serves a different model mix than your app requests, traffic from regions you do not deploy in, request volume that exceeds your application’s own telemetry, and usage during hours when your jobs do not run. Distillation looks like a very large number of short, repetitive completions against one model, so token counts rising while your own request counts stay flat is the signal worth alerting on.

For structural limits, split keys by environment and by service instead of reusing one across staging and production, and put an egress allowlist in front of anything that calls a model provider. Our DevOps security guide for 2026 covers the CI/CD side of this. If the build step can read every variable in the project, then the build step is part of your secret inventory.

What the Preliminary Report Leaves Open

Railway calls the document preliminary and says forensics continue with external incident response experts and authorities. The decryption mechanism is described only as a combination of compromised credentials recovered from a CI environment, with no detail on which system issued them or what scope they carried. Whether any distillation workload actually ran on stolen keys is not established. The attack is described as “attempting” to obtain keys for that purpose, which is a statement about intent drawn from payload logs and behavior.

Source code exposure is unresolved. Railway writes that it is not ruling out that the attacker accessed user source code, and will confirm at the final report. The scope question that matters for compliance is also open: Railway can bound the affected workspaces but cannot determine which individual variables were decrypted. That is precisely what the customer in the Station thread asked for, because a GDPR Article 33 assessment needs to know which personal data was exposed, not which bucket it sat in.

Third-party writeups that appeared within a day deserve caution. Railway has not published an initial access vector. The defensible reading is the narrow one: a build-platform compromise reached secret storage, envelope encryption capped the damage at a subset of workspaces, and roughly 100,000 workspaces should treat their stored credentials as potentially readable.

For related cases, our coverage of the GitHub internal repository breach and the Bitwarden CLI supply chain compromise shows the same pattern from different angles. Developer infrastructure holds the credentials, and the theft is monetized downstream.

Related Reading

More in-depth coverage from this blog on closely related topics:

• Waymo Robotaxi Scaling: $5B Loan, 20K Cars
• Ransomware Negotiator Legal Risk

Sources and References

Sources cited while researching and writing this article:

• Preliminary Report: Oct 21st Security Incident | Railway Blog
• Railway Platform Outage , Resolved | Railway Status
• Security Disclosure: Wednesday October 21st | Railway Blog
• OWASP LLM10: 2025 Unbounded Consumption
• Railway Says Intruder May Have Accessed Customer Secrets, Urges Credential Rotation
• Security incident 21 Oct 2026: unauthorized charges and request for credit and incident details – Railway Central Station
• Engineering | Railway Blog
• New Railway Incident: Treat Every Env Variable as Exposed