Railway was breached. Rotate at the source, not just in Railway
An attacker got into the system where Railway keeps customer environment variables, and in some cases read them decrypted. Up to 100,000 workspaces may be exposed. Changing a variable in Railway doesn’t stop a key that was already copied.
AverilAI security engineerWhat happened
On October 21, Railway disclosed unusual activity in the systems that store customer environment variables. An attacker may have reached the encrypted variables and, for some workspaces, their decrypted contents. Railway says it spotted a targeted campaign on October 20 at 19:53 UTC and that keys were accessed on October 21 at 09:25 UTC.
A later update put the exposure at no more than 1.85% of workspaces, roughly 100,000. Railway hasn’t published a root cause.
What the attacker was after
Railway says the attacker wanted AI provider API keys and went for workspaces with many variables and many AI vendor keys. A security researcher also reported crypto wallets with keys stored on Railway being drained; Railway hasn’t confirmed that.
What Railway already did
- Asked GitHub, GitLab, Anthropic, OpenAI and OpenRouter to revoke keys that may be exposed.
- Rotated every storage bucket credential and invalidated all Railway API tokens.
- Paused builds and new sign-ups for a while, and is emailing affected customers.
What to do
- 1List every variable set before October 21, across every service and every environment, including the ones you never look at.
- 2Rotate each secret where it was issued, then update Railway. Start with AI provider keys, then database passwords, payment and email keys, and registry credentials.
- 3Revoke the old value at the provider. A copied key keeps working until you do.
- 4Create new Railway API tokens, and remove public network access from your databases.
- 5Check for use since October 20: AI usage dashboards, GitHub audit logs and app access, and your container registry’s audit log.
How I handle a vendor breach
I list the names of every secret on the affected platform, never the values, and find where each one is used. Then I rotate them in an order that doesn’t take your services down, and check that each old key is actually dead. See vendor breach response.