Revolut says a missing server nobody was alerted to stopped failover in its card outage

Revolut says a power cut in one Google Cloud data centre in London stopped some of its databases failing over on 20 October because one of five voting servers in each affected group was already offline and no alert had fired, according to its technical overview of the outage, published on 13 November. Card payments were working again about six hours 40 minutes after the power failure began at around 06:00 UTC (7pm in New Zealand).

Gergely Orosz, who writes The Pragmatic Engineer newsletter, flagged the post on X on 14 November. He said Revolut had no monitoring on database nodes going down and that its five-node layout left no zonal resilience once one node failed. He said on 10 November that Revolut had stayed silent for three weeks, then responded and promised a postmortem after he told it he’d publish what was publicly observed.

Revolut shared the postmortem of their global cards outage of 5 hours. Some interesting (alarming?) notes:

1. They had no monitoring on database nodes going down (!!)

2. Across 3 regions, they ran 5 nodes, with #1. So a single node going down meant no zonal resillience (cont'd)

— Gergely Orosz (@GergelyOrosz) November 14, 2026

Two details differ from Revolut’s own account. Revolut says card authorisations and in-store payments were back by 12:40 UTC, about six hours 40 minutes in, and that all remaining services returned by 17:15 UTC, so the outage ran longer than five hours. It also says the affected databases run across three zones in Google Cloud’s London region, with replicas in another region for disaster recovery, not across three regions. The missing alert Revolut describes is narrower than no monitoring at all: it covers one failed coordination server in a group.

What failed

Revolut says it uses coordination servers to decide which copy of a database may accept new transactions. The servers work in groups of five spread across three zones, and at least three in a group must agree before a decision stands. That majority vote is called a quorum. A tool called the failover manager, installed with every database copy, asks the group which copy is the primary, the one that takes writes, and promotes a replacement in another zone if a zone goes down.

Revolut says some routine server replacements had failed to restore all five members of a group before the outage. Those groups kept working on four, so customers saw nothing, and Revolut says it wasn’t alerted that a member was missing while there was still time to restore it. When Zone B lost power, two more servers in each of those groups went offline. Two remained, which can’t form a majority of five, so the failover manager couldn’t promote a new primary.

Revolut says the groups were unevenly placed. Its deployment process favoured one zone when it placed the first server in each group, and it’s changing that rule.

Why recovery took until the evening

Revolut says its primary instance for visualising database metrics was deployed in Zone B, so engineers diagnosed the database layout through logs, command-line utilities and ad hoc monitoring instances built in the surviving zones before they could start fixing it. Some of its monitoring tools were offline, and early reports suggested several cloud regions, including its backup region, were affected. Once Revolut confirmed the power failure was confined to Zone B and that Zones A and C were up, it kept restoring services in those two zones under its recovery plan for a single-zone outage.

After the voting servers regained quorum, Revolut says some PostgreSQL clusters stayed read-only because replication lag, the delay before a copy catches up with the primary, blocked automatic failover. Engineers had to check the state of each copy and choose a safe recovery path. Revolut says it’s working with Google Cloud to understand a “suspected infrastructure brownout” and how it affected database recovery.

The recovery came in steps. By 12:40 UTC card authorisations and in-store payments worked again; from 15:45 UTC customers regained access to the app, updated balances and transfers; and by 17:15 UTC all remaining services were back. In New Zealand that’s 1:40am, 4:45am and 6:15am on 21 October. Revolut says backup processing let some card payments go through during the outage, and some payments made then took longer to appear in the app. It says all accounts, customer funds and data stayed safe.

What Revolut is changing

As of 13 November, Revolut says it has fixed the degraded coordination servers so failover works again, strengthened its database monitoring so it stays available in a single-zone outage, and deployed changes to process card reversals immediately once the backlog from stand-in processing, the backup route that approves card payments while the main systems are down, clears.

Work still under way, per Revolut: monitoring of coordination-group membership, voter health and failure tolerance to warn when quorum is at risk; end-to-end monitoring of its alerting pipelines; warnings when it nears cloud resource quotas or runs short of the network addresses a zonal failover needs; zonal failover tests extended to cover the conditions seen on 20 October and run more often, including a check that services can return under load; and sturdier supporting tools that need less manual work when a zone goes offline.

Revolut writes that “while the initial trigger was an external infrastructure failure, our architecture should have absorbed it seamlessly.” Its post doesn’t say how many coordination groups were short a member, how many databases they served or whether affected customers will be compensated. I’ve asked Revolut’s press team those questions and will add its answers here.