Workday takes some customer systems offline after a problem found in backup testing

Workday placed some customer systems into maintenance mode on Friday 13 November US time (Saturday 14 November in New Zealand) after a problem turned up while it was testing its backup systems, the San Francisco Chronicle reports. Employees at affected companies couldn’t log in to check pay or request time off. Workday hasn’t published a restoration time or confirmed that every customer is back.

Workday, the Pleasanton, California software company that serves more than 11,500 organisations, said the problem left some information in an inconsistent state and could cause errors when people opened accounts or finished tasks, Times Now reports. It apologised, said it was investigating the best way to restore service and promised updates every 30 minutes. As of Friday afternoon it hadn’t given an estimate for full restoration.

What the customer notice says went wrong

CTC News reports, citing a customer notice, that Workday found the problem while validating a weekly maintenance window for its disaster-recovery tenants. A disaster-recovery site is the standby copy of a system that takes over if the main one fails, and a tenant is one customer’s own Workday environment. The notice says services at the standby site unexpectedly connected to production databases, the live ones holding customers’ real data. That left the data in an inconsistent state and produced errors when customers opened tenants or ran jobs, integrations and tasks.

Workday then put affected production tenants in its WD5 environment into maintenance, and later some WD1 tenants, CTC says. Workday’s own status page for customers sits behind a Workday Community login, so the notice reaches the public second-hand.

Hiring-software maker Ashby says on its status page that Workday’s API service for the WD5 data centre was refusing all connections. Ashby couldn’t send data to or receive data from Workday for customers hosted there, so pushing candidates to Workday failed and syncs didn’t complete. Ashby rated its own incident minor. That matches WD5 as the first environment in CTC’s account.

How widely it hit

KRON4 reported more than 2,900 Downdetector reports at 3:15pm Pacific time on Friday, and some users saw the message “Workday is currently unavailable” at login. The impact varied by employer. CTC says some customers kept access while others lost it entirely.

CTC lists the work that stalled for affected staff: time tracking, benefits enrolment, hiring and internal approvals, hitting employees, managers, HR teams and job applicants. Customers with urgent deadlines were told to use their business continuity plans, and CTC says some employers were close to open-enrolment or year-end deadlines when access stopped.

The Chronicle lists Target, Walmart and Nordstrom among Workday’s customers, per Times Now. An anonymous source told The Newsground that Walmart, Target, Chipotle and Skechers were among the affected companies, and Times Now repeated the list in its headline. Walmart and Target hadn’t commented publicly at the time of writing.

Is it back?

Workday hasn’t said that service is restored. The monitoring site Down for Everyone or Just Me logged a problem of about two hours on Friday based on user reports, and it shows no current problem. CTC notes that monitors showing Workday’s public endpoints as reachable don’t show that each customer’s tenant works, because each tenant is its own environment. The restoration question stays open until Workday or an affected employer says otherwise.

It isn’t Workday’s first outage. KRON4 reported in November 2023 that a power outage at one of Workday’s data centres interrupted service, and Workday said then that it had restored service for 99% of affected customers.

A separate fault from Microsoft and the FAA

Workday’s problem followed Microsoft’s Outlook and Microsoft 365 outage by a day and landed on the day of the FAA’s nationwide ground stop. Each has its own stated cause. Microsoft blamed a network capacity upgrade, Transportation Secretary Sean Duffy blamed overloaded routers for the FAA stop, and Workday points to its backup-system testing. Homeland Security Secretary Markwayne Mullin said he saw no reason to connect the FAA stop to the Microsoft outage.

The reports don’t say how many Workday customers were affected or when every tenant will be back. Workday hasn’t published a full cause report.