LONDON (AP) — Fastly, the company that was hit by a major outage that caused many of the world’s top websites to go offline briefly, blamed the problem on a software bug that was triggered when a customer changed a setting.
The problem at Fastly meant internet users couldn’t connect to a host of popular websites including The New York Times, the Guardian, Twitch, Reddit, and the British government’s homepage.
A company executive said in a blog post late Tuesday that the outage was caused by an undiscovered bug included in a software update and the company is trying to figure out why it wasn’t detected during testing.
Nick Rockwell, Senior Vice President of Engineering and Infrastructure, said the bug was detected within a minute and the company was able to return 95% of its network to normal within 49 minutes.
“This outage was broad and severe, and we’re truly sorry for the impact to our customers and everyone who relies on them,” he wrote in the blog.
Going forward, Rockwell says Fastly is deploying its bug fix across its network and it’s conducting a “complete post-mortem” of the processes and practices it followed during the incident.
“We’ll figure out why we didn’t detect the bug during our software quality assurance and testing processes,” wrote Rockwell. “We’ll evaluate ways to improve our remediation time.”
In the end, Rockwell says the company should have anticipated the outage.
“We provide mission-critical services, and we treat any action that can cause service issues with the utmost sensitivity and priority,” he wrote. “We apologize to our customers and those who rely on them for the outage and sincerely thank the community for its support.”
