Why did ChatGPT, WhatsApp and X suddenly crash? The untold story behind the Outage…

A “latent bug” was to blame for a software bug that Cloudflare released on Tuesday morning, and which knocked out the infrastructure provider’s systems and prevented some services from working, including ChatGPT, WhatsApp, Canva and X. Cloudflare detected a latent bug in the company’s “core infrastructure” during a “routine configuration change,” and discovered that the bug triggered the failure, said the company in a status update.

Cloudflare had a latency issue at 8:00 a.m. ET, which the provider updated on its status page, but Cloudflare said that it found a cause and had deployed a fix. After that, the system “quickly stabilized,” and a few hours later, the company confirmed on the status page that it had applied the patch.

Cloudflare CTO Dane Knecht also confirmed that the failure was not due to an outside attack on the company’s infrastructure.

So, what exactly is a latent bug?

A latent bug (also called a hidden bug) is a dormant error in infrastructure that a change or particular condition can uncover in certain software or hardware. This could include the use of a different operating system or the addition or removal of new code, or simply a change in configuration. They are called latent bugs because they lie dormant and don’t get exposed during usual or expected use cases and testing.

In this case, the latent bug in Cloudflare’s bot-mitigation systems was triggered by a “routine configuration update.” This triggered a series of performance issues that eventually resulted in a partial network collapse.

Why this matters

The sheer number of platforms and services affected in the Cloudflare incident is an indicator of how much of the Internet is built on a handful of key providers: Cloudflare alone provides infrastructure to around 20 % of all websites on the Internet. The company has data centres in 330 cities, and customers span 13,000 networks from major telcos and cloud platforms to individual enterprises.

When one of these “keystone providers” has an issue, it can have repercussions across a wide swath of the internet. “The major cloud providers are enormous utilities that everyone on the internet now runs on,” says Larry Cannell, senior director of digital trust and security at Fortinet.

When services were interrupted, for some users, this translated into slowed page loads, service interruptions, login problems and a pervasive sense of the web not working. For businesses, this event was another reminder that a provider considered by all accounts to be rock-solid can be vulnerable to, and can experience, latent bugs.

Cloudflare itself summarized the incident on a status update: “We failed customers and the broader internet.”

What’s next for Cloudflare?

Cloudflare has promised to provide a full incident report and to improve systems internally to avoid this kind of incident. The fact that this incident was caused by a routine update is likely to lead many operators, developers and IT managers to consider their own degree of reliance on third-party infrastructure, and to take steps to improve redundancy and testing measures.

It may seem like the Cloud is ethereal, but it’s built on very real (and sometimes quite fragile) foundations, and hidden bugs in those networks can bring even the largest platforms to a halt.