Global CrowdStrike disruption brings to mind lessons from 2022 Rogers outage

Hey there, time traveller!
This article was published 19/07/2024 (453 days ago), so information in it may no longer be current.

TORONTO – Canadians who woke up Friday to a worldwide technology outage that disrupted airlines, hospitals and banks may be thinking back to a widespread telecom outage from two years ago.

The July 2022 outage of Rogers Communications Inc.’s networks lasted more than 24 hours and affected more than 12 million customers while paralyzing communications across a number of sectors, including health care, law enforcement and the financial industry.

At the heart of the massive disruption on Fridayis CrowdStrike, a cybersecurity firm that provides software to scores of companies worldwide. CrowdStrike said the problem occurred when it deployed a faulty update to computers running Microsoft Windows, noting the issue behind the outage was not a security incident or cyberattack.

Friday’s outage put into perspective yet again how reliant Canadians are on certain technology systems. The IT issue was trigged by a software update, while in Rogers’ case, a configuration error during a network upgrade led to the outage.

Experts have said Friday’s outage serves as an example of the risks inherent to companies relying on common technology providers amid market consolidation.

“All of these systems are running the same software,” cyber expert James Bore told The Associated Press.

“We’ve made all of these tools so widespread that when things inevitably go wrong — and they will, as we’ve seen — they go wrong at a huge scale.”

CrowdStrike has said a fix is on the way. Still, chaos deepened hours after the problem was first detected.

Friday’s disruptions began when a faulty update was pushed out from CrowdStrike for one of its tools, Falcon. In a statement about the situation, the company said the defect was found “in a single content update for Windows hosts” — noting Mac and Linux systems were not affected.

The consequences of this kind of technical problem have been far-reaching due to the fact so many companies that rely on CrowdStrike for their security needs with Windows as their operating system.

While not everyone is a CrowdStrike client, it is one of the leading cybersecurity providers, particularly in the transportation and banking sectors that have a lot at stake in keeping their computer systems working.

Airlines unable to access their systems cancelled and delayed flights around the world Friday, including for Canadians. Delays were expected to last up to 72 hours for some travellers.

Both British Columbia and Newfoundland and Labrador reported their health systems were affected, as did multiple Toronto-area hospitals.

Two years ago, Canadians had a similar reminder of the interconnectedness of modern technology during the Rogers outage. In that case, communications were affected, which meant many 911 services couldn’t receive incoming calls and debit transactions were paused when Interac was knocked offline.

Several hospitals reported being affected by the outages at the time.

Small business owners were among those hardest hit by the outage, which left them unable to process debit card payments.

A report on the Rogers incident by Xona Partners Inc., which was delivered to the CRTC earlier this month, said the outage was caused by a configuration error during a network upgrade, prompting a flood of data to core network routers, which crashed.

The network failure could have been prevented if the core network routers had been configured with an overload limit, the report said.

Rogers has sought to strengthen the resiliency of its networks since the outage by separating the IP core for its wireless and wireline networks and improving the processes for change management and incident management.

The telecommunications company said it completed a full review of its networks and implemented all recommendations contained in the independent report.

In September 2022, about a month after the Rogers outage, Canada’s major telecom companies reached a formal agreement to “ensure and guarantee” mobile roaming and other mutual assistance in the case of a future major outage.

Rogers said Friday its networks were unaffected by the global disruptions related to CrowdStrike.

— With files from The Associated Press

This report by The Canadian Press was first published July 19, 2024.

Companies in this story: (TSX:RCI.B)