Software glitch caused multi-provincial 911 outage: Bell

A 911 outage that affected Manitoba and four other provinces for about 90 minutes last month was caused by a software failure, according to Bell Canada’s report to a federal regulator.

An abridged version of the report, which was made public Friday, did not say how many callers were affected when Bell’s next-generation 911 platform stopped receiving new calls and transferring new calls to local 911 communications centres due to the glitch.

The outage, which lasted from 2:13 a.m. to 3:40 a.m. central time, was not caused by a cyberattack or any change activity, Bell said.

“Rather, the outage was caused by a latent software issue that triggered an unexpected condition in certain network components within the (next-generation 911) network,” Philippe Gauvin, Bell’s assistant general counsel, wrote in the report, which was published on the Canadian Radio-television and Telecommunications Commission’s website.

Bell told the CRTC a permanent fix was implemented that morning.

The outage affected 79 communications centres, or public safety answering points (PSAPs), and 62 originating phone network providers in Manitoba, Ontario, Quebec, New Brunswick and Prince Edward Island, the report said. That day, Bell told reporters Nova Scotia was also affected.

The report said callers from landlines that still rely on a legacy copper wire network within Bell’s 911 footprint were unable to get through if the call was destined for a public safety answering point that had not yet transitioned to next-generation 911 service.

Missed call logs were provided to all relevant public safety answering points, “and these PSAPs confirmed that there were no critical public safety incidents reported with respect to these calls,” Gauvin wrote.

Manitoba’s two primary PSAPs, which dispatch first responders to an emergency, are located in Winnipeg and Brandon.

Gauvin said calls from internet protocol-capable networks were redirected by originating carriers to their third-party operators, who then verbally relayed each caller’s location and dispatch information to the appropriate PSAP.

The backup system worked for the “vast majority” of calls by Bell customers, the report said. Bell believes most of the other carriers successfully rerouted calls, although it is waiting for information from some companies.

Portions of the report were redacted, which Gauvin said was necessary to prevent a “malicious actor” from identifying or exploiting weaknesses in 911 infrastructure.

The report said Bell is conducting an end-to-end review of its next-generation 911 infrastructure following the outage. The telecom told the CRTC it has taken steps to redirect 911 calls from legacy copper wire network customers to a third-party operator if an outage occurs again.

Some automated alarms did not function during the outage. Bell said it has improved monitoring and alarming systems.

Bell did not notify PSAPs, other telecoms, the CRTC and governments about the outage until after service was restored.

This happened “due to delays in determining the exact scope of the impact and assembling the contact information for all targeted entities,” Gauvin wrote, adding Bell’s operational focus during the event was on restoration.

The company has since prepared distribution lists and pre-drafted message templates for notifications in the event of future outages. Some contact addresses Bell had collected from PSAPs prior to the Sept. 17 outage were not functioning that day, the report said.

Premier Wab Kinew indicated Sept. 22 the province is open to all potential options, including switching 911 providers, after multiple recent outages in Manitoba. Kinew has said he raised concerns with Bell Canada CEO Mirko Bibic and Prime Minister Mark Carney.

“Bell takes its responsibility as a 911 Service Provider with the utmost seriousness and is committed to the reliability, resilience, and continuous improvement of emergency calling services,” Gauvin wrote in the report published Friday.

Montreal-based Bell has positioned next-generation 911 service as more reliable than the “legacy architecture” currently used in Manitoba. The telecom previously said it expects Manitoba’s PSAPs to move to the new system by Dec. 31.

chris.kitching@freepress.mb.ca