The recent report commissioned by the CRTC and conducted by Xona Partners has shed light on the significant 2022 Rogers outage that impacted over 12 million personal and business customers for more than 26 hours. The outage affected both wireless and wireline services before services started being restored.
Xona Partners detailed the findings of the report, highlighting the events leading up to and during the outage, as well as the measures Rogers has implemented to address network deficiencies. The report was based on an independent review of responses from Rogers, multiple rounds of questions, meetings with technical and management staff, and information provided in response to the CRTC’s request for information (RFI).
So, what exactly caused this infamous outage? According to Rogers, the outage was triggered by an update issue that they attributed to their partner Ericsson. Xona Partners explained that Rogers’ wireless and wireline networks share an IP core network used to manage and route data traffic internally and externally. The outage occurred during a phase of upgrading this core network in July 2022, disrupting both types of services.
The specific cause of the outage was an error in configuring distribution routers within the IP network. Removing an Access Control List policy filter from these routers led to an overwhelming flood of IP routing information into core network routers, causing them to crash due to capacity overload.
The absence of overload protection on core network routers could have prevented this failure. The configuration error stemmed from a change management oversight by Rogers staff during a network upgrade phase. This oversight went unnoticed due to insufficient scrutiny and testing after previous phases were successful.
Recovery efforts were prolonged due to a reliance on a failed management network during the outage preventing remote troubleshooting access. Lack of redundant connectivity from alternative providers necessitated physical visits to remote sites for troubleshooting.
To prevent future outages of such magnitude, Rogers implemented critical changes such as improving router configurations, creating a separate management network for troubleshooting, adding backup connectivity from third-party providers at key sites, validating router configuration changes with new tools, and separating IP core networks for wireless and wireline services.
Moving forward, recommendations include considering low earth orbit satellite constellations for backup connectivity at remote sites and emerging direct-to-cell constellations for emergency 9-1-1 calling.
The CRTC funded Xona Partners $230,000 for a forensic technical review of the Rogers outage. This incident disrupted over 12 million Canadians and impacted payment networks relying on mobile services for debit/credit card processing.
If you were affected by the Rogers outage two years ago, share your experience with us! How did it impact you? Let us know in the comments below.
