Dear Carrytel Customers,
We sincerely apologize for the network outage that occurred on 1st November 2020 at 2:45pm EST impacting some of our cable customers in Ontario. We are completely aware that this outage had a significant impact on our customers as during this COVID-19 pandemic, many Canadians rely on their home internet to work, study and spend time with loved ones. We are so sorry about the negative impact this outage may have caused. Please give us a chance to explain what happened this past Sunday.
Below is the timeline of events which occurred from the time of realization of the outage to the time of service restoration:
1st November 2020
2:45pm: Carrytel senior technician reported outage and started to investigate.
2:50pm: The outage cause has been isolated to our Cable Provider's facility impacting some customers in Ontario, provider has been informed, Carrytel senior technician dispatched to the data center to collaborate.
4:28pm: Cable Provider's Network Operations team detected a potential fiber cut approximately 15km away from the Head End but the exact location was still unknown.
5:54pm: Initial onsite tests were conducted. Fiber technicians and planner collaborated to further isolate the fault and create a restoration plan. Carrytel’s senior technician was standing by in the data center.
7:50pm: The issue was further escalated to corresponding Network Partner’s teams
11:54pm: Network technicians on site had not located any physical damage. Technicians were aware of current track work/construction being completed in the area and were en route to further investigate.
2nd November 2020
2:54am: Technicians located two faults close to the main track and began formulating a plan of action to safely complete further work as it is a live track. More crew members were dispatched to assist with the restoration process.
5:59am: The fiber contractor was provided access to start the restoration work. All teams remained onsite and were exhausting all efforts to restore services.
9:51am: The Network teams still had not located the second fault and investigated to isolate it. Further testing continued.
10:54am: The second fault had been located. The restoration process started.
12:19pm: The restoration process was fully completed.
Our network team monitors our service performance closely 24/7, in order to minimize our response time under any circumstances. However, as the fiber cut happened on our Network Partner's infrastructure, the collaboration caused delay of update to our customers. We tried our best to update you with as much information as we could in real-time through our website and social media channels. And we hope that this letter provides you a better explanation of how fast we respond to outage and how we collaborate with our backend providers when resolving network issue.
Our growth can't continue without your support. We truly appreciate your support during this time.
Carrytel
November 2nd, 2020