On April 18, 2018, at approximately 2:20/1:20 PM EST/CST, Element 74’s monitoring software alerted our Network and Support teams of a data center network connectivity issue affecting our web servers co-located at Cogeco/Peer 1’s San Antonio Data Center. A portion of Element 74’s websites were impacted by this outage.
Our network operations partners at Cogeco/Peer 1 Networks and their 3rd party backbone Internet providers immediately began to troubleshoot this issue with the utmost priority. In the following time period, it was discovered that the data center’s Main Fiber Optic Backbone had been damaged, causing a complete loss of Internet connectivity. No other systems were affected, including data center security, emergency backup power, or climate control.
Approximately 6:20/5:20 PM EST/CST the damaged segment of the fiber optic trunk was identified and the estimated repair time was 4 hours.
By 7:30/6:30 pm CST/EST the fiber optic backbone connection was repaired and web traffic to the servers resumed while repairs were finalized. By 9:00/8:00 PM EST/CST traffic began to normalize, and website operations were confirmed to be stable.
Please note that integrity of customer and website data and other website assets were not impacted during the service interruption. The actual web servers were running in their normal state, their connection to the internet was just unavailable. We have confirmed through overnight and continuous monitoring that all websites and web services hosted on Element 74’s servers in the San Antonio Datacenter are now fully operational and available to Internet users.
Our team appreciates your patience and understanding in this incident, and apologize for any inconvenience.
Element 74 Support Team