CenturyLink f-up
https://www.geekwire.com/2018/report-huge-centurylink-outage-caused-bad-networking-card-colorado/ Can someone comment on how was it possible for a single piece of networking equipment f.up network and phones in multiple state, including 911 lines for two days?
While gear can randomly fail or act up, I don’t believe a single network management card could cause this sort of outage. Personally I think there are other factors not disclosed both publicly and internally. I think there may have been some sort of configuration change that was made (intentional or not) that caused the issue. Since propagation can take a long while, the impact of the change wasn’t fully realized until much later and not all at once. Probably starting with smaller local outages and grew to the behemoth outage as propagation went on. Just my thought.
They are cheap as fuck with their infra, run it way oversubscribed, behind patch levels, and use plenty of trash gear which even the pics show with racks of super micro garbage. Not really a shock.