Cloudflare issued a blog post explaining how Verizon sent a large chunk of the internet offline this morning after it wrongly accepted a network misconfiguration from a small ISP in Pennsylvania. The outages affected Cloudflare, Facebook, Amazon, and others. The Register reports: For nearly three hours, network traffic that was supposed to go to some of the biggest online names was instead accidentally rerouted through a steel giant based in Pittsburgh. More than 20,000 prefixes — roughly two per cent of the internet — were wrongly announced by regional U.S. ISP DQE Communications: this announcement informed the sprawling internet’s backbone equipment to thread netizens’ traffic through one of DQE’s clients, steel giant Allegheny Technologies, a rerouting that was then, mindbogglingly, accepted and passed on to the world by Verizon, a trusted major authority on the internet’s highways and byways. And so, systems around the planet automatically updated, and connections destined for Facebook, Cloudflare, and others, ended up going to Allegheny, which black holed the traffic.
Internet engineers suspect that a piece of automated networking software — a BGP optimizer called Noction — used by DQE was to blame for the problem. But even though these kinds of misconfigurations happen every day, there is significant frustration and even disbelief that a U.S. telco as large as Verizon would pass on this amount of incorrect routing information. The sudden, wrong, change should have been caught by filters and never accepted. […] One key industry group called Mutually Agreed Norms for Routing Security (MANRS) has four main recommendations: two technical and two cultural for fixing the problem. The two technical approaches are filtering and anti-spoofing, which basically check announcements from other network operators to see if they are legitimate and remove any that aren’t; and the cultural fixes are coordination and global validation — which encourage operators to talk more to one another and work together to flag and remove any suspicious looking BGP changes. Verizon is not a member of MANRS.