On Saturday, Chief Strategy Officer Kyle York of Internet routing company Dyn DNS posted a detailed account of the massive attack that kneecapped the Internet, beginning early Friday morning.
Starting at approximately 7:00 am ET, Dyn began experiencing a DDoS attack. While it’s not uncommon for Dyn’s Network Operations Center (NOC) team to mitigate DDoS attacks, it quickly became clear that this attack was different (more on that later). Approximately two hours later, the NOC team was able to mitigate the attack and restore service to customers. Unfortunately, during that time, internet users directed to Dyn servers on the East Coast of the US were unable to reach some of our customers’ sites, including some of the marquee brands of the internet. We should note that Dyn did not experience a system-wide outage at any time – for example, users accessing these sites on the West Coast would have been successful.
After restoring service, Dyn experienced a second wave of attacks just before noon ET. This second wave was more global in nature (i.e. not limited to our East Coast POPs), but was mitigated in just over an hour; service was restored at approximately 1:00 pm ET. Again, at no time was there a network-wide outage, though some customers would have seen extended latency delays during that time.
News reports of a third attack wave were verified by Dyn based on our information. While there was a third attack attempted, we were able to successfully mitigate it without customer impact.
York went on to describe the incident as a “sophisticated, highly distributed attack involving 10s of millions of IP addresses.”
He said the company’s investigation of “the nature and source of the attack” is ongoing, but “with the help of analysis from Flashpoint and Akamai,” Dyn has determined that one source of the attack was a huge number of devices “infected by the Mirai botnet.”
Flashpoint and Akamai are Internet security companies. As Flashpoint explains, the Mirai botnet is basically an army of electronics transformed into a zombie horde of Internet weapons by a computer virus. The masters of the virus can activate this army at their pleasure, causing the infected devices to begin attacking targeted websites with Distributed Denial of Service (DDoS) attacks – the Internet equivalent of jamming a company’s phone lines with crank calls, so they can’t talk to their legitimate customers.
What makes Mirai, and the attack on Dyn DNS, really scary is that most of the attacking devices aren’t actually computers:
Mirai botnets were previously used in DDoS attacks against security researcher Brian Krebs’ blog “Krebs On Security” and French internet service and hosting provider OVH. Mirai malware targets Internet of Things (IoT) devices like routers, digital video recorders (DVRs), and webcams/security cameras, enslaving vast numbers of these devices into a botnet, which is then used to conduct DDoS attacks. Flashpoint has confirmed that at least some of the devices used in the Dyn DNS attacks are DVRs, further matching the technical indicators and tactics, techniques, and procedures (TTPs) associated with previous known Mirai botnet attacks.
While Flashpoint has confirmed that Mirai botnets were used in the October 21, 2016 attack against Dyn, they were separate and distinct botnets from those used to execute the DDoS attacks against “Krebs on Security” and OVH. Earlier this month, “Anna_Senpai,” the hacker operating the large Mirai botnet used in the Krebs DDoS, released Mira’s source code online. Since this release, copycat hackers have used the malware to create botnets of their own in order to launch DDoS attacks.
Brian Krebs, the security consultant who was the unhappy record-holder for DDoS attacks until the assault on Dyn, explained on his blog that Mirai bots are using a new attack technique that sacrifices anonymity for raw power.
The bot horde attacks DNS servers with a tidal wave of properly formatted Internet address requests – the process that allows human users to type easily-understood website names like “krebsonsecurity.com,” trusting their machines to instantaneously work out the proper route for communicating with the desired servers. These requests can be traced back to their source fairly easily, but the attackers don’t care, because the virus-infested sources are everywhere.
That’s literally how senior security advocate Martin McKeay described the system, as quoted by Krebs: “Someone has a botnet with capabilities we haven’t seen before. We looked at the traffic coming from the attacking systems, and they weren’t just from one region of the world or from a small subset of networks – they were everywhere.”
How does one create a botnet encompassing millions of devices around the world? Krebs quoted security analysts who said Mirai, and the imitation malware that has been unleashed since Mirai’s source code was published, roams across the Internet of Things, looking for devices with easily penetrated security, especially devices with no password, or that use the default password provided by the manufacturer. Other devices have security flaws which Mirai and its hellish stepchildren are programmed to exploit.
In a new post on the Dyn attack, Krebs cited research that suggested Mirai primarily infests “compromised digital video recorders (DVRs) and IP cameras made by a Chinese hi-tech company called XiongMai Technologies,” whose components are “sold downstream to vendors who then use it in their own products.”
“It’s remarkable that virtually an entire company’s product line has just been turned into a botnet that is now attacking the United States,” observed Flashpoint research director Allison Nixon.
Also remarkable is Krebs’ warning that most of these compromised devices are unfixable. Mirai-style botnets infiltrate many of these devices using secret passwords the consumer is unaware of, and cannot change – they are effectively back doors coded into the firmware.
Flashpoint’s Nixon worried that the Internet of Things is inherently dangerous, because it has so many unfixable security flaws. Even if firm industry-wide security standards were established by an international association, as Krebs recommends, and consumers aggressively sought to purchase only standards-compliant devices, it would take a long time for them to replace all of the old, vulnerable devices.
Krebs also floated the idea of forcing the companies that produced these vulnerable, unfixable devices to pay for a global recall, which would be incredibly expensive and difficult to enforce, given the international nature of the production chain. And then there’s the possibility Chinese companies did this deliberately, to create vulnerabilities the Chinese military planned to exploit, before hackers discovered them and began creating public-domain botnet software. China is very big on invoking sovereign immunity to protect state-owned enterprises (i.e. nearly all of them) from expensive foreign court decisions.
Security expert Bruce Schneier worries that “someone is learning to take down the Internet,” and even these last few huge attacks are just reconnaissance missions for larger actions to come:
The attacks are also configured in such a way as to see what the company’s total defenses are. There are many different ways to launch a DDoS attack. The more attack vectors you employ simultaneously, the more different defenses the defender has to counter with. These companies are seeing more attacks using three or four different vectors. This means that the companies have to use everything they’ve got to defend themselves. They can’t hold anything back. They’re forced to demonstrate their defense capabilities for the attacker.
I am unable to give details, because these companies spoke with me under condition of anonymity. But this all is consistent with what Verisign is reporting. Verisign is the registrar for many popular top-level Internet domains, like .com and .net. If it goes down, there’s a global blackout of all websites and e-mail addresses in the most common top-level domains. Every quarter, Verisign publishes a DDoS trends report. While its publication doesn’t have the level of detail I heard from the companies I spoke with, the trends are the same: “in Q2 2016, attacks continued to become more frequent, persistent, and complex.”
There’s more. One company told me about a variety of probing attacks in addition to the DDoS attacks: testing the ability to manipulate Internet addresses and routes, seeing how long it takes the defenders to respond, and so on. Someone is extensively testing the core defensive capabilities of the companies that provide critical Internet services.
Schneier notes such tactics are “common practice in espionage and intelligence gathering,” and don’t seem like “something an activist, criminal, or researcher would do.” He said he has seen data suggesting China was the origin point for many of the probing attacks, but notes it is “possible to disguise the country of origin.”
Gizmodo’s William Turton warned on Friday that the Dyn DNS attack marked “the beginning of a bleak future,” and the ripple effect damaging major Dyn users like Spotify and Twitter demonstrated how “frightfully fragile” the Internet has become.
“Some think the attack was a political conspiracy, like an attempt to take down the internet so that people wouldn’t be able to read the leaked Clinton emails on Wikileaks. Others think it’s the usual Russian assault. No matter who did it, we should expect incidents like this to get worse in the future. While DDoS attacks used to be a pretty weak threat, we’re entering a new era,” Turton wrote.
When the “Internet of Things” was newborn, critics worried about the amount of traffic that would be generated by countless devices – cameras, thermostats, home security systems, digital video recorders, and just about anything else manufacturers decided to patch into suddenly-ubiquitous home wi-fi networks with cheap wireless transmitters – and also feared the devices could be taken over by hackers. The danger that seems to have been largely overlooked is that hackers would turn this vast legion of devices into attackers, weaponizing the Internet of Things into a parasite that can devour its host.
There will eventually be a solution that clears up a good deal of the danger. Perhaps it will take a few years for higher security standards to be implemented and propagated to the consumer base, a process aided by the low cost of these Internet-enabled gadgets. That means whoever has been probing the Internet and learning to take it down entirely – breaking the back of the entire system, instead of just shutting down a few specific websites – has a fairly narrow window of time to unleash whatever massive horror they’ve been planning.