Shodan me the Memcache!!
Terrible puns aside, two near-ELEs (extinction level events) occurred in late February and early March that should have garnered the attention of every security practitioner.
On February 28, 2018, GitHub was hit with a 1.35 Gbps sustained DRDoS (Distributed Reflected Denial of Service) attack. A "reflected" attack occurs when the attacker spoofs the victim’s IP address as the requesting address so the server believes the victim made the request and return/attack packets are sent to the victim. While large DDoS attacks are not necessarily new and were almost expected, the GitHub attack deviated from past DDoS attacks in that it did not require a BotNet. The large DDoS attacks that hit DynDNS (1.2 Tbps) and Krebs on Security (665 Gbps) used a Mirai-variant botnet that leveraged vulnerable and poorly secured IoT devices. The GitHub attack used a little-known process called memcached, a memory object caching system primarily found in Linux systems. GitHub alerted their DDoS protection company Akamai which scrubbed the traffic for malicious packets and restored GitHub’s service back to normal only 18 minutes into the attack.
Memcached uses well-known port 11211 and can carry traffic over TCP or UDP. When implemented correctly, memcached is a very useful for storing small chunks of arbitrary data from database calls, API calls, or page rendering. This protocol was designed for internal use only, bound to localhost, and never be exposed to the open internet. According to Shodan, misconfigured and poorly secured servers have exposed nearly 88,000 unique machines with memcached exposed to the public internet without any authentication.
These 88,000 machines were again used on March 5, 2018 to launch a 1.7 Gbps sustained DRDoS attack against a US-based service provider. Surprisingly, no outages were reported as a result of this attack. This fact is a real testament to the DDoS mitigations in place that the service provider implemented in anticipation of such an ELE.
Unlike Mirai which leveraged a BotNet, memcached has no vigilante malware such as BrickerBot or Hajime to take over over and shut down vulnerable devices. Memecached presents several other problems representing an escalation in the arms race of DRDoS malware creators and DDoS protection services:
Memcached runs on UDP, so no authentication is required and the victim’s IP address can easily be spoofed as the destination address
The amplification ratio of a memcached is 51000:1 meaning an attacker sending 1 malicious packet can generate 51000 response/attack packets to the victim
In true Mirai fashion, the attack code has been made public and sample attack scripts are surfacing on the web. Over 15,000 memcached-based DRDoS attacks have been observed since the 1.7 Gbps ELE. A security research firm has claimed to have found a “killswitch” where a victim can simply send a command to an attacking service to shutdown the memcached process. The GitHub attack originated from tens of thousands of servers originating from over 1000 ASNs, so this “killswitch” may not be feasible during an attack, but could lead to a vigilante group to shut down vulnerable servers.
Treating the symptom of these ELEs would to simply throw more bandwidth at the problem. Treating the disease would require systems administrators to follow basic security practices to disable any unneeded services, do not expose services to the internet that do not require it, and block all outgoing ports except for those required.