Better to be Lucky Than Good? After Not Petya, Shipping Company Maersk Saved by Power Outage
Nearly a year after NotPetya brought down many Ukrainian and multinational organizations, additional information is coming to light including a miraculous story regarding Danish shipping firm Maersk. In a piece authored by Andy Greenberg for Wired, many off the record sources describe what happened before, during, and after the ransomware attack cost the company nearly $300 million USD. The attack only took two hours, the recovery took ten days, but both the cyber and real-world effects were felt for months.
Roughly a month after the WannaCry outbreak that crippled computer networks around the world, it is believed that a Ukrainian accounting software company ME Doc (similar to Intuit in the US, the maker of TurboTax and other accounting software) had their update servers compromised and a malicious update was pushed to all ME Docs customers. The malicious update contained a remote access trojan that allowed the attacker to send commands to affected machines. Any organization in Ukraine doing business or filing taxes would have ME Doc software on their computer.
On the morning of June 27, 2017, the attacker released their endgame, a modified version of the popular Petya ransomware variant with a few virulent enhancements to significantly assist with spreading. Security researchers dubbed the new variant “NotPetya, because while it had code in common with Petya, NotPetya had no recovery mechanism, even if the ransom was paid; it was purely written to be destructive.
While the initial outbreak was confined to Ukraine, it quickly spread beyond its physical borders reaching the United States, Tunisia, France, Germany, and other counties in Western Europe. Multinational organizations such as Merck, Federal Express / TNT, Maersk, and Mondelez all reported losses in the hundreds of millions of US Dollars. Total damages were calculated to be over $10 billion USD globally. In the cyber world, there are no borders and an attack on one country could easily translate into an attack on everyone.
Maersk reports being responsible for 20% of global shipping capacity. Roughly translated, that means a Maersk ship is entering a port somewhere in the world every 15 minutes, 24 hours a day, 7 days a week, 365 days a year. On the day of the attack, Maersk employees noticed something strange with their computers. They were randomly rebooting, not unusual when the IT department was pushing out patches, but this time the computers would boot up to a blank screen or to a cryptic message that their files had been encrypted and that payment would need to be made to recover their data. Employees looked around their office space and watched computers blank out one by one until every visible screen went blank. Employees ran into conference rooms, datacenters, and shipping terminals, doing everything they could to unplug computers from power or physically disconnect them from the network to prevent further spread and damage. It took two hours to finally stop the spread of NotPetya, but the damage had already been done to tens of thousands of machines. Ships were turned away from ports, security gates could not be opened, transport trucks lined up for miles, and everything from payroll to shipping manifests would need to be processed by hand using pen and paper. Panic could not even begin to describe the environment at Maersk in the following hours and days following the attack.
Once the initial panic subsided, IT administrators came to a startling realization. While backups for many mission critical servers had been located, no one could locate a backup of a domain controller. An initial assessment came to the conclusion that every primary and backup domain controller (approximately 150) had been lost in the attack. This backup strategy works well if one or a handful go down, but does not take into account the possibility that every domain controller is wiped simultaneously, which occurred in the NotPetya attack. This presented the team with a monumental problem, as all other recovery efforts would be futile if the domain controllers were not restored from backup, as they contain the information for every Maersk user and computer. That would have meant building their entire IT infrastructure back from scratch, which could take months with losses amounting in the billions of dollars. The recovery team, frantic to locate a single working copy of the domain controller, hit a stroke of luck when they phoned a datacenter in the West African country of Ghana. A power outage in Ghana knocked their copy of the domain controller offline and it was not reconnected to the network at the time of the NotPetya attack. That single existing copy of the domain controller would play a crucial role in the recovery of the multinational shipping company’s IT systems. After some addition logistical problems were overcome, involving slow internet, travel and visas, a hard drive containing the image made it to Maersk’s IT headquarters in London and the recovery process began.
Within days, IT systems were coming back online. Combining with the partially restored systems and the ingenuity of employees often using personal Gmail accounts, WhatsApp, and Microsoft Excel to continue operations, cargo was beginning to move again.
Although NotPetya came a month after WannaCry, which shone a spotlight on the previously-patched Windows vulnerability EternalBlue, the authors of NotPetya combined EnternalBlue with MimiKatz, a Windows-based password stealer. NotPetya would only need to find a single unpatched machine to exploit EnternalBlue, then use MimiKatz to steal credentials stored in RAM to spread and infect machines that were already patched against EternalBlue. Combining these two spreading methods, NotPetya was credited with being the most virulent strain of malware to date.
It is often said that it is better to be lucky than good, and in this instance, Maersk got extremely lucky. The problem with luck is that it is unpredictable and this could have very easily gone the other way where Maersk lost every domain controller in existence without any backups. It is also said that an ounce of prevention is worth a pound of cure. Since organizations cannot count on “Luck” to be their cybersecurity strategy, putting in security controls such as a Cloud Sandbox with quarantine to analyze unknown files no matter where a user resides. Connecting users to applications instead of entire networks, as traditional SSL VPN does, will also limit the attack surface remote users have against corporate assets. With proper security controls, “Luck” will not need to be a part of an organization’s cybersecurity strategy.