I received a call on 12/24/19 early Christmas Eve from a company that was hit by Ransomware. It appears that the attack happened around 3:00 a.m. on 12/24/19. This company was a client of the Managed Service Provider (MSP) Synoptek. As you may have heard, hackers compromised Synoptek and used Synoptek's Remote Monitoring and Management (RMM) Tool to deliver Sodinokibi Ransomware to a subset of Synoptek's clients. The Ransomware attack infected the company's servers and workstations and took their network down. At the time, Synpoktek wasn't giving timely updates about how they were going to address the "Security Incident." The company reached out to us for suggestions on how to recover from the attack.
We took a look at their Veeam backup infrastructure, and their backup files appeared to be Ok. I suggested copying off the backup files to another storage device to see if they could restore from the backup files. They were running ESXi Servers for their virtualization platform. If you need an inexpensive ($1300) ESXi host for testing, check out https://www.adscon.com/blogs/news/here-is-your-cheap-mini-me-portable-vsphere-esxi-6-5-host. The news of the attack first hit Reddit https://www.reddit.com/r/sysadmin/comments/ef2egh/synoptek_issues/ on 12/24/19. On this thread, someone mentioned that Synoptek paid the Ransom for an undisclosed amount. When we heard of the Ransomware payment, we suggested that the company wait until they received the decryption keys before attempting any restore. The decryption keys worked and the company was back up and running around 12/27/19.
Fortunately, the company was able to recover relatively gracefully from the attack. In the future, here are some steps to protect against this type of attack.
- Two Factor Authentication. We now recommend two-factor authentication for all clients. Make sure both your MSP and your company uses two-factor authentication for all accounts.
- Firewall with Malware Scanning. Use a firewall that includes malware/anti-virus/intrusion scanning. The company that was infected did not have a firewall with this additional protection.
- Follow the 3-2-1 Backup Rule. 3 copies of your data, on 2 different media with at least 1 copy off-site and offline. Companies may perform backups, but a lot of companies DO NOT have offline backups! If you get infected with Ransomware and your only backups are online and encrypted, you will not be able to restore from those backup files.
- Create a Protected Management Network with Restricted Access. All management of your ESXi/Hyper-V Servers should be performed from a separate protected Management Network. Your backups should be run exclusively on this management network. There should be a separate vLAN and Firewall rule that restricts access into this network. Use a separate Active Directory Domain with different credentials and two-factor authentication to access the Management Network.
- Hacking Recovery Plan. Have a recovery plan in place before you are hit with an attack. We feel that this plan should be part of your Disaster Recovery Plan. You don't want to figure out how to recover during an actual attack.