The War On Ransomware: 4 Lessons Learned From The Trenches
By Derrick Wlodarz, FireLogic Inc
It was a Friday afternoon this past August, with the staff at my IT Managed Services company FireLogic winding down operations leading into the weekend. Out of nowhere, a call comes in from a new client we have never worked with asking for an SOS. A few emails and phone calls later, and we finally had a clear picture of what the situation looked like.
This client is a midsize vendor for the aerospace defense industry that was infected with ransomware via one of their core servers, and this malware was suspected of spreading to most systems at their primary location. In their talks with a security incident response firm, the decision was made by management to “nuke and rebuild” the entire IT infrastructure from scratch. My firm, FireLogic, was called in to provide the boots on the ground to help with re-imaging their entire PC fleet – a task which took almost two weeks to complete including a bevy of overtime work.
The client did not have much of a Disaster Recovery (DR) strategy in place that could be relied upon to mitigate these types of situations. Furthermore, the security software in use was sub-par and not universally deployed, and core network firewall hardware was old and outdated. To say the least, numerous facets of IT operations were neglected, and this firm ended up paying the greatest price possible as a result.
While we’ve done firefighting for a few other ransomware incidents, none were on this scale both in scope and the size of the firm attacked. These attackers essentially sent this company back to the IT stone age after all was said and done – to the point where the staff was forced to do work on pen and paper for extended periods.
Here are some of the core lessons I took away from this incident as we continue to hone our proactive stance toward this type of threat across our own managed client base.
- Disaster Recovery & Business Continuity Are “Need To Have” Business Requirements
One aspect that all major ransomware incidents seem to have in common these days is the fact that Business Continuity (BC) and Disaster Recovery (DR) are either not present, never tested, or incapable of being relied upon. This client we came in to save was no different.
While my company has kept DR as a top-of-mind item across our client base for years, we’re now moving steadily toward a universal Datto rollout for fully managed clients. Their platform offers Business Continuity for disasters where we can have clients up and running in under an hour. Plus, they have ransomware detection built in that detects if a backup set is tainted, so we can take the proper steps for remediation.
Managed IT service providers need to ditch the idea that they will be able to prevent 100 percent of malware. Having a DR strategy that employs proper planning and IT solutions is the new reality we all need to recognize. Anti-malware software and IDS/IPS alone aren’t enough.
- Patching Cannot Be An Afterthought
Another avenue that most major ransomware incidents tend to take advantage of is unpatched software and hardware. I get it – IT support teams are stretched thin, with more on their plates than ever before, and patching can be a headache to schedule and implement—but these excuses usually come down to firms simply not investing in platforms that can automate this age-old sore point.
To solve this for our managed client base, we have standardized all system patching to be handled by Datto RMM and uphold a very aggressive patching schedule for OS and third-party software. On the network side, we consider Meraki our gold standard. Their gear is fully cloud-managed with firmware updates being deployed on an automated, zero-effort basis.
The more manual your patching process is, the less likely it will be enforced -- no matter how great a team you may have.
- Reduce Or Eliminate The Biggest Weak Links: On Prem Servers
Another all-too-frequent commonality for clients who have been hit by a ransomware attack is the reliance upon on-premises servers. There is no less than a plethora of options for ditching these servers, ranging from full blown Software as a Service (SaaS) platforms to more familiar Infrastructure as a Service (IaaS) cloud-hosted virtual machine environments. Clients who are sustaining fleets of physical servers on-prem without the requisite attention to patching and security oversight, are putting their entire IT infrastructure at unnecessarily high risk to ransomware.
There may be situations in which on-prem systems cannot be migrated to the cloud; being an advocate for a hybrid approach is perfectly OK. Often, though, physical servers are being left operational as a matter of complacency of the “status quo.” This is a tough mindset to overcome without a burning business requirement or worse: an incident causing a radical rethinking of IT strategy.
A good way to help justify re-evaluating the usage of on-prem servers is through risk assessments. These are extremely helpful in regulated industries, like healthcare with HIPAA compliance realities, where risk assessments are required yearly. These structured analyses can help place objective ratings on acceptable risks or expose those risks which need outright remediation. For example, we’ve been leveraging a firm called HIPAA Secure Now for our healthcare client risk assessments for many years with great success.
We’re frequently involving telecom master agent TBI to help evaluate cloud-first options, analyze risks, and most of all get our clients the best pricing possible. Don’t go it alone if you aren’t experienced in navigating the cloud vendor landscape.
- Only End-User Education Can Truly Defeat “The Enemy Within”
Organizations will easily spend countless sums to help protect against the enemy at the gates. And rightfully so. But how many of your clients are just as concerned about the “enemy within” – their own end users? While on-prem servers that are poorly patched are the perfect attack surface for ransomware actors, they usually don’t get in on their own. They’re almost always aided and abetted by internal users who just don’t know any better, either by action or inaction.
There are some excellent vendors offering internal-facing training programs which actually place a premium on the competency of users with baseline security concepts that should be shared concerns across an organization’s departments these days. Training that ditches the bland PowerPoint slides and mere platitudes and instead embraces proactive situation-based assessments plucked from real-world examples are all the rage now.
Patching is paramount, next generation IDS/IPS is critical, and moving to a cloud-first strategy is wise in an era of unfettered ransomware. But technical safeguards alone can’t be the end-all-be-all when it comes to a modern ransomware mitigation stance.
The War On Ransomware Is Achievable
A comprehensive ransomware strategy is the only realistic way forward for today’s organizations. Many MSPs are sadly still pushing purely-technical safeguards for a problem that is multi-faceted in both scope and risk level, depending on the client. If you’re relying on a status quo which hasn’t been battle-tested in the face of complex ransomware attacks, you’ll likely be in for quite the surprise when they do hit.
Using a mixture of reducing on-premises attack vectors; leveraging objective risk assessments; enforcing tough patching routines; and standardizing on Business Continuity platforms have been keenly effective in keeping our clients at the forefront of this war. Beware of any vendors that claim to have the sole silver bullet— any plan with a chance of winning needs a layered approach that is equal in technical ammunition as it is on end user education.
About The Author
Derrick Wlodarz is a seasoned IT Specialist who owns Des Plaines, IL-based Managed IT Service firm. He has more than 12 years of IT industry experience across the private and public sectors, with numerous technical credentials from Microsoft, Google, and CompTIA. He specializes in providing SMB clients with managed IT support, consulting, and training. Derrick is a long-serving member of CompTIA's Subject Matter Expert Technical Advisory Council that shapes the future of CompTIA exams across the world. FireLogic is a partner of TBI, a master agent who assists partners through their upmarket technology and marketing needs so they can communicate their expertise.