Why MSPs Must Plan For And Expect IT Security Failures

By Terry Cole, Cole Informatics

Recreational pharmaceuticals notwithstanding, nothing really alters your perception of the world like experience. As MSPs charged with implementing security measures that effectively protect our clients from harm, our industry would like to think there’s a set of solutions out there that we can arrange to provide perfect protection. In reality, though, experience always seems to have at least one extra lesson.

Recently, all those alarming news stories about just how vulnerable business technology is to a determined attacker were made real for one of our clients. I'm sharing this tale in the hope that our experience might teach you not only how to solve one particular security issue, but also to recognize the necessity of putting plans in place for when every defense you have comes up short.

Here’s the story: the public sector client in question had budgeted out (along with all the hurdles that entails for a government entity) a robust array of security solutions with the goal of achieving complete security protection. Nevertheless, an unauthorized party was able to access the client’s Microsoft Office 365 account and use it – at a minimum – to send phishing emails to a discrete list of email recipients.

We protected this long-time client with a list of operational and security initiatives that we fully intended to be comprehensive. In practice, while we certainly made a “best effort” and checked all the boxes we knew to check, this experience just reminded us that there’s always more attack surface to cover. This is the dilemma: there’s always more money you can ask your client to spend – until there isn’t.

Let’s run down the checklist of safeguards we had in place protecting this client:

Firewall And Related Services – Check.

The client has a state-of-the-art Palo Alto Networks firewall, with all conceivable services deployed and configured to hedge against both internally- and externally-initiated threats. Any click on any link leading anywhere is under protection.

SIEM Services/SOC Monitoring – Check.

For the last five months, the client has been under the watchful eye of a well-known vendor's Security Information & Event Monitoring (SIEM) and Security Operations Center (SOC) solution, covering the logs of the firewall and all servers required for network operations.

DNS Management And Control – Check.

For many years, we've had aggressive DNS security in place, originally with Umbrella from OpenDNS (later Cisco). For the past six months, we’ve switched to DNSFilter, which we believe is an improvement. Every day, this DNS filtering catches potential dangers that otherwise easily pass through the other defenses.

Advanced End-Point Security – Check.

As early adopters of SentinelOne, then Deep Instinct, and now Cylance, we've kept the client’s desktops and core servers protected as aggressively as anyone can from an “antivirus” perspective.

Traditional Antivirus – Check.

To the degree it adds value, we’ve had traditional antivirus deployed in addition to advanced end-point security, simply because we’re able to so at no additional cost to the customer or ourselves. In my professional opinion, however, the efficacy of traditional AV is dubious.

Email Protection – Check.

Over the years we’ve always used a top-name email protection service, initially Reflexion, and for the past few years MailProtector, through which all of the client's inbound and outbound email flows. We trust this solution to do what it advertises and have years of experience observing it work.

Dark Web Monitoring – Check.

For the past nine months, the client’s internet domain and core site IP address has been proactively monitored for signs of compromise. There have been none reported.

NOC Monitoring – Check.

For years, leading network operations teams have perpetually monitored the client’s desktops and servers.

Security Awareness Training – Check.

All of the client’s personnel have taken at least one information security awareness training course in the past six months. We plan to dial this up to monthly trainings with mandatory participation (driven by the client’s newfound resolve following its security event).

Even with all of this in place, attackers found a box we didn’t have checked.

After becoming aware that phishing emails were being sent from the client’s accounts, we investigated and found that we needed to add a license called “Microsoft Azure Information Protection P1” for every user simply to view the Office 365 sign-in logs. Since getting budget approved on the fly and fast enough to identify the source of the issue wasn’t realistic, we added this license using a trial offer from Microsoft. Viewing the logs, we then found logins for specific users in locations as far afield from our Tennessee-based client as New Jersey, Virginia, and, you guessed it, Nigeria!

This experience led us to discover the security gap we missed: the Office 365 license deployed by the client didn’t include the ability to use geo-fencing for logins, which Microsoft calls “Conditional Access”. By adding this feature, attackers attempting to login from New Jersey are now denied access, let alone Nigeria.

That said, the real lesson of this experience is that there will always be another gap. As IT service providers, our intent is to be perpetually aware of any threats or exposures, and we deploy myriad tools to that effect. Realistically, however, it’s just not possible to have our attention be everywhere, all the time, even with the best of tools. Protection, and even detection, is never perfect. That’s why preparing for failure with data backups, disaster recovery plans, and other such measures is so important: so that the lessons that come with experience don’t have to be such hard ones.

About The Author

Terry Cole is the Founder of Cole Informatics, an IT and business telephone professional services and support company in West and Middle Tennessee.