6 Keys to Surviving A Ransomware Attack: MSP Tip Sheet

I recently had the pleasure of co-hosting a webinar with Huntress Labs CEO Kyle Hanslovan covering a topic on every MSP’s mind — ransomware. Specifically, ransomware incidents where attackers have compromised MSPs, used their credentials to hijack the MSP’s legitimate software tools, and abused those tools to deploy ransomware across their entire client base.

Kyle has personally worked with nearly 40 MSPs who have been victims of these attacks. As a result, he’s seen and heard firsthand what owners and technicians go through, and learned what separates the ones who are able to successfully recover from the ones who don’t.

If you missed the live webinar, you can watch a recording any time here.

Update: We’re also hosting a follow-up webinar on Thursday, October 17 at 1pm EDT with incident response (IR) specialists from TrustedSec. They’ll be taking questions and sharing real-world examples of what working with an incident response (IR) provider during a ransomware incident looks like. You can save your spot by signing up here. 

Before we dive into Kyle’s takeaways, a quick overview: Ransomware attacks targeting MSP networks have been quickly ramping up since June. They’ve affected MSPs big and small, triggering downtime and disruptions for a wide mix of MSP clients ranging from 22 Texas municipalities to hundreds of dental offices across the U.S.

It’s important to note that, with the exception of the February attacks involving an outdated ConnectWise ManagedITSync plugin for Kaseya VSA, none of these incidents appear to have involved attackers exploiting a vulnerability in MSP software. Instead, they involved the abuse of weak or stolen credentials, which attackers used to hijack whatever RMM or remote access tool the MSP was using.

For that reason, enabling 2-factor authentication is widely seen as an effective deterrent to these attacks. Many vendors (including NinjaRMM) have either already made or are in the process of making 2FA mandatory for their users.

Note: If you’re using RMM or remote access software and you haven’t setup 2FA, stop reading and do that now. It’s truly not a matter of if you’ll see one of these attacks, but when. 

While we’re talking prevention, here are two additional recommendations that are super basic, yet incredibly effective. In fact, they can help keep you safe from the majority of today’s most common ransomware attacks:

You can find a bunch of additional things you can do to harden your systems and keep yourself safe in our 2019 MSP Cybersecurity Checklist, but the above are three simple, super critical things you absolutely have do, baseline.

With that, I’ll get off my soapbox. Because what I really want to share are six things Kyle said you should focus on now to make sure you’re prepared to act quickly if/when $#*% does hit the fan.

But first…

Two Reasons You May Not Be Ready

One of the big things to understand about these attacks is that they aren’t like others you may have prepared for. They’re an entirely different class, with a different scope and different implications. Even if you have well-documented procedures for helping your clients deal with ransomware incidents, those are only going to be, at best, partially applicable here.

Why? Two reasons:

1) It’s not them, it’s you

Unlike previous incidents, these ransomware attacks aren’t originating in isolation on one of your client networks. They’re being launched from inside your organization. That means you have to have an entirely different set of procedures designed to lock down your own tools and techs until you can identify, isolate, and contain that compromise. Otherwise, if you launch right into trying to remediate client systems, attackers may simply encrypt them all over again until cut off their access at the source.

2) Everything hits at once

Here’s another point Kyle made during the webinar that really resonated with folks:

You can likely handle one of your clients getting infected. What you likely can’t handle is ALL your clients getting infected at once. 

One of the things that has made these attacks particularly devastating for MSPs is the scale. Imagine how stressful and chaotic it can be dealing with one customer infected with ransomware. Now imagine that x 10, 20, 50, or 100. Even larger MSPs can be easily inundated and overwhelmed by the sheer volume of emails and phone calls they have to respond to, the number of systems they have to clean, the amount backups (god-willing) they have to restore.

For many, like the small husband-and-wife-run shop PM Consultants, it can all be too much. As detailed in this piece from ProPublica, in the midst of a ransomware outbreak PM announced to its clients via email that it would be closing.

The company said it had been “inundated with calls” on the morning of the ransomware attack, “and we immediately started investigating and trying to restore data. Throughout the next several days and into the weekend, we worked around the clock on recovery efforts…. However, it was soon apparent the number of PCs that needed restoration was too large for our small team to complete in any reasonable time frame.”

“Receiving hundreds of calls, emails, and texts to which we were unable to respond,” PM ultimately  had to throw in the towel, advising customers, “At this time, we must recommend you seek outside technical assistance with the recovery of your data.”

So, knowing that these attacks are a completely different ballgame in terms of implications and scale, how can you prepare? Here are six keys Kyle highlighted that can make the difference between your MSP going under due to one of these compromises, or coming out battered but still breathing on the other side.

1) Be ready to lock down your own tools

If you have more than one customer notifying you of a ransomware infection at the same time that’s unfortunately a big warning sign you’re dealing with one of these attacks. The first thing you need to confirm or rule out is whether your own infrastructure has been compromised and your RMM and/or remote access tools are actively being hijacked.

2) Divide and conquer

During a live incident, a lot of things have to happen very quickly, often in parallel. The only way to stay afloat is to leverage everyone to the full extent possible. And the way that happens is by ensuring everyone knows their role and the key things they’re responsible for BEFORE a crisis hits.

At a high level, key responsibilities fall into three buckets:

Major responsibility #1: Critical decision making

Owner: CEO

Setting the course in terms of:

Essentially, you need someone providing strong leadership, making tough decisions, and directing people, and that’s really hard to do if that person is also trying to be directly involved with restoration or perform other roles at the same time.

In the moment of crisis, it can be difficult for some owners to stay in this lane and avoid drifting to technical activities they’re more comfortable with. But in working with owners who have gone through these attacks Kyle stresses that often the best thing an owner can do during an outbreak is to delegate and get out of their technicians’ way. There are simply too many other things that need to happen that no one else on the team can do. A vacuum in leadership can be disastrous.

To give you a place to start with, here’s an example checklist that CEOs/owners can reference. You’ll want to adapt this to fit your company’s specifics.

Major responsibility #2: Communications

Owner: Sales and marketing team, or, if none, CEO and other non-technical staff

The second key need during a live incident is communications. As soon as ransomware hits your affected clients are going to need to know you’re aware of the situation and you’re actively working on it. Otherwise, if they feel like you’re ignoring them or that you’re not in control of the situation, they’re going to panic. They may even to try to assert control, themselves. That’s when things can quickly go from bad to worse and when lawyers can start really getting involved.

To prevent all that from snowballing, you need to have people on your team dedicated to establishing and maintaining clear communication channels. That includes:

Major responsibility #3: Containment & restoration

Owner: Lead technician

Last but certainly not least, the third major responsibility you need to assign is obviously stopping the attack and getting your clients back up and running. That starts with locking down your own accounts and tools, then locking down affected clients, as well. The initial goal is to stop the bleeding and ensure the attack is contained before diving into active restoration. In fact, it’s strongly advised to engage with an incident response specialist immediately following initial containment efforts (more on that below).

In addition to isolation and backup recovery, there’s also forensics gathering and other investigation activities to consider. Depending on how many infected clients you’re dealing with, you’ll likely want to divide these tasks up among your techs. On top of that, you may also have clients that haven’t been affected, but who still need usual support. If everyone has a clear understanding of who’s responsible for what heading into a ransomware situation, the team will be able to move much more quickly and effectively when it counts.

3) Develop SOPs

Knowing who’s responsible for what is half the battle. The other half is knowing step-by-step what those responsibilities actually dictate you do. To act quickly in the middle of a ransomware $#*% storm everyone on your team needs to have standard operating procedures they can immediately turn to that remove as much of the guesswork for them as possible.

Some extremely basic rules for SOPs:

4) Be ready to triage

As Kyle explains, one of the hardest parts of responding to a client ransomware outbreak is the fact that, in most cases, you can’t help everyone at once. As difficult as it may be or as callous as it may sound, you initially have to prioritize some clients over others. To do that, you’ll need a system of assessing need and other factors to determine who gets your attention first.

Kyle compares it to the triage training he received in the military (with obvious differences).

The chart below was modeled after START, a system developed to guide medical triage in mass casualty situations. There may be different questions or a different flow that’s more appropriate for you and your clients. The point is, determining how exactly you’ll handle client triage is something you need to be thinking about.

5) Leverage outside help

Chances are, you don’t have the necessary staff on hand to recover from a client-wide ransomware outbreak quickly or effectively. The people you do have are great. They’re extremely smart and capable. But they likely aren’t all security experts, and they likely can’t clone themselves at will. The sooner you acknowledge that a situation like this will require outside help, the sooner you can go about getting the appropriate phone numbers on your speed dial.

Here are a few of the external resources you’ll need to be able to count on and who you should lay the groundwork with now:

6) Prep now so you don’t have to pay or pray later

The recurring theme of the webinar and this post is it pays to be prepared for the worst.

The good news is these latest attacks appear to be thwarted by enabling 2FA, so everyone should absolutely stop what they’re doing, audit their environment, and confirm 2FA is enabled across the board now.

But that’s still not a replacement for having a well thought-out and practiced incident response plan in place. Attacks evolve. Defenses that work today will be bypassed tomorrow. You need to adapt your defenses, but you also need to be ready to react.

Don’t get discouraged by thinking planning has to be a massive task. Kyle suggests a great way to get started is by taking an afternoon to sit down with your team and simply asking the question, “Ok, so how would we deal with this?”

It can be informal. Grab some beers and a whiteboard. Get talking. You may be surprised by how many action items you’re able to quickly come up with. From there, chip away. Security isn’t a one-and-done activity. It’s all about making incremental progress, circling back, and making a little more.

Take advantage of the time you have now, when your house isn’t on fire. For every minute of prep you spend, you’ll potentially save yourself hours of chaos and headaches if and when disaster does strike.

What’s Next?

If you liked this post or thought it was useful, please consider sharing it with others. You can watch Kyle and I discuss this topic in even more detail by watching a recording of our webinar here. As mentioned at the top, we’re also hosting a follow-up webinar on Thursday, October 17 at 1pm EDT with incident response (IR) specialists from TrustedSec. Sign up here for the chance to ask them any questions you have about working with an IR provider. It’s certain to be an interesting discussion.

Jonathan Crowe is director of content & brand strategy at NinjaRMM. Read more NinjaRMM blogs here.