Transcript:

139 audio export

Rich: [00:00:00] This episode of MSP Chat is brought to you by MSP Mastered. If you like co-host Erick Simpson’s Tip of the Week, you’ll love the comprehensive growth advice Erick and his team provide at MSP Mastered, your go-to resource for overcoming business challenges, improving service efficiencies, selling more profitable MRR agreements, and increasing the value of your MSP business.

From sales and marketing, to service delivery, to hiring and retaining high performing talent, MSP Mastered offers access to over 90 online master classes, 150 on-demand webinars, and 250 advanced MSP tools resources, along with regular group coaching sessions and unlimited strategic email support, all for one all-inclusive membership [00:01:00] fee.

Unlock your true MSP potential by joining MSP Mastered today. And three, two, one, blast off, ladies and gentlemen. Welcome to another episode of the MSP Chat Podcast, your weekly visit with two talking heads talking with you about the services, strategies, and success tips you need to make it big in managed services.

My name is Rich Freeman. I’m one of your co-hosts on this show. I’m also Chief Analyst at Channel Mastered, the organization responsible for this show. I am joined this week, as I am every week, by your other co-host, our CEO and Chief Strategist at Channel Mastered. His name is Erick Simpson. Erick, did you have a, a nice Memorial Day weekend?

Erick: Had an awesome Memorial Day weekend Rich. Any day that we get to, take a break and recharge is always welcome. How about you?

Rich: I agree with everything you just said, and I will add, as a Seattle resident, that when the weather cooperates as well, that is two big thumbs up. So the we- we are still officially in summer for a few [00:02:00] more weeks, and it was actually summery here over the weekend in Seattle, which was great.

So yeah, I had a great weekend too.

Erick: It was summery there, but it was raining here in Southern California. Oh, wow. So we had a flip there. Yeah, but that’s, … But hey, I’ll take it any day of the week.

Rich: Exactly. Exactly, yeah. Rain plus a day off, sure, versus the alternative, right? I’m with you right there.

Let’s dive into our story of the week, and it’s going … it concerns something I wrote about in my most recent post on Channelholic, my blog. The point of what I’m gonna get into here isn’t so much about the product as what the product kinda signals about something that’s going on out there in AI for managed services and beyond.

So I’ll give you a quick profile of a company called Clarify. C-L-A-I-R-I-F-Y, Clarify. And this is a very young company, a startup. I first met with them a couple of years ago at Pax8’s Beyond event. At the time they were creating something for MSPs, and the idea was you’re a busy MSP. [00:03:00] Between meetings, you pull out your phone, and you’ve got stuff flying at you from multiple directions, texts, emails, tickets, et cetera, and you’ve gotta very quickly prioritize and triage, and it’s all in different systems.

And this was gonna be one feed for all of that, and it was gonna help you kinda stay on track basically through your day. What the founders of that company have figured out since then is that MSPs aren’t the only people on planet Earth facing that kind of issue. Pretty much any leader at any organization could use that kind of AI tool, and obviously the addressable market gets a lot bigger when you start looking past MSPs to end user executives of all kinds.

So that’s what the Clarify product does now. Now it’s gone beyond creating this, this feed that is helping you triage and prioritize and so on. It’s still kinda doing that. It now though kinda plugs into all of the tools in your stack, or if you’re selling this to a customer, it’s gonna plug into all the tools [00:04:00] in their stack.

And it’s going to perform basically the functions of an agentic or digital chief of staff which is an interesting thing, by the way, ’cause I mentioned Pax8’s Beyond event, and this was what Pax8 was telling everybody there. If you wanna get started in agentic AI, build a chief of staff agent.

Use it for yourself, sell it to your customers. Here’s a tool actually that kinda comes with that functionality out of the box. What’s interesting about it, and I think telling about it though, is that you can ask yourself, of course, what’s to stop somebody from creating a digital chief of staff in Copilot or with Claude or with ChatGPT?

And the answer I got from the co-founders of Clarifai basically is nothing. You can go ahead and do that, but the answers you get will not be as good basically as the answers that we’re going to give you. Because those tools they know a lot about everything, but not quite enough about you.

That the thing about [00:05:00] Clarifai, according to its co-founders, it is very executive specific in terms of how it works and what it keeps track of, and it’s very specific to you and your data graph. And so because it has narrower but deeper context about you, it provides you with better answers than you can get from a generalized LLM kind of a tool.

And this echoes for me with an interview that I did with Pinar Ormaci, the CEO of Lexal, a company we’ve talked about on the show before. They’re building a, a documentation solution of the future that goes beyond that basically to create a very complete data graph for an MSP that helps them utilize AI more effectively.

And when I interviewed her at ChannelCon just a few weeks ago, she was kinda talking about an experience that she personally had with Claude where it gave her some bad information. And her diagnosis of what had happened essentially is that- Claude has too much context. It doesn’t know enough about [00:06:00] her, it, and it knows too much about things that aren’t her and her business.

And you get better answers, she said, from a system like Lexal because of its focus and its its depth specifically on you. And so I think this is something we’re see- If you’re reading the headlines, you know that there’s a bit of a momentum shift away from the SaaSpocalypse narrative right now.

All of a sudden, SaaS doesn’t look like a doomed business. And not for everyone everywhere, but there are certainly use cases that are proving out in software as a service, and a lot of them have to do with things like two factors that I called out in a Channelholic post earlier this year: risk and focus.

And risk is all about compliance risk, security risk, et cetera. You don’t necessarily want a vibe code in EDR. But the focus part is what we’re talking about here. Y- you want … If you’re using AI tools to run a business, you want them to have very specific focus and context on your function and your industry [00:07:00] and and your data, your context.

That’s what’s gonna get you the best results. And so as a, an investor as an end user, as you’re looking at tools, that’s something I think you should be looking at. I think as an MSP as well, as you’re evaluating the tools and that you’re going to use to run your business, the AI tools specifically, I think you wanna be considering this, this complexity depth question basically, and making sure that the tools you are entrusting important business processes to and leaning on for leadership input and so on, are very specific to you and your business and managed services, and can therefore give you better answers than you’re gonna get if you’re just relying on Claude or ChatGPT.

Even if you go through the work of grounding and training those, which is a lot of effort, versus going out and just buying a Clarify. I think you’re looking you should as an MSP, as you’re thinking about the AI tools that you implement, be thinking about depth of [00:08:00] context

Erick: Very interesting, Rich, because I think, what…

from my perspective, it’s great to hear about a tool that’s gonna, it’s already built to learn as much about me as possible because, you and I have shared our journeys in vibe coding and working with these different LLMs recently. It feels to me like the early days of when I did some, some programming and some, access databases and SQL database things and things like that, and it’s like this blank canvas where you have to create everything that you want that platform to do.

I feel the same way with the AIs that I’ve been using. It– Just trying to teach them about me, about my background, about my voice, about what, what’s a priority for me. And it’s when you’re de- a developer, Rich, it’s never finished, right? The s- the application, the solution is never finished.

There’s always something… And that’s what I feel it’s like working with, these platforms on our own, trying to get them to do what we want. It’s never [00:09:00] finished. There’s always something else, and there’s always something that has to be retrained. So I like the idea that, Clarifai and others, that I’m sure will come behind them are focused on that thing that says, “I’m going to…

i’m built to learn everything about you,” rather than me having to figure out how to train it to learn everything about me. That’s one. So that’s cool. Number two because it’s built for executives and leaders, the general anonymized data that it’s going to collect from all of its users to help all of its users is also very interesting because, we talk a lot on the program, Rich, about, the, how the AI influence on the tools that MSPs use, and we talk a lot about, PSA and documentation and these other things.

Here’s something now that is completely different. It’s pointed to executive decision-making and, understanding all the channels where data can flow into that executive. And the decisions that you’ll make as an executive are much different than decisions that you’ll [00:10:00] make as an engineer in some cases, right?

So that’s really interesting to me as well, where you’re getting the best of both worlds. And then is it just a system of record like a chatbot now, and do you know anything about the roadmap where it may become a system of action where you can have it, s- check in for your flights and, order, the flowers for, your significant other, and do those things where you’re seeing the email come in and it asks, “Hey, should…

Do you want me to take action on that?” Will it become a system of action as well?

Rich: So I can’t get too deep. I don’t have the answer to that in depth basically, but it already has certain system of action functionality. The way one of the co-founders described it to me basically is it– a, a tool that would- will do what a human chief of staff would do for you if you had a human chief of staff.

And and some of that includes communicating stuff out to the organization I assume without knowing, stuff like travel might fall within [00:11:00] that. So if you think about the stuff that you would have a chief of staff handle for you, the, the the term one of them used, which I thought was great, was this is a tool that will protect the leader’s time.

That’s what a good chief of staff does- Yeah … is handle the stuff that can be handled so that there’s more time for the decision-maker to make decisions. So there is some of that functionality in there now. I don’t know exactly how much more of it will be coming down the road.

Erick: Yeah. Just clearing the noise of all the information flow from all the different, applications, email, ticketing systems, texts, social media, just prioritizing that, clearing the noise, right?

That’s super helpful.

Rich: And so I wanna point out something I probably should have pointed out before quickly, just for anyone in the audience here who’s wondering how does it know so much about me and, how can it make these give me these smart responses because of that. They– the founders of this company have invested a lot of time and effort in integrations, and not like sort of bespoke…

it’s [00:12:00] not like they have this giant menu of integrations. They’ve created a natural language prompting kind of system that will connect with all of the tools and the sta- with whatever LLMs you want and with all the tools in the stack, and then it can access and draw on information from all those places and collate that to, to give you the answers that it gives you.

The other thing, I wanna circle back to the point you made about, developers understanding that work is never done. I think it gets to a really kind of key point. Yeah, I have a post coming up in Channel Hawk about tech debt, and it just gets to this question of how, you can vibe code all sorts of stuff, but where do you wanna be applying your time and effort?

It, do you want to be investing serious time and effort in maintaining a system like that? Or if you can outsource it, is that maybe a better decision? And I tend to think of that as the how you decide whether, to vibe code or not to vibe code, basically. If there’s something, an opportunity where software can do something important and [00:13:00] powerful, and there really isn’t a tool, or the tools available out there just aren’t gonna work for you for some reason or other, then that might be a good situation to vibe code.

But if there is something out there that can do the job and then you don’t have to worry about that your developers’ work never being done, that probably makes a lot of sense. And that, again, is another reason why the SaaSpocalypse doom and gloom might be a little overblown.

Erick: Yeah. I agree.

It’s, build or buy. And, y- I get the feeling… what’s that line from Al Pacino in The Godfather III? He’s like, “Every– I thought I was out. They pull me back in.” It’s when you start vibe coding, it’s like you get to a point, Rich, and I know you felt it too, where it’s like, my goodness, if I’d have known it was this much work, I ne- I never would’ve started down this road.

I just would’ve done something different. But you’re in too deep now, and you’ve got to see it through.

Rich: As I alluded to previously, officially summer continues through September 22nd if I’m not mistaken. But there is an expiration date on summer, and it is approaching, [00:14:00] and expiration dates, as it happens, have something to do with your tip of the week, Erick.

Erick: Oh, nice tee-up, Rich. Thank you for that. Yeah. We’re talking… We’re gonna talk about pricing today. So MSPs as a former MSP, I dealt with this challenge, and I know that a lot of MSPs out there are s- always trying to keep ahead of, making sure that they are pricing correctly for the…

to achieve or realize the target profit margins, gross profit margins that they are forecasting for. And Rich, a, a lot of MSPs, and I’m guilty of this too, is like we would let, Or how long’s it been since you since you raised your prices? And, in the early days we were like, “Oh, it’s been way too long.”

And I’ve spoken with MSPs through the years, Rich, that have said things like, “Oh, we’ve never raised our prices in fill in the blank of number of years.” And I’m telling you, I had a partner tell me It was double, early, small double digits of years they had not raised their price. Now, of course, that was…

These were [00:15:00] the old days. MSPs today are much more on top of this stuff. But, I wanna just have this, I want to insert this thought in the back of everybody’s mind. It’s like a milk, your milk goes bad after a certain date, so you- there’s an expiration date. So you’re, when you’re pricing your services, that pricing is good for the term of that first year of the agreement.

And I’m gonna be clear about this, Rich. Whether it’s a three-year, four-year, or five-year, I don’t care how long the, the actual agreement is written for, that pricing should hold for a specific period of time, because you should not be held hostage for things that you cannot control as a business owner.

Meaning, the cost of your third-party tools and services going up year over year. You’re signing annual agreements i- in most cases, right? For the services that you’re consuming and adding into your solution, like your third-party subscriptions and services. Maybe it’s three years to get better pricing, but those prices do go up, too.

It’s like, think of streaming, Rich. My [00:16:00] goodness, how many times a year are the streaming companies gonna keep raising our prices? Two? Three? Who knows. We’re we need our streaming. We need Wi-Fi. We need internet. We need this stuff, right? So here are three things to think about. There’s an expiration date.

Write your agreement so that you have the ability to adjust that pricing based on things that you cannot control. Third-party adjustments, economic pressure, right? I remember when there was a, a, when I was in MSP, like there was a fu- you know, fuel crisis, and the cost of gasoline went up. Not as high as it’s up right now, right?

But if you’re driving and you’re rolling trucks to locations, you’re probably already charging something for those fees. Maybe you included those within a certain radius, but now, fuel is too expensive, so you’ve gotta pass those costs onto your clients. So three quick tips, Rich, to kinda stay ahead of this and make sure it doesn’t slip by First of all, identify all of your [00:17:00] client agreements that have not been trued up or price increased in the last 24 months.

Those are your first low-hanging fruit opportunities. Create a nice little letter thanking those clients for their service, and this doesn’t have to wait for renewal time, or it might be, right? We’re getting close. It’s Q4 right around the corner, so you might start thinking about this, sending out these letters.

And it’s not a ask permission first, it’s ask forgiveness afterwards kind of a thing, where you’re saying, “Hey, we’ve appreciated your relationship. We love working with you. Unfortunately, we’re gonna have a price increase,” January 1, let’s say, “and your new pricing will be X.” And just let them know that’s what it is, and you’ll probably have a small single-digit percentage of clients that might, wanna talk to you about that.

Okay, talk to them about that, but they’re not going to sway you from raising their price. Maybe, you might be willing to reduce that price increase a little bit [00:18:00] for these folks just because they reached out to you. But in general, this is your new pricing, and folks that aren’t, haven’t been trued up in many years, it’s gonna take you a couple of price increases to get them close to maybe where your prices are for a client that you would sign up next month.

And if the gap is that far, chances are you’re losing more money on that client than it might take time for you to recover to your current pricing. They may be on your shortlist of replacement as you bring on new higher profit clients. Check out the last 24 months, who hasn’t had a t- price increase, first target.

Tip number two: compare your current cost of service delivery against even your current pricing. You as I did, “Oh, this is a great solution. We’re gonna add it to our bundle, and we’ll just work it out later,” or, “That’s a cost of doing business.” There is no cost of doing business when you’re adding things to your stack that you’re reselling to a client.

If you’re adding something to your [00:19:00] stack, you’ve made a decision to do that- If their price needs to be increased because you’ve you’ve added a new, better XYZ for them, then let them know that. “Hey, we made an executive decision. Here’s what it is.” Now, I’m not talking about something that costs hundreds of dollars more a month.

If it’s a couple of bucks per user, work it out, but make sure that you’re comparing your current cost of service delivery, including your labor and all your third-party costs, and don’t forget to add your tools in there, too. What does it cost for your PSA solution? What does it cost for other things that are that you’re using to deliver service to clients?

Make sure you’re distributing that or including that cost into some of these agreements as well. And then third tip, build annual pricing reviews into your renewal process. As you’re getting closer to renewal time, if you’re doing a, a QBR process, you’re probably not doing it with every client, but make sure that you’re touching base with all the clients that [00:20:00] you’re not having QBRs with and letting them know that you’ve done a review, you wanna sit down with them and go through their current needs, where you want them to be moving forward into the new year, and, “Oh, by the way, we are having a price increase, so look out for that on your, January invoice

Rich: Yeah.

Two quick thoughts and a follow-up maybe on that, that last point there. So first of all the, this idea, ’cause this is a well-known, long-standing phenomenon. And in fact, one of the, one of the co-founders of Clarify, the product we were talking about before, was talking about how MSPs are stack build, right?

And, they find some shiny, new, interesting tool and they add it to the stack, and then the question is, am I gonna increase prices or eat margin on that? And I kinda you’re, what you’re hinting at in a way, Erick, is one of the questions, maybe the most important question you should ask yourself when you’re deciding will I or will I not add this tool to the stack is, am I or am I not willing to increase [00:21:00] prices in response to adding this to the stack?

And that’s gonna tell … The answer to that question, in a sense, is gonna tell you how valuable this tool really is or isn’t. Is it worth the time and the effort and the money and so on? If it’s not something that you feel like you can go to a customer and explain, “Here’s why you’re paying two bucks more,” then maybe that isn’t really a tool you need to add to the stack.

The other thought that kinda popped into my head is I think everything you just said is totally and completely true and rela- relatively straightforward with respect to the services MSPs have traditionally delivered. With the AI services that people are still kinda trying to figure out, and especially the pricing- Yeah

how do we price this? I think figuring out a way basically to give yourself even more flexibility than you were talking about to adjust prices over time and to set that expectation with the customer, I think that’s gonna be an important thing. And then the last bit is I s- suspect I know the answer to this question.

I wanna make sure I know the answer to this question. [00:22:00] Do you advise MSPs, in their agreement with customers, to actually build a price increase in there? At the end of this term, prices will go up by X percent. And that way you don’t have to set a to-do every, 12 months or 24 months, whatever, and true up the stuff you forgot to.

It’s just part of the process.

Erick: Okay. I’ll take the last one first, Rich. So yeah, I have templates over at MSP Master that have a clause in there that says, should the need to raise prices, occur, then there’s a couple of variants, right? We will reach out to you and let you know what that looks like.

During, really challenging economic times, I have a version of that clause that basically says, not to exceed cer- certain percent. But again, you put an expiration date on that, right? So yes I definitely want MSPs to have some flexibility and the ability to increase prices no ma- at any time for any…

for justifiable reasons, [00:23:00] right? If you start doing it too often though, then, that’s not– that’s gonna backfire on you, right? When you’re talking about, AI or something new, like a new solution that isn’t part of your normal stack, right? It’s easiest when you’re just truing up to what the current stack is with these clients or maybe even, a higher level of a cybersecurity, benefit or platform or something like that.

That makes a little bit of sense. But when you’re talking about I- AI and things like that, those– Because you’re also learning on your time and hopefully, a little bit of your client’s time I want partners to think about those types of opportunities as pilots that they that they recruit their clients into and have skin in the game as well, and the client is going to help defray the cost of that pilot until you figure out how to scope it do the assessments, deliver some…

So it’s almost like, vendors will create a early adopter program for a new product [00:24:00] feature or something like that with their partners, Rich. Same concept with MSPs with their clients. “Hey, we’re starting to roll out AI. I want to invite you to our early adopter program. It’s a p- it’s a, a pilot program, and here’s how it works.

And if you’ll agree to defray some of our, just hard costs, then– and help us roll it out and test it and then when we do the full rollout, you’ll get super special love and pricing just because you were part of my early adopter program.” Something like that, right? And then I had one last thought now that it escapes because the one I was trying to hang on to, but you gave me three questions there.

Yeah. I’ll leave it there. We- we’ll come back to it in a future episode folks. And just rest assured, as soon as we stop recording, Erick’s gonna pound the desk, “Damn it, I know what it was now.” Yeah. And I’m sure it was good too, so once you do remember it bring it back up on a future episode.

Rich: But for now, folks Erick and I are gonna take a quick break. When we come back on the other side, this is gonna be good. We’re gonna be joined by [00:25:00] Oli Thordarson of Alvaka Networks. Oli is a longtime MSP. He has also for a number of years now been a ransomware recovery specialist, somebody who parachutes in for another MSP for an end user who has been victimized by ransomware, and deals with that very complex complicated real-time kinda situation.

And what I am looking forward to is getting a perspective on the current situation in ransomware, not so much from a security vendor or a security expert, or even just an MSP who has watched clients be victimized by it, but by somebody who, an MSP who has real upfront expertise in ransomware.

And that’s exactly what we’re gonna get from Oli in just a few moments, so stick around. We are gonna be right back

And welcome back to part two of this episode of the MSP Chat Podcast, our [00:26:00] spotlight interview segment, where we are very pleased to be joined by the CEO of Auvicka Networks. His name is Oli Thordarson. Oli, welcome to the show.

Oli: Thank you very much. Glad to be here.

Rich: As I said before the break why we are so excited to have you on the show is you are both an MSP and a ransomware recovery expert, which means that you have a perspective on ransomware right now that is even more relevant really to the folks in our audience maybe than a, a security vendor’s standpoint or a cyber insurer’s standpoint.

But for folks who don’t know you, don’t know Auvicka just tell me a little bit about yourself and about the the various things you do for customers at Auvicka Networks.

Oli: Yeah. So Auvicka has two primary revenue channels. The first one is our MSP services. It’s more mid-market oriented, tends to be co-managed type services in the mid-market.

You could call that our legacy business if you want. And then six years ago, we created [00:27:00] a whole other group within the company focused on doing ransomware recovery, and we set out a five-year plan, which we totally exceeded that plan. Yeah, ransomware just turned out to be a much bigger problem than we anticipated that it would be.

And so anyway, that’s the faster growing part of the business. And I expect it to continue to grow because the bad guys just don’t seem to be abating.

Rich: So one of the reasons I so much enjoy going to security conferences like the RSA Conference and Black Hat and so on, is that it gives me a chance to sit down with the s- sort of security practitioners as opposed to the channel chiefs, for example.

I get to get a glimpse from what it’s like in the trenches to deal with some of the threats and incidents that these companies are seeing, and it’s always super interesting and enlightening. So we wanna talk, broadly with you about ransomware, but I would love to kinda kick things off a little bit by y- just help us understand y- the, [00:28:00] the ransomware recovery part of your business.

You guys are like, smoke jumpers. You’re jumping into a situation where someone has been compromised, and it’s real time. And help… Give us a sense, what happens after you get the call the, the bat light goes out or whatever, and you guys come in. What are you doing? Who are you interacting with?

What are the, the functions that you’re performing that in a certain sense only you guys can perform because of the experience you have?

Oli: Yeah. It’s a fascinating segment. To succeed in that, in, in that ransomware recovery space, you have to be approved for work by the cyber liability insurers if you’re gonna do cyber insurance-paid cases.

We do self-insured clients as well. And then you gotta also buil- develop a reputation with the breach counsel firms. They need to feel comfortable working with you, that you’re an experienced firm doing ransomware recovery. So that’s kind of the backdrop. But your metaphor of us being smoke jumpers is a really [00:29:00] good one.

For most people on this podcast, if you’re an MSP It, I’m guessing that it takes six weeks to six months from the first contact that you have with a prospect to a contract. Again, it’s typically six weeks to six months. It could probably vary a little bit in there, but that’s, I think, a pretty reasonable number.

In the ransomware recovery business, this afternoon we might be signing a client that’s never heard of us as of this moment in time, and we’ve never heard of them. The first contact to contract in that space is about six hours. And, everything comes in, it’s an emergency. And the smokejumper part, when we hire our guys to work on these ransomware cases, we specifically tell them that the first five days or so that they’re on a case, they’re gonna be working 16 and 20-hour days.

They’re gonna work through Saturday, Sunday. They’re gonna work through Thanksgiving or Christmas if it’s during that time of year. And that’s what they sign up for. Yeah, and we drop [00:30:00] them in behind the the, cybersecurity fire line, so to speak, and leave them there for about three weeks typically.

The bulk of the work is done in the first five days or so, but there’s usually two, three weeks of mop-up work to be done after that. And then they come off the case or off the job, and we give them a bunch of time off to recover because, yeah, these guys pour into it 100%, 110%.

Erick: Wow. That is smoke jumping and almost SEAL Team Six-ing the stuff there, huh, Ole?

All right. Pretty much, yeah. I have two questions here, so I’m gonna make Rick- Rich nervous. First one is, do those customers then that you’ve rescued out of this, do they then become ongoing customers after the fact? And second question is, you guys are experts. What are the biggest mistakes that MSPs that don’t have your expertise, making when there’s a ransomware incident and they have to respond for their clients?[00:31:00]

Oli: Yeah. First up on the mistakes that are commonly made most MSPs walk in the door and think that the solution is to wipe all the systems and reload everything from scratch. And there’s a few problems around that. One is it’s very time-consuming and expensive. That doesn’t necessarily need to be done.

Actually, it rarely needs to be done. Once in a while it needs to be done, but rarely. In that process, a lot of forensics evidence is lost. So if breach council lawyers or forensics pros that come in to figure out how, what happened, how it happened, and was data exfiltrated and that sort of thing, all that data is lost.

So those are probably a couple of the biggest mistakes that I see made. And I forget what the first question was that you asked. It was, do those customers typically become clients later?

Yeah, so we are definitely a ride a boom business on our ransomware recovery side of [00:32:00] the, the operation meaning that we come in after the, explosion’s happened and everything’s destroyed.

So we work them through the whole recovery remediation process, and our goal is, yes, to swing them around to the left side of the boom or on that timeline to where we’re putting them on a good cybersecurity hygiene program going forward to mitigate the chances of a re-attack Thanks, Lily. Not too far off script, huh, Rich?

Rich: No. No. And I’m gonna stay pretty close to script here as well right now, but th- I’m gonna add a little bit of a nuance in there. I am under… I could be wrong, and you’ll correct me if I am, but I’m under the impression that at least some of the time the, the folks that you’re dropping into the the fire zone there basically are in touch with the people responsible for the ransomware.

And I’m just curious because there are some very, very strong opinions in the industry about whether you should or shouldn’t negotiate [00:33:00] with ransomware practitioners, talk to ransomware practitioners. There are absolutists in both directions. There from your perspective how much contact do you have, first of all, with the people responsible for the ransomware?

And then second of all, what’s your take on negotiating?

Oli: Yeah. So any information that I provide here and that we get is actually secondhand information. We are not a ransomware negotiator or a threat actor payer firm. There are firms in the space that specialize just doing ransomware negotiation and others that specialize in the payment facilitation, and there’s a handful that maybe do both.

But yeah, it’s it’s a specialty and of its own, and you do wanna generally negotiate with… enter into… You wanna enter into negotiation with the threat actor. I think that’s considered, generally considered best practice. And the reason why is the moment that you start entering into negotiation with the threat actor, they usually stop [00:34:00] trying to further encrypt the system.

If they’re doing harassing phone calls to executives at the com- at the victim company or employees of the victim company, or in some cases key customers, they’ll call them and make disparaging remarks about the security of the operation and all the information they’ve stolen. When you start that negotiation process, they usually abate that or stop that harassing while…

And so it, what it does is it buys you time to get the system hardened and cleaned up that they… to mitigate further damage that they could potentially be doing. And then the other reason for negotiating is in about 60% of the cases that we’re in, the clients think they’re in a recoverable state.

They think they have good backups, but they don’t. Yeah, starting that negotiation is, some businesses without paying a ransom, they’re just simply not gonna recover. It could be that you don’t think you’re gonna go into a ransomware negotiation and pay it, but it’s from our experience, it’s a high likelihood or possibility that [00:35:00] will actually be an eventual thing that happens.

Erick: Oli, you’ve been doing this a lot longer than a lot of folks in the industry. So thinking about, just the advent of how the landscape has evolved since you started doing this kind of work to what it is now and what happening in the future, how… Tell us about that journey and how it’s evolved and how you’ve had to adapt along the way, you and your team.

Oli: So prior to January 2000, we had a very small handful of experiences with ransomware. Either… A, a couple of them were with existing clients at the time and and two or three of them were from call-ins from the web, from our web advertising that we do digital marketing. And at in January 2020, that’s when we decided to make this a focused business, and it is very much a specialty.

We’ve developed our own IP. [00:36:00] We have all these processes. We have this 300-point punch list of things that we do when we walk in the door on a new case. We’ve hired specific talent that is experienced at doing this work, and over the five, six years that they’ve been doing it, they just get better and our processes get better and our IP gets stronger.

So anyway yeah, it is very much a specialty business, and I do keep it separate. That operation is very separate from our managed services business.

Erick: What can you tell us about the advent of the threat actors being emboldened and empowered with tools that are now available, like AI and things like that, Ole?

Oli: Yeah. So my, my latest big shocker was I think about three weeks ago maybe it was, yeah, I think it was about three weeks ago when the Enable vulnerability cropped up. That could have been a lot uglier than it was. The lucky part, if you can say that, anything was lucky about it, was the threat actor group was just not very experienced.[00:37:00]

So there are threat actor groups out there that are very strong, very powerful. They’re experienced teams and they’re gonna own the system. They actually prefer, if you’re gonna get hit and this is gonna sound weird, I would rather get hit by one of the really strong, smart, powerful organizations because they do a pretty thorough job of encrypting and locking the system down, but they’re also the most recoverable.

It’s the inexperienced gangs that come in and do wacky stuff. Sometimes they’ll even use two or three different encryptions, and it makes decryption nearly impossible. So yeah, these threat actors are out there. They’re using AI tools. They’re living off the land using tools like Enable. I think the latest one is there’s been a lot of talk about FortiBleed, and FortiBleed has not really, I think, blown up as big as it could potentially blow up.

But one of my guys was just telling me that the latest thing is that evidently it’s been [00:38:00] detected that FortiGate Sniffer has been installed on about 12,000 firewalls to to do credential harvesting and, basically cap- packet capture and credential harvesting from there.

And it- it’s this new persistent threat on the system, and you really can’t remove it unless you reinstall the firmware on the firewall, as, as my guys are telling me. It’s a pretty serious remediation on the FortiGate. So yeah

Rich: ransomware I on, on the other side of what I do, on the, the channel haul, like the journalist side of what I do I get press releases probably three, four times a week from security vendors informing me that they’re publishing a new security research paper in a few days, and the results show that there’s a lot of ransomware out there.

And they’re probably frustrated by the fact that I don’t jump at the opportunity to report this news. But of course, you might not be the first person today to present that kind of data. Ransomware [00:39:00] has been an issue for a long time. We’ve all known about it. We’ve all been dealing with it for a long time.

There are all sorts of tools out there for dealing with it that MSPs have been putting into place. It’s a big picture question, but why after all this time Oli, do you think ransomware continues to be as big a problem as it is?

Oli: Yeah, it, and it’s gonna continue to be a big problem. And the best analogy or comparison to this sort of crime is the illegal drug trade.

For decades, countries, United States and many other countries, have been trying to block the importation of tons of illegal drugs daily, weekly. I’m not sure exactly how many tons a week come across. But the drugs sh- are– these are, sometimes they’re big shipments, and they have to come in trucks and all that, and then it has to be [00:40:00] dis- the product has to be physically distributed within the country, and then payments are received, and that money needs to be laundered because of all the banking controls against m- drug money laundering.

If– The way I look at it is if law enforcement and borders of countries can’t stop that, how are they gonna stop the guys that have nation-state support somewhere around the world in the, Eastern European countries, Russia, Ukraine? And with a wink and a nod, their governments are giving them a thumbs up, going, “You– What you’re doing is cool to the West.

You’re being disruptive to our political and economic foes.” And and they can secret themselves in through, routers and firewalls and get into systems and then- … then download their payloads and do the damage and then get paid in digital currency that’s, harder to track and, it’s not, subject to money laundering laws the way, US greenbacks are.

Yeah. The, it’s a… If we can’t stop physical drug trade, we’re not gonna stop [00:41:00] digital crime.

Erick: That’s sobering, Willie. Let’s talk a little bit about what you like to offer MSPs to do more of to reduce ransomware exposure. Now it’s firewalls, it’s phishing, it’s all this.

So what, what could MSPs do to reduce their customers’ and their own exposure to ransomware?

Oli: Yeah. And MSPs n- really need to watch out themselves, too. If you think you’re safe, I would encourage you to adopt some paranoia and second-guess yourself because it does happen to people that think that they’re in a good spot.

But first off, what are the recommend- the short list of recommendations that I have are for you and your clients, make sure you have good immutable backups. Nothing is more discouraging than walking in the door and, you talk to the client, they go, “Oh, yeah. Good thing we got backups.”

And then we go look at the [00:42:00] backups, and the backups were deleted or the backups were encrypted or the backup jobs, nobody was ever checking the error logs, and the backup jobs haven’t been running for 27 months. We see those things. Or the crown jewels of the company is on a SQL Server, and a year and a half ago they installed a new SQL Server, but nobody added it to the backup job, things like that.

And yeah, all of a sudden a recoverable situation is now not recoverable, at least not from backups. Managed EDR is a really strong solution. There are some good solutions out there from Blackpoint and Huntress and others and, the SentinelOnes, the CrowdStrikes, the Microsoft Defender.

Those tools are really powerful. Then keep your patching up to date at your client sites. Y- almost every case we walk into, patching is a mess. People think they’re doing a good job on it, they’re not. And then segmenting the networks, and then good cyber hygiene around your email.

[00:43:00] Good email filtering for spam and malware will help block a lot of that stuff coming in before the users can click on it. And then of course there’s security awareness training. But … And it’s, I would recommend those things pretty much in that order. There’s other things I could recommend, but you, if you c- you know, walk through that list, you’re gonna be in a pretty good spot for yourself and your clients

Rich: digital crime is not going away.

Unfortunately, there’s no such thing basically as a ransomware-proof stack. But you were just talking about some of the elements of a stack that should be in place. They’re not like the answer, but they should absolutely be in place. So I guess it’s a two-part question. What does the most ransomware-proof stack possible look like?

And as an MSP in our audience is evaluating those tools, i- is there anything… you mentioned Blackpoint and Huntress before. There, there’s something about those solutions that leads you to believe they’re doing that function particularly [00:44:00] well. How should folks evaluate the tools that they’re building into their stack specifically around ransomware?

Oli: Yeah. So the managed EDR is really important. And you gotta have somebody watching it 24/7. I think some MSPs are trying to be the SOC and monitor the events. I think that’s a pretty big challenge. Most MSPs are pretty small and having the coverage, seven days a week, 24 hours a day, or when you walk in the morning, there’s always some sort of disaster going on with a client or something urgent that you gotta react to.

So are you getting to, look at those logs right away on, on event logs on what’s going on? I would really recommend sourcing out to a, a service provider like Blackpoint or Huntress. Both in our experience have done a really good job at catching threats and incidences before they really become something significant.

And I will tell you that there are some [00:45:00] SOCs or MSSPs out there with some big marketing budgets and big names and quite frankly, I’m not very impressed. We’ve had situations where we walk in on a case and we find out that they do have a managed EDR and a SOC, and we call the SOC up and go…

thinking, “Hey, let’s call these guys up and see what they’re seeing or what they know.” And we get answers like what are you talking about? I, I’m not seeing anything. I don’t know.” “I don’t know.” Or it’ll be several days after the incident that we’re calling them and they go, “Oh yeah, we were just about to call you guys about that.”

Again, another bogus answer. So anyway, the first two that I the two that I mentioned we’ve had really good experiences working with them. So anyway that’s a personal testimony on those guys. And then I think, what was the other part of your question?

Rich: Are there particular evaluation criteria stuff that folks should be thinking about as they’re building an anti-ransomware stack?

How do you s- through what, which [00:46:00] tool and which category is gonna be the right one?

Oli: Yeah. So this’ll be a little bit of a repeat of what I said before. Make sure you have a recoverable position, that you’ve got those immutable backups. And not all backups are immutable. There are some backups out there where the vendor will tell you, “Oh, we’re immutable.”

But really, you gotta start asking them questions about how are you making it immutable and so that a threat actor can’t still get to that data. We’ve seen some pretty ingenious things that the threat actors have done to get to allegedly immutable backups. And like I said, the managed EDR stack is really important.

This is not really rocket science. It’s, have recoverable backups, have a good managed EDR, do your patching in a very thorough and disciplined manner have cybersecurity or have good email hygiene for your users, and segment the network and and good [00:47:00] SAT security awareness training.

That’s a fairly limited stack that a lot of MSPs are already doing. If you just do those things right, you will be in a really good spot. And what I talk, you’ll never hear me talk about preventing ransomware attacks. I always use the term mitigate the risk of attack because outright defense, 100% defense is not really language I would use because you can do a lot of things right and you just miss one thing for a week and or it’s, something that you don’t even control.

It’s a supply chain breach. And something happens, yeah, it’s, you gotta just look at mitigating the damage or mitigating the risk and then mitigating the damage should it occur.

Rich: Oli, super interesting conversation. I really appreciate you taking some time out to to share your thoughts and experiences with us.

For folks in the audience who wanna get in touch with you, follow up on this conversation, learn more about [00:48:00] Olvaca learn more about your ransomware business specifically, where should they go?

Oli: Certainly on LinkedIn. You can hit our website. If you send a, fill out a web form, say you wanna talk to me or or whatnot.

But I’ll even give out my email address. It seems like everybody in the world has it anyway, and they’re sending me stuff. It’s pretty easy. It’s Oli, that’s O-L-I, @Olvaca. It’s down, I think, in my signature there on the lower left of the screen there. Olvaca. It’s basically [email protected]. And I am pretty good about responding to emails.

Rich: All right. Oli, thanks again very much for joining us. Folks, Erick and I are going to take a quick break right now. When we come back on the other side, we’re gonna share some parting thoughts about this interesting conversation. We’re gonna have a little fun. We’re gonna wrap up the show. Stick around.

We will be right back.

Welcome back, part three of this episode of the MSP Chat pod- podcast. And one last thank you [00:49:00] to Ole for taking some time out of the fire zone to talk with us a little bit about what he and his team do in response to ransomware incidents. All sorts of stuff that we could get into there. I…

One thing I, I’ve spoken about ransomware and incident response with lots and lots of people. This is the first time I’ve ever heard somebody say that one reason that they will quite often make contact with the threat actor is because it buys them time and can actually reduce some pressure on the victim, where maybe the victim is getting harassing phone calls and so on.

Interesting angle on that. I think it’ll be interesting to see in the, the wake of this episode going live if we get any input from people about what Ole was saying in terms of, needing to pay that ransom requ- And I just my two cents I have always been reluctant to be an absolutist about this in terms of just never pay the ransom.

I [00:50:00] understand the logic of that, and I can just hear my former, my buddy and former podcast co-host Matt Whitlock from ChannelPro in my ear basically saying there’s a lot of ransomware out there because people pay the ransoms, and if people stop paying the ransoms, period that it’s just not a good business for criminals.

I totally get it, but I also understand why some businesses really need to. And that gets to the other thing, and I can’t believe we’re even talking about this, Erick, but if anything jumped out at people in the audience, it is the absolute essential criticality of having immutable backups in place for every customer that you work with.

And I, I was talking about getting lots of security research. I get managed services research. I… Th- there’s often a number in there that says, whatever, 70, 75% of MSPs offer backup services to their clients. How could there possibly be anyone not doing that? If you’re in our audience and you are not 100% sure every customer you work with [00:51:00] has immutable backups in place, do that.

Stop listening, disconnect from whatever device you’re on, go get that done immediately

Erick: Yeah, man, when the bad actors can get in there and delete backups then you have nowhere to go. It’s gotta be absolutely immutable. The other thing that struck me, Rich, was just the, it’s that immediacy of when, the thing happens and he says, “Hey, sometimes selling, an agreement or a project six weeks, six months, this is like hours.”

Somebody reaches out and we have to be ready to go, right very quickly to respond to our client’s needs. Wow, that is, … that is an extra level of of readiness and potential anxiety as you’re just flying in and trying to gather as much data and information, as quickly as possible.

And, Ole and his team has, have been doing this for so many years that they’ve got that down to a [00:52:00] science. That is a rare skill to have and, no wonder they are so busy

Rich: Yeah. Yeah, exactly. They’re so busy because they have the talent to do this and because there are a lot of people still getting victimized by ransomware.

Unfortunately. Yeah. Folks, that leaves us with time for just one last thing, and I’ll tell you up front, this one is going to be a little challenging in terms of our ability to discuss this one last thing and do it in good taste. But somehow or another, I just couldn’t resist this.

And I say this as somebody who right now as I record this, I’m wearing not one but two devices that are tracking biometrics for me. I’ve got the Oura Ring on, I’ve got the Apple Watch on. I’m a guy who likes to collect metrics about various aspects of biological performance. But even I- Oh … I’m not quite so sure I’m gonna be an early adopter of what is not yet a commercial product, but could very well be at some point.

And what I’m talking about here is smart underwear. And what this particular smart underwear… [00:53:00] So th- this is something that researchers at the University of Maryland have actually developed. And in particular, what this smart underwear is designed to track is your flatulence. And the idea basically Erick, is that if we only had good, deep, solid data on, flatulence, real world flatulence data, we…

There’s a lot we could learn about gut health. You could imagine a world in which people with a lot of disposable income are being encouraged to go out and buy a commercially available smart underwear system so that just I’m learning about my sleep quality every morning and tracking my exercise statistics, I could get good, solid data on my gut health via a, a pair of undershorts that are carefully watching my flatulence.

Erick: Big capture device, if you will, Rich.

Rich: Yes.

Erick: Yeah. I wanna pass on that myself, so

Rich: Yeah. Yeah. [00:54:00] What? Yeah. Pass. There you go. Yeah. Yeah. Folks, that is all thankfully probably you’re thinking to yourself, that is all the time we’ve got for you this week on the show. Thank you very much for joining us.

We’re gonna be back in a week’s time with another episode for you. Until then, I will simply remind you, as I always do, this is both a video and an audio podcast, which means that if you’re listening to us right now, but you’d like to check us out on video, you can go to YouTube. You’re gonna find us there.

If you’re watching us on YouTube, but you’re into audio podcasts, go to Spotify, Google, Apple, wherever you get them. You’re gonna find us there, too. And wherever you find us, please subscribe, rate, review. It’s gonna help other people find and enjoy the show, too. This show is produced by the great Riley Simpson, part of the team with us here at Channel Mastered, where we help vendors build, grow, and optimize thriving MSP channels.

You can learn all about the many ways we do that with our clients at our website, which is located at www.channelmastered.com. Channel Mastered has a sister organization called [00:55:00] MSP Mastered. That’s Erick and his team working directly with MSPs to help them grow and optimize their business. Learn more about them at www.mspmastered.com.

So once again, thanks for joining us. We’ll see you in a week. Until then, folks, please remember, as we always encourage you to, you can’t spell channel without MSP.