Episode 135: Ding Dang Right

 

Transcript:

 

Rich: [00:00:00] This episode of MSP Chat is brought to you by MSP Mastered. If you like co-host Erick Simpson’s Tip of the Week, you’ll love the comprehensive growth advice Erick and his team provide at MSP Mastered, your go-to resource for overcoming business challenges, improving service efficiencies, selling more profitable MRR agreements, and increasing the value of your MSP business.

From sales and marketing, to service delivery, to hiring and retaining high-performing talent, MSP Mastered offers access to over 90 online master classes, 150 on-demand webinars, and 250 advanced MSP tools resources, along with regular group coaching sessions and unlimited strategic email support, all for one all-inclusive membership [00:01:00] fee.

Unlock your true MSP potential by joining MSP Mastered today. And three, two, one, ball. Blast off, ladies and gentlemen. Welcome to another episode of the MSP Chat Podcast, your weekly visit with two talking heads talking with you about the services, strategies, and success tips you need to make it big in managed services.

My name is Rich Freeman. I am chief analyst at Channel Mastered, the organization responsible for this show. I am also joined physically side by side for the second week in a row, how awesome is that, by your other co-host, our CEO and chief strategist at Channel Mastered. His name is Erick Simpson.

Erick, how you doing?

Erick: Doing well, Rich, considering it is the final day of ChannelCon 2026 this year here in San Diego, GTIA’s signature event

Rich: And it has been an amazing, not quite 48 hours for us here at the show. It’s been a really terrific show, actually. A great turnout, a lot of interesting content.

I’ve been doing tons of [00:02:00] interviews for my blog, Channelholic. We’ve done, obviously, as you have seen already, some great podcast interviews, including one coming up a little bit later in the show with Hexie Xiao from Bumblebee who’s gonna go deep on agents. The actual mechanics of when you can, can’t shouldn’t employ agents to address customer use cases.

That’s coming up a little bit later. But first, let’s dive into our story of the week which actually comes out of a Channelholic post that I published a few days ago as we’re recording this right now. It’ll have been out about 10 days by the time you’re seeing this episode of the show, and it is about what I was loosely referring to in that piece as the AI pricing Niagara that MSPs are approaching right now.

And I borrowed that metaphor from Robin Ody who’s an analyst at Omdia. And the idea basically is pretty much everyone in the tech industry right now is thinking pretty hard about pricing AI and [00:03:00] tokenomics token economics, and so on. The, the … In the SaaS world in particular, they’re thinking about that a lot, along with the SaaSpocalypse.

According to Robin Ody, Jessica Davis, his colleague at Omdia, MSPs aren’t thinking along those lines quite as universally as everyone else is. And in fact, they did a survey at Omdia, where they asked a bunch of MSPs, “What are your pricing plans going forward? How are you gonna price your services in the AI era?”

And about 60% of them said that they were exploring one model or another that we can chat about just quickly here. But 40%, fully 40% of the MSP surveyed said, “I’m just gonna keep billing per user And I think if there’s one thing we know, Erick, it’s that the per user model that has defined managed services pricing for a couple of decades at this point probably isn’t going to work on its own, at least permanently at this point, or for very much longer.

And the reason, of course, is AI is making businesses more productive. [00:04:00] They’re getting more done with fewer people. If you are if your average customer today has 20 or 25 employees, it’s not unthinkable it’s gonna be more like 15 or 20 in a few years. There, there was a story in The Wall Street Journal last week about the rise of million-dollar companies with one employee.

You’re gonna see smaller businesses. You’re gonna see fewer use- users in need of the s- general services that MSPs provide. So what, what can you, how should you bill going forward? One of the models that has been under consideration is I’ll combine the remaining per user charges that I’m doing with some kind of AI consumption charge based on token consumption The issue there being not so easy to measure and predict and budget for on your side.

And also token prices continually fall over time. I spoke to the president of RapidScale, a very big business with a big anthropic resale business, and he said it… reselling tokens [00:05:00] is just really not a business people wanna get into. In fact we do it because we have these big clients, but there’s very little margin there.

So what are some of the other options? There’s a lot of talk about outcome-based pricing. The question then becomes: what is the outcome that you are pricing against? The obvious candidate would be tickets closed, but for one thing all of these service desk automation tools out there may be imperiling the future of the ticket.

To the degree that these systems, as they get smarter, proactively eliminate issues before a ticket is ever produced, tickets might be an endangered species just the way user counts aren’t a viable model. And then there’s this whole kind of realm in there where, okay, maybe I’m billing for autonomous resolutions or proactive resolutions.

Or even if I’m billing for traditional tickets, you can imagine yourself getting into arguments with a, a customer about should I pay you for that ticket which in my eyes wasn’t exactly resolved to my satisfaction?” So at… No one that I’m aware [00:06:00] of has come up with a really good, solid, reliable, easily measured little disputed outcome pricing basis for MSPs.

That leaves a model that is perhaps m- more viable from a certain perspective, but also demanding. And the president of RapidScale, Duane Barnsoy, I interview. At that company, the way they do it is employing what they call an Accenture-esque model- … where basically they estimate if you’re spending $15,000 on this workflow now, using AI we- we’ll get that down to 10,000, and you’re gonna share some of those savings with us, and that’s how we get paid.

Again, it takes a lot of sophistication to make that calculation, what are the savings going to be, and ensure that you’re getting decent margins. So the long and the short of it Erick, is just that there is a pricing question out there for MSPs. 40% of MSPs aren’t even asking it, and the other 60% don’t have a solid answer yet.

Erick: Just when we thought we had this pricing thing [00:07:00] solved, Rich. Past is prologue. We started with, having trouble pricing our services and getting clients to, to say yes and delivering value and overcoming any invoice haggling and all this other stuff.

And we said, “You know what? Let’s make it easy. Let’s make it flat fee.” Then we started seeing the subscription impact of all the different third-party services that we are including in our portfolio, and bundles of service and delivering to clients. So then we said, “Okay, maybe per user, per desktop.”

I remember, per device was when we started our MSP. It was like per device thing, and we kinda… We’ll figure out the user thing later. Let’s measure it. And of course, that first year of our MSP we lost our shirts on that pricing model, and then figured it out. But then, came the cloud, and then came backup solutions.

Then we were starting to see this little bit of, okay, maybe we need to, inject a little bit of consumption-based pricing along with our per user or per device pricing. I think that, before AI came along, Rich, I think, pretty sure that the majority of the MSPs that I speak to, at least in my mind too, we kinda [00:08:00] had it settled around a per user with maybe there’s additional billing that comes from consumption, like the M365 stuff, the backup stuff things like that.

And so then, MSPs were probably doing a, a two billing cycle per month kind of invoicing scenario, where you’re billing in advance for all the flat fee stuff, but then maybe a week or two after you get all of the consumption numbers figured out, then you’re billing clients on top of that.

Now we’re being faced with this, tokenization, tokenomics, token burn thing and consumption. And it is very difficult, Rich, as for MSPs just to get their accounting in order and the invoices correct and getting clients to pay them on time with… And trying to keep clarity around those invoices, which I’ll be talking about in a second.

But then to add an additional kind of a, a needed service that business owners want- Asking MSPs now to prove [00:09:00] not only the value of all the other things that they’re delivering, but then to come up with some other type of, ROI, so value-based pricing model, it is … Even for, organizations that are not in our space, that is a very mature operating model, and it takes a lot of trust and confidence.

So MSPs are gonna need all the help they can get trying to figure this out, and I know vendors are, trying to figure it out as well, as we talk to a lot of vendors that are also trying to figure out what’s the right pricing model for the services that we’re delivering to MSPs and through MSPs to their customers?”

I bet many MSPs, Rich, are thinking you know what? I’m just gonna have the client sign up for whatever the platforms are. They’re gonna get that bill directly, so I don’t have to worry about it and I don’t have to fight them over delivering that value,” which means that MSPs are gonna leave a ton of money on the table, and a ton of uniqueness and distinction and differentiation in being that trusted advisor for those clients.

Rich: It … They’re still … [00:10:00] Even if they do that they’re still gonna need to bill for something-

Erick: Yeah …

Rich: on some kind of basis.

Erick: Yeah.

Rich: And and the thing is I … you look at the consumption model or the outcome model. Part of what made managed services work and has made it popular with the end users is the predictability of it.

Yep. And that’s the beauty of per user pricing as well. As long as I know what I’m paying per user, and I can just look around the office and count the users-

…

Rich: I know what I’m paying every month. Whereas if you add a consumption-based element, that gets harder. If you add an outcome-based element, I don’t know how many outcomes,

Erick: Yeah

Rich: Th- and it gets to I think a larger point that I’ve been thinking about since I wrote this piece a little bit, which is I don’t know what the answer is But I believe it’s gonna be the end users that decide. And just because it worked this way with managed services doesn’t mean it’ll work this way again.

But in the managed services era, people like you came up with this flat rate IT concept. The end users decided, “I like this.” And therefore, more people had to follow in your footsteps and switch [00:11:00] to the managed services model. And then as more and more IT providers did that, the vendors were forced to align their pricing to your business model.

So I don’t know what the right answer is. I think the end users will decide, the MSPs will follow- … ’cause that’s where the business is going, and then the vendors will follow the MSPs.

Erick: Yeah, you used the term predictability, and that’s the way we sold it initially when we were trying to figure it out.

And it was about, hey you’re able to budget now for your IT investments over time. And only when there’s something that is outside of the scope of our agreement, which we tried to include as much as possible there or a project or something was there additional billing.

And that was part of the ease of getting a client to go, “Oh, okay, I can put a budget number in here for the entire year and know that, outside of any extraordinary circumstances, I’ve got this covered. I can, I’m, I have that, that, that certainty.” And now it’s a whole different world.

And [00:12:00] I like the outcome-based model, but like I said, it’s gonna take a very sophisticated mature MSP organization to figure out how to deliver services and build them based on an outcome-based model. And like you said, Rich, not all clients are gonna like one or the other type of billing. So it’s gonna be a minute before the industry figures this out.

And you mentioned, the vendors coming along and supporting MSPs after- because we were lobbying them to say, “Hey, guys, we need you to change your pricing model ’cause this is how our business operates.” And to their credit, they made that switch and helped us out. Now, it’s gonna be a very unique different, I would say a different perspective because, like I said, even the vendors are trying to figure out what the correct pricing model is.

Should we go in debt in order to grow our indirect MSP sales channel, or … Which, that, that could have all kinds of bad, ramifications 12 months down the line. Or, how do we share that cost? We’ll be [00:13:00] keeping an eye on this, Rich, ’cause I, you know, you and I are both very, very interested in it, and it throws a completely different dynamic into things, where MSPs just haven’t been faced with this type of a consumption model where it is completely unpredictable.

So I’ll give an example. Let’s say, offline backup, right? Cloud backup and things like that. The easy way for MSPs to price it is to price it if it’s between this much and this much data volume, then it’s this flat fee, knowing that, they’re keeping it within a tier that they can that they can manage.

But when it exceeds that, then they sell them the next biggest bundle or storage volume of backup. And so that was a way to kinda keep it contained. But with, with- AI, who knows when a user just goes deep and spends two days just, in some vibe, coding session, and all of a sudden, it’s thousands of extra dollars in token [00:14:00] burn.

Rich: Yeah.

Erick: So it’s tricky.

Rich: And it’s also a very good segue into your tip of the week, ’cause we were talking about predictability and surprises when it comes to invoices, and I don’t think people enjoy surprises when it comes to invoices.

Erick: You’re ding dang Rich. No client likes to look at an invoice and be surprised at something that they can’t understand, right?

MSPs have done a great job of, eliminating line item things on the invoicing and all that. We understand that, the value of what we deliver has a, a, has a cost to us. We’re gonna make our 60, 70%, target profit margin a- as much as we can, and then we wanna reflect to the client, you’re getting all of this service and all of this value for one flat fee a month, outside of, sm- small variables, not AI yet, right?

To make it simple for a client to see and understand the invoices is key number one. So we don’t want a client ever to be surprised by what’s in an invoice or by, [00:15:00] like the delay of getting invoice. That’s another thing. Some clients just are habitual late payers. They’re slow paying these MSPs.

And think about it, Rich, if you’ve got a very complicated invoice and they haven’t paid the last two invoices, and then they… Y- your- you’re- you’re saying, “Please pay your invoice. Please pay your invoice.” And they sit down and start looking at it, and if it’s difficult for them to figure out what’s on the invoice, it’s just gonna delay that invoice getting paid even further.

So a couple of tips here for MSPs suffering from this. Ask me how I know, ’cause we went through it as well. Just review your invoice billing for clarity. Try to strip down the verbiage of what’s in there. You don’t want a seven-page invoice if you can have a two-and-a-half-page invoice, right? If the client has been a client for a while, they know what you’re delivering.

Keep it as simple as possible. Don’t put any surprise additional l- items on the invoice unless you [00:16:00] have sat down with that client, they’ve agreed to this, upgrade or additional subscription of something, and you are very clear in letting them know that this is gonna appear on your next invoice, and it’s gonna look like this.

So they now see that. The third tip would be, Rich, if you have something that’s coming come, up on the invoice that may not be a recurring charge, so they, they understand it, they see it, and then they’ll see it from then on. Maybe it’s a one-time thing. Maybe it’s a purchase. Maybe it’s some hardware.

Maybe it’s a service. Something like that. You’re gonna have that discussion. If it’s a larger ticket item that’s gonna stand out on the invoice, you’re gonna be talking about that during your strategic business reviews with your client. You know you’ve got your roadmap. Okay, it’s August now. We’re gonna begin deploying this solution that was approved, and budget was approved, so just want you to know that it’s going to appear on your next invoice.

Here’s what it’s going to say, so just so you know. We’re good? We’re good. So we have a c- a conceptual agreement with the client so [00:17:00] that they understand that if something is new that’s gonna be added to the invoice, and it’s gonna be a recurring charge- Let them know that, let them understand it so that there’s no kinda, ‘Hey, I’m gonna wait till I see Erick, in a couple of weeks when he comes in for my QBR before I ask him about this line.

I’m delaying that invoice payment.’ And just overall, simplicity is the key. And sometimes, clients are just auto-paying those invoices without an invoice review because it’s an ACH or a credit card that runs for MSPs and things like that. Super, super important to follow these three tips because if they ever have a, a, a time when they’re sitting down and looking at those invoices they’ve been paying all year, you don’t want them to have any uncertainty or questions about what it is that you’ve been charging them

Rich: And I assume that basically, if there’s a guideline coming out of this advice, it is that you always check with the customer.

If there’s anything out of the ordinary one month to the next, [00:18:00] you always let the customer know this is coming. And I say this thinking in part about the example you gave before where there’s a certain sort of standard amount of backup storage allotment, but maybe somebody goes over that.

It’s written into the contract. You’ve explained it to the customer before. You don’t want that to surprise them. So you’re gonna tell them, “Hey, this month- … this is gonna happen. Let’s maybe talk about changing your plan or whatever if it’s gonna be an, a recurring issue. But just so you know,” there are MSPs smart MSPs.

As I understand it, this is a good thing to do, but they have a standard price increase built into their contract annually. Annual

Erick: increase.

Rich: Yep. A few percent. The… Again it’s signed. It’s already agreed to, but you probably don’t want that up… Yeah.

Erick: That’s another great point, right? Yeah.

‘Cause we as consumers, we see that all the time. Why did my bill go up all of a sudden?” “Oh, it was the automatic thing” and “Oh, yeah.” And then ping them and they say, “Oh, yeah, it’s in your, EULA or whatever.” I’m like, “Okay. I didn’t read, 55 pages of, minus- minuscule text.”

The, the good thing to do, A, we are all in business together with our clients. We wanna make sure that we’re [00:19:00] delivering service at such a high quality rate so that clients will want us to continue doing that and being their strategic partner. So we wanna let them know, “Hey, y- on your January invoice, our annual price increase will be X percent.”

And I remember the days, Rich when partners would be hesitant to raise their price, and we suffered through that for the first year, but then we did it kinda regularly. And, as long as it’s not an a- as long as it’s not an egregious price increase, right? Today’s con- today’s business owners are used to that.

All of the things that we pay for, as human beings have automatic price increases. Heck, streaming services will do it, 18 times a year it feels like sometimes, right? So yeah. There’s always something that you wanna make sure that you let the client know to look out for if it’s something different than what they’re used to.

And, on that topic, Rich, if you have to raise your prices before your annual price increase- That’s okay too. Cl- the cl- your clients are raising their prices to keep up with the cost of doing business today. You just have to sit down [00:20:00] and have a conversation with them and let them know, “Hey, our prices from our upstream vendors, right?

The supply chain prices have been going up, so unfortunately we’re g- we’re gonna have to raise our prices just a tiny bit just to cover our costs.” And typically, clients will be okay with that. And the ones that, complain about that, you just have a side conversation with them directly. And I don’t, and I’m not suggesting you don’t raise their prices, but you may, shave a percent off just so that they feel good about it.

Or you just determine that, okay, they’re on the short list for when, I can replace them because they just don’t fit my model anymore.

Rich: But the key takeaway, again, is no surprises. No surprises. And so just always contact the customer if a number is going to change month to month. The worst possible outcome basically is they say, “Yeah, I know.”

“Look get off the phone. I’ve got stuff to…” That is the worst case. And the best case basically is they thank you for letting them know, and you avoid an upset customer that you then have to soothe. So just no [00:21:00] surprises. Better safe than sorry. Let them know if there’s gonna be any price change.

Erick: That’s it.

Rich: All right. So Erick and I are gonna take a quick break. When we come back on the other side, we will be joined by Hexi Xiao of Bumblebee. He will introduce himself and the company, Bumblebee, if you’re not familiar with it. But he’s coming up with us onto the show because we got a chance to attend a session that he did here at the GTIA ChannelCon event about agents, and agents that he has attempted to build that did not do the job they were meant to do, other agents that were more successful.

If people are telling you that agents are the cure-all answer to every business need, they are incorrect, and we’re gonna learn why, both why agents can be incredibly powerful and where their limitations are. And that is all coming your way after the break. Stick around.

Welcome back to part two of this episode of the MSP Chat Podcast, our spotlight interview segment, our last little bit of [00:22:00] content from GTIA ChannelCon 2026. We are very pleased to be joined by Hexi Xiao. He is the CEO of Bumblebee. He also is someone who delivered a very interesting session here at the show yesterday that I attended and that we get to bring some of your way here.

It’s all about agents and agentic AI, and getting past the high-level general conversation about that, and really start wrapping our hands a little bit around how to build agents and when maybe not to build agents. But first Hexi, welcome to the show. Thank you. Thanks for having us, having me.

We have known each other a little bit for a while now, but for folks who are new to you and new to Bumblebee, tell them about those. Yeah. Bumblebee is a AI automation platform for MSPs and MSP’s clients. We help MSP build … Save time and grow revenue by automating the repetitive tasks that you and your clients have.

And we offer integrations [00:23:00] to all the systems you already use, so it makes the building process simpler and faster. So you have plenty of hands-on experience yourself building agents. Part of what attracted me to the session yesterday was the title of the session, which was Agents That Flopped.

And so the, the primary subject, although we, w- you got into agents that did the job great, but you focused a lot of time on agents that you thought would be great and then you went out and built and discovered that they didn’t actually deliver what you expected. That’s right. For very interesting reasons that say some things about the, the possibilities and the limits to those possibilities with agents.

Yes. In terms of just your thinking process, before we get into some of that what were you hoping that the attendees in that audience would take away from that? Were you h- trying to help them think a little more systematically about when to build agents? … Just share some of your trial and error with them?

What were you thinking? So starting about earlier this year, [00:24:00] we started hear MSPs discussing and asking questions like, “How can I use AI for my clients?” This is the year where the technology underneath has gotten good enough to prompt that kind of conversation, and MSPs don’t know how to start. So we at Bumblebee pioneered a bit earlier to try to solve a lot of the use cases through automation and agents, and I figured I want them to walk away with some theory, or more like framework that they can take to apply to their own use cases.

Every MSP interface hundreds of businesses, and if they can have a set of questions they take to ask their clients or ask themselves when they talk with their clients whether there’s an opportunity, we will accelerate the adoption of AI for the small, medium businesses as we uncover good use cases that deliver concrete value.

Erick: Hexie, the implication of some of those experiences you [00:25:00] shared was that MSPs have a lot of pain points that they would like to automate but can’t yet. So did I get that right? And is that a temporary condition? Will AI eventually catch up to help them solve some of these other pains, or are there some things that just won’t be able to be automated?

What do you think?

Rich: Some of them are temporary, some of them are not. We keep a very close tab on where AI’s progressing and how it expands its scope of capability to do more things. A, a very concrete example is one of the criteria we ask when building agents is what amount of data, what’s the volume of the data you need to chew through in order to successfully deliver an outcome?

That used to be a very big constraint. When the context window of a language model is 200,000 tokens, which is when we about started to build agents about a year ago, that is about a million characters, which [00:26:00] gets consumed very quickly if you process large documents. Now, a year later, we have Cowork, which is based on a sandbox environment that can handle a lot of data and aggregation.

So now data volume is not as big of a problem as long as the structure of the data is consistent. The AI is agentic enough to know where to look for each of the piece, that it can combine them without consuming the context window. So to bring it back to concrete, layman’s language, the ex- the capability of AI is expanding, and we shall see improvements continuously where the scope of automation grow over time.

So data volume wa- there were four sort of variables. When you’re thinking now about what does or doesn’t lend itself to agentic automation, there are four variables that you consider, that you learned the hard way you need to think about. Data volume was [00:27:00] one of those. Yep. The other three were consistency, verifiability, and repeatability.

We can’t dive into them as deep as you did on s- stage yesterday, but give people a high-level sense for why all four of those variables are now top of mind for you in terms of deciding what works and what won’t. I’ll talk about the lessons we learned and failures we encountered on a high level-

so you get an idea of why we start making some of these criteria a, a must-ask questions. So early on when we tried to build a, a client profitability analysis agent that does simple math calculation of running through your time entries, calculating how many hours you’re spending on this client versus how much you’re billing, what we’re really doing is a very simple math add- addition problem.

So my initial hunch was it’s easy to do math as long as we have access to the data, right? And we integrate with PSA. But what wound up happening is we have to fit [00:28:00] every single time entry into the context window for the math to take place. And that just overloaded the LLM- … at the very beginning.

Yes, there’s optimization you can do. Yeah. But as you improve your system, your client’s appetite grows, too. So they ask, “Okay, is this client profitable in the past year? What about the past three years?” So every time we fixed something- Yeah … their appetite grew again and we broke something. So data volume largely makes or break a automation if you wanna do that.

So that’s how we learned the data volume being one of the important criteria. The next criteria is about data consistency. So there’s a lot of alerts, tickets MSPs get that are sometimes travel alerts. The two devi- the same device logged in here and then all of a sudden logged in there the next time.

That doesn’t seem right. These are the situations where vendor don’t really have the true answer. The true answer lies somewhere between the client and the tech team. [00:29:00] It’s usually buried somewhere in a comment somewhere where they’ve told the IT team that they’re going on vacation. It was not documented, and it was not logged somewhere.

So when the next person pick up this alert ticket and ask the client goes, “I don’t know how I’d already told you this. Are you doing your job?” That’s when it started to become a pain- painful enough time that the MSP wants to automate it, and they wanna find a way to get to the answer if it’s already buried in their system.

So- That’s where when we started implementing this use case, we thought, “Okay, we’re gonna break this down into smaller chunks,” right? We handle one ticket at a time when we’re looking for the answers, and that helps get around the issue. Now, the, the problem is the truth lies in different parts each time.

It’s sometimes buried in the ticket, sometimes in the transcript, sometimes in the email. It’s very hard to know where, and because the structure of the answer is not consistent, AI hallucinates. It works, but it doesn’t work consistently, and if it doesn’t work consistently, it cannot be [00:30:00] automated. It has to be on a human-verified basis, and if you don’t succeed more than 90% of the time, there’s no point in even trying at all.

So these are failures that, led us to believe data volume and consistency are very important to as the prerequisites of of building automation, and similarly, quickly, verifiability and repeatability. Yeah. Verifiability came from a, a different experience where we build something.

So the, the meta use case we have is invoice auditing. That’s where we learned this case. So MSP gets billed by their vendors from may- various sources, from the distributors, and then they add a margin to bill their clients. Sometimes it’ll match up perfectly, and- MSP have to do a manual audit, eye by eye, pull up these entries.

He’s smiling. Erick is smiling.

Erick: Ask me how I know.

Rich: They do it eye by eye and that is absolutely [00:31:00] the perfect use case for machine. It’s not sexy, but it’s mission critical. I don’t wanna do it, but I have to. It is the perfect example to give to an AI. And AI, we know it’s not great at math, but it can compare numbers.

So ideally, all the prerequisites exist. The data volume is there. You can break it down per client so you’re just not overloading NLM, and the data consistency’s there. You’re always finding invoice in the same place. You’re finding the price in the same field, so we can tell AI exactly where to look.

So we thought this would be a great use case. What we’ve learned is that while we’re able to reduce the time for audit by 80%, from five days to about one day over a few iterations, so it’s still pretty big value add, we can’t get to the fully autonomous phase, because verifying invoice discrepancies requires human inputs.

It’s every time there’s a new edge case, because the issues you found, you fixed already last month, so this month it’s always new [00:32:00] use cases. So it doesn’t automate at scale, because the human verification process is not scalable in this particular use case. And to translate this into MSP’s language, when you’re trying to deliver an automation for your client, when you’re setting the expectation, you need to know, are we aiming for putting human in the right place, or are we aiming for a full-blown automation that are fully autonomous?

Because for things that are not verifiable, you can’t promise that. There’s still gonna put people doing it, so-

That’s, yeah, very important.

Erick: So it’s very catchy that the title of your session, right? Agents That Flopped and Why. What you just described to me sounds like I reduced the need for a human being to spend five days down to one day maybe.

That’s not a flop to me. That’s a very market improvement, especially for MSPs. So can you give us a couple of examples [00:33:00] of some agents that you worked on that you thought were gonna succeed and actually did flop, and why?

Rich: So there was another use case we didn’t talk about in the session that was a complete no-go. So Threat remediation is a use case we wanted to automate because it has a few characters that demands for automation. Number one, when a threat comes in, there’s usually a playbook that you run a, a list of scripts, right?

To lock down a tenant, to reset MFAs. It’s like a fixed number of steps. Number two, it is extremely urgent. So if you can do something within five minutes because it’s programmed versus 50 minutes, that makes a big difference. Sure. And more importantly that is mission critical. It is a ideal use case to automate.

As when [00:34:00] your team is constrained by human resources and you have five security incidents, it’s like, what are you gonna do? You can’t support it, right? So the demand is variable. So for demand is variable, it’s mission critical, and it’s very urgent. And we tried. The problem we run into is the blast radius is quite large if you get it wrong, and AI doesn’t always get things right.

So we try to have AI generate PowerShell scripts that run and work, but we don’t have the trust to give … hand the keys to an AI and say, “Hey, I want you to mitigate this threat. I want you to lock down a tenant and execute things step by step.” So it didn’t work at all. It’s it’s not even a flop.

It’s like it just didn’t work- Yeah … from the get-go. And very quickly we gave up the use case. So the lessons we take away from that is for something that is business critical but also easy large blast radius, if it [00:35:00] has the chance to get a lot of things wrong, like- The, the higher the risk

complete business discontinuation, if you lock down a tenant incorrectly, then you shall not automate it still. Yeah. So you spent the first third, half of the session talking about agents that flopped, and then once you had introduced people to those four critical variables, you did talk about some use cases that do work.

So let’s flip the script and talk about the s- the stuff that people can actually accomplish, and maybe we’ll begin with the MSP as sort of customer zero, as as they say. Looking internally to the MSP and their workflows and business processes, what- what’s a a good example of an agentic use case that will satisfy those four criteria and produce good results?

So we did not mention the fourth criteria which is repeatability, so I’ll briefly touch on it before we dive into why these four criteria score good on some use cases. So the fourth criteria is repeatability. We had a similar example where- We deliver [00:36:00] some value for an automation for a CPA firm, actually.

They do cost allocation for different books. What we’ve found is this is a use case that meets the first three criteria. So it has… W- we’ve sectioned the data volume to scope down to each client. We scoped out the data structure to be, dealing with the same books, right? These are similar cost CSV format that is the input of the automation, and then the output is also a fixed format.

But more importantly, we have verifiability because we have months of books that were done in the past by human that we can pass to a machine for verification. So we don’t even need human in the loop for verification because we have the finished report that are audited, final. So if I build my model or workflow using last year’s data and I test it on Jan, Feb, March, April, May, I can know whether this is working well enough or [00:37:00] not to tell if it could work the June.

So this is great. Having all these three checks, we build successful automation for one book. But when it comes down to scaling to more books, this become very difficult. It is precisely because even though they look semantically similar, the actual, like, cost allocation process for different company is very different.

The way they document their hours is very different. The way they do their projects is very different. So we have to almost rediscover these rules each time when we build automation per book, so it doesn’t scale very much across. And then very soon we’ll talk about a use case that it does scale across, and we’ll see why that makes that difference.

Erick: Then let’s talk about that use case. Okay. Now you’ve got me really dialed in. Yeah.

Rich: All right. The favorite use case that works with AI is ticket triage and ticket documentation. If you think about it from all of the four axes, the data volume is quite low, just per ticket. The data [00:38:00] consistency’s there.

Th- there’s four fields you need to update for each ticket. And the verifiability’s there. If AI misdocuments something or if the AI miscategorized something, when the human picks up, they go, “What the is the AI doing?” “Let’s fix that.” So you get immediate feedback to your agentic system to know that, oh, it has made a mistake.

Let me correct my decision-making process so it doesn’t make the same mistake again. So it is something that a human already naturally does as a part of their, working a ticket, is the verification of the previous step. It meets all three criteria. And ticket triage is fail… Like repeatable, like every ticket goes the same process, right?

So that is the favorite use case and therefore you already see wide adoption from within, their own house to custom solutions, point solutions to the PSA are coming up with those solutions themselves. You can see, like this will be continue… c- completely automated soon.

Erick: Yeah. I think partners, MSPs are really trying to get the most out of AI for their own, to take care of [00:39:00] the… To- to take care of the noise, right? That they’re spending all this human capital so that they can, reallocate the human resources to do more valuable services, right? MSPs aren’t folks that are gonna easily release their staff because they’re gonna be replaced by AI.

They’re gonna repurpose that, those staff- Yeah … especially if they’re lower cost staff, which is where the huge opportunity is for ticket triage, right? But MSPs are also really keen on tr- trying to figure out how to deliver these types of services to their customers.

Rich: So

Erick: you gave us a use case that was a no-go, was the accounting, because the books and everything, and it’s just not going to work.

Are there a couple of use cases that you’ve seen that do meet all of your four criteria that then MSPs might be able to see that as an opportunity to introduce agents and managing those agents for those types of businesses?

Rich: So we’re early [00:40:00] on that journey. Where we’re seeing the most success is in internal implementation, but we do have a few pilots that are going on with a few MSP partners to work with their clients.

We don’t know how it’s gonna turn out. It might be a flop, it might be a success. We’ll see. We’ll sure use those scoring criteria. So one example is a let’s see which one to use best. Let’s say personal injury law, there’s a customer intake process that there’s a set of questions you need to ask.

It is absolutely the scaling bottleneck because it’s business-critical to answer the phone when somebody calls in, but at the same time you can’t staff your desk 24/7 to intake. So there’s a, a reason to build an automated system for that purpose. I think for that use case I anticipate decent level of success because we’re dealing with a fixed amount of data that is also structured, right?

There’s a set of questions you ask: “Are you okay? Whose fault is [00:41:00] it? Where is this with zip code?” Et cetera, et cetera. So I think we can build a system that processes the data well. It’s pretty repeatable and not super verifiable, so that’s, I think that’s where it’ll come up a little bit short is the verifiability.

Whether the answer is correct, you still need a human in the loop to apply the same judgments. But at least answering the phone, like the data collection process can get decently automated through voice agents or through some of the other systems, as long as you deliver that same kind of experience.

So that would be a kind of use case that I feel like is interesting. And that’s a great lead-in to something that I wanted to ask about because at the end of the session you opened it up to the audience, give me a use case from your business and let’s evaluate it. Is this a good fit or not?”

And like in, in the personal injury lawyer case we were just talking about, it’s like verifiability, maybe. In, in these examples that you were evaluating live on stage, sometimes you would say, and there’s no official scoring system for this, but you were like, “This feels like a [00:42:00] 6 out of 10.

This feels like an 8 out of 10.” Those cases where you check off all four boxes and it’s clear this is a fit might be relatively rare. There are gonna be these, a lot of these edge cases. So what kind of advice can you offer an MSP, if they come up with a 6 out of 10 or an 8 out of 10 thought about an agent?

When should they just go ahead and and proceed with that? Or when should they back off ’cause it’s gonna be a flop? If the score is super low, you already anticipate problems. You should refrain. Ideally, the way you should go about it is you take your client sets, work through their industry, ask AI what are the top automation use case for them, score them internally and talk to them and validate that the scoring is correct, and then you work with one or two to pioneer these use cases.

You set expectation upfront that we wanna try this out. We think it can help you. We don’t know how well it’s gonna work, but I think this [00:43:00] is big enough of a pain point for you to give it a try. So I would go after problems that people are willing to pay for. That is usually an indicator of significant importance for a business that would justify the kind of R&D budget that MSP would get to, to build these agents, and I think that is a good starting point.

Another approach, which is very different from my approach, would be to tackle the horizontal capabilities. And for example, every company that have a sales team needs a sales assistant of sorts. Like in MSP’s example, your sales team most likely don’t know the model of the firewalls you’re selling. They don’t know.

They have no idea. But they don’t even know if you sell firewall sometimes. There’s a partner who’s like, “I don’t know if we sell firewall. I’ll text my husband right now.” But it is those moments that if you can answer those questions on the spot, then you can close the deal or move the deal forward a bit more, right?

[00:44:00] So imagine those scenarios. These are- today, people are typically done through sales training to memorize those things. But come on to, to the sales staff, firewall don’t mean anything. So If they had a assistant that have access to the knowledge base of the product catalog they sell, quickly they can ask that assistant that answers immediately and be able to say, “Hey, yes, we do sell firewall.

Here are like a few models of the brands. Do you have a preference?” Or they can drive the conversation a different direction on those fleeing moments. Those would be, like a sales assistant would be something that generally scales horizontally across all companies. So you can say, “Oh, we offer a sales assistant or a HR Q&A bot.”

These are like general things that work across that. And we offer that. Maybe you can deliver it, implement it through a co-pilot implementation or something on top of, you know, Microsoft Suites and offer that as a service. That is another approach that the market tends to go. It’s a different approach from us.

We like to find a concrete pain point that are [00:45:00] worth paying for. We deliver it, and we scale that use case. Instead of saying we offer this and have them try it out. But it’s, different approaches.

Erick: Yeah, I like that, the sales approach, because I think a lot of… what you were saying is some sales teams they tend to sell the thing that it’s easiest for them to sell, and sometimes I think with an agent it can also say, “Oh, listen, you should bundle it with these other two or three things,” right?

And now you’re getting a larger sale rather than just a point product or transactional sale. But I also like your approach where you’re actually solving something. Like you said, Hexi, it’s something that someone is willing to pay for because if we can solve that for them, it’s gonna save them so much time, energy, money, frustration, right?

So let’s say an MSP is going out there and just having conversations with their clients, like determining whether, what their appetite is for AI. They know they’re using it already, and some of their… Maybe the leadership doesn’t know that some of the staff are using it.

That’s why we get all this data leakage and stuff that we hear about. [00:46:00] But let’s say that a client that the MSP has had for a couple of years, a good client, and says here’s my problem.” And the MSP, maybe it works with you and you figure out it’s probably not going to work, right?

There’s no real good way for us to prove, measurable value for them to invest in. How should an MSP faced with that approach the conversation with their client? Because that’s a situation that MSPs don’t usually get themselves into. It’s ’cause it’s, they don’t, they typically are selling things they know about.

But if they come back and they say, they have to come back and say it’s not really going to be able to be enhanced with AI or an agent,” how do you navigate a conversation like that? What do you guys do?

Rich: So we would typically, on a situation where- A complex problem comes and we feel like we can’t tackle it.

We first dissect whether we can simplify into smaller problems- … and [00:47:00] solve a portion of it. A

Erick: portion of it,

Rich: And whether that portion solution would mean something. If you… Again, bringing back to MSP’s example, L1 ticket resolution is a large category, and generally it entails password resets, user onboarding, a bunch of things.

If you aggregate them all together, it’s hard. But if you break down to, okay, the, we do the triage, and then we have another thing that does the user onboarding, we have another thing that does this other thing, does the ROI justify the spend? And if so let’s get on a pilot. That is one approach.

The other thing you wanna think about is you need to keep a repository understanding of what AI is good at, right? AI is good at retrieving information. AI is good at engineering things. AI is good at reverse engineering things that have a predefined outcome. If you can verify the outcome programmatically at scale, generally AI can figure out the internal workings in between.

So [00:48:00] understanding what AI’s good at, you start looking for parts of the problem that AI can tackle and, dig deeper into how much does that part cost. And that’s generally how we would approach it. Even with the CPA use case when we watched their cost allocation process, we broke down and say, “Hey, the very first step is a repeatable process across every single client.

We’re gonna strip it out to build a standalone workflow that works for everybody. And then we’re gonna build custom workflow for each book that you have trained on the past books you’ve done for this client.” “And each of this workflow might use a different integration, one with ConnectWise, one with Autotask, and then we’ll incrementally get there step by step.”

Erick: So you’re eating the elephant, maybe not the whole elephant at once, but you’re eating it bite by bite, and then you’re orchestrating all the different workflows and agents then to deliver kind of a larger outcome.

Rich: Correct. Correct. Correct. Yeah. That’ll take some building experience for an MSP to feel comfortable saying, “Hey, we can do portion, [00:49:00] but we also can do the whole thing.”

Hexi it was a really interesting session, really interesting conversation here. We thank you for taking some time out to speak with us. For folks in our audience who have some questions, wanna talk with you more about your experiences with agents, or they just wanna learn about Bumblebee, where should they go?

Oh, you can come to hirebumblebee.com. It’s our website that has a scheduling link that you can get on my calendar and we can chat more, or you can just add me on LinkedIn. That’s another option. All right. Hexi Xiao, CEO of Bumblebee, thank you for joining us on MSP Chat. Folks, Erick and I are going to take a quick break here.

When we come back on the other side, we’re gonna share some final thoughts about this very interesting conversation we just concluded, have a little fun, wrap up the show. Stick around. We will be right back.

All right. Welcome back to part three of this episode of the MSP Chat Podcast. [00:50:00] And thank you again to Hexie Xiao from Bumblebee for joining us here on the show. Erick, I found that to be not just a really interesting conversation, but a really enlightening and useful one. In addition to being the chief analyst of Channel Mastered, I am a journalist.

I’ve got a publication called Channel Hawk. I talk to vendors all week long every week. Needless to say, they are continually talking to me about their the AI functionality that they’re embedding in their products right now. Everybody is incredibly excited about the potential of agentic AI, and they are all serving me pitchers full of their Kool-Aid, basically.

And no matter how hard I try to resist it- … I wind up sipping some anyway. And it was just really interesting and useful to talk to somebody who actually works in agentic AI and get a feel, a real-world feel, for both the incredible power and potential of agents, and the also very real limitations, some of which, as, as Hexie revealed, are [00:51:00] not necessarily temporary limitations.

Some, he was talking about how over time you’re gonna see the models get more powerful and the context windows will get bigger, and there’ll be certain things it can’t do now that it will be able to do. But there are also some s- stumbling blocks, and there are some issues that may always be issues.

We may never get to the day when agentic AI is the magic easy button solution to every issue that an MSP and their clients runs into. And having some basis for understanding where it is useful and where it isn’t, and where it might not be at any point in the future is is really interesting.

And the other quick thing that I’ll say is, we talk- It’s a, the huge thing in the MSP channel right now that you’re seeing all these service desk automation tools come around from the big established vendors, startups. Every last one of them includes ticket triage functionality, and from a certain point of view, of course they do because that is the thing that every MSP needs and wants help with obviously.

So that’s where they begin. But we just learned another [00:52:00] reason why these tools always start with ticket triage. It is a perfect use case for agentic AI.

Erick: It’s absolutely very mind-expanding to have a conversation with Hexie, as you mentioned, Rich, who is in it, doing it. And to learn that He’s got four kind of areas that he uses to identify whether a, a workflow or an agent is a good idea.

And I’ve got them down here. So let me see. There, there was… And I’d never heard this before.

Rich: No one had.

Erick: Right? Data volume. These are all the four criteria that he judges whether or not we should try something or not. Data volume, the amount of data; consistency, the data consistency; the data verifiability; and repeatability.

You know- For someone like me, an engineer, [00:53:00] that makes perfect sense. Oh, I see. If I’m gonna sit there and identify whether or not I should do a thing, I’m first gonna measure it against the blockers. How successful… what’s my measure of success? And what was interesting was even if you score highly in three of those and you don’t score very highly in another one, then it may not deliver the kind of ROI that the end user or the client or the MSP expects for you to do that thing.

So that was really interesting how he looked at it from a, is it worth the, the investment and time, energy and money to return the ROI? Which is exactly how MSPs should be talking to their clients about anything that they deliver to them, right? But then the other thing that I also took away from it was just because we can’t eat the whole elephant in one bite, we might be able to eat bits and bites of the elephant.

So breaking down that larger problem into smaller chunks that then we can be [00:54:00] successful at, and then do the sum of those smaller chunks also qualify within these four parameters for success? I… as someone like me that, you know, like I said I think very logically about things, I’m like, wow, MSPs can follow that logic and reasoning, and we can express that logic and reasoning to our clients.

And having, a, a strategic, partner like Bumblebee that delivers it that way, I think that’s the trifecta right there. And now we’re delivering real value to our clients, MSPs are receiving the same value, and everyone is speaking from the same code book, if you will. It’s… and having those difficult conversations with a client where the MSP understands how to qualify are these four things true before they promise anything is also super helpful.

Rich: Yeah. Yeah. No, absolutely. It, it is a problem for an MSP today to go into a customer, the customer says, “I would love it if you could automate this particular workflow.” I- it’s risky, [00:55:00] frankly, to just say, “Nope, can’t be done,” right? ‘Cause there are so many other people who will come in and say they can do it.

It’s even worse though, obviously, to overpromise and underdeliver. You do not wanna say, “You bet I’ll fix that for you,” and then fail to do that. And so to know how to analyze, to assess whether or not you’re gonna be able to build an agent that just fixes that problem, and then also to know what to do if the answer is no then how do we maybe start to break it up into pieces that we can…

That’s that’s really important knowledge and skills for MSPs to have right now.

Erick: Yeah, I agree. No- nobody wants to be change ordered to death, and especially not an MSP’s clients, and MSPs don’t like doing it either. And frankly, sometimes we’re not good at it. We’re not good at going back to the client saying, “Hey, this kind of project, is veering off the tracks here.

We’re gonna have to implement change management and charge you more money.” It’s an awkward conversation. The better o- option is to, assess properly first, and then be completely, candid and direct with a client about what, is possible, what [00:56:00] may not be possible. And then let the client decide whether it makes sense for them to invest in that pilot.

Rich: Totally agree. And folks, that is all the time we’ve got for you this week. This is our last little appearance, our last little bit of content to for you coming from GTIC ChannelCon 2026. But we will be back in a week’s time with another episode of the show for you.

Erick: Until then- Rich, pause One last thing.

Oh, fuck. Okay. That’s okay, but you said there’s a clean break. Okay. Riley can pick it up. Yeah. Because you said, “Okay, folks, that’s all.” So now’s a clean break, so now, okay.

Rich: Got it.

Totally agree. And it leaves us, folks, with time for just one last thing, and it’s coming to us from Dunellen, New Jersey where your worst nightmare recently came to life. That is, of course, a shrub with a badge. What am I talking about there? Folks, you probably have your phone with you when you’re driving.

I certainly hope you’re not consulting it too closely while you’re driving, ’cause that’s not safe. Don’t do that. And in fact, [00:57:00] it is also illegal in many places, including apparently Dunellen, New Jersey, where the police are out looking for distracted drivers who are paying too much attention to their phone, and they have figured out a creative way to spot them without the driver knowing the, the police is out there looking for them.

And we’ve all seen speed traps of various kinds and where the police will hide. I am personally not familiar with any speed trap situation in which a police officer dressed up as a shrub to avoid being detected, and yet that is what they have been doing in Dunellen. There is a link to a news story about this in the show notes.

I encourage you to go there- … and look at the picture of this police officer c- covered in greenery watching drivers go by. So ye- yeah, so once again, don’t drive distracted, but if you do, understand i- in this day and age, it’s not just AI that is surveilling you. Every tree and shrub you drive by might potentially be surveilling you as well.

Erick: #ticketedbyapine.

Rich: Insult [00:58:00] to injury, folks. Don’t let it be you next. Speaking of next, we are gonna be back next week with another episode of the show for you. Until then, I will simply remind you this is both a video and an audio podcast, which means that if you’re listening to us right now, but you’d like to check us out on video, go to YouTube, look up MSP Chat.

If you are watching us but you’re into audio podcasts, go to Spotify, Google, Apple, wherever you get your audio podcasts. Look us up there, too, and wherever you do find us please subscribe, rate, review. It’s gonna help other people discover, enjoy the show just like you do. This show produced, i- is produced by the great Riley Simpson, part of this the team here with us at Channel Mastered, where we help vendors build, grow, and optimize thriving MSP channels.

You can learn about the many ways we do that at our website, which is located at www.channelmastered.com. Channel Mastered has a sister organization called MSP Mastered. That’s Erick and his team working with M SPs directly to help them grow and optimize their business. You can learn more about that [00:59:00] at www.mspmastered.com.

So once again, we thank you for joining us. We’ll see you in a week. Until then, please remember, as we always ask you to, you just can’t spell channel without MSP.