Deciding what to hand off to Customer Agent isn't really a setup problem. It's a trust problem of figuring out what you're willing to let AI own, and what still needs a real person no matter how good the tool gets. I found that out for real in June, when our full-time CS person had to take an unexpected leave of absence.
Customer Agent picked up the tickets that would've buried the skeleton crew we had left, so the team we did have could keep their heads above water instead of drowning. And it kept getting better the whole time, because every ticket it touched that stretch was basically free training data.
Getting to that point wasn't easy, though. Setting up skills felt overwhelming at first. There were a lot of options, a lot of decisions around tone and guardrails, and it took us a minute to find our footing. That's what this piece is about: not the setup, but the judgment calls that came after it.
Which workflows should you trust Customer Agent with first?
We'd actually beta tested Customer Agent early on and gone live with it once already. We didn't step away because it was bad. We stepped away because we'd just signed with a different helpdesk provider (Gorgias) right before the Customer Agent beta opened up to us. Bad timing, not a bad product.
Coming back to it, so much had changed. New skills, new training options, new ways to handle guardrails. It was more than I remembered from the first time around, and it took a minute to get my bearings again. What got us moving was starting with the basics: order status, "where's my package," return policy questions. Those felt safe because the answer already existed somewhere in our data. It's retrieval, not judgment. If you want to see the full list of what's available out of the box, the Understanding Skills Help Center article covers it. Ours activated as soon as we connected Shopify.
Once those first few skills were live, expanding got easier. Training stopped feeling like a big project once we weren't starting from scratch anymore.
What does "promising" actually look like with Customer Agent?
For us, promising wasn't a dashboard stat, at least not at first. It was that June stretch. Watching Agent absorb the volume that would've buried our thin coverage, with no backlog forming, is when it clicked for me.
That's also when I started actually watching the Performance dashboard instead of just the ticket queue: percent resolved by AI, and which escalation rules were firing and how often. Most of our order-status and return-policy conversations never needed a human at all. We were also watching those tickets like hawks that whole stretch, not just to make sure they got answered, but to catch where Agent needed sharpening and feed that straight back into training.
What should you never hand off to Customer Agent?
There are three things we don't budge on:
- VIPs. Talking to a real person is one of the actual perks of being in our top community tier. Not something we're willing to cut.
- Anyone who's clearly frustrated. Talking someone down is a judgment call, not a lookup.
- Weird edge-case order issues. That's exactly where a wrong answer has real consequences, and there's no clean answer sitting there waiting to be retrieved.
How do you set escalation rules Customer Agent won't override?
We've got a couple of rules Agent never gets to override, set up in Guidance: obvious frustration means handoff, VIP status means handoff. No exceptions on either.
A clean handoff looks like Agent catching one of those signals right away and passing the conversation along with full context, before the customer has to repeat themselves. Managing Customer Agent settings covers how handoff destinations work if you want to see the options.
A messy one is what we used to deal with on our previous AI support tool, where it would just keep going and sometimes flat out make things up, like products we've never sold in our lives. That's the failure mode our rules exist to avoid.
How has my thinking about AI support changed?
I went in bracing for that same hallucination mess, honestly, since our previous tool was the freshest bad experience we'd had with AI support. It hasn't happened. I genuinely don't worry about it anymore, which isn't something I expected to say this early on.
What also caught me off guard was realizing how much of that early overwhelm was a getting-started problem, not an actual limitation of the tool. Being able to adjust Agent in plain language, through chat, made a huge difference here. Once the basics were live, anyone on my team could train or tweak a skill, even one they didn't build, and it stopped feeling like a whole project every time. We still run it through the Agent Preview tab (How to test and train your Customer Agent) every so often to catch anything before shoppers do.
Wrapping up
Additional resources:
Read more about customer service strategy on the blog: 8 ways to transform customer service strategy in 2026
Checkout more of my articles on the community 👇
What was the first thing you handed off to Customer Agent? Is there a category you've decided is staying human no matter what? And what does your escalation setup actually look like?
Reply below!