Your customer gave up. Your dashboard called it a win.

Imagine a customer asking your support bot to change a delivery address. It explains where the settings are. The customer cannot find them on their phone, closes the chat and decides to deal with it tomorrow.

Your report shows another resolved conversation.

That is an illustrative failure, not a claim about a particular company's customers. But it exposes a buying question that deserves more attention than the chatbot's personality: what evidence lets your system call something finished?

A customer who has stopped arguing can look remarkably similar to a customer who got what they needed. If your measurements cannot distinguish them, celebrating the number is premature.

The sales pitch has moved from answers to outcomes

On October 7, Fin announced Fin 4, including expanded customer-agent capabilities, operations automation and human review. That is the supplier's description of its release, not our test result. Fin 4 announcement.

More capable support software is welcome. A system that can safely fix an account problem is considerably more useful than one that can describe the fix. It also makes the definition of success more consequential. An answer, a completed change and a satisfied customer are related, but they are different things to measure.

Read the definition underneath “resolved”

Fin's published rules distinguish confirmed from assumed resolutions: after an answer, a customer may confirm satisfaction or leave without seeking more help. The rules also say an unanswered clarifying question does not count, and a resolution is deducted if the customer returns to that conversation needing assistance. Fin outcome definitions.

Those qualifications matter. It would be unfair to describe the product as simply charging for every abandoned chat. An assumption can be a useful reporting signal; demanding a thank-you from everyone would miss plenty of successful interactions.

The mistake is promoting that signal into a stronger claim than it supports. If you want to know whether an address changed, inspect the address record. If you want to know whether the customer felt helped, silence leaves you with uncertainty. Keep that uncertainty visible.

This applies to a human support team as well. Closing a ticket is an administrative event. You still need to know what was accomplished.

A better percentage can describe the same amount of work

Fin also documents a reporting change that excludes conversations where it had no opportunity to answer from the resolution-rate denominator. That raises the rate without changing the resolution count, automation rate or billing. Fin performance-metric update.

There is a reasonable analytical purpose here: measure performance on work the system could attempt. The buyer's job is to preserve the distinction. A higher percentage after a definition change does not, by itself, show that more customers were helped.

Before comparing two months or two vendors, write down which conversations enter the calculation, which leave it, and what event earns a success. Otherwise you can spend a meeting congratulating arithmetic.

Follow one request all the way through

Return to the hypothetical address change. For an eligible order, I would ask the implementation team to demonstrate the whole path in a test account:

1. Establish who is asking. Confirm that this person can change this order. Knowing an order number is not automatically sufficient authority.
2. Check whether the change is still allowed. An order already dispatched needs a different response from one waiting to ship.
3. Make the permitted change and read it back. The authoritative order record should contain the requested address. A confident sentence in the chat is not that record.
4. Handle uncertainty honestly. If the update times out, verify its state before repeating it. Tell the customer what is known and what still needs checking.
5. Keep a usable handoff. When a person must intervene, pass the request and relevant context to someone who can act, with a clear next step for the customer.

The test should include an ineligible order and a failed update. A system that correctly refuses an unsafe change and gets the right person involved may have served the customer well, even though it did not complete the change automatically.

You do not need to give an AI unlimited authority to get this benefit. Define permitted actions, verification and approval boundaries first. The language model can help interpret the request; your application still needs to enforce the rules.

Look at the conversations that went quiet

For an existing rollout, start with a manageable review sample. Include assumed resolutions, confirmed resolutions and handoffs, spread across common request types. Treat the first review as a diagnostic exercise, not a statistically precise verdict on the whole service.

For each case, record what the customer wanted, what the system said, what actually changed and what remains unknown. Where you have a legitimate way to link records, look for the same issue returning through another channel. A new email may belong to an old problem.

Keep the reporting categories separate: answer provided, action verified, customer confirmation, human handoff, unresolved and unknown. Some overlap. That is more informative than forcing every interaction into one green box.

Then fix the cause you find. Perhaps the bot lacks permission to act. Perhaps a help article points to the wrong screen. Perhaps the handoff lands in an inbox nobody owns. Buying a more articulate model will not automatically repair any of those.

The work worth paying for

I want AI support that earns the confidence its dashboard displays. That takes connected systems, sensible permissions, tested recovery paths and measurements tied to the customer's task. Better prose is only one part of it.

If your support numbers look healthy but customers keep coming back with the same problem, tell Rosecraft which platform you use and which request keeps going wrong. We can help scope the investigation and build the missing integration, workflow or verification.

Ready to hear what needs fixing, even if the dashboard says everything is fine? Send us a message. Bring the workflow; we will start with what it actually does.