Many AI pilots in insurance end in the same place: everyone agrees the demo looked good, and no one can say whether it actually worked. That happens when the metrics get defined after the pilot instead of before it.
Start with accuracy on your own data, not a vendor's. A system that performs well on generic examples can still struggle on your specific policy language, your forms, your edge cases. Define accuracy against a sample of your real cases, reviewed by your own people, before you start.
Define what "good enough" means for this specific use case. The bar for a customer service suggestion that a person reviews before it goes out is different from the bar for something that reaches a customer directly. Match the metric to how the output will actually be used.
Measure consistency, not just correctness. Ask the same question, or a close variant of it, more than once. If the answer changes meaningfully between runs, that's a real problem even when any single answer looks fine.
Track how often the system flags uncertainty, and whether that matches how often it should. A system that never expresses uncertainty is not necessarily more accurate — it may just be less honest about its limits. Under-flagging is often a bigger risk than over-flagging.
Measure the human cost, not just the output. How much time does your team spend checking or correcting the system's work? If verification takes almost as long as doing the task manually, the pilot hasn't actually saved anything yet — even if the output looks impressive.
Agree on a stop condition, not just a success condition. Decide in advance what result would tell you to pause and rework the approach, so the pilot doesn't quietly become a permanent process because no one set a bar to fail.