“Can’t I just use Claude for this?”
We ran the experiment. The answer is no. Here’s the analysis.
168 conversations · Claude Opus 5 · Q3 2026
Introduction
46% of the insights Claude pulled from our customer calls were wrong.
If you’re using Claude (or any general-purpose LLM) today to analyze customer conversations at scale, you cannot trust the conclusions.
In a focused experiment across 168 customer calls, Claude surfaced insights that were wrong 46% of the time. Wrong insights that fed into feature groupings, that shaped a roadmap, that became work for product, engineering, and marketing teams.
Here’s what happened.
The experiment
We took 168 customer conversations and ran an experiment. We wanted to answer the question every product team eventually asks:
Based on customer requests, what should we build next quarter?
We used the same model (Claude Opus 5), the same prompts, the same conversations.
Two reports came back. They didn’t agree.
The reports
Each report ranked the things customers wanted most. Here’s the roadmaps provided by Claude and Format:
| # | Claude | Format |
|---|---|---|
| 1 | Manual/Paper Process Pain Position the product as a "paper replacement". | Automated Status Notifications Automated alerts for overdue items and failed checks. |
| 2 | Reporting & Data Export Invest in platform-level reporting infrastructure. | Structured Data Import CSV import that unblocks 10+ customer migrations. |
| 3 | System Integrations Build more integrations with customers' systems. | Self-Service Portals Portals for external users. Blocks 12+ migrations. |
| 4 | Inspection Management Better scheduling and mobile tools for inspections. | Standardized Reporting Region-specific annual reports for each product line. |
| 5 | Automated Communications Build automated notifications across the platform. | Location & Mapping Integration Map integrations for location data in the field. |
You’ll notice that in these two roadmaps, there is a different #1 priority, three of Claude’s five items don’t appear on Format’s roadmap, and three of Format’s five don’t appear on Claude’s:
| Claude | Format |
|---|---|
| Different #1 priority | |
| #1Manual/Paper Process Pain | #1Automated Status Notifications |
| In both reports, but with different prioritization | |
| #5Automated Communications | #1Automated Status Notifications |
| #2Reporting & Data Export | #4Standardized Reporting |
| Only on Claude’s roadmap | |
| #1Manual/Paper Process Pain | — |
| #3System Integrations | — |
| #4Inspection Management | — |
| Only on Format’s roadmap | |
| — | #2Structured Data Import |
| — | #3Self-Service Portals |
| — | #5Location & Mapping Integration |
If you’re a product or engineering team deciding what to build next quarter, these two reports point you in different directions.
Remember that we have the 168 calls, the evidence is already there, so one of these reports must be wrong.
So we went back to the original transcripts and audio and checked the work.
Where it breaks
Analyzing 168 conversations has to go through six steps, whether you do it by hand or with an LLM. Claude handles four of them well. It fails at two.
| # | Step | Claude |
|---|---|---|
| 1 | Define the question | |
| 2 | Read every conversation | |
| 3 | Extract insights | |
| 4 | Aggregate into themes | |
| 5 | Count, order, prioritize | |
| 6 | Draw conclusions |
Claude is excellent at grouping, ranking, and writing (steps 4 through 6). That’s what LLMs are built for.
The problem lies in steps 2 and 3: actually reading each conversation and extracting the right insights. At 168 conversations, Claude skipped over many conversations and simply skimmed the others. That’s the worst place for the failure to land, because every theme, every ranking, every conclusion downstream is only as good as what those two steps produce.
In summary, perfect aggregation of garbage produces polished, confident garbage.
Fact-checking Claude
We took every one of Claude’s conclusions and traced each supporting insight back to the transcript it came from. When Claude said “customers want X,” we looked for the moment a customer actually asked for X.
Often, we couldn’t find it. It wasn’t there. Across 168 calls, this happened thousands of times.
Exhibit A
Prospect:“Do you handle data export to Excel?”
Rep:“Yes, let me show you.”
Prospect:“Oh, that's great. We'd definitely want that.”
Claude counted this as:
Demand for data export
What it actually is:
A discovery question during a demo. The feature already exists. The prospect is reacting to learning about it, not asking for it to be built.
Exhibit B
Prospect:“We'd need pricing by end of January. Our budget process kicks in February.”
Claude counted this as:
Customer need
What it actually is:
Procurement logistics. The customer needs pricing, not a product feature. This is how they buy, not what they want to buy.
Exhibit C
Prospect:“Right now we use a spreadsheet for all of this.”
Claude counted this as:
Manual-process pain
What it actually is:
Context. The prospect is describing their current setup in passing, not complaining, not asking for an alternative.
All three exhibits are the same underlying error: Claude treats what was said as a stand-in for what the customer needs. A careful reader catches this. But at 168 conversations nobody is reading carefully, because if they were going to read them anyway, they wouldn’t have asked Claude in the first place.
When we walked Claude’s most prominent themes back to their source data, 46% of the supporting insights were wrong.
Each wrong insight becomes a drop in a bucket. The bucket gets a label: “reporting demand,” “manual process pain.” The label becomes a theme. The theme becomes a top priority. The top priority becomes a recommendation the product team acts on.
The takeaway
If you use Claude, or any general-purpose LLM, to analyze customer conversations at scale, you cannot, and should not, trust the conclusions.
This isn’t Claude failing at its job. Claude is excellent at grouping, ranking, and writing — it just isn’t built to read 168 conversations and reliably separate real product demand from noise.
The fix is not a better prompt. The fix is a purpose-built system where every pain, request and signal is pulled from each conversation as it was said. Nothing is summarised away and every item links back to the exact moment in the call, ticket or email. Then the extracted items are grouped, deduplicated, counted and ranked across every conversation. The output is insights and numbers you can trust.
That’s what Format does and that’s the difference between the two reports.