Product
Turn thousands of customer reviews into roadmap priorities, and validate feature ideas before building.
Updated 4 days ago · 10 sources
Proof over hype
- Enterprise
Ticketmaster
Ticketmaster's client support team needed live context during ticket sales, like how a show was tracking against a comparable past event, but getting that meant filing a request and waiting for someone to run a manual data pull. On a live sale, that wait left a client's question just sitting there.
Support could answer live sales questions immediately instead of waiting on manual data pulls, and Muehlenkamp shipped four more feature prototypes the same way within three months.
The takeaway A prototype (a clickable fake, not real code) isn't there to look good in a review. This one's job was to settle a real design argument, side panel versus takeover, with actual power users before a single engineer wrote code — the difference between a demo and a decision tool.
How they did it
Product manager Brian Muehlenkamp had already spotted Databricks' natural-language query tool and realized the fix wasn't a new report, it was surfacing that tool inside the dashboard support staff already had open. Before committing engineering time, he used Figma Make to test the actual interaction: should the AI chat take over the screen, or sit in a side panel a rep could glance at while still watching the live dashboard? Built with Ticketmaster's own design patterns, the prototype felt close enough to the real product that power users could react to it honestly rather than imagining a mockup, and they confirmed the side panel, because it let them ask a question without losing sight of the sales numbers they were already watching. That real validation, not a slide deck, was what got engineering to commit. The assistant shipped internally as 'The Wizard': support staff get sales numbers the moment a client asks, instead of filing a request and waiting. Muehlenkamp went on to prototype four more features the same way over the next three months, and leadership liked the process enough to ask him to teach it to the rest of the organization.
Tool used Figma
- Mid-market
Algolia
Every pricing page test at Algolia needed a developer to build the variant, so most test ideas quietly died in a backlog instead of ever running.
A 15% lift on the target metric, with an estimated six-figure effect on sales pipeline.
The takeaway The real bottleneck wasn't test design, it was the developer queue. Removing that step, not making the test itself smarter, is what took this from weeks to hours.
How they did it
Website optimization manager Brittany Cole used Amplitude's website conversion agent to go from idea to a running test without filing a ticket for engineering. The agent generated multiple working variants of the pricing page for her to preview, and she iterated on the appearance and functionality directly with it instead of through a build queue. 'I had a ready-to-launch experiment in a matter of hours instead of what would have been weeks, maybe months,' Cole said — she was sharing working preview links with stakeholders the same day she came up with the idea, instead of waiting on a sprint.
Tool used Amplitude
- Enterprise
LIFULL Co.
At LIFULL, only people who could write a database query could actually investigate why a metric moved, so everyone else had to interrupt the data team to ask about every odd number they noticed.
Investigation time per initiative fell from about 90 minutes to about 15.
The takeaway This wasn't really an analytics story, it was a bus-factor story: the company stopped depending on the two or three people who could write a query, which matters more than the minutes saved on any single investigation.
How they did it
LIFULL's Yuuki Sasaki connected Amplitude to Figma, Jira and Confluence through AWS, so the AI could pull in product specs and team documentation alongside the usage data, and let anyone ask questions in plain language — investigating a spec, checking a funnel, or diagnosing a KPI drop — instead of writing a query. Kotaro Inoue said the effect went beyond speed: 'We've significantly reduced our dependency on specific individuals, and the team can now run the growth cycle at a quality level approaching that of our experts' — the bottleneck stopped being which two or three people on the team actually knew how to query the data.
Tool used Amplitude
- Enterprise
Duolingo
Duolingo wanted to run more product experiments than its engineers had time to build, against a backdrop of social-media claims that AI could write 'anything you want in five seconds.'
A measurable but explicitly modest per-engineer productivity increase, the company's first in years, while continuing to run hundreds of concurrent A/B tests.
The takeaway A CEO on a public earnings call had every incentive to talk up an AI number, and he still called it 'not humongous.' That restraint is worth more than a company that claims 10x and won't show its math.
Duolingo Q1 2026 earnings call transcript · May 2026
How they did it
Duolingo rolled AI tools into its engineering and product teams and kept running its usual large slate of concurrent A/B tests. On the Q1 2026 earnings call, CEO Luis von Ahn reported something the company hadn't seen in years: a real increase in output per engineer. He was careful not to oversell it: 'The increase is not humongous, but it is kind of the first time we have seen an increase on a per-capita basis in years' — and he explicitly pushed back on the bigger claims circulating online, noting that the promise of programming 'anything you want in five seconds' hadn't shown up at Duolingo's actual scale.
- Learn from thisSmall business
METR
Everyone on an engineering team says AI makes them faster. Nobody had actually timed it.
First round: 19% slower with AI. Second round: 18% slower for original developers, 4% slower for new developers, both within the margin of error.
So: Time the same task on your own team, with and without the tool, before you buy — and if your team refuses to do the task at all without AI, that refusal is itself a data point worth taking seriously.
The takeaway The selection-bias problem cuts both ways: developers who most love a tool self-select out of any fair test of it, which means both blind faith and blind skepticism about AI productivity are probably measuring the wrong people.
How they did it
METR, an independent AI research non-profit, ran a randomized controlled trial in early 2025: experienced open-source developers, paid $150 an hour, worked on tasks from their own real projects, each task randomly assigned to 'AI allowed' or 'AI not allowed.' The result surprised even the researchers, who had expected a speedup: developers were 19% slower with AI, not faster. METR reran the study in August 2025 with 57 developers, most of them new to it, across more than 800 tasks in 143 repositories, at a lower hourly rate. The second round came out close to even — original developers 18% slower, new developers 4% slower, both within the study's own margin of error. METR's own explanation for why this contradicts what developers report about their own speed is a set of selection effects baked into the study itself, not a claim that AI never helps: developers increasingly refused to work without AI at all, even for study pay, which pushed the highest-uplift users out of the sample entirely; 30 to 50% of participants admitted avoiding tasks they expected AI to speed up the most, specifically because those tasks were hard to measure fairly; and participants doing AI-allowed work sometimes produced different documentation and testing than they would have by hand, muddying what 'the same task' even meant. METR's own conclusion is that the study likely understates real gains for the developers most invested in using AI well, not that AI has no effect.
Try this today
Rank your feedback by theme
20 minA short list of what customers actually asked for, with a real quote behind each one.
Copy the prompt
You are a product researcher. Below is customer feedback my team collected from support tickets, sales calls and emails. Group it into themes. For each theme give four things: the problem in the customer's own words, the number of separate items that mention it, one exact quote, and whether it reads as a bug, a missing feature, or a confusing design. Rank themes by how many items mention them, not by how strongly they are worded. Mark any theme with only one item as 'single mention'. Add nothing that is not in the text. Put anything you cannot place under 'Unclear'. Feedback: [PASTE FEEDBACK]
One check first. Check the sample size and read two raw quotes yourself. A summary is not the same as talking to a customer.
Pressure-test your spec
15 minThe open decisions and missing cases in your spec, written down before anyone starts writing code.
Copy the prompt
You are a senior engineer reviewing a product spec before your team commits to it. My spec is below. Do three things. First, list every decision the spec leaves open, hardest to work around first. Second, list the edge cases it does not cover — what happens when data is missing, the user is brand new, or the action fails. Third, write the five questions you would ask me in the review. Quote the line that prompted each point, or write 'not stated' if the spec is silent. Do not suggest new features and do not rewrite the spec. Spec: [PASTE YOUR SPEC]
One check first. It finds gaps, not whether the thing is worth building. Your customer evidence still decides that.
Rehearse cutting a feature
15 minYou walk in knowing the three hardest objections and roughly how you will answer each.
Copy the prompt
You are playing my head of sales in a meeting. I will tell you which feature I am cutting this quarter and why. Argue against me with the reasons a sales leader really uses: revenue at risk, a named account, a promise already made. Give me your three strongest objections, hardest first, one at a time. After each answer I give, say whether it would satisfy you and what still bothers you. Do not be polite and do not agree early. Stay in role until I write 'stop'. The cut: [DESCRIBE THE FEATURE AND WHY YOU ARE CUTTING IT]
One check first. The model is guessing at what your stakeholder cares about. Ask them first, then use this to rehearse.
Specialized tools, and what to ask vendors
| Tool | What it does | Setup | Best fit |
|---|---|---|---|
| Dovetailfree plan, then a custom quote | Stores your interviews, support tickets and sales calls in one place, then summarises the themes it finds. | Under a weekIT sign-off | You have more customer conversations than anyone on your team has time to read.Skip it ifYour interviews are not recorded or transcribed. The tool has almost nothing to read and will find patterns in four notes. |
| Mazequote only; AI interviewer is enterprise | Runs tests where real users try your design, plus interviews where an AI asks the follow-up questions. | WeeksIT sign-off | You test designs often and the write-up is the part that slows your team down.Skip it ifSensitive or regulated subjects, where an AI asking follow-ups will miss what a trained researcher would hear. |
| Productboardfree plan, then from $19 per maker | Pulls feature requests out of support, sales and email, groups them into themes, and links them to roadmap items. | Under a weekIT sign-off | Requests arrive in five different places and nobody can name your top three themes.Skip it ifFewer than a few hundred requests a quarter. A spreadsheet and one hour of reading beats it. |
| Figma Makeincluded from $16 per Figma seat | Turns a written description into a clickable prototype in minutes. | Under a week | You need to show an idea to engineers or customers before anyone commits build time.Skip it ifAnything headed for production. The generated code is for showing an idea, not for shipping to customers. |
| Amplitudefree to 2M events, then quoted | Answers questions about your product data in plain English, and can build an A/B test (two versions shown to real users to see which does better). | WeeksIT sign-off | Your product already sends usage data and your team asks the same five questions every week.Skip it ifYour usage data is a mess. The agent answers confidently from bad numbers and nobody catches it. |
Questions to ask before you buy
Dovetail — 7 questions to ask them
Also used by. Atlassian, Shopify, Canva, Breville and Deloitte are named on Dovetail's own site
- Where are our customer interview recordings stored, in which country, and who inside your company can listen to them?
- When your AI gives me a theme, can I click straight through to the exact quote and timestamp it came from?
- What does it do with a project that has only four interviews? Does it still report a theme, and does it warn me?
- How do you stop one loud customer's opinion being written up as if many people said it?
- If we cancel, what do we get back, in what format, and how long do you keep copies of our recordings?
- Is our data used to train any model, yours or a third party's, and can we switch that off?
- Show me a customer who stopped using you and tell me what they said on the way out.
Maze — 6 questions to ask them
- When your AI asks a follow-up question, who wrote the rules behind it, and can I read them first?
- How do you stop the AI interviewer leading people toward the answer we were hoping for?
- Where are the session recordings and transcripts stored, and who at your company can watch them?
- With eight participants, what does your report say about confidence, and does it warn me at all?
- Which AI features sit behind the enterprise tier, and what does that tier cost for a team our size?
- Can we pull out the raw video and transcripts if we leave, and how long do you keep copies?
Productboard — 6 questions to ask them
- When your AI merges two requests into one theme, can I see why it decided they were the same?
- How many AI credits does a normal quarter of our feedback volume actually burn through?
- What happens to a request that is badly worded, very short, or written in another language?
- Can it tell one request from a large customer apart from the same request from fifty small ones?
- If we leave, do we export the notes, the themes, and the links between them, or only the raw notes?
- Which of your customers has our request volume, and can I speak to their product lead directly?
Amplitude — 6 questions to ask them
- When the agent answers a question, can I see the query it ran and the events it counted?
- What does it do when our tracking is wrong or missing? Does it say so, or answer anyway?
- Who reviews an experiment the agent sets up before it goes live to real customers?
- How many events does our current traffic generate, and what does that cost past the free tier?
- What data leaves our account when the agent answers a question, and where does it go?
- Show me a customer whose agent rollout stalled, and tell me what went wrong for them.
What everyone is asking
- NewJul 2026
Best AI for user research synthesis (turning interview notes into themes)?
Dovetail for pulling themes out of interviews and tickets you already hold. Maze if you also need to run the sessions. Both need recordings before they do anything.
What to watch for
The comparison is published by Maze, so read it as a vendor's view. Its March 2026 survey of about 500 practitioners is the more useful half.
Also worth a look. Dovetail, Maze, UserTesting, Productboard Spark
- NewJul 2026
Does AI actually make my engineers faster?
Measured, not much. METR's trial found experienced developers 19% slower with AI. A repeat round found 4%, inside the error bars. Surveys of executives say something else.
What to watch for
Marty Cagan cites Atlassian's 2026 survey: 89% of executives say AI made work faster, and 6% can point to a return.
Also worth a look. METR randomized trial, executive surveys
- NewJul 2026
Can AI-generated users replace real customer interviews?
Synthetic users (AI-invented customers, not real people) agree with you too easily. Researchers use them to draft screeners and pilot a study, then test with real people.
What to watch for
In a May 2026 survey of 150 researchers, 97% used AI somewhere in their work and 8% used AI-generated participants regularly.
Also worth a look. AI-generated participants, recruited real users
- NewJul 2026
Best AI for sorting customer feedback into a roadmap?
Productboard Spark if requests arrive as text from many places. Amplitude if you want feedback sitting beside usage data. Both need real volume to earn their price.
What to watch for
Canny wrote that comparison and ranks itself first. It does list real weaknesses for each tool, including price floors and volume minimums.
Also worth a look. Productboard Spark, Enterpret, Canny, Amplitude AI Feedback, Thematic
- NewJul 2026
Can a product manager build a prototype (a clickable fake) without a designer?
For showing an idea, yes. Product managers at Ticketmaster, ServiceNow and Affirm built their own in Figma Make. What comes out is a demo, not shippable code.
What to watch for
Both sources are vendor blogs, so treat the numbers as the vendor's. The people and companies named in them are checkable.
Also worth a look. Figma Make, Amplitude AI Agents
Worth following
Lenny Rachitsky
Writes a product, growth and career newsletter with more than 1.2 million subscribers.
weekly or more · newsletter
Why them
Deeply researched advice aimed at product leaders and founders, not theory about the craft.
Teresa Torres
Author and coach who runs Product Talk, a site about making better product decisions.
a few times a month · blog and podcast
Why them
She teaches continuous discovery (weekly customer conversations instead of one big study a year) with worked examples.
Marty Cagan and SVPG
Silicon Valley Product Group partners writing on product strategy, discovery and how teams are run.
a few times a month · blog
Why them
Cagan is blunt about AI speed that does not produce better outcomes. His July 2026 piece says so.
Mind the Product
A large product management community and publication, now owned by Pendo.
near-daily · publication
Why them
Near-daily posts and podcasts from working product leaders, and it names who is speaking.
Paweł Huryn — The Product Compass
Writes a product management and AI newsletter with more than 135,000 subscribers.
several times a month · newsletter
Why them
Step-by-step playbooks and prompts you can run the same day, rather than opinion pieces.