A five-star review and a one-star pile-on can both be honest
Almost every tool review you read is a first-try test. Almost every complaint is about the twentieth try.
Here is a thing that looks like a contradiction and isn't.
Search for reviews of a popular AI app-building tool and you will find blogs scoring it 4.7 and 4.8 out of 5. Directly above them in the same results page, you may find the company's own review profile sitting at something closer to 1.4. Same product. Same month. Both pages written by people who are not lying.
I ran into exactly this recently, and the resolution turned out to be more useful than the verdict.
The tool was Bolt.new. I gave it one paragraph containing six numbered requirements, the identical brief I had already handed to two competitors, so the results could be compared rather than merely described. Two of the six were written specifically to be fumbled: one asked for a number that has to be derived rather than displayed, and one asked for a state that leaves a running total without leaving the page.
It got all six on the first attempt. No follow-up prompts. And it did more than clear the bar. When I opened the source, the persistence layer was a separate module rather than logic smeared into a component, and the key it wrote to storage was versioned, the sort of thing you do because you expect the shape of the data to change later. Nobody asked for that. When my brief asked for browser storage, it wrote back that it had a hosted database available but was using browser storage because that is what I had specified. A product declining its own upsell in writing is rarer than it should be.
The whole thing cost 100,000 tokens, about a tenth of what the free plan hands out in a month.
That is a genuinely good result, honestly measured, and I would stand behind every sentence of it.
It is also almost completely useless as a prediction of what your month with this tool will feel like.
Because look at what that test actually measured: one carefully specified request, answered correctly, first time. And now look at what the angry reviews are about. Not the generated code, which barely comes up. They are about tokens vanishing into debugging loops, the same fix attempted five times, then billing, then trying to get money back. Every one of those complaints lives in the space after the first try.
A first-try test cannot reach that failure mode. Not because the tester is careless or bought, but because the scenario being tested is structurally the one the product handles best. You write a clean prompt, you get a clean result, you write it up. The review is accurate and the number is real and the experience it describes is the narrowest, kindest slice of the product's life.
So the two scores are not in conflict. They are measurements of different events, both reported as if they were measurements of the tool.
I want to be straight about the limits of my own side of this. The one-star profile blocks automated access, so I could not read those reviews myself; I am going on how that page is summarised in search results, which put it at about 1.4 out of 5 across roughly two hundred reviews in early August 2026. What I could read was the highest-ranking discussion thread and the competing write-ups that engage with the complaints, and they point the same way. I also checked six other reviews of this product for any mention of that score. None of them mentions it at all, which is its own small finding.
The habit I would like to leave you with is not to distrust reviews. It is one question, asked before you get to the verdict: what did this review actually do?
If it describes one prompt and one result, you have learned something real about the tool's best case and nothing at all about its worst. If it reports a cost, take it as a floor rather than a typical month, because my build was cheap precisely because it was right first time, and iteration is where the meter actually runs. If it never mentions a price, a limit or a second attempt, it is a demo write-up wearing a review's clothes.
And if a product's own reputation page is ranking above the blogs praising it, that gap is not noise to be explained away. It is the most interesting fact on the page. One clean build cannot reach whatever produced it, and the review that pretends otherwise is the one to stop reading.
I wrote up the whole test, the brief, the six requirements, the code it produced and the number it cost, in the full Bolt.new review, including the parts I could not verify.
