The Audit We Described And The Audit We Run

Every product on this site carries a Confidence Score out of 100, built from five factors worth twenty points each. Our methodology page has always told you what those five factors measure. On every one of the five, it described a different measurement than the one we actually perform.

The page said Source Diversity counts how many independent sources mention a product, across editorial, community and retail channels. What that factor actually scores is where the reviews live and whether they can be trusted to be real: ten points if the review section runs on an independent review platform or a marketplace that only accepts verified purchases, five if at least half the visible reviews carry a purchase-verification badge, five if a public reviews policy is linked from the review widget itself.

The page said Platform Integrity asks whether the seller is legitimate, and listed secure checkout and clear return policies. What it actually scores is whether the sentiment reads like real people: an average between 3.5 and 4.9 stars, because a flawless rating is a warning rather than a reassurance; at least two low-star reviews describing a real functional problem; a brand that answered a neutral or negative review in the last six months; and top reviews that vary in structure and vocabulary, the way genuine writing does and templates do not.

The same gap ran through the other three. Not one of the five published descriptions named the questions behind it.

Where the wrong descriptions came from

They were never a drift or a typo. In an earlier version of this system, the five factors were renamed to fit the field names our data uses. The rename table paired each new name with a short explanation invented to make that name sound sensible, and stated in the same breath that the underlying method had not changed. That was true: the method never changed. The explanations described the new words, not the questions.

Our internal specification retired those explanations. Nobody removed them from the site. They have been published continuously since, through several redesigns of the page that carried them, because every reader and every review of that page saw five confident sentences that read perfectly well on their own.

How it was caught

It was not caught by looking at this page. A separate job was queued to rename the factor labels on about fifty of our lab records so they matched the field names. Before running it, the check that precedes work of that kind asked what the site currently tells readers those names mean. That question is what found this. The rename is now on hold behind it, because relabelling records to match a definition we were about to correct would have turned an inconsistency into a contradiction, published on purpose.

A second reader was then set on the finding with instructions to break it. It found that we had declared one of our machine-readable files clean after checking a single line of it, and that the same wrong descriptions were sitting in that file too. That correction is included here.

What changed, and what did not

The five factor descriptions on our methodology page now state the actual questions and the points each one is worth. The trust page, which published four of the five and omitted the fifth entirely, now publishes all five. One more line went with them: the description of what a high Confidence Score means was still written in the retired vocabulary, on both pages, and it is corrected too.

Both machine-readable files carry the same question sets. Correcting those turned up five more places where our wording was looser than the rule it described, including a review threshold off by one and a six-month window that had gone missing. All five are fixed here too.

No score moved. This corrects what we said the questions are, not what any product scored against them.

What the second reader found in this correction

Every correction here was checked by a second reader whose only instruction was to break it. It broke it. The first version of this fix got all five descriptions right on the page a person reads and five details wrong in the files a machine reads. It published a rule about score adjustments our own specification has not finished settling. And it certified that a piece of the old wording was gone from the site while that wording sat twenty-five lines below the text we had just corrected.

None of it reached the site. A pass that exists to correct false published statements had written five new ones, and would have shipped them under a receipt saying it had not.

The uncomfortable part is that the honest version is the less impressive one. “We count how many independent sources mention this product” sounds like research. “We check whether the reviews run on a third-party platform with verified purchases” sounds like plumbing. The second one is what we do, and it is the one that can be checked.