Our Own Rulebook Contradicted Itself About One Number

A Confidence score on this site is five factors, twenty points each, added together. That is what we publish, and it is deliberately checkable: a reader can read the five questions on our methodology page, add them up, and see whether our number is honest.

Our internal rulebook also carried two adjustments that we never published. One of them took fifteen points off the total when a product had a large pile of reviews and an almost perfect star rating, on the reasoning that a rating that uniform is weaker evidence rather than stronger.

The problem was not the reasoning. It was that the same section of our own rulebook both allowed that subtraction and forbade it. One paragraph said fifteen points come off the sum. Three paragraphs later the rule said that once the sum exists, nothing multiplies it, caps it or adjusts it by hand. Our own publishing gate takes the second view and fails any record whose score does not equal its five factors, so a record that obeyed the first rule would have been stopped by our own software.

The ruling: the sum wins. A Confidence score is the exact sum of its five factors and nothing moves it afterwards. The suspicion behind the old penalty survives, because it was never wrong: a wall of identical five star reviews is weaker evidence than a mixed one. It now gets answered inside the factor that measures exactly that, which means it shows up in a number a reader can see instead of as a deduction nobody was told about.

The second adjustment is the one we would rather not have found. It promised that a small quiet maker with no press coverage would not be punished for coverage that was never going to exist, by moving those points onto two other factors. That rule could never run. Moving the points pushed a factor above its cap, and no factor can hold more than twenty points, not in our specification and not in the software that stores it. So the protection existed on paper and never once in the data.

We have retired it rather than leave it sitting in the rulebook reading like a promise. What that costs, plainly: until we settle a replacement, a product whose outside coverage genuinely does not exist is scored on the evidence that does exist, and the absence is written into the trust explanation instead of being quietly made up for somewhere else. That can mean a lower Confidence score for exactly the kind of small maker this site was built to find. We would rather publish a number we can explain than a number that flatters.

One more thing belongs here, because leaving it out would make this entry tidier than the truth. Some records already published describe the old subtraction, and they are now queued for re-check rather than quietly reworded. The reason a number exists is part of the number. Rewriting an explanation while leaving the score untouched would invent a calculation nobody ever ran, which is a worse thing to publish than the inconsistency we found.