What a 90 percent confidence should mean

Every decision Sixth Sense makes carries a number between 0 and 100. That number decides whether the robot acts or asks. So it matters what the number means.

A definition you can check

A confidence score is calibrated when it matches the outcome rate. Take all the decisions the model scored at 90. If the score is calibrated, about nine in ten of them were right. Take the ones scored at 60: about six in ten. The score is a forecast, and a forecast is good when it comes true as often as it claims.

This is a definition you can check with data you already have. Nothing about it depends on how the model works inside.

Why an uncalibrated score is worse than no score

An uncalibrated model can be accurate and still dangerous. Imagine a model that scores every item at 95 and is right four times out of five. A threshold of 85 lets everything through, including the one in five it gets wrong. The number looked precise and meant nothing.

The opposite failure is quieter. A model that never scores above 70 sends every item to the operator. Nobody gets hurt, and nobody needed the robot.

A threshold only works on a number that means something. That is why we treat calibration as a requirement, not as a nice property.

How we check it

The audit log records, for every decision, the confidence and what happened next: the robot acted and nobody objected, or an operator confirmed, corrected or blocked. That log is a calibration dataset.

We group decisions by score, in bands of five or ten points, and compare each band’s score with the share of its decisions that turned out right. Plotted, this is a reliability diagram. In words: the 90 band should sit near 90, the 70 band near 70. Where a band sits below its score, the model is overconfident there. Where it sits above, it is underconfident.

Calibration is checked per site and per task. A model that is well calibrated on one plant’s items can be off on another’s, because the items are different. Recalibration is a small adjustment on top of the model, not a retraining, and it is redone as items change.

From calibration to a threshold

Once the score is calibrated, the threshold becomes a plain question: how many mistakes per hundred are you willing to let through on this task before a person looks? A threshold of 90 accepts about ten. A threshold of 98 accepts about two and sends more cases to the operator.

The right answer depends on what a mistake costs. Rejecting a good vial costs a vial. Shipping a damaged carton costs a customer. Tasks with cheap mistakes get a lower threshold. Tasks with expensive ones get a higher threshold and, often, a hard rule that blocks the action outright below a floor.

What we do not claim

Calibration drifts. New products, new lighting, a new supplier: each shifts the items the model sees. We check calibration continuously against the log rather than once at installation, and we treat a drifting band as a signal to look at the items, not only at the model.

The score is honest as long as someone keeps checking that it is.