When it says 80%, is it right 80% of the time?

The testThe central claim: the probabilities are calibrated. We need real labels to check it, so this uses the public UCI SMS Spam Collection (5,574 text messages with human labels). The demo scores a 500-message sample spread across the file (~12 s, under a cent); the full run is one flag away and gave the same picture. One yes/no question per message, "is this spam?", with no examples and no training.

How to read itThe reliability chart groups messages by the probability the model gave and plots the actual spam rate in each group. Perfect calibration sits on the dashed diagonal; a dot below it means the model was more sure than it should have been. Dot size is how many messages are in the bin. ECE is the average gap weighted by bin size.

What to watchExtremes vs the middle: most messages get a near-0 or near-1 answer, and those are honest. The grey zone is where the gaps live. The threshold sweep is what you'd actually use to pick a cut-off. With 500 messages the middle bins hold only a handful each, so read their gaps loosely. The "mistakes" at the bottom include a few that are the dataset's own labeling errors.