Main Facts: The Illusion of Certainty in Large Language Models When Horace He and a team of researchers at the Thinking Machines Lab initiated a straightforward computational experiment in September 2025, they were testing a foundational assumption of modern artificial intelligence. They sent the prompt "Tell me about Richard Feynman" to the massive Qwen3-235B model […]
A Deep Dive into Calibration of Language Models: Platt Scaling, Isotonic Regression, Temperature Scaling
# Introduction A model that says it is 90% confident should be right 90% of the time. When that relationship breaks down, you get a miscalibration problem. The model’s scores stop telling you anything useful about reliability. For large language models (LLMs), miscalibration is widespread. A 2024 NAACL survey found that confidence scores diverge […]
A Deep Dive into Calibration of Language Models: Platt Scaling, Isotonic Regression, Temperature Scaling
# Introduction A model that says it is 90% confident should be right 90% of the time. When that relationship breaks down, you get a miscalibration problem. The model’s scores stop telling you anything useful about reliability. For large language models (LLMs), miscalibration is widespread. A 2024 NAACL survey found that confidence scores diverge […]