A Calibration Benchmark for LLM Beliefs Across a Taxonomic Hierarchy by DanRKAlex

·Nuno Sempere··

TL;DR: I’ve built a few small cal­ibra­tion bench­marks to test whether LLMs rep­re­sent un­cer­tainty well, not just whether they’re ac­cu­rate. The most re­cent one found in­con­sis­ten­cies in some LLMs’ pre­dic­tions/​guesses in ad­di­tion to and ir­re­spec­tive of their de­vi­a­tion from the ground truth. The test is some­what un­der­pow­ered (n=54×6) but I still found the re­sults in­ter­est­ing.Back­ground: I’m an economist by train­ing (MS Iowa State) who moved into data sci­ence (MS UMi...

Read full article →

Related Articles

Anthropic IPOs on November 19 2026?
Jack · Manifold Markets · 3d ago
Opus 5.5 Forecast: The Road to an AI Deal
Alvin Ånestrand · LessWrong · 3d ago
How long will the FAA ground stop last?
Elijah Ravitz-Campbell · Manifold Markets · 6d ago
When will the FAA Nationwide Ground Stop be lifted?
Jack · Manifold Markets · 5d ago
Will I think all AI hell broke loose by EOY 2027?
Eliezer Yudkowsky · Manifold Markets · 6d ago