A Calibration Benchmark for LLM Beliefs Across a Taxonomic Hierarchy by DanRKAlex

TL;DR: I’ve built a few small cal­ibra­tion bench­marks to test whether LLMs rep­re­sent un­cer­tainty well, not just whether they’re ac­cu­rate. The most re­cent one found in­con­sis­ten­cies in some LLMs’ pre­dic­tions/​guesses in ad­di­tion to and ir­re­spec­tive of their de­vi­a­tion from the ground truth. The test is some­what un­der­pow­ered (n=54×6) but I still found the re­sults in­ter­est­ing.Back­ground: I’m an economist by train­ing (MS Iowa State) who moved into data sci­ence (MS UMi...