Toward a Theory of Evaluability

Apple ML Research··

Machine learning models, particularly large foundation models, are increasingly evaluated along multiple axes that are relevant for their usage. In addition to accuracy measures appropriate for the specific task of interest, measures capturing safety, calibration and fairness properties are often of interest. The classical approach to these problems assumes that the test set has been held out from the training process, an assumption that is increasingly harder to justify for large foundation mod...

Read full article →

Related Articles

OpenAI’s o1 correctly diagnosed 67% of ER patients vs. 50-55% by triage doctors
donsupreme · Hacker News · 6mo ago
Accelerating Gemma 4: faster inference with multi-token prediction drafters
amrrs · Hacker News · 6mo ago
A couple million lines of Haskell: Production engineering at Mercury
unignorant · Hacker News · 6mo ago
Using “underdrawings” for accurate text and numbers
samcollins · Hacker News · 6mo ago
ProgramBench: Can language models rebuild programs from scratch?
jonbaer · Hacker News · 6mo ago