Enough is Enough: Measuring Diminishing returns to benchmark size with Item Response Theory

This is a speedrun sort of project completed at the start of the Pivotal Fellowship with inspiration and mentorship from SecureBio. It will probably be largely unrelated to what I'll be working on with them for the rest of the summer, so I thought I'd throw it here now. The writeup is messy, and the appendix was written with much help from Claude, although I stepped through it to make sure I thought all the math was correct. For a little background on IRT and how it gives a notion of the "Fisher...