Measuring agent capabilities on open-ended AI R&D tasks

TL;DR. We measure agent capabilities on four long and open-ended research tasks. Frontier models have been improving with an overall doubling time of around one year since 2024 (based on an exponential fit). Agents perform better at engineering-heavy tasks compared to more conceptual tasks (between 7 and 16.8 months median doubling time), and model capabilities are improving slower on the latter.Epistemic status: This analysis is based on a small set of tasks and we rely heavily on LLM-grading (...