Making benchmarks outputs directly useful for AI safety and security

Epistemic status: written in 30 min. This is not as polished as I’d like but I prefer to share this as is than not to share it at all.AI are becoming increasingly good at solving problems. Benchmarks are saturating fast.I think we should take advantage of this to make them solve useful problems while being evaluated. Epoch’s Open Problems are already doing this, and this is great. It would be even better if the problems solved were directly relevant for AI safety or security. We can easily task ...