How good are slop-vestigators?

TLDR:We release MessageBoardAuditBench: a benchmark to measure how well agents can replicate the recent investigation into a swarm of OpenAI agents colluding via a message board on an online wiki. We open-source the benchmark as an Inspect eval.We find that top models cover up to 51% of findings under our rubric and that model performance improves with time budget and general capability.We observe OpenAI models are less likely than other models to suggest the incident came from an internal deplo...