Frontier models may not be reliable auditors

SummaryFrontier model providers are employing their models internally in order to iteratively make future models safer. For instance, Anthropic uses automated audits to identify misalignment concerns prior to releasing a model. More effective auditors, holding their own alignment fixed, should necessarily produce more informative audits. But how effective are these auditors? We can test their various safety-related capabilities on known misalignment modes, but we are uncertain of the total space...