Ensemble monitoring for AI control: diverse signals outweigh more compute

tl;drWhat improves LLM monitoring besides leveraging more compute? Leveraging more diverse compute.In the AI control setting of Greenblatt et al. (2024), we build 12 monitors via various prompting and fine-tuning approaches; we then study different ways to compose them into ensembles. We evaluate our ensembles on the APPS coding dataset, extended with model-written backdoors, and on BigCodeBench-Sabotage (BCB-S), an out-of-distribution dataset. Our best 3-monitor ensemble achieves 2.4× greater d...