Improving Audit Realism With Inference Time Compute and Deployment Scaffolds

tl;dr: We introduce critique refinement, a method to iteratively improve the realism of auditor outputs in Petri, and DISH (Deployment Imitating SWE-agent Harness), a method to audit a model within a coding agent scaffold to ensure the model receives a real system prompt, tools and scaffold injections. These techniques compose to triple realism win rate and substantially reduce unprompted verbalized eval awareness in Petri. Scaling critique refinement leads to additional improvements in realism ...