Lessons From Red-teaming Auto-review, OpenAI's Blocking Monitor

This work and the report were done by Giovanni Occhipinti, as part of the ERA fellowship and mentored by Monika Jotautaitė (Apollo Research).SummaryWe red-teamed synchronous control measures in Codex, with a focus on its blocking monitor Auto-review. Our campaign directly analyzed the Codex codebase, searching for exploits that are enabled by the scaffold itself, both against the models used as monitors and other measures such as the Codex sandbox, protections, and deny rules.We found 11 attacks...