Towards Alignment Auditing for RL Environments

Thesis: Auditing what RL environments reward is a promising and actionable direction for improving frontier-model alignment. These environments provide a concrete point of intervention: their prompts, sandboxes, and graders can be inspected and revised when they reward behavior we do not intend to teach. Embedded evaluators are a valuable first step, but auditing practices need to scale with the volume and complexity of training and draw on expertise beyond a small group of AI researchers. My fo...