How to give embedded evaluators the support they need: Lessons from the finance industry

Epistemic status: Extrapolating from my experience as a model validator at a bank testing machine learning models, a role that I see as similar to internal risk teams at AI labs, and my interactions with US Federal Reserve Bank regulators, who perform a function that has a lot in common with the proposed embedded evaluators. All views expressed here are my own.Summary: Dario Amodei’s We Must Pace the Frontier proposes a three-step plan to pace AI progress: (1) invite independent review (embedded...