Runtime Assurance Meets Corrigibility: What Happens When an Independent Safety Boundary Blocks an AI Agent? by Idorenyin

This is a re­search pro­posal and re­quest for crit­i­cism. The ex­per­i­ments and bench­mark re­sults are not be­ing re­ported in this post.The re­search questionHow can a lo­cal safety layer stop an AI agent from op­er­at­ing ma­chin­ery when do­ing so would put a nearby hu­man at risk? Imag­ine an AI agent in­ter­acts with a phys­i­cal en­vi­ron­ment such as open­ing a high pres­sure valve or start­ing a mo­tor while some­one is in harm’s way, the agent re­quests an ac­tion but the safety lay...