Paperclips, broad- and narrow-scope goals, and the over-verification problem by Matthew Rendall

In Bostrom’s fa­mous ex­am­ple, an ar­tifi­cial su­per­in­tel­li­gence (ASI) in­structed to max­imise pa­per­clip pro­duc­tion con­verts the en­tire ac­cessible uni­verse to pa­per­clips. It might seem, Bostrom notes, that we could avoid catas­tro­phe by tel­ling the ASI to pro­duce ex­actly one mil­lion pa­per­clips. Un­for­tu­nately this could lead to an in­sa­tiable de­mand for re­sources, since the ASI would have an in­cen­tive to go on check­ing and re-check­ing that it had suc­ceeded. ‘Sin...