Paperclips, broad- and narrow-scope goals, and the over-verification problem

Published on June 28, 2026 12:38 PM GMTIn Bostrom’s famous example, an artificial superintelligence (ASI) instructed to maximise paperclip production converts the entire accessible universe to paperclips. It might seem, Bostrom notes, that we could avoid catastrophe by telling the ASI to produce exactly one million paperclips. Unfortunately this could lead to an insatiable demand for resources, since the ASI would have an incentive to go on checking and re-checking that it had succeeded. ‘Since ...