Paperclips Can't Fight Back (Why Many Ultimate Goals Should Lead to Pervasive and Diverse Intelligence)

This post is crossposted from my Substack, Structure and Guarantees, where I explore how formal verification and related ideas might scale to more complex intelligent systems. Here I argue that instrumental convergence, usually presented as an alignment gotcha, may have a surprising upside: across many competently pursued ultimate goals, it may push toward spreading intelligence throughout the universe while preserving individuality.There’s a decently common thread in thinking about the future o...