Some “coherence arguments” are two different things in a trenchcoat

The “two kids in a trenchcoat” trope, in case you haven’t heard of it. Image sources: 1, 2tl;drI go through three “coherence arguments”, related to (1) market competition, (2) AI self-reflection and self-modification, and (3) AI training algorithms. I argue that each of these implies a pressure for (something like) “being self-consistent”, and also a pressure for (something like) “only caring about long-term eventual outcomes, with ruthless indifference to anything else”. But the latter pressure...