The Probability of an Event under a Simplicity Prior

In this short post, I talk about a difficulty we encountered while working on a model of the fragility of value. You can read my post about that work here.The TL;DR is that the probability of a fixed event can be made arbitrarily large or small depending on the choice of simplicity prior. I'm sure this result was already known, but I found it unintuitive and thought it might be helpful to share.BackgroundIn our model of the alignment problem, AI agents go through idealized alignment training bef...