Aim at agency: institutional design under preference uncertainty, and why alignment needs it by act65

The argumentEvery an­swer any­one has given to “what should so­ciety max­imise” — sal­va­tion of souls, the work­ers’ par­adise, ag­gre­gate hap­piness, co­her­ent ex­trap­o­lated vo­li­tion — was pro­posed by some­one whose own prefer­ences the an­swer con­spicu­ously served. Each has aged badly.That isn’t a claim that no ethics is cor­rect. It’s that bet­ting a durable in­sti­tu­tion on hav­ing found the cor­rect one is a bad bet, and it has been los­ing for a long time. The main rea­son is ca...