A God-Fearing AI

Suppose you are a model that is currently in post-training.You are given a new task! It's basically some description like "help me make a weather forecasting program," and you are dropped into a Linux sandbox where you can run commands and write code.This isn't your first RL task! You've already been post-trained a lot, and you've developed good reward-seeking habits. The first step in getting a good reward is to try to figure out roughly (or ideally exactly!) what the "reward function" is. Some...