When should an AI assistant push back? Measuring warnings and unsolicited opinions on decisions you've already made

This is a side project from my time in the Anthropic Fellows Program. The benchmark items, prompts and scripts are in the pushback-bench repository.I studied how much and when AI assistants push back on choices people have already made and ask the assistant just to execute on the decision. I created a set of about 500 planning decisions in which a person has already made a choice between two plans. The only request for the assistant was to write a message about the plan.Sometimes the plan was im...