Constitutional AI Widens Narrow Secret Loyalty of LLMs

TL;DRRecent work has shown that it is possible to instill Secret Loyalties in models, which trigger them to output responses favoring a certain principal(ex: a politician, a corporate entity etc) whom they are loyal to. We show that Constitutional AI (CAI) can widen a model’s Narrow Loyalty (i.e. trigerring in very specific contexts) making it adaptable to conversational context. We also report that a CAI fine-tuned model’s ability to dodge Black-Box audits remains at par with the Narrow model.M...