Two: Ajeya Cotra on accidentally teaching AI models to deceive us

About this episode

Imagine you’re an orphaned eight-year-old whose parents left you a $1 trillion company, with no trusted adult to guide you. You have to hire a smart adult to run that company, guide your life the way a parent would, and administer your vast wealth. You have to hire them based on a work trial or interview that you design. You don’t get to see any resumes or do reference checks. And because you’re so rich, tonnes of people apply — for all sorts of reasons.

Ajeya Cotra argues this peculiar setup resembles the situation humanity finds itself in as we train very general and very capable AI models using current deep learning methods. Ajeya was a senior research analyst at Coefficient Giving at the time of this interview, and she now works at METR (Model Evaluation & Threat Research). 

As she explains, this eight-year-old faces a challenging problem. In the candidate pool there are likely some truly nice people, who sincerely want to help and make decisions that are in your interest. But there are probably other characters too — like people who will pretend to care while you’re monitoring them, but intend to exploit the job to enrich themselves as soon as they think they can get away with it.

Like a child trying to judge adults, at some point humans will need to judge the trustworthiness and reliability of machine learning models that are as goal-oriented as people, and greatly outclass them in knowledge, experience, breadth, and speed. Tricky!

Can’t we rely on models' performance during training tasks to guide us? Ajeya worries this won’t work. The trouble is that three different sorts of models will all produce the same output during training, but could behave very differently once deployed in a setting that allows their true colours to come through. She describes three such motivational archetypes:

• Saints — models that care about doing what we really want
• Sycophants — models that just want us to say they’ve done a good job, even if they get that praise by taking actions they know we wouldn’t want them to
• Schemers — models that don’t care about us or our interests at all, who are just pleasing us so long as that serves their own agenda

In principle, a machine learning training process based on reinforcement learning could spit out any of these three attitudes, because all three would perform roughly equally well on the tests we give them, and ‘performs well on tests’ is how these models are selected.

But while that’s true in principle, maybe it’s not something that could plausibly happen in the real world. After all, if we train an agent based on positive reinforcement for accomplishing X, shouldn’t the training process produce a model that just does X and doesn’t have complex thoughts and goals beyond that?

According to Ajeya, this