Published on April 2, 2025 3:59 AM GMT
A key step in the classic argument for AI doom is instrumental convergence: the idea that agents with many different goals will end up pursuing the same few subgoals, which includes things like “gain as much power as possible”.
If it wasn’t for instrumental convergence, you might think that only AIs with very specific goals would try to take over the world. But instrumental convergence says it’s the other way around: only AIs with very specific goals will refrain from taking over the world.
For idealised pure consequentialists — agents that have an outcome they want to bring about, and do whatever they think will cause it — some version of instrumental convergence seems surely true[1].
But what if we get AIs that aren’t pure consequentialists, for example because they’re ultimately motivated by virtues? Do we still have to worry that unless such AIs are motivated by certain very specific virtues, they will want to take over the world?
I’ll add some more detail to my picture of a virtue-driven AI. Such an AI could still be a competent agent that often chooses actions based on the outcomes they bring about. It’s just that that happens as an inner loop in service of an outer loop which is trying to embody certain virtues. For example, maybe the AI tries to embody the virtue of being a good friend, and in order to do so it sometimes has to organise a birthday party, which requires choosing actions in the manner of a consequentialist.
There’s no reason that the ‘virtues’ being embodied have to be things we would consider virtuous. All I’m interested in is the nature of being an agent that tries to embody certain traits rather than bring about certain outcomes.
(I’m interested in this question largely because I’m less and less convinced that we should expect to see AIs that are close to pure consequentialists. Arguments for or against that are beyond the intended scope of the question, but still welcome.)
-
Although I can think of some scenarios where a pure consequentialist wouldn’t want to gain as much power as possible, regardless of their goals. For example, a pure consequentialist who is a passenger on a plane probably doesn’t want to take over the controls (assuming they don’t know how to fly), even if they’d be best served by flying somewhere other than where the pilot is taking them. ↩︎
Discuss