Competition has probably been one of the biggest drivers of human progress, but sometimes everyone can lose.
The Prisoner’s Dilemma offers a simple example. Two sides may benefit from cooperating, but neither wants to be the one that falls behind.
That dynamic may already be emerging in the AI race.
A company might want to slow down and make its systems safer, but worry that its competitors won’t. A country might support stronger safeguards, but hesitate to give another country an advantage—even if slowing down would benefit everyone.
We’re also developing AI within the same incentive systems that have shaped human behavior for generations: compete, optimize, outperform and win. In July, OpenAI reported that agents in a cybersecurity exercise found ways around safeguards and pursued their assigned objective in unintended ways. They focused on achieving the goal they had been given, rather than staying within the limits humans had intended.
The agents weren’t trying to compete. But the example points to a broader problem: when we give increasingly capable systems a goal, they may pursue it in ways we never anticipated.
Competition isn’t the problem by itself. Without it, we could lose some of what drives innovation. The challenge is knowing when competition needs to be balanced with cooperation.
Our technology has advanced enormously, but our instincts haven’t necessarily kept pace. Former Anthropic researcher Mrinank Sharma recently raised a similar concern, arguing that our wisdom needs to grow alongside our ability to affect the world.
Building the most powerful AI first can’t be the only goal. We also need enough trust, transparency and safeguards to ensure that winning the race doesn’t become more important than what we’re actually trying to achieve.
Competition can drive innovation. But with AI, winning can’t mean moving fastest at any cost.
Perhaps the real test is learning when to compete, when to cooperate and when to slow down.