The context you need, when you need it

When news breaks, you need to understand what actually matters — and what to do about it. At Vox, our mission to help you make sense of the world has never been more vital. But we can’t do it on our own.

We rely on readers like you to fund our journalism. Will you support our work and become a Vox Member today?

Soldiers can refuse to commit war crimes. Can AI?

The Pentagon wants AI models that can’t say no.

Who gets to tell the military “no”?

That question has become even more important after the Pentagon pushed AI companies to loosen limits on how their technology may be used in war — including how much human control must remain over autonomous weapons. When Anthropic said no, it lost a major government deal. OpenAI stepped in, embracing the Pentagon’s more flexible standard instead. Now, thanks to new reporting from The Intercept, we’re getting a better look at what those negotiations actually entailed.

The OpenAI-Pentagon agreement included specific language on the type of AI models the DOD would receive: “OpenAI models that are designed for national security use cases and have minimal refusal rates.” That phrase — “minimal refusal rates” — means that the Pentagon is seeking models that rarely, if ever, tell it no. (According to The Intercept, a Justice Department lawyer representing the Pentagon confirmed this language, before backtracking only a few hours later.)

On the surface, that provision might seem reasonable. The military is a strictly hierarchical institution. When your senior gives you an order, you do it. Leaders generally have more battlefield wisdom and experience than their subordinates, so the rank and file can usually assume that their commanders know what they’re doing and follow along. Plus, in the high-stress, time-sensitive environments of most military operations, there is little room for discussion, explanation, or alternative proposals.

So, if soldiers can’t refuse an order, why would a machine be allowed to? Who is a robot to tell its commander no?

There’s just one issue. Soldiers can refuse an order — when the order is illegal. In fact, they are legally required to. But by introducing technologies with a limited ability to challenge commands, the military may be making it harder to stop war crimes before they are committed.

Humans have a duty to disobey

As outlined in US domestic, military, and international law, soldiers are only obligated to follow lawful orders, and they have a duty to refuse unlawful ones.

The Department of Defense’s Law of War Manual is unambiguous: “Each member of the armed services has a duty to: (1) comply with the law of war in good faith; and (2) refuse to comply with clearly illegal orders to commit violations of the law of war.” Obeying an illegal order — as in a directive that violates the........

© Vox