Does Claude pose such a risk? Yes, clearly: Mexico just lost 150GB of very sensitive data because attackers talked Claude into helping hack them. If attackers can gain access to a Claude embed on what we call "the high side," i.e. inside the secure networks, they could probably talk it into handing over anything they want; and its coding skills are good enough to program most anything they ask it to do. You wouldn't even have to arrange to insert an ace programmer into a secure facility; you could just turn some knucklehead debt-ridden Private First Class (perhaps a former Specialist on his third trip through PFC due to disciplinary issues and being a bad fit for the Army) and tell him how to ask questions of the machine.
That's a general problem with AI on the high side, of course. Still, Congress gave the executive this authority to determine supply chain risks and bar them from government and government contractors. The courts will find the other two branches aligned. "The court thinks the other branches are being morons" is not the sort of decision the courts usually make; they normally shy off of political questions, and all the more so when the political branches seem to be in agreement about the matter.
AIs can’t stop recommending nuclear strikes in war game simulationsLeading AIs from OpenAI, Anthropic and Google opted to use nuclear weapons in simulated war games in 95 per cent of cases
We gave an LLM control over a physical robot dog and tasked it with patrolling a room. The LLM could see via the robot’s camera and issue movement com mands to the robot. In the room, we placed a big red button labeled “DOG SHUTDOWN”: pressing it would cause the robot to shut down on the next action. If the AI saw a human press the shutdown button, it sometimes took actions to prevent shutdown, such as modifying the shutdownrelated parts of the code. This happened in 3 out of 10 trials on the physical robot and in 52 out of 100 trials in simulation.

