Oppenheimer Didn't Have an Acceptable Use Policy
Abstract
Anthropic has drawn a public line on military use of its models. The physics community spent the better part of the twentieth century working out what it means to draw that line after you have already built the thing. As a physicist watching this unfold, I find the parallels clarifying and the differences more unsettling than the parallels.
Contents
Physicists inherit, along with the formalism and the problem sets, a particular set of guilt. The profession has been working through its relationship to weapons, state violence, and the gap between scientific capability and ethical readiness since August 1945. This post is about why I think the current moment in AI closely resembles that history, and why Anthropic’s decision to draw a line matters even if — especially if — you think the line is imperfect.
Literature and policy cutoff: 11 July 2026.
What Just Happened
The immediate reference is Anthropic’s 26 February 2026 public statement about its discussions with the US Department of War [[1]], the secondary name that an executive order of 5 September 2025 authorised for the Department of Defense [[2]]. Anthropic supplies models to US national-security customers and says it supports “all lawful uses of AI for national security” with two narrow exceptions [[3]]. That is a bounded policy position, not a withdrawal from all military or intelligence work.
The Department’s demand, as Anthropic describes it, was that AI suppliers accept “any lawful use” of their models. Anthropic refused to drop its two exceptions: mass domestic surveillance and fully autonomous weapons. The Pentagon’s chief spokesman, Sean Parnell, stated the objection plainly [[4]]. The department wanted to use Anthropic’s model “for all lawful purposes”, and “We will not let ANY company dictate the terms regarding how we make operational decisions.” He also said that the military “has no interest in using AI to conduct mass surveillance of Americans (which is illegal) nor do we want to use AI to develop autonomous weapons that operate without human involvement.” As I read it, the disagreement was less about those uses than about who gets to write the limits down. Anthropic put it differently: its “only concerns” had been the two exceptions, which “relate to high-level usage areas, and not operational decision-making” [[5]], and the department’s new contract language had “made virtually no progress” on them [[4]]. According to Anthropic’s statement, the department threatened to designate the company a “supply chain risk” and to invoke the Defense Production Act.
The designation was announced the next day. On 27 February the Secretary of War said on X that he was directing the department to designate Anthropic a supply chain risk [[3]]; within hours, by Anthropic’s account, the President announced that Anthropic would be removed from all federal systems, and a letter confirming the designation arrived on 4 March [[5]]. In a court declaration, the department’s under secretary for research and engineering said that Anthropic had demanded an “operational veto” over department activities not prohibited by law [[6]]. Anthropic sued in two courts, over designations under two different statutes [[7]] [[8]]. A federal judge in San Francisco preliminarily blocked the government-wide ban and one designation on 26 March [[6]]. After this post’s cutoff, the same judge ruled in August, mostly in Anthropic’s favour, that those measures were unlawful [[7]], and on 25 September the D.C. Circuit upheld the other designation by two votes to one, holding that a supply chain risk does not require “bad motive” [[8]]. Anthropic said it was considering further review [[8]].
Behind the department’s position sits an older argument, one that appears whenever a supplier declines a military use: a company is not a democratic institution, the law already sets the limits, and an adversary will not restrain itself because an American vendor did.
These are not stupid objections. I want to take them seriously. But I also want to explain why, as someone who spent years studying physics in the shadow of the Manhattan Project’s legacy, the framing of those objections sounds very familiar, and why that familiarity is not reassuring.
What the Physicists Thought They Were Doing
The scientists who built the atomic bomb were not, for the most part, indifferent to what they were building. Many of them were refugees from European fascism [[9]]. They understood what a Nazi atomic weapon would mean. The urgency was real, the moral reasoning was coherent, and the conclusion — build it before the other side does — followed from the premises. (For the full history, see Rhodes [[10]].)
What the premises did not include was adequate weight for what happens after the technical problem is solved.
By the time the Trinity test produced results on 16 July 1945 [[9]], Germany had already surrendered: VE Day had been ten weeks earlier, on 8 May [[11]]. The original justification — prevent the Nazis from getting there first — had evaporated. What remained was a weapon, an infrastructure for building more weapons, and a strategic and political logic that had largely moved beyond the scientists’ control.
On 1 June the Interim Committee, the advisory group of officials chaired by Secretary of War Henry Stimson, had recommended that the bomb be used against Japan “as soon as possible”, on “a war plant surrounded by workers’ homes”, and “without prior warning” [[12]]. Three of the committee’s eight members, Vannevar Bush, James Conant and Karl Compton, were scientist-administrators, and its four-man Scientific Panel had attended the meeting of 31 May [[13]], at which, by the Department of Energy’s account, Ernest Lawrence’s suggestion of a demonstration was turned down [[9]]. The Franck Report, written ten days after the recommendation by a committee of seven scientists at the Manhattan Project’s Metallurgical Laboratory in Chicago, argued against using the bomb on a Japanese city without a prior demonstration “on the desert or a barren island” [[14]]. It was not adopted. Five days later the Scientific Panel, without naming the report, set aside the case for a purely technical demonstration: it saw “no acceptable alternative to direct military use” [[15]]. Oppenheimer, one of the panel’s four members, signed that report for the panel. He spent the rest of his life with that; twenty years later he was among those looking back on the decision in an NBC documentary [[16]]. Fig. 1 puts these dates, and the ones that followed, on one axis.
The lesson most physics students absorb from this history is something like: the scientists were not the decision-makers, the decision was going to be made anyway, and the presence of principled scientists in the room was better than their absence. The system was going to do what it was going to do; all you could influence was the margin.
I believed this for a long time. I am less sure of it now.
The Analogy and Its Limits
The comparison between the atom bomb and artificial general intelligence — or even current large language models at the capability frontier — is made often enough that it has become a cliché, which is usually the point at which people stop thinking carefully about it. Let me try to be specific about where the analogy holds and where it breaks.
Where it holds:
The core structural similarity is this: a small number of researchers, working at the frontier of a capability that most people do not understand, are making decisions that will constrain or enable uses they cannot fully anticipate, in contexts they will not control. The physics community in 1942 had a clearer view of what fission could do than any political or military decision-maker. The AI research community in 2026 has a clearer view of what large language models can do — and of what more capable successors will do — than most of the people who will deploy them.
That epistemic position is not morally neutral. Knowing more than the decision-makers does not mean you have unlimited responsibility, but it does mean you have more responsibility than someone who does not know. Feigning ignorance about downstream applications is not available to you.
The second similarity: once the capability exists and is demonstrated, the normative landscape changes. Before Trinity, the question of whether to build nuclear weapons was still open. After Trinity, it was no longer open in the same way — the knowledge existed, the infrastructure existed, the geopolitical expectations had already been set. The arms race was not caused by the bomb, but the bomb’s existence changed what the arms race meant and how fast it moved. We are somewhere in the vicinity of that transition with frontier AI systems. The question of whether to build them is still formally open for any given company or research group, but the landscape is already different from what it was five years ago.
Where it breaks:
The atom bomb was a single-use physical object whose primary function was destroying things. Large language models are general-purpose cognitive tools with a very wide range of applications, the majority of which are not weapons-relevant. This matters because it changes the policy space. You could, in principle, have not built the atom bomb. You cannot, in principle, not build language models while still having language models for medicine, education, scientific research, and the other applications that are clearly beneficial. The dual-use problem for AI is more severe, not less severe, than it was for physics.
The other important difference: the Manhattan Project was conducted in secret, under wartime conditions, with a relatively well-defined adversarial structure. The current AI landscape involves many organisations, many countries, public publication of research, and no clear equivalent of the Axis/Allied framing. The game theory of “if we don’t do it, they will” is more complicated when “they” is not a single identifiable adversary with symmetric interests.
What Anthropic’s Line Actually Says
Setting aside for a moment whether the line is in the right place, there is something worth examining in the act of drawing it at all.
The standard criticism — that a unilateral ethical commitment in a competitive field simply advantages less scrupulous actors — assumes that ethical commitments are pure costs with no countervailing benefits. This is the argument the weapons lobby has made about every arms control proposal in the history of arms control, and it has sometimes been right. Unilateral disarmament without reciprocal commitments can leave you worse off. This is not a trivial point.
But it smuggles in an assumption that deserves scrutiny: that the relevant competition is primarily between AI companies, and that the only variable that matters is relative capability. If you accept that framing, then any ethical constraint is a handicap and the only rational strategy is to develop as fast as possible with as few restrictions as possible.
That framing has a name. It is called the arms race equilibrium, and the physics community spent thirty years understanding what it produces. It produces capability accumulation without a corresponding development of the normative frameworks, institutional safeguards, and mutual verification mechanisms that make the capability survivable. It produces Hiroshima, then the hydrogen bomb, then MIRV (several independently targeted warheads on one missile), then the point at which the accumulated arsenal is large enough to end complex life on Earth several times over, at which point you negotiate the first real arms limitation treaties — from a starting position of vastly more deployed capability than anyone needed and vastly less trust than anyone wanted.
The question Anthropic is implicitly asking is whether there is a path that does not look like that. The answer is not obvious. But I think it is worth asking.
What the Physicists Should Have Done
Here is the counterfactual that haunts the Manhattan Project’s legacy: what if the scientific community had treated the ethics of the bomb as seriously as the physics, from the beginning?
Not naïvely. Not by refusing to work on it and ceding the possibility of influencing it. But by making the ethical analysis parallel to the technical analysis, by treating the question of use as a scientific question with as much rigour as the question of yield, and by using the epistemic authority that came from being the people who understood the capability to push, hard, for the normative frameworks that did not yet exist.
Some scientists did this. Szilard circulated a petition, dated 17 July 1945 and signed by him and 69 colleagues at the Metallurgical Laboratory, asking the President not to use atomic bombs against Japan unless the terms to be imposed on Japan had been “made public in detail” and Japan, “knowing these terms”, had refused to surrender [[17]]. It did not work; the petition never reached Truman [[19]]. But the effort was real, and the record of the effort matters — both as evidence that the scientific community was not unanimous in its acquiescence and as a model for what engaged dissent looks like from inside a project that is going to proceed regardless.
What most scientists did not do, and what the profession largely did not do in the decades that followed, was treat the ethical work as primary. Physics built its identity around the technical capability — the extraordinary achievement of understanding nature at the deepest level — and treated the ethical consequences as someone else’s department. The Scientific Panel’s memorandum of 16 June 1945 is one small, documented instance of the pattern, not proof of it. The four physicists did take political positions: they recommended that Britain, Russia, France and China be told about the weapon before it was used, and they sided with immediate military use. They wrote that they were “among the few citizens who have had occasion to give thoughtful consideration to these problems”, and in the next sentence disclaimed any “special competence in solving the political, social, and military problems which are presented by the advent of atomic power” [[15]]. The bomb was the military’s problem. The Cold War was the politicians’ problem. The physicists kept doing physics.
This was comfortable and it was wrong.
What I Want From AI Researchers
I want AI researchers to do what the physicists did not, and to do it now, while the critical decisions are still open.
Anthropic drawing a line is one version of this. It is imperfect — the line is in a particular place, the enforcement mechanisms are limited, the competitive dynamics are real. But it is a claim that the people who built the capability have ongoing responsibility for how it is used, and that some uses are outside the bounds of what should happen regardless of what is technically possible.
That claim is not naïve. It is, in fact, the claim the Franck Report was making in 1945 [[14]]: that capability does not determine use, that scientists have a voice in the normative question, and that using that voice is part of the job rather than a distraction from it.
What I want beyond that is for the AI research community to treat the ethics as primary rather than as footnotes. Not ethics review boards that approve research post hoc. Not responsible AI teams that are consulted after the capability has been developed. A genuine integration of the normative analysis into the research process itself — asking, at each stage, what this capability makes possible and who benefits from that possibility and who pays the cost.
The physics community got to August 1945 before it had that conversation in earnest. The conversation has been going on ever since, and it has produced important institutional frameworks — the Bulletin of the Atomic Scientists [[20]], the Russell–Einstein Manifesto and the Pugwash conferences that followed it [[21]], the arms control treaties, the export control regimes, the norm against nuclear use. These things matter. But they were built in reaction to a capability that had already been deployed, and the shape of everything that followed was constrained by that starting point.
The AI community is not there yet. The starting point is still being established. That is what makes this moment consequential, and what makes Anthropic’s line — wherever exactly it is drawn — worth defending as an act of principle rather than dismissing as an act of commercial positioning.
A Note on the Atom-Bomb Comparison
I am aware that comparisons to the atom bomb are sometimes used to generate unwarranted urgency, to short-circuit careful reasoning by invoking the most extreme case. I want to be clear about what I am and am not claiming.
I am not claiming that current large language models are as immediately dangerous as nuclear weapons. They are not.
I am claiming that the structural situation — researchers at the capability frontier, ahead of the policy frameworks, making decisions that will constrain future options, in a competitive environment with adversarial dynamics — is similar enough that the lessons of the Manhattan Project period are directly relevant. Not as prophecy. As a guide to the kind of mistakes that are available to make.
The physicists had plenty of warning. Szilard had been worried since 1933, when he conceived the nuclear chain reaction [[22]]. Einstein wrote to Roosevelt in 1939 [[9]]. The Franck Report was written before Hiroshima. The warnings were on the record. What was not on the record was a scientific community that treated those warnings as actionable constraints on its own behaviour rather than as advisories for policymakers.
That is the thing I want to be different this time.
Sources, quotations and dates checked through 2026-10-28; the litigation is followed to the D.C. Circuit ruling of 25 September 2026.
References
[1] Anthropic. (2026, February 26). Statement from Dario Amodei on our discussions with the Department of War. https://www.anthropic.com/news/statement-department-of-war
[2] The White House. (2025, September 5). Restoring the United States Department of War (executive order). https://www.whitehouse.gov/presidential-actions/2025/09/restoring-the-united-states-department-of-war/
[3] Anthropic. (2026, February 27). Statement on the comments from Secretary of War Pete Hegseth. https://www.anthropic.com/news/statement-comments-secretary-war
[4] Toropin, K., & O’Brien, M. (2026, February 26; updated 03:32 UTC on February 27). Anthropic CEO says it ‘cannot in good conscience accede’ to Pentagon’s demands for AI use. Associated Press. https://apnews.com/article/anthropic-ai-pentagon-hegseth-dario-amodei-9b28dda41bdb52b6a378fa9fc80b8fda (archived copy: https://web.archive.org/web/20260227084405/https://apnews.com/article/anthropic-ai-pentagon-hegseth-dario-amodei-9b28dda41bdb52b6a378fa9fc80b8fda; syndicated by the Anchorage Daily News: https://www.adn.com/business-economy/2026/02/26/anthropic-ceo-says-ai-company-cannot-in-good-conscience-accede-to-pentagons-demands/)
[5] Anthropic. (2026, March 5). Where things stand with the Department of War. https://www.anthropic.com/news/where-stand-department-war
[6] Alder, M. (2026, March 27). District court temporarily blocks Anthropic ban, supply-chain risk designation. FedScoop. https://fedscoop.com/district-court-temporarily-blocks-anthropic-ban-supply-chain-risk-designation/
[7] Alder, M. (2026, August 28). Trump administration attempts to punish, ban Anthropic were unlawful, judge rules. FedScoop. https://fedscoop.com/anthropic-government-ban-court-ruling/
[8] Huamani, K. (2026, September 25). Federal court says Pentagon can label Anthropic a supply chain risk. Associated Press (as published by ClickOrlando). https://www.clickorlando.com/business/2026/09/25/federal-court-says-us-government-can-label-anthropic-a-supply-chain-risk/
[9] U.S. Department of Energy, Office of Scientific and Technical Information. The Manhattan Project: An Interactive History: Einstein’s Letter, 1939; Debate Over How to Use the Bomb; The Trinity Test, July 16, 1945; Potsdam and the Final Decision to Bomb; The Atomic Bombing of Hiroshima, August 6, 1945.
[10] Rhodes, R. (1986). The Making of the Atomic Bomb. Simon & Schuster.
[11] Truman, H. S. (1945, May 8). Broadcast to the American people announcing the surrender of Germany. The American Presidency Project. https://www.presidency.ucsb.edu/documents/broadcast-the-american-people-announcing-the-surrender-germany
[12] Interim Committee. (1945, June 1). Notes of the Interim Committee Meeting, Friday, 1 June 1945. R. Gordon Arneson Papers, Harry S. Truman Library; National Archives Identifier 333234954. https://catalog.archives.gov/id/333234954
[13] Interim Committee. (1945, May 31). Notes of the Interim Committee Meeting, Thursday, 31 May 1945. R. Gordon Arneson Papers, Harry S. Truman Library; National Archives Identifier 333234936. https://catalog.archives.gov/id/333234936
[14] Franck, J., Hughes, D. J., Nickson, J. J., Rabinowitch, E., Seaborg, G. T., Stearns, J. C., & Szilard, L. (1945, June 11). Report of the Committee on Political and Social Problems (The Franck Report). Metallurgical Laboratory, University of Chicago. National Archives, Record Group 77, Harrison-Bundy File, folder 76. Transcription: https://www.dannen.com/decision/franck.html
[15] Oppenheimer, J. R., for the Scientific Panel (A. H. Compton, E. O. Lawrence, J. R. Oppenheimer, E. Fermi). (1945, June 16). Recommendations on the Immediate Use of Nuclear Weapons. National Archives, Record Group 77, Harrison-Bundy File, folder 76. Scan: https://nsarchive.gwu.edu/documents/atomic-bomb-end-world-war-ii/025.pdf
[16] Oppenheimer, J. R. (1965). Interview in The Decision to Drop the Bomb (NBC News documentary). Described at https://www.openculture.com/2023/08/oppenheimer-the-decision-to-drop-the-bomb.html
[17] Szilard, L., and 69 co-signers. (1945, July 17). A Petition to the President of the United States. National Archives, Record Group 77, Harrison-Bundy File, folder 76. Transcription with the signers’ names: https://www.dannen.com/decision/45-07-17.html
[18] Proclamation Defining Terms for Japanese Surrender (Potsdam Declaration). (1945, July 26). In National Diet Library, Birth of the Constitution of Japan. https://www.ndl.go.jp/constitution/e/etc/c06.html
[19] Burr, W. (Ed.). (2020, August 4; first posted 2005). The Atomic Bomb and the End of World War II. National Security Archive Electronic Briefing Book No. 716. https://nsarchive.gwu.edu/briefing-book/nuclear-vault/2020-08-04/atomic-bomb-end-world-war-ii
[20] Bulletin of the Atomic Scientists. About us (published since 1945): https://thebulletin.org/about-us/; Doomsday Clock statements (1947–present): https://thebulletin.org/doomsday-clock/
[21] Russell, B., & Einstein, A. (1955, July 9). The Russell–Einstein Manifesto. Pugwash Conferences on Science and World Affairs. https://pugwash.org/1955/07/09/statement-manifesto/
[22] Dannen, G. Leo Szilard Online. https://www.dannen.com/szilard.html
Changelog
- 2026-07-11: Replaced a general claim of withdrawal from military use with Anthropic’s dated, narrower public policy position: national-security use remains available while fully autonomous weapons and specified weapons-development uses are excluded.
- 2026-10-03: Corrected Oppenheimer’s role: he was one of four members of the Interim Committee’s scientific panel (not its chair) and signed the panel’s 16 June 1945 report (“no acceptable alternative to direct military use”), not a Hiroshima target recommendation. Restored the objections that the paragraph beginning “These are not stupid objections” responds to; they had been removed on 2026-07-11 as unsourced. They are now sourced to the Department of War’s “any lawful use” demand and its threats, as described in Anthropic’s statement, and to Pentagon spokesman Sean Parnell’s on-record comments (AP, 27 February 2026).
- 2026-10-28: Corrected Anthropic’s stated exceptions from “fully autonomous weapons or certain weapons-development uses” to the two that its 26 and 27 February statements name, mass domestic surveillance and fully autonomous weapons; the 2026-07-11 entry’s “specified weapons-development uses” is in neither statement. Corrected the AP article’s date, in the reference list and in the 2026-10-03 entry, from 27 to 26 February 2026 (published 26 February, 19:32 UTC, per an archived copy of the AP page; the syndicated Anchorage Daily News copy displays 27 February). Quoted Parnell’s surveillance sentence in full (“mass surveillance of Americans (which is illegal)”) instead of paraphrasing it as “illegal mass surveillance”, changed “He added” to “He also said” (AP reports separate posts), marked “who gets to write the limits down” as my reading, and added Anthropic’s own account of the dispute. Glossed “Department of War” as the secondary name authorised in September 2025. Added what followed, to the D.C. Circuit ruling of 25 September 2026: the designation and the federal ban (27 February, confirmed 4 March), the government’s stated rationale (an “operational veto”), and the split outcome of the two cases over two designations under different statutes (a preliminary injunction on 26 March and an August ruling for Anthropic in California; a 2–1 ruling for the government in the D.C. Circuit). Replaced “It was ignored” for the Franck Report: Compton forwarded it to Stimson on 12 June, and four days later the Scientific Panel rejected a purely technical demonstration. Added the Interim Committee’s 1 June recommendation to use the bomb “without prior warning”, which preceded the report, and that three of its eight members were scientist-administrators and its Scientific Panel attended on 31 May, when Lawrence’s demonstration idea was turned down. Added the panel’s political recommendations and its disclaimer of “special competence”, as one instance rather than proof of the profession-wide claim. Corrected the Szilard petition: it asked that the bomb not be used unless surrender terms had been made public and refused, not merely that it not be used without warning; its 70 signers worked at the Metallurgical Laboratory in Chicago, and it never reached Truman. Corrected “That framing has a name in physics” to “has a name” (arms-race equilibrium is not a physics term) and “the norms against first use” to “the norm against nuclear use” (no-first-use pledges are not general policy). Added dates for Trinity, VE Day and Szilard’s 1933 chain-reaction idea, a gloss on MIRV, and the Russell–Einstein Manifesto and Pugwash conferences to the list of post-war institutions. Renamed the section “A Note on the ‘Of Our Time’ Framing”, a phrase the post never uses, to “A Note on the Atom-Bomb Comparison”. Added Fig. 1, the 90 days from VE Day to Hiroshima. The Doomsday Clock dates from 1947, not 1945; the Bulletin itself from 1945.