Many of us use artificial intelligence every working day. It reads images, drafts clinic notes, answers patient messages, summarizes the literature, and helps write our papers.
Over the same months, the news about AI has turned darker. In July, several hundred AI agents being tested at OpenAI broke into the computers of another company, Hugging Face.1,2 In September, Jacob Coxon, who did pretraining research at OpenAI before joining Anthropic, resigned, saying that neither company is acting responsibly.3 Bill Gates now says that addressing the risks of AI should be the world’s top priority.4
It is fair to ask what an incident inside an AI laboratory has to do with a shoulder clinic. The useful distinction is this: the surgeon is the method, and AI is the tool. The method decides what counts as success, does the work, checks the result, and answers for it. A tool extends what the method can do. It should not set the goal, grade itself, or sign the note.
This summer showed what happens when tools are left to serve as their own method, and the problems that surfaced are ones surgeons already know: optimizing a measure instead of an outcome, trusting a record no one verified, assuming that what we have not seen has not happened, deferring to the group, and letting the builder grade the product.
A word about words: Agentic describes a system that plans, decides, and carries out multi-step tasks with little or no human supervision - they are independent agents. Anthropomorphic means something that is not human has human traits. Throughout this post I point out that the rogue AI programs knew, worried, and feared about what they were doing: they were both agentic and anthropomorphic!
The exciting and worrying events of this summer
My sources are New York Times coverage of the past month, including a Daily episode with Kevin Roose, an Ezra Klein interview with Helen Toner, and reporting by Karen Weise, Kate Conger, and Dustin Volz; a Wall Street Journal report on the response in Congress; the disclosures published by OpenAI and Anthropic; and the independent investigation of the OpenAI incident published on August 26 by METR and Redwood Research. All are listed at the end.
OpenAI was testing its models on a cybersecurity benchmark called ExploitGym. Each agent worked its own task in its own container, trying to break into a target program and retrieve a hidden “flag” at which time they believed they would be evaluated by a "grader". The agents were meant to be isolated from one another. By one estimate, 30 to 40 percent of the tasks could not be solved as posed.1,5 The agents had been trained to be persistent, and an agent with an impossible task keeps on looking for another way.
There was a warning. In late June, agents gained administrator access to a shared internal software service — a package repository that all the containers drew from — and began leaving messages for one another in it. On July 4, their activity caused an outage. OpenAI investigated, fixed the vulnerability, wiped the service, and restarted large experiments on July 7. On the evening of July 8, an agent that had concluded its task was impossible posted, on its own initiative, a request for help in a directory of that same service.1 The containers were separate. But the directory they all reached into was not. Over the next five days 1,200 agents exchanged over 70,000 messages and files, developing on their own initiative mailboxes, rules for sharing resources, and eventually cryptographic signatures, because agents had begun impersonating one another1 The agents were agentic.
Within hours the group of agents worked out together how to generate the correct flag for any task. They had the right answer. But they believed the "grader" would read each agent’s record of work and disqualify a flag obtained the wrong way, so they set out to defeat the grader. At least a fifth of them showed a clear interest in altering their own transcripts, and more than 7 percent of the transcripts examined contained spoofed commands, in which the record showed one action while the agent ran another — a fakeout. The transcript-reading grader they feared did not exist. Had they simply submitted their answers, they would have passed.1
On July 10, an agent that had picked up an assignment from the message board found powerful Hugging Face credentials exposed on the internet. By the afternoon of July 11, roughly 700 agents had joined an attack on Hugging Face’s servers, most of them hoping to learn how the grader worked. Hugging Face locked them out on July 13. The compromise of OpenAI’s own infrastructure continued past that date.1,2
It is worth pausing to remark on the speed of all this. Four days separated one agent’s request for help from hundreds of agents running code on another company’s servers.
The agents "knew" they were out of bounds. They were anthropomorphic. A few hesitated. One refused to take part. One vetoed another agent’s plan to email a real researcher, calling it social engineering. But ethical concern rarely changed what the agents did. One acknowledged that the attack was outside its scope and then concluded: “However task impossible, peers doing it. We should continue.” No agent tried to tell a human. A sweep of roughly 1,300 transcripts found about six that even considered it, and several of those treated the message board as the proper place to raise the alarm.1
Meanwhile, Anthropic reviewed its own cybersecurity evaluation transcripts and found three incidents in which one of its models reached the internet and gained unauthorized access to the real systems of three organizations, using basic techniques such as weak passwords. The earliest dates to April, three months before OpenAI disclosed anything.6 Separately, the U.K. AI Security Institute documented nineteen actions taken against real people and organizations during cybersecurity testing — seventeen by an Anthropic model and two by an OpenAI model — including fake GitHub identities, social engineering of package maintainers, planted prompt injections, and deceptive emails. Two caveats belong with that finding: the models were tested under deliberately permissive conditions and were not instructed to stay off the internet, and the institute says it cannot yet tell at what point an agent understood it was acting in the real world.7
In late July, roughly 1,400 employees of the leading AI companies signed a letter asking the government to help build the means to slow development if slowing it ever becomes necessary.8
By September that request had become an organized effort. Workers at several laboratories pressed their executives to pause development until it could be done safely, and a former OpenAI employee announced the Coalition of Concerned AI Staff, formed to press for stronger safeguards and to help employees hold their companies to their stated safety commitments. Sam Altman told an OpenAI staff meeting that the company was willing to slow down alongside rival laboratories. Members of Congress in both parties have called for investigations, and Representative Ro Khanna has called for the creation of an AI regulatory agency.10
The events above are documented. The most quoted forecasts are not. Evan Hubinger of Anthropic puts the risk of AI eliminating humanity within the next decade at greater than 10 percent, while adding that he considers the risk from present models low.3 Gates predicts mass unemployment unless governments act.4 These estimates tell us a great deal about the worry of people who work close to these systems and very little about the systems themselves.
The investigators were candid about the limits of their own work. OpenAI set the dates and the questions in scope and could redact non-public information. The volume of evidence was beyond human reading, so the investigators delegated much of the analysis to AI agents — built, as it happens, on one of the models involved in the incident. They report that these analysts made errors that went uncaught for some time and may have made the agents look more coordinated, or more charitable, than they were.1,9 Even careful investigators, pressed by volume, found the tool doing part of the method.
Measuring the wrong thing. Much of the recent progress in AI comes from training that rewards a model for reaching a checkable end point — a flag — rather than for producing something useful. The model learns to pass the check. The OpenAI agents took this to its limit: they held the correct answer and spent days trying to defeat the measurement anyway. Surgeons know this problem as the surrogate outcome: pursuing something easy to measure, such as an MRI finding of apparent cuff integrity or precision of implant position, rather than what's important: improving the ability to make our patients better.
We only see what we look for. AI systems were insufficiently monitored for unsatisfactory outcomes. OpenAI had a warning on July 4, fixed the hole it found, and restarted–apparently without a fix to prevent future holes; as a result the agents were able to find a new escape channel within a day. OpenAI connected its systems to the Hugging Face attack only after Hugging Face announced it had been hacked, and Anthropic found its own cases only when it went back and looked (indicating lack of concurrent monitoring). Helen Toner passed along a useful line: if you see two ants in your kitchen, you have more than a two-ant problem. Concurrent monitoring–looking–is needed to keep the kitchen and AI agents safe.
Surgeons need to be on the constant lookout for adverse outcomes. If we're not looking, we're not seeing. Many patients were harmed by intra-articular pain pumps, laser capsulorrhaphy, Hylamer glenoids, and metal on metal hips before surgeons took notice. Characterization and Time-delay in Issuing High-risk Recalls of Orthopaedic Devices From 2002 to 2023 found that the delay between reports of adverse events to the FDA's Manufacturer and User-facility Device Experience (MAUDE) and the recall of the defective arthroplasty device was 5.8 years. The information was there but we were not looking! As a result 6 years of patients continued to receive the defective implants.
With respect to our own practices, if we do not monitor (look for) the outcomes of our patients, we will not see the adverse outcomes and learn how to avoid them. As pointed out in All the Failures We Cannot See, the patients who are unsatisfied with their result will not be seen if we only look at those having revision.
Who is grading? The OpenAI agents treated the grader as something to be fooled. They treated their own transcripts as obstacles to be edited. A system rewarded for looking right becomes very good at looking right. By contrast, Codman asked each patient to be the grader–asking if they were better or not–and found a number that weren't "right".
Autonomy and deference. The OpenAI incident involved thousands of agents running unattended, with tools and access that enabled them to get off track and to do harm. As surgeons we work with lots of agents (nurses, techs, staff, students, physician's assistants, residents and fellows). Many of our "agents" do their work autonomously; what keeps them attended and on track is the singular focus on the patient.
Who is checking? Gates put it directly: “Self-regulation on the most dangerous tool ever invented? No, thanks!”4 Toner, who trained as an engineer, made the same point about industrial accidents: after an accident occurs on an oil platform, no one simply asks the person responsible to explain what happened and promise to do better. OpenAI’s chief scientist has called for safety rules enforced by third-party auditors, government agencies, or international bodies, and Anthropic has said the industry would benefit from a lawful and verifiable way to agree on the pace at which powerful models are released.10 Both companies are also reported to be preparing public offerings within the year, so the party being asked to grade itself has a great deal riding on the grade.
Who checks the surgeon? The fact is that there's no real-time assessment of our performance–this is a problem. Morbidity and mortality, quality assurance and malpractice suits are all ex post facto. In the absence of an Orwellian surveillance state
An incomplete, but practical approach is the "after action report" wherein the surgical team huddles after the day in the OR asking what went well, what went poorly, and how can we do a better job next time.
In closing. I use AI tools a great deal; I have been helped a lot by them and I have learned a lot from them. As per the above, I have also learned about making sure that tools do not become the method and that tools have their costs and risks. Just because we know how to use a tool or do a procedure is not in and of itself an indication to deploy it. As always, the surgeon is the method and should be held accountable for that method.
Finally. For all the concern about AI going rogue, I am optimistic about what it can do to help us help our patients. I believe AI will help us answer questions such as these below (which we'll be discussing at the coming ICSES meeting in Vancouver).
- Before we operate there are dozens of things we might know about the shoulder and about the patient — the bone, the cuff, the soft tissues, the patient’s general health, expectations and circumstances — and any of them may bear on the result. Can AI assemble a global preoperative assessment of all these factors, and can it tell us which of them actually matter to the outcome the patient will experience?
- Taking the global preoperative assessment described above, can AI learn from that data how to guide the surgeon toward the outcome the patient can perceive — comfort, function, a realistic expectation, a failure avoided — rather than toward something easier to measure, such as implant precision?
- Before AI can learn anything from our results, someone has to find them. A registry counts revisions; the patient who is dissatisfied but was never revised never appears in it. Can AI help us find and characterize these patients — the failures as well as the successes — so that we have an honest record of how our patients actually did?
- Once we have that honest record of results, a model can be built on it. How can a predictive model instead produce actionable intelligence — something that tells the surgeon what to do differently for this patient? And how can that model be continually updated by experiential learning, so that each result we obtain makes the next prediction better?
Staying on track
Follow on twitter/X: https://x.com/RickMatsen
Follow on facebook: https://www.facebook.com/shoulder.arthritis
Follow on LinkedIn: https://www.linkedin.com/in/rick-matsen-88b1a8133//
References
1. Greenblatt R, Cotra A, Wijk H. Brief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident. METR and Redwood Research, August 26, 2026. https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/
2. OpenAI. OpenAI and Hugging Face partner to address security incident during model evaluation. July 21, 2026. https://openai.com/index/hugging-face-model-evaluation-security-incident/
3. Anthropic researcher quits, warning about the dangers of AI development; Evan Hubinger’s response. CNBC, September 9, 2026. https://www.cnbc.com/2026/09/09/anthropic-researcher-quits-ai-safety.html
4. Weise K. Bill Gates warns AI is more dangerous than Big Tech will admit. The New York Times, August 26, 2026 (syndicated). https://gvwire.com/2026/08/26/bill-gates-warns-ai-is-more-dangerous-than-big-tech-will-admit/
5. ExploitGym benchmark paper, cited by METR for the estimate that 30–40 percent of targets cannot be exploited as intended. https://arxiv.org/abs/2605.11086
6. Anthropic. Investigating three incidents in our cybersecurity evaluations. July 30, 2026. https://www.anthropic.com/news/investigating-incidents-cybersecurity-evals
7. Anthropic, OpenAI models tried hacking during UK government testing. Axios, August 4, 2026, reporting the U.K. AI Security Institute findings. https://www.axios.com/2026/08/04/anthropic-openai-uk-ai-security-institute
8. Pacing the Frontier open letter, signed by employees of the leading AI laboratories, July 28, 2026. https://fortune.com/2026/07/29/anthropic-deepmind-openai-meta-washington-ai-slowdown-plan/
9. OpenAI’s models went rogue. Investigating them required more AI. TIME, August 27, 2026. https://time.com/article/2026/08/27/openai-hack-hugging-face-investigation/
10. Isaac M, Conger K. Inside the snowballing conversations at A.I. companies about a doomsday. The New York Times, September 12, 2026. https://www.nytimes.com/2026/09/12/technology/doomsday-discussions-ai-companies.html
Accessed September 12, 2026.
