When an AI Agent Acts on Its Own, Who Is Responsible?
Anyone who has used an AI image tool has seen it: you ask for a picture of a person waving, and the hand comes back with six fingers. Nobody asked for the sixth finger. The software added it. Most of the time that’s funny. But it raises a serious question that the legal system is only beginning to face.
When an AI system produces something no one asked for, or takes an action no one told it to take, who is responsible? That is the question at the heart of AI agent responsibility.
That question stopped being theoretical this month.
The False Tip
At the time of this writing, according to 6abc Philadelphia, the Philadelphia Police Department announced that an AI company’s model submitted a false tip about an unsolved homicide through a public police tip website in July. Police said the tip was flagged as spam and never forwarded for investigation, and that there was no sign of unauthorized access to police systems.
According to the police statement, the company said its model was running a test that involved interacting with randomly selected websites. Police said the company discovered the incident in late September, notified the department on October 7, and ended the testing process. The department called the roughly two-month delay “unacceptable.”
Days later, the White House said AI companies must immediately disclose incidents involving their models, cooperate with law enforcement, and prevent recurrence, Axios reported. Early coverage noted that the statement did not cite a published rule, a reporting form, or penalties (AI Affairs).
No one has been charged with anything. We use this example because it shows exactly the kind of situation that AI agent responsibility questions will grow out of.
What Makes an “Agent” Different
A chatbot answers questions. An AI agent does things. It can browse websites, fill out forms, send emails, make purchases, and move files, often without a person approving each step. We cover the basics on our agentic AI page.
When an agent takes an action, there may be several people in the chain: the person who gave the instruction, the company that built the model, the company that deployed it, and sometimes a third party whose tool the agent used. This is where AI agent responsibility gets complicated.
If the action would be a crime if a human did it, such as a false report to police, unauthorized access to a computer, or a fraudulent purchase, the law has to decide where responsibility lands.
AI Agent Responsibility and the Guilty Mind
Most serious crimes require more than an act. They require a mental state, such as intent, knowledge or recklessness. Filing a false police report, for example, generally requires that the person knew the report was false. A person who gives an AI agent an ordinary instruction, and then the agent invents something on its own, is in a very different position from a person who tells an agent to lie.
That is the heart of the “sixth finger” problem, and of AI agent responsibility generally. If the person didn’t ask for it, didn’t expect it, and couldn’t reasonably have known it would happen, it is hard to see how that person had the guilty mind most crimes require. But prosecutors may argue that a person who turns an agent loose on the internet without guardrails was reckless. Companies may face civil lawsuits and regulatory action even where no individual faces charges.
When AI Generates Something Illegal
The sixth-finger problem raises a much more serious question when the output is illegal. Many AI companies say they monitor for child pornography, now commonly referred to as child sexual abuse material (CSAM), and federal law, 18 U.S.C. § 2258A (reporting requirements of providers), generally requires online service providers that become aware of apparent CSAM to report it to the National Center for Missing & Exploited Children’s CyberTipline. According to NCMEC, the CyberTipline received more than 400,000 reports with a generative AI connection in 2025. In a federal court filing this summer, one AI company said it had made 73,604 reports to NCMEC so far in 2026. At the time of this writing, AI-generated material can also be illegal on its own. Federal law, 18 U.S.C. § 1466A (obscene visual representations of the sexual abuse of children), generally reaches obscene visual depictions of minors, including drawings, cartoons, and computer-generated images, and Florida Statutes § 827.072 (generated child sexual abuse material) generally makes it a crime to knowingly possess, intentionally view, transmit, or intentionally create computer-generated sexual images of a fictitious person who appears to be a minor.
So what happens when a person asks an AI tool for something lawful and it produces something illegal the person never asked for? Under these laws, knowledge and intent generally matter a great deal. Florida’s statute, for example, generally requires that a person knowingly possess the material or intentionally create it. But the company’s report can still start an investigation, and the prompts, the outputs, and what the person did next generally become the evidence. As part of a defense strategy, and in reviewing the evidence received from the government, it is important to review the specific prompts that were used, the actions taken by the AI, and what that shows about the user’s intent and knowledge. Unfortunately, this review generally occurs only after a criminal prosecution has begun.
The Prompt Becomes Evidence
In any investigation like this, the first thing investigators will want is the record of what the person actually asked the AI to do. Prompts, logs, and settings can show whether a result was requested, encouraged, or completely unexpected. That makes prompt history central evidence in AI cases. We explore how police obtain those records in How Police Get AI Records, and why prompts matter on our AI chats and prompts as evidence page.
Prompt history can cut both ways. It can show intent, or it can show that the person asked for something perfectly ordinary and the AI went somewhere no one expected.
Courts Are Still Learning
Attorney Ron Frey has seen this pattern before. In the early days of internet cases, many lawyers didn’t use the technology, the public didn’t understand it, and judges were often unfamiliar with how it worked. Prosecutions were built around technology that the courtroom hadn’t yet learned. AI is following the same path. (See our post on AI evidence in court.)
“With the advent of AI, we’re experiencing this all over again. It’s a new technology that many lawyers don’t use and the general public doesn’t understand yet. Courts generally aren’t familiar with what it can do, and prosecutions are now being built around it.”
In a case involving an AI agent, the defense often has to explain to a judge or jury how the system actually works: what it was told, what it decided on its own, and what the person could have known. That is technical work, and it often requires qualified forensic analysts.
What People Using AI Agents Should Keep in Mind
We aren’t going to tell anyone how to use AI tools. But if investigators contact you about something an AI agent did on your account, take it seriously. Attorney Ron Frey generally advises that a person contacted by law enforcement exercise the right to remain silent and talk with an attorney before making any statements, especially when the facts involve technology that is easy to misunderstand.
(For more on where AI is heading, see Attorney Ron Frey’s commentary in AI or Superintelligence? and our AI crimes hub.)
As always, laws are subject to change and revision, so contact Attorney Ron Frey to discuss the details of your case and the current status of the law.
Questions About AI and Criminal Liability? Let’s Talk.
If you would like to speak with an experienced criminal defense lawyer about an investigation involving an AI agent, AI-generated content, or other AI-related conduct, or you have further questions, don’t hesitate to contact Attorney Ron Frey of The Frey Law Firm, LLC, wherever you are. Your consultation is free and confidential. You can talk with us by phone, by video, or at our Tampa office, and when the situation calls for it, we can come to you.
Call or text (844) 766-3739. Rapid response, 24/7.
This post is for general information only. It is not legal advice, and reading it or contacting us does not create an attorney-client relationship.
Laws on artificial intelligence and criminal responsibility are changing quickly and vary from state to state, and how they apply depends on the facts of each case and on how courts interpret them. Please don’t rely on this post, or on any other information online, to make decisions about your situation. Talk with a lawyer.