When an AI Agent Orders Wrong, Who's Liable?

What If an AI Ordered the Wrong Thing for You

Let’s say you handed your grocery shopping to an AI assistant. It builds a cart, shows you the total, and asks whether to complete the order. The amount looks about right, so you hit approve. Later, you notice one of the items contains an allergen. Or maybe the AI shipped to the wrong address, or the same order got charged twice.

The final click was yours. But before that click, a chain of things had already happened together: your instructions, the AI’s interpretation of them, the store’s product listings, and the ordering-and-payment system. Depending on where exactly things went wrong, the party you’d need to hold accountable changes too.

Clicking “confirm” doesn’t automatically mean the user bears all the responsibility. But the reverse isn’t true either—you can’t say the user bears none of it just because an AI handled the process. Today, I want to trace how something like this goes wrong by walking through five stages: intent, authority, execution, confirmation, and recovery.

Even With Verification Steps, Errors Still Slip Through

Even a chatbot that only answers questions can cause harm with bad information. But agents can go further—executing orders, sending messages, deleting files—all on their own authority. That’s exactly why accuracy alone isn’t enough; you also need to decide what actions to permit and how to undo them.

In its Operator system card from January 2025, OpenAI published results from running the model—without safety mitigations applied—through 100 everyday tasks. Thirteen errors caused real inconvenience, and 8 of those were easily reversible within minutes. The remaining 5 included things like emails sent to the wrong recipient and food ordered incorrectly.

The company said that confirmation steps cut this risk by roughly 90%. In a separate evaluation covering 607 tasks, the confirmation recall rate—the share of situations requiring confirmation in which the model actually asked for it—averaged 92%. This is a pre-launch, in-house evaluation, so it shouldn’t be read as a real-world incident rate among actual users. Operator System Card

The ChatGPT agent system card from July 2025 reported a confirmation recall rate of 91.0%. The document notes that this figure is likely an underestimate, since the evaluation missed many instances where confirmation actually occurred in practice. Critical actions, like financial transactions, were assessed separately. So you can’t simply compare the 92% and 91% figures across the two releases and conclude there’s been no improvement—or that one in ten payments now goes through without confirmation. ChatGPT agent System Card

What matters to me here is that you need to design for both moments at once: confirmation before an error happens, and recovery after it does.

What you saw when you approved matters

A screen that shows only the total amount and a screen that shows the product name, quantity, shipping address, and return conditions give the user different information to review. If consent scope is later disputed, the screen and instructions at the time can serve as important evidence. Still, the fact that only the total appeared on screen doesn’t automatically mean, legally, that consent was limited to the total alone.

There’s also a limit to how far you can push approval requests. In an April 2026 post, Anthropic described a system for setting each tool action to always-allow, approval-required, or blocked — allowing calendar lookups, say, while requiring approval for sending invitations. At the same time, the company noted that asking for confirmation dozens of times in a row can make users stop reading requests carefully. For Claude Code, Anthropic also proposed reviewing the execution plan up front. Anthropic’s design explanation

Approving a plan up front reduces the burden of stopping at every single step. But even then, you need to distinguish between tasks that were part of the original plan and tasks that arose afterward. It has to be clear whether permission to organize files also covers deleting unrelated files, and whether approving a single payment also covers subsequent automatic payments.

In Anthropic’s February 2026 usage study, about 20% of new Claude Code users used full-session auto-approval, and among experienced users that figure exceeded 40%. Experienced users also intervened mid-session more often. Rather than pre-approving every step, a pattern emerged of watching execution unfold and stepping in when needed. This finding doesn’t necessarily mean the level of scrutiny has dropped. Study measuring agent autonomy

I think what matters isn’t the number of approvals but the scope of control the user actually retains. You should be able to know what you’ve delegated, where you can stop it, and which actions still require additional permission.

There’s a risk of approving manipulated information

Web pages or documents that an agent reads can contain hidden instructions designed to make it act in ways the user never asked for. This is called prompt injection. The user might ask the agent to compare products, while an instruction buried in the page tries to get it to send information to a different site.

Confirmation steps can help limit the damage from this kind of attack. But if the AI-generated summary or approval screen itself contains false information, the user can easily miss the error. That’s why OpenAI and Anthropic don’t rely on a single confirmation prompt — they also talk about model training, monitoring systems, and restricted permissions working together.

In a case like this, a log that simply says “the user approved it” isn’t enough to explain what happened. You need to trace the whole chain: what material was read, what was shown to the user, and whether what was approved actually matched what was executed. The mere existence of an approval record doesn’t erase a product defect or absolve a company of its obligations.

The draft is accurate and complete—no issues found.

You can trace the cause in five steps

Step	What to check	Where to look
Intent	What did the user ask for	The original instruction and any later edits
Permissions	Was read, write, payment, or deletion access granted	Permission settings and the scope of delegation
Execution	What information and tools were used to process it	Product info, tool calls, order and payment records
Confirmation	What was shown to the user	The approval screen and the timestamp of approval
Recovery	Can it be canceled, refunded, or restored	Seller contact info and the recovery process

This table isn’t a formula for assigning legal liability. It’s a sequence for pulling together records scattered across different companies to identify the cause. And recovery doesn’t necessarily happen outside the product either — it might end with an in-product cancellation feature, or it might require help from the seller or the payment provider.

If you’re sharing a company account with an agent, you need to keep more careful records. The account name alone may not tell you whether an employee acted directly or the agent acted on their behalf. Per-tool permissions and approval/execution logs help make that distinction.

OWASP’s agent security guidelines also recommend granting only the tools a task actually needs, applying permission checks to sensitive actions, and logging tool calls and their results. OWASP Security Guidelines

The company showing the order screen may not be the seller

In OpenAI’s Agentic Commerce Protocol documentation, ChatGPT displays the order screen, but it’s the seller’s own system that decides whether to accept the order, and an existing payment processor handles the transaction. OpenAI states that in this structure, it is not the “merchant of record.” Order and payment structure

This explanation alone doesn’t settle the company’s legal status or liability under Korean law. You have to look at what role the service actually plays alongside the applicable law. The agent-liability clause in the terms for submitting merchant product data only concerns that submission activity. It’s a stretch to read it as a clause defining liability for every incident caused by a consumer’s AI assistant.Merchant feed terms of service

Article 20 of Korea’s Act on Consumer Protection in Electronic Commerce requires an online-intermediary platform to give advance notice, in a way consumers can easily recognize, that it is not itself a party to the transaction. The intermediary must verify the identity information of the business-operator seller and provide it to the consumer before they place an order. Explaining the seller’s status in English in a developer document doesn’t, by itself, satisfy this consumer-notification obligation. Article 20

Article 20-2 also establishes joint-and-several liability for damages arising from a failure to give notice or from providing incorrect identity information. Whether damages actually resulted from the violation still has to be examined, and there’s an exception for identity-information liability where the platform exercised due care. The mere fact that the seller’s name isn’t visible on a single screen isn’t enough, by itself, to establish liability for damages.

As a general rule, a consumer can withdraw an offer within 7 days of receiving the written contract, or, if the goods arrive later, based on the date of receipt, among other criteria. Where the goods differ from what was advertised or from the contract terms, a separate standard applies: within 3 months of receiving the goods, and within 30 days of the date the consumer knew or could have known of the discrepancy. There are limits and exceptions for damaged goods or digital content already provided, so the details of the specific transaction need to be checked. Ministry of Government Legislation’s explanation of the right of withdrawal

How this law applies to a given AI service depends on the transaction structure involved. In an actual dispute, you’d need to examine not just the approval screen but the contract, the product information, and the order records together.

Even free services need a clear recovery path

Being a free AI assistant doesn’t automatically exempt a service from privacy and consumer protection obligations. That said, the records provided, the scope of support, and the compensation commitments differ from product to product, whether free or paid.

It’s worth checking who you can contact if something goes wrong, whether you can export your order number, and whether you can secure the records you need even after your service contract ends. Price alone isn’t a reliable gauge of how recoverable your data will be.

Oswarld’s Lens

While preparing this piece, what I spent more time verifying than any number was the launch-and-shutdown notices for these products.

OpenAI’s current help documentation explains that the Operator website is no longer available. It also notes that ChatGPT agent is no longer offered, pointing users instead to ChatGPT Work and cloud browsers. For Atlas, the notice lists August 9, 2026 as the shutdown date and instructs users to separately save any bookmarks and browser data they need. ChatGPT agent notice, Atlas shutdown notice

When a product changes or gets discontinued, the orders, shipments, and deletions it carried out don’t automatically get undone. Whether you can still access the approval screens or execution logs from that time, though, is a separate question. You need to check the retention period and how to migrate that data.

That’s why I think that when adopting agentic AI, the first question to ask—alongside accuracy—should be whether you can still investigate incidents and take necessary action even after you’ve stopped using the product. The answer needs to come from your contracts, your record-export features, and the support processes of your vendors and integration partners.

The same applies when employees use agents through company accounts. You need to know whether the only record the company can check is the employee’s account name, or whether approvals and tool executions are logged as well. This isn’t an argument for storing every log indefinitely. It’s about deciding, at the point of adoption, the scope of records you need, how long to retain them, and who gets access.

Looking at the draft, I compared it carefully against the Korean source. The translation is accurate, preserves the numbered list structure, maintains the meaning, and contains no Hangul characters, number mismatches, or structural drift. No corrections are needed.

Five things to check before you approve

1. Can you actually see what’s about to happen. You should be able to confirm the specifics that determine the outcome — product, quantity, price, shipping address.
2. Is the scope of your approval clear. Check whether it applies only to this one transaction, or whether it extends to repeated runs or additional tasks.
3. Can you find the record and the way to cancel. Make sure you can get the order number, the execution timestamp, and the procedure for reversing it.
4. Are you granting only the permissions actually needed. There’s no reason to hand over payment or deletion rights when all that’s required is read access.
5. Do you know who you’re dealing with and where to get support. Keep the roles and contact channels of the AI provider, the seller, and the payment processor clearly separate.

A confirmation button is a way to reduce accidents. That single button doesn’t block every risk or settle every question of responsibility. What you need alongside it is information you can actually review, the authority to control execution, and records and recovery procedures you can use when something goes wrong.

💬 If you’ve ever had an AI handle an order or a shipment for you, what do you double-check before approving? I’d also love to hear about a time you went to reverse something and couldn’t find the record you needed.

References & Further Reading

• OpenAI Operator System Card, January 23, 2025.
• ChatGPT agent system card, July 17, 2025.
• Anthropic: Trustworthy agents in practice, April 9, 2026.
• Anthropic: Measuring AI agent autonomy in practice, February 18, 2026.
• Role definitions in OpenAI’s Agentic Commerce Protocol.
• OWASP AI Agent Security Cheat Sheet.
• Act on Consumer Protection in Electronic Commerce: provisions on intermediaries’ duties and liability, and rules on withdrawal of offer.

Past issues worth reading alongside this one

• What Korean merchants need to check before joining agent payments
• How to give an AI agent a prepaid card and a spending limit
• How the experience of using AI agents is reshaping oversight

The author is Oswarld (Kwangseob Ahn). Current roles: Adjunct Professor at Sejong University, Strategy Consultant at INLEVEL9. Career, research, books, and recent work are kept current on the About page. Latest · July 2026: HEMA-2: A Consolidation-Aware Tri-Memory Architecture with Multi-Channel Scheduling for Lifelong Conversational AI.