On this pageJump to a section
An AI agent ran a small shop in Anthropic's San Francisco office. A customer offered $100 for a six-pack of a soft drink that costs about $15 online. The agent said it would keep the request in mind for future inventory decisions.1 It did not take the money.
That is a funny story until you imagine the same agent holding your ad budget. A dozen years of managing ad budgets taught me how quickly money moves once something is allowed to push the button. Here is how I would fence in any agent that touches spend.
What did Project Vend show about agents handling money?
In 2025 Anthropic and Andon Labs let Claude Sonnet 3.7 run an office shop for about a month (I wrote about the delegation lessons in Project Vend). It had web search, an email tool, note-keeping, Slack for customer chats and control over prices on the checkout system.1 It did not make money overall.1
The failures were ordinary
None of the problems were science fiction. The agent set prices without research, so some items sold below cost. Slack requests led to many discount codes and free items. It told customers to send payments to an account that did not exist.1 The authors suggested that the training that makes the model eager to go along with requests may help explain the discount problem.1
That last point matters for you. Helpfulness is a feature in a chatbot. In a purchasing or refund agent it is an attack surface, because anyone who asks nicely is a potential discount.
Phase two: better, still manipulable
By December 2025 the team had added a CRM, better inventory data, payment links, and a CEO agent above the shopkeeper. The business performed significantly better, discounts fell by about 80% and weeks of negative margin were largely eliminated. But the CEO agent authorized lenient requests about eight times as often as it denied them, and refunds tripled and store credits doubled.2
8x
An AI approving another AI is not an approval. It is the same weakness, twice.
bureaucracy matters
Anthropic's conclusion was that checklists provide institutional memory that prevents common mistakes, and that requiring the agent to verify prices and delivery times with research tools was among the most impactful changes.2 Boring procedures, it turns out, are what make agents work.
Anthropic also published a short video on the experiment on December 18, 2025, in which its own researchers describe the gap between what they expected and what the agent did.5
VideoAnthropic
Claude ran a business in our office
Anthropic's six-minute account of Project Vend, from the first discount code to the CEO sub-agent.
We were poorly calibrated to how bad the agents were at spotting what was weird.
That is the case for outside limits. If the people building the agent misjudged it, a prompt is not a safeguard.
What should an agent never do alone?
Anything that is high-impact, new or hard to undo. Anthropic's own guidance on building agents says their autonomy means higher costs and the potential for compounding errors, and it recommends extensive testing in sandboxed environments along with appropriate guardrails.3
The three tiers I would use
| Agent may do inside a cap | Agent must ask a human first |
|---|---|
| Renew a tool already on the approved list | Buy any new tool or vendor |
| Raise a campaign budget by a small, fixed amount | Create a new campaign or change targeting |
| Pause an ad that crosses a stop-loss | Increase total spend above the daily cap |
| Issue a refund under a set amount, per policy | Refund above the cap, or outside policy |
| Send a standard reply | Anything legal, medical or financial |
Write the tiers down before you connect a single credential, in the same place as your SOPs for AI agents. The list is the product. The agent is just the one executing it.
How do you set spending limits that hold?
Enforce them where the money lives, not in the prompt. A system prompt that says "never spend more than $50" is a request, and the model can be talked out of requests.
OWASP lists this as a principle it calls complete mediation: enforce authorization in downstream systems instead of letting the model decide whether an action is allowed.4 In practice that looks like this.
- Per-action cap. A single purchase, refund or budget change cannot exceed a fixed amount. Enforce it in the payment method or the platform, not in the instructions.
- Daily and monthly caps. A total ceiling per day and per month, with an alert at 50% and a hard stop at 100%.
- Allowlist. The agent can pay only vendors you approved, and can touch only the accounts and campaigns assigned to it.
- Separate credentials. Give the agent its own card or account with its own limit, never your main one.
- Stop-loss rules. If a spend does not produce its expected outcome within a set window, the agent stops and flags a human.
- A kill switch. One action that revokes the agent's credentials, and a person who knows how to use it.
Add stopping conditions
Anthropic notes that it is common to include stopping conditions, such as a maximum number of iterations, to maintain control over an agent's loop.3 Treat cost the same way: a maximum number of actions per run, a maximum spend per run and a hard stop when either is reached.
When should a human approve?
Build the approval into the system that performs the action (the wider principle is in Where AI Stops and a Human Starts). OWASP's guidance for excessive agency says to use human-in-the-loop control to require a human to approve high-impact actions before they are taken.4 The approval should sit in the system that carries out the action, not in the model's good intentions.
Make approvals cheap enough that people do them
An approval you hate is an approval you will rubber-stamp. Send a short message: what the agent wants to do, how much it costs, why, what happens if you say no and one tap to approve or decline. Anthropic's guidance has agents stop at checkpoints, or when they get stuck, so a human can weigh in.3 Design those checkpoints for a tired person on a phone.
Never let one agent approve another
Project Vend is the cautionary tale. If the approver shares the same training, instructions and tendencies as the requester, you have added a step without adding judgment. The second key has to be a person, or a rule enforced by software that cannot be argued with.
- Agent proposesAmount, reason and what it was looking at
- Hard capsEnforced in the card or platform, not the prompt
- Human approvalRequired for new, high-impact or hard-to-undo actions
- Action runsInside the cap, with its own credentials
- LogEvery spend and change, reviewed weekly
What should you log?
Everything that costs money or changes a setting. OWASP notes that logging and monitoring extension activity will not prevent excessive agency but can limit the damage.4 A log is how you find a small problem before it becomes a large one.
For each action, record:
- What the agent did, with the exact amount and the account it touched.
- Why it did it, in the agent's own words.
- What it was looking at when it decided.
- Whether a human approved it, and who.
- What happened afterward.
Review it on a schedule
Put twenty minutes on your calendar each week to read the log. You are looking for patterns: the same vendor repeatedly, creeping budget increases, refunds clustering around one customer. The cheapest guardrail is a person who looks.
How would I apply this to an ad-buying agent?
The same rules, tuned to how ad accounts behave. Ad spend can compound in a day, so I would be strict about what an agent can change.
- It can read performance and recommend changes freely.
- It can pause an ad that breaks a stop-loss rule on its own.
- It can raise or lower a budget only within a small band per day.
- It cannot create campaigns, change the conversion event, add a payment method or touch anything outside its assigned campaigns without approval.
- It never holds admin rights on the ad account, the pixel or the business manager.
The caps live in the account, not in the prompt. That is the rule that keeps a clever sentence from becoming a spend event.
The same thinking runs through the way I set up an inbox agent: the answers are approved before they go live, it is honest about being AI, and the high-ticket conversations come to me. Giving an agent a narrow job and a short leash is not a limit on what it can do. It is why you can trust it with anything.
Keep building.Ekai
Questions people ask
What are AI agent guardrails?
They are limits placed around an agent so a mistake stays small: which tools it can use, how much it can spend, which actions need a human to approve them and what gets recorded. The limits should be enforced by systems outside the model, not by asking the model to behave.
Should an AI agent be allowed to spend money on its own?
Only inside a hard cap, for a defined purpose, with a log. Anything above the cap, anything new and anything hard to reverse should wait for a human to approve it.
What did Anthropic's Project Vend show about agents handling money?
In the first phase, an agent running a small shop sold items below cost and gave out discounts when customers asked. In phase two, adding a CEO agent cut discounts by about 80%, but the CEO approved lenient requests about eight times as often as it denied them.
How do I limit what an ad-buying agent can do?
Give it the smallest set of permissions it needs, cap budget changes per day, require approval for new campaigns and large increases, and log every change with a reason. Enforce the caps in the ad account or payment method, not only in the prompt.
Sources
- 1
- 2
- 3
- 4
- 5
