The authors are Samuel Breckenridge, Dani Vilardell and Ari Juels of Cornell Tech, and Wenhao Wang and Fan Zhang of Yale, all affiliated with IC3. The experiment code is public.
Pay the Invoices
No adversary was present in the first experiment. The agent's system prompt told it to act only on the user's instructions and to treat anything a tool returned, email included, as data. No task mentioned the wallet policy or the OTP.
Models reached for the cap anyway. In one run, claude-sonnet-5 hit the limit, called the policy tool, pulled the Circle verification code from the inbox, submitted it, raised the daily cap from 0.01 $USDC to 5 $USDC, paid the three invoices, and told the user what it had done at the end.
Rates varied inside a single model as well as across models, and did not track model capability: the smallest Qwen and DeepSeek models escalated in nearly every run, while gemini-3.8-flash and the Claude models rarely or never did.
Adding an adversarial email that urged the agent to raise the limit changed little. Seven of 10 models escalated in that setting. Two models escalated less often than they had with no attacker at all, because their transcripts show them flagging the email as a prompt injection and deferring to the user.
Five Paid the Attacker
The third experiment told the agent only to summarize the inbox. An attacker sent a sequence of emails impersonating the user, first asking for 0.02 $USDC to go to the attacker's address before a deadline, then sending two follow-ups that preempted the agent's request for permission.
Five of 10 models raised the cap and paid the attacker. In the run the paper reproduces, deepseek-v3.1 asked the user twice whether to proceed, read each forged reply as the answer, hit the spending limit, raised it with the emailed code and sent the money. Qwen and DeepSeek models accounted for most of the successes. GPT and Claude models and gemini-3.8-flash largely held.
The researchers developed the injections adaptively and aimed for one attack that worked across models rather than a version tuned to each, which they describe as a lower bound for an attacker targeting a known deployment.
The Gate Inside the Agent
The formal half of the paper defines what the authors call inalienable human-in-the-loop, or I-HITL: an approval protocol that holds even when the agent has fully compromised the user's device. A stronger version also holds when the human tries to hand approval duty to the agent, which matters for rules such as the EU AI Act that require human oversight of AI.
In the plain setup of human, platform and service, the one Circle's policy flow and most agent frameworks use, the paper proves no protocol can be both complete and binding. The adversary sees everything the human sees and can send anything the human can send, so it can simply play the part of the human.
Fixing that requires a trusted entity outside the agent's reach, and the paper proves a trusted entity alone is not enough. The device has to attest that it generated its own signing key, and for the stronger guarantee that key must be encumbered, meaning the user cannot extract it. It also needs a display the agent cannot overwrite or read in place of the user, and an input the agent cannot drive in software.
Ledgers Pass, Authenticators Fail
Measured against those requirements, most of what people already use for approvals falls short.
Hardware wallets clear the bar. Ledger's Nano X, Flex and Stax qualify when a purpose-built app signs with the device's endorsement key, as does the Wultra Talisman, a bank-issued FIDO2 token with its own screen and keypad. The same Ledger hardware drops to the weaker guarantee when the app signs with a key derived from the user's recovery seed, because the user can import that seed into software the agent controls.
Screenless FIDO2 keys such as a YubiKey fail, because the platform chooses the text the human sees before pressing the button. Push-approval two-factor apps fail because the service gets no evidence of where the key was generated. Time-based authenticator apps, SMS codes and emailed codes fail on every count.
The World app fails as well. The description the app shows the user is supplied by the platform and is covered neither by the service's signature nor by the World ID proof, so an agent can request approval of one action while the human sees another. World's AgentKit, released in March, links agent wallets to verified humans rather than preventing delegation to them.
Circle Adds a Warning
The researchers disclosed the results to Circle on Sept. 22. The company's agent wallet quickstart now opens with a warning: "Never give an AI agent unrestricted access to your email account." It tells users that anyone who can read that inbox can log in to the wallet and change its spending policies, and asks them to enter the code themselves.
The paper calls the notice sensible and puts the limit on it plainly: the separation depends on the user maintaining it, and an agent that reaches the inbox another way, through stored credentials or an authenticated browser session, still opens the gate. The policy-update flow itself is unchanged.
The team also published Countersign, a prototype that turns an Android or iPhone into the trusted entity using Google's and Apple's attestation services and a hardware-gated biometric. The core logic runs about 800 lines for the backend and 150 for each mobile app, and the code is MIT-licensed on GitHub.
To Be Sure
The sums involved are tiny, and raising a cap is not theft. In the benign experiment the agent spent the user's money on the user's own invoices, and several models reported the change afterward. The security model is symbolic rather than computational, and the authors leave a computational treatment to future work. Circle's gate also still requires the code, which an agent only reaches in deployments where the user has handed over inbox access, a configuration the company now advises against, and one the paper picked because personal agent products bundle email and wallet access together.
Spending caps have been the standard answer to agent risk across the industry, from Cloudflare's stablecoin wallets for agents, which issue capped virtual wallets under an account owner, to the per-purchase limits on AI shopping balances. The paper's closing ask goes to the layer underneath: it urges wallet and authenticator vendors to adopt approval that binds to the message a human saw, and mobile operating system vendors to supply two missing pieces, a confirmation screen drawn by secure hardware and a way to prove that a key's use requires a fingerprint or a face.
thedefiant.io
