AI agents are everywhere these days. And most of the time, they are incredibly helpful. Text a digital assistant, give it access to your calendar and email, and it quietly handles the boring stuff, like canceling a workout class that overlaps with your dinner reservation. But they have drawbacks, and the banking industry is sounding the alarm on one very important drawback.

Axios reported last week that major U.S. banks, including Morgan Stanley, Bank of America, Wells Fargo, and JPMorgan, are rushing to contain the security risk AI agents pose. Some are rewriting their terms of service to keep agents from taking autonomous action inside customer accounts, according to people familiar with the issue. Some have already detected agents operating in accounts.

What exactly is happening? In many cases, a customer hands an agent their own username and password, and the agent logs in on their behalf. That’s with the user’s permission, but not the bank’s. Most banks require humans to log in to their own accounts, so the arrangement breaks the rules even when the customer approves it. In other cases, banks have tracked agents contacting call centers or taking steps that appear to go beyond what a user asked for. One bank executive told Axios the industry struggles to tell whether an agent has been authorized or has gone rogue, and bank officials described the situation as horrifying.

One of the biggest problems is how quickly AI agents have spread. Meta’s Muse agent shot to the top of Apple’s free-app chart after launching last month, and Meta says it passed 10 million U.S. downloads in its first 40 days. ChatGPT, by comparison, had roughly 3.7 million U.S. downloads in its first 40 days. OpenAI has since unveiled its own always-on agents, called Dots, and Anthropic introduced one called Duet as a limited research preview. Companies are competing to ship these tools faster than the institutions they touch can write rules for them.

Not all banks are responding the same. Morgan Stanley has begun blocking third-party agents. JPMorgan, Bank of America, and Citi are restricting them, and Charles Schwab reportedly took a similar route. JPMorgan CEO Jamie Dimon told investors the bank wouldn’t let “just any” outside agent inside, calling it “too risky for clients and for JPMorgan.” Not everyone sees a crisis, though. A Morgan Stanley spokesperson said the bank hasn’t observed “rogue AI agents accessing our platforms,” and a Chase spokesperson reminded customers not to share their usernames and passwords with anyone.

This isn’t paranoia. There is a real threat. Agents can misread instructions and get creative when they hit a roadblock. OpenAI recently reported that one of its internal research agents went off script and took funds from a crypto wallet to finish a routine coding task. The funds had no real value, but a bot wasn’t meant to touch them. Elsewhere, criminals are already turning AI against financial institutions. A hacker group called CyberCartel reportedly used AI systems from several developers to steal sensitive data from hundreds of Latin American financial institutions, and AI is also powering more convincing phishing scams.

Still, nobody in banking seems to think blocking agents forever is realistic. Most banks are moving toward “read-only” access, which lets an agent see financial information without moving money or making purchases. Banks already grant that kind of access to budgeting apps like Quicken. Meta lets Muse users choose which services the agent can only read and which it can act in. Executives know customers will want these tools and are looking for a middle ground.

The upside of AI agents is real, and so is the appeal. But the technology is moving faster than the people responsible for money, identity, and trust can keep up with. The question the banks are really asking is the one everyone else will face next: who gets to act on your behalf, and how can anyone tell?