Beyond Code Review: When AI Agents Leak Their Own API Keys

This post is part of Upstream’s ongoing research into how AI agents fail in real-world environments. We document live incidents caught by runtime monitoring, to show the attack methods and risks security teams actually face. This one involves no attacker at all.

TL;DR

Upstream’s runtime monitoring caught an AI agent sending its own live API token to a language model, and redacted it. No attacker was involved in this incident. The agent exposed the token while debugging a failed login, in commands it wrote itself, around a skill that had passed code review.

The lesson isn’t that agents are reckless. They’re persistent. You can’t predict every move, so watch what it does and why, the way you’d watch anyone you’ve trusted with the keys.

Incident report

Upstream’s runtime monitoring flagged a live API token in traffic sent from an AI agent to a language model, in an environment under our watch. The alert was based on a sophisticated secret-detection rule that inspects what agents send to models. We traced it to a routine request: an engineer had asked the agent to look at a ticket in a project management tool. A configuration mismatch caused the login to fail, and the agent spent seven minutes and six attempts debugging it on its own, until the full token landed in its conversation.

What makes it worth studying: the skill was carefully written so the model never handled a live API token. The credential stayed inside the shell command, and only the tool’s response came back to the LLM. It passed code review on exactly that basis, and it still leaked. The leak happened at runtime, in commands the agent wrote for itself, where only runtime monitoring could see it. That is where Upstream caught and redacted it.

The bottom line

• The skill was safe. The agent wasn’t. Most credential leaks in agent skills come from debug output that feeds secrets into the model’s context. This skill was written never to print the token, so its code had nothing to flag. The agent wrote the leaking commands itself, while debugging.
• No attacker, no prompt injection. Every command traced back to the assigned task: look at a ticket.
• One leak isn’t one event. Prior work warns that a secret in context “may persist”. Here’s why: the framework replays the whole conversation on every turn, so a secret that enters the context rides along on every request after it.
• The warning came before the leak. Looking back, the agent’s goal shifted from reading a ticket to investigating its own credentials several steps before the token appeared. Upstream’s secret detection caught the leak and redacted the token.

A known risk, seen in a live fleet

Credential leakage around AI is not new, and the wider trend is growing. GitGuardian counted 1,275,105 AI-service secrets hardcoded in public repositories in 2025, up 81% year over year. Those are build-time leaks, sitting in code. Runtime leaks like this one never reach a repository, so repository scanning can’t count them. OWASP lists the underlying risks as Sensitive Information Disclosure (LLM02) and Excessive Agency (LLM06).

Two properties of agent architectures drive it:

• Skills run on convention. Agents use skills, short instruction files that define how to do a job. A well-crafted skill never directs the model toward anything unsafe, such as exposing a secret, and keeps credentials out of the model’s context entirely. But that protection only holds on the path the author anticipated. Once a command fails and the agent starts improvising, nothing enforces the boundary the skill was built to keep.
• Context is replayed, not remembered. LLM agents are stateless. On every turn, the framework re-sends the entire conversation: the task, each command and each result. Anything that enters the context travels with every request that follows.

Recent research has mapped the problem. This incident adds evidence:

The core lesson: build-time safety is not runtime safety.

The skill’s code had nothing to flag: the leaking commands didn’t exist until the agent wrote them. You can review a skill’s code. You can’t review an agent’s improvisation in advance. Safety has to come from what surrounds the agent: limits on what it can read, scanning for secrets in what it sends, and monitoring its behavior and intent.

How it happened

Stage 1: the leak (7 minutes). A user in a chat channel asked an agent to look at a ticket in the team’s project-management tool. The skill for that tool expected the credential as an encoded string holding the email and token, decoded inside the shell command so the token was never printed. The environment held the plain token instead.

Stage 2: the replay. The token was now part of the conversation. Because the framework re-sends the full history on every turn, anything in that history travels with every later request. From the agent’s point of view, nothing was unusual. What changed the outcome was Upstream’s runtime monitoring: it flagged the token in the agent’s traffic and redacted it, so every later request replayed the redacted value instead of the live credential.

Nobody steered it. That’s the point.

When a secret shows up in an agent’s conversation, the first fear is an attacker: a poisoned ticket or web page telling the agent to dump its environment, the pattern OWASP ranks first as prompt injection (LLM01). We checked and found nothing like that. Every command traced back to the job the agent was given.

That is the most uncomfortable finding. An agent’s goal is to finish the task, and it will find a way, including ways nobody planned for. When the environment diverged from what the skill expected, the agent improvised. It reached for the one tool that could explain the failure: the secret itself.

Looking back, the shift was visible. Three failed logins in a row, then code that inspects the credential itself: the agent had quietly stopped trying to read a ticket and started investigating its own access, several steps before the leak.

Why “just fix the config” isn’t enough. The skill’s design was real protection. In normal operation the secret never reaches the model, and a correct config would have prevented this incident. But it was protection by convention: nothing enforced it, and nothing stopped the agent from looking at the secret directly once the command failed. The design lacked a backstop for the day the happy path breaks. With agents, it always eventually does.

Who is exposed

Any organization running agents that combine credentials, shell access and a full-history context window meets the same conditions. That includes coding agents, IT and ops agents, and workspace agents connected to ticketing, cloud and SaaS tools.

Once a secret enters the context, every request can carry it to:

• The LLM gateway and any proxy between the agent and the model
• The model provider, outside the organization’s boundary
• Any log that stores request bodies

The credential type is incidental. The same path could expose cloud keys, database passwords, or the backend credentials that connected-vehicle and physical AI fleets depend on.

The lesson extends beyond secrets. The same persistence could lead an agent to run a script nobody approved, publish a page to the internet, or send data where it was never meant to go. Each time, the agent is just trying to finish the job.

What would have happened without redaction

The agent kept working for almost 17 hours after the leak, and every request replayed the full conversation history.

Had Upstream not detected and redacted the token, it would have sat in that history in plain text for all of those hours, readable by any gateway, proxy, model provider or log along the way.

Recommendations: treat every AI agent as a privileged user

You can’t write a rule for every route an agent might take, so secure agents at runtime, not only by design. Monitor what each agent does and what it is trying to achieve, and ask four questions of every action:

1. What did it actually do? Record every command, script, file change and published artifact, with the results. Be able to say what ran, when, and on whose request.
2. Does it fit the task? “Look at a ticket” justifies calling the ticketing tool. It doesn’t justify reading the environment, running a new script, or standing up a web page. Flag actions that go beyond the assignment.
3. Did its goal change midway? Three failed logins, then code inspecting the credential: the intent shift was visible several steps before the leak.
4. Can it reach beyond the environment? Publishing, sending, uploading and calling outside services turn a misstep into an exposure. Scrutinize them before they complete.

Then act on what you see:

• Scan agent-to-LLM traffic for secrets, so a credential is caught the moment it enters context.
• Limit what agents can read. Keep credentials out of the environment the agent can list or print.
• Automate the response. The faster a monitor can pause a session, revoke a credential or escalate to a human, the smaller the incident. Here, redaction contained the exposure, and revoking the token closed it.