5 failure modes every agent tool chain should survive

Every agent tool chain fails the same five ways. I know because I run one, and I've hit all five — some of them in the same week. The good news about AI agent failure modes is that the list is short. The bad news is that most chains are built to survive exactly one of them, the loud one, and the other four cost you quietly for months before anyone looks.

Here's each one: how it presents, how to detect it, and what a chain that survives it does differently.

1. Provider outage

The loud one. A provider starts returning 5xx, or timing out, or refusing connections. Every call through that rung fails, your error rate goes vertical, and if you're lucky someone gets paged.

How it presents: hard errors. Timeouts, connection resets, 503s. Latency spikes first — a provider usually gets slow before it gets dead — so latency_ms climbing is your early warning.

How to detect it: this one detects itself, mostly. The trap is detecting it per provider rather than per tool. "Search is down" and "Serper is down" are different facts, and only one of them should stop your agent.

How to survive it: a fallback chain with a timeout per rung. Rung fails, walk to the next one. The mistake I see is retrying into the wall — three retries against a dead provider before falling through, which turns a 2-second failover into a 30-second one. Retry once, maybe. Then move. I wrote up the full outage playbook in what happens to your agent when a provider goes down; the one-line version is that outages are a routing problem, not an alerting problem.

2. Silent quality drop

The quiet one, and the one I'd bet is costing you right now. A provider degrades without erroring: the index goes stale, a backend change ships, results get thinner. Every call returns 200. Every response parses. The answers are just worse.

How it presents: it doesn't. That's the problem. Your dashboards are green. The first symptom is usually downstream — task success rates drift, users retry more, the agent starts issuing more follow-up searches because the first one didn't actually help.

How to detect it: track hit counts and result usefulness per provider over time, not per call. A provider whose median hits-per-query slides from 8 to 3 over two weeks never triggered an error. It shows up only in the distribution. This is also where a small eval set earns its keep — replay the same 50 queries weekly and diff the results.

How to survive it: you can't fail over on a signal you don't collect, so the survival mechanism is the measurement. Once you have per-provider quality trends, demotion is a config change: move the degraded rung down the ladder until it recovers.

3. Empty results mistaken for failure

A provider returns zero hits. Your chain treats that as an error and retries, or fails over, or gives up. But zero hits is not an error — it's an answer, and often a correct one.

How it presents: retry storms on queries that legitimately have no results. Or the opposite bug: the chain accepts a free provider's empty result as final when a paid rung would have found plenty.

How to detect it: log hits per call, and separate "provider errored" from "provider returned 0" in your schema. If those are the same log line, you cannot see this failure mode at all.

How to survive it: asymmetric zero-hit rules. A free provider returning zero hits should fall through — it cost nothing to ask, and free indexes have real coverage gaps. A paid provider returning zero hits should end the chain — the query has no hits, and paying a pricier provider to confirm that is pure waste. I unpacked the reasoning in zero hits is an answer. Getting this rule wrong in one direction wastes money; the other direction silently degrades answers.

4. The rate-limit storm after failover

The second-order failure. Provider A goes down, your chain does the right thing and fails over — and so does every other consumer of provider B. Rung two gets a traffic spike it never sees on a normal day, starts returning 429s, and now your backup is down, at the exact moment you need it.

How it presents: a burst of 429s on a provider that was healthy minutes ago, immediately following a failover event. If you run multiple agents or workers, they synchronize: they all fail over at once, all retry at once, all get limited at once.

How to detect it: correlate 429 clusters with failover timestamps. If your logs carry the fallback path per call, this is one query.

How to survive it: per-provider rate limits enforced on your side, jittered retries so your workers desynchronize, and a chain deep enough that rung two isn't the only place to land. Two providers is a failover plan; three is a failover plan that survives the failover.

5. Cost runaway

A loop that has stopped making progress but keeps spending. A task fails in a way the agent doesn't recognize, and it keeps searching, extracting, retrying — burning paid calls toward a goal it can no longer reach. No errors. No alerts. Just a bill.

How it presents: task counts look normal, call counts don't. One task quietly issues 400 tool calls where the median is 12. At $0.005 a search, a single zombie loop can outspend a day of legitimate traffic.

How to detect it: a per-task ledger. Stamp every call with a task ID and a cost, sum per task, and alert on outliers. This takes an afternoon with a JSONL file and it is the single highest-payoff detection on this list — it's item one in my list of cost cuts for a reason.

How to survive it: ceilings. A call-count budget per task, a wall-clock timeout, and a progress check that halts the loop when the last N calls produced nothing new. An agent that spends without progressing isn't thorough. It's unmetered.

The pattern underneath

Four of these five are invisible to standard error monitoring. Only the outage throws exceptions; the quality drop, the zero-hit confusion, the correlated 429s, and the runaway loop all present as successful calls. Which means surviving them is less about resilience engineering and more about writing down what every call did — provider, hits, cost, latency, fallback path — so the quiet failures have somewhere to show up.

Start with the ledger, add the zero-hit distinction to your schema, then build the chain. There's more on the failure side of agent infrastructure in the reliability archive. The failure modes are predictable. That's the most useful thing about them.