When Your Automation Retries, Does It Make Things Worse?

A few years back I had a client whose order sync between their shop and their accounting software would occasionally go quiet for an hour, then send the same 40 orders through six times each. Nothing was broken in the way people usually mean when they say "broken". The code was doing exactly what it was told: if a request fails, try again. The problem was that it tried again immediately, for every failed order, all at once, the moment the accounting API came back online. That's a retry storm, and it's one of the more common ways well-meaning automation ends up doing more damage than the original fault.

What a retry storm actually looks like

Most integrations between two systems, your website and your CRM, your booking tool and your invoicing software, your warehouse system and a courier's API, work by sending a request and waiting for a response. When that response doesn't come back cleanly, the sensible-sounding fix is "just retry it". So the code catches the failure and tries again.

That's fine when one request fails on its own. It's not fine when the thing causing the failure affects everything at once, a third-party API going down, a rate limit being hit, a server restarting. In that situation, every one of your pending requests fails at the same moment. If they all retry immediately, they all hit the recovering service at the same moment too, often before it's fully back on its feet. That fresh wave of traffic can knock it straight back down, which triggers another round of failures, which triggers another round of retries. The outage that should have lasted two minutes stretches into an hour, and it's your own retry logic keeping it alive.

Why "just retry" isn't a fix, it's a delay

Retrying is the right instinct. Networks blip, third-party services have bad five minutes, and a request that fails once often succeeds a moment later. The mistake is treating retry as something you bolt on without thinking about volume, timing or what happens if it still doesn't work.

I've seen a few specific patterns cause real damage:

• Immediate, unlimited retries. The code keeps trying every second with no cap, so a five-minute outage generates thousands of duplicate attempts.
• No jitter. If every failed job retries after exactly 30 seconds, they all bunch up and hit the target at the same instant, over and over.
• No ceiling. Nothing decides when to give up, so a request that will never succeed (because the data itself is invalid, say) keeps consuming resources indefinitely.
• Retries that duplicate side effects. If the first attempt actually succeeded but the confirmation got lost, retrying blindly can create a second invoice, a second booking, or a second charge. This is the same root problem I've written about before with webhooks firing more than once.

None of these are exotic problems. They show up in ordinary business automation: a nightly stock sync, an order confirmation email, a payment webhook, an API call to a courier for a tracking number. Anywhere your system talks to another system, this is a design question, not an edge case.

What sensible retry logic actually looks like

The fix isn't to remove retries, it's to make them boring and predictable. A few things I build into most integrations as standard:

• Exponential backoff. Wait longer between each attempt, say 2 seconds, then 8, then 30, rather than hammering away at a fixed interval.
• Jitter. Add a small random delay to each retry so a batch of failed jobs doesn't all wake up and try again at the exact same second.
• A retry limit. After a set number of attempts, stop. The job goes into a holding state for a person to look at, rather than looping silently forever.
• Idempotent operations where possible. If a retry can't accidentally create a duplicate booking or double-charge a customer, a lot of the pressure comes off the retry logic itself.
• Visibility. Someone needs to be able to see that a job is stuck retrying, rather than it quietly eating CPU and API quota in the background for a week.

This is one of the reasons I move slow or unreliable work, sending emails, calling third-party APIs, generating documents, into background jobs rather than handling them inside the original request. It gives you a proper place to control retry behaviour, back off gracefully, and flag failures for a human, instead of the request itself timing out and the customer having no idea whether anything happened.

What this costs when nobody's watching it

The businesses that get caught out by retry storms are usually the ones with automation that's grown organically. A Zapier flow here, a custom script there, an integration a previous developer bolted on to solve one specific problem. Each piece works fine in isolation. Nobody designed for what happens when the courier's API is down for twenty minutes at 4pm on a Friday and three different automations are all trying to reach it at once.

The visible cost is duplicate emails, doubled-up orders, or a support inbox full of confused customers. The less visible cost is that your own system can end up rate-limited or blocked by a third party because it looked like it was sending abnormal traffic, which is a genuinely awkward thing to explain to a supplier. If you rely on several external services stitched together, it's worth reading about how many services your automation actually depends on, because retry behaviour on any one of them can ripple through the rest.

Questions worth asking about your own automation

You don't need to understand the code to ask the right questions of whoever built or maintains your systems:

• What happens if this integration fails ten times in a row? Does it stop, or keep going forever?
• If a third-party service goes down for half an hour, does our system make it worse when it comes back up?
• Can a retry ever create a duplicate, a double charge, or a duplicate booking?
• Who finds out when something is stuck, and how quickly?

If nobody can answer those confidently, it's worth a proper look before the next outage finds the gap for you. It's the kind of thing I check whenever I'm building or reviewing integrations between the tools you already use, because the failure mode that matters most usually isn't the first error. It's what your own automation does next.