An API integration can be working normally one minute and fail the next. A vendor may be temporarily unavailable, a network request may time out, an access token may expire, or a business rule may reject a record. The technical cause matters, but the operational response matters just as much. Without a clear process, a small team can spend hours guessing, retry the wrong request, or create duplicate work while trying to recover.
A practical failure runbook gives people a shared sequence for deciding what happened and what to do next. It does not need to be a large incident-management system. It needs to be specific enough that the person on duty can make a safe first decision.
Start by defining what counts as a failure
Write down the signals that should start an investigation. They might include a queue that has not moved, an unusual number of failed requests, a timeout reported by the integration, or a customer-facing workflow that has not completed within its normal window. Include both technical and business signals. A successful HTTP response does not always mean the business action completed correctly, and a delayed response may still mean the remote system accepted the request.
For each signal, record where someone should look first: an application log, a queue dashboard, an email notification, or a record in the source system. Keep the first check short. The goal is to establish whether the problem is isolated to one record, affects one endpoint, or is blocking the entire integration.
Classify the failure before retrying
Do not make “try it again” the default response. Classify the error into a small set of useful categories:
- Temporary service or network failure: a timeout, connection reset, gateway error, or service-unavailable response may clear after a short delay.
- Rate limiting: a service may be asking the client to slow down. If the response includes a Retry-After value, the integration should respect it rather than immediately sending more requests.
- Authentication or configuration failure: an expired credential, changed endpoint, missing permission, or malformed configuration needs investigation, not repeated retries.
- Validation or business-rule failure: the receiving system understood the request but rejected its data. Fix or review the record before sending it again.
- Unknown outcome: a timeout after a write request may mean the receiving system completed the action but the response was lost. Treating it as an ordinary failure can create a duplicate.
This classification is more useful than copying every error into a single “integration failed” bucket. It ties the next action to the likely cause.
Make retries bounded and deliberate
Retries are appropriate for some transient failures, but they should have clear limits. Use a small maximum attempt count, increasing delays between attempts, and a cap on the total retry window. Adding jitter—a small random variation to the delay—can prevent multiple workers from retrying at exactly the same moment.
Retry only operations that are safe to repeat, or make them safe with an idempotency key, a unique business reference, or a precondition that lets the receiving system recognize the same request. Reading a record is usually easier to repeat than creating an invoice, sending a customer notification, or charging a payment method. The exact rule depends on the systems involved, so document it per endpoint or workflow.
When the retry limit is reached, move the work to a visible failure state. Preserve the original request reference, response code, timestamp, and attempt count. A person should be able to see what needs review without searching through raw logs.
Separate recovery from diagnosis
A runbook should tell the responder which actions are safe immediately and which require investigation. For example, a temporary read failure may be retried automatically, while a failed customer update may be held for review. This separation prevents a well-intentioned recovery action from changing evidence or making the original problem harder to understand.
Include a short checklist:
- Confirm the affected workflow and time range.
- Identify whether the issue is isolated or widespread.
- Check the last successful request and the first failed request.
- Classify the response and determine whether the request’s outcome is known.
- Apply the documented retry, hold, or escalation action.
- Record the result and link it to the affected business record.
Define escalation in business terms
“Escalate to IT” is not enough. State who owns the next decision and when to involve them. An operations lead might handle a single rejected record. A developer or integration owner might investigate repeated authentication failures or a changed response format. A business owner may need to decide whether a backlog can wait, whether a customer should be contacted, or whether a manual workaround is acceptable.
Set thresholds that are understandable without pretending to know a universal number. Examples include repeated failures across multiple records, an unresolved queue approaching a business deadline, or an unknown outcome on a high-impact write. The threshold should reflect the workflow’s consequences and recovery time, not just the HTTP status code.
Close the loop after service is restored
Recovery is not complete when the next request succeeds. Check whether failed records were replayed, held records were reviewed, and downstream systems now agree. Look for duplicates, partial updates, and records that were skipped because the integration stopped at the first error.
Afterward, make one small improvement while the details are fresh. Add a missing alert, clarify a retry rule, capture a better request identifier, or document a response that the team previously treated as unknown. A runbook becomes valuable through these small revisions.
A useful first version can fit on one page
Start with the integration’s most important workflow and document its signals, failure categories, safe retry rules, escalation owner, and verification steps. Test the runbook with someone who did not write it. If they cannot determine what to check or what not to repeat, simplify the instructions.
Reliable business integrations are not created by retries alone. They combine predictable software behavior with a human process for the cases that software cannot safely resolve. Next step: Schedule a short consultation to identify the next useful improvement.