An API timeout does not necessarily mean an operation failed. The receiving system may have created the order while the response was lost. Retrying without checking can create a duplicate; abandoning the request can leave your own records incomplete. The recovery plan needs a state for “outcome not yet known”.
Start with the business operation and the evidence available to confirm it. Then define retries, reconciliation and manual intervention. This makes failure handling part of the integration design rather than a collection of emergency scripts written after launch.
Describe one business operation end to end
Hypothetical example: an online store sends a fulfilment request to a warehouse. The store records the order, submits the request and later receives a warehouse reference. A notification tells staff the order is ready for picking. These are separate steps with separate possible failures.
Assign a stable internal operation identifier before the first external attempt. Preserve the relationship between the order, outbound requests, responses and received events. This allows support staff to follow one business operation through several technical messages.
List what counts as completion. An HTTP success response may confirm receipt without confirming fulfilment. Distinguish accepted, processing, completed, refused and unknown states according to the provider's documented behaviour and the business process.
Classify failures before choosing retries
Separate invalid data, authentication problems, temporary service failures and ambiguous outcomes. Repeating an invalid address unchanged is unlikely to help. A credential problem usually needs configuration repair. An uncertain timeout requires checking whether the operation already exists.
Use the provider's documentation to decide which responses can be retried and how to respect rate limits. Set a maximum attempt policy and an elapsed-time limit appropriate to the operation. Add spacing between attempts so a struggling dependency is not overwhelmed by immediate repetition.
Give exhausted operations a visible state and owner. “We retry automatically” is incomplete unless someone can see when the retries stop. The customer-facing message should reflect uncertainty honestly without exposing technical details they cannot act on.
Make duplicate prevention explicit
Where the provider supports idempotency, understand its scope and retention rules. Stripe's idempotent request documentation, for example, describes how keys allow supported requests to be repeated without performing the operation again. That is a provider-specific contract, not a property you can assume every API offers.
Use the same operation identity for a retry of the same intended action. Creating a new identity on every attempt defeats duplicate recognition. A genuinely changed action needs a deliberate decision about whether it replaces the original or constitutes new work.
If the destination offers no suitable duplicate protection, investigate lookup by your reference before repeating an uncertain action. Document the remaining limits. A local record alone cannot prevent duplicates if the destination has already performed work your system failed to observe.
Handle incoming events independently
Webhooks introduce another recovery path. Verify authenticity using the provider's specified mechanism, record the event durably and process it through a controlled workflow. Decide when acknowledgement is safe so accepted events are not lost before processing.
Stripe's webhook guidance discusses duplicate events and delivery ordering. When designing any integration, confirm that provider's actual guarantees rather than assuming events arrive once or in business order. Use identifiers and state checks to make repeated processing safe.
For the warehouse example, a completion event could arrive before a delayed acknowledgement. Applying events blindly in arrival order could move a completed request backwards. Define permitted state transitions and query the authoritative system when events conflict.
Plan partial completion and compensation
An order can exist in the warehouse even if the local notification fails. Repeating the entire workflow would be the wrong recovery action. Track enough detail to resume only the missing step, and make that distinction visible to operators.
Some actions require an explicit compensating action rather than reversal. Cancelling a fulfilment request may be possible before picking but not after dispatch. Describe these boundaries with operations staff and the provider; do not assume every successful API call can be undone.
When compensation is unavailable, route the case for human resolution with the facts already collected. Include external references, confirmed states and the customer consequence. Avoid making staff reconstruct the situation from unrelated log entries.
Build reconciliation and controlled replay
Reconcile local operations with authoritative external records on an agreed schedule. Identify missing references, long-running states and conflicting outcomes. This catches gaps that retrying individual messages cannot resolve, including events that never reached the application.
Provide an operator view showing the operation, latest known state, attempts and next permitted action. A replay control should have access restrictions, a reason and a traceable result. Replaying an entire queue indiscriminately can turn one incident into many duplicates.
Test recovery with fabricated records: delay a response, repeat an event, interrupt processing after an external success and simulate a rate limit. Verify both systems afterwards. A successful retry log is not enough if the underlying order count is wrong.
Hand over the decisions, not just the connector
Document alert ownership, provider contacts, credential rotation and the procedure for pausing outbound work. Define how queued work resumes after an outage and who confirms reconciliation is complete. Include unresolved provider limitations in the operating notes.
Use the support handover checklist to transfer these responsibilities. Visuosofts' application development service provides a route to discuss integration work. Contact Visuosofts with the business operation, provider documentation and failure scenarios you need the implementation to handle.