An API integration has stopped: investigate and recover without duplicate work

Describe impact in business terms

Identify which work cannot proceed. Are customer updates missing, draft documents absent or planning information outdated? Record the suspected starting point and the last demonstrably correct result. A server error at ten o’clock does not prove every transaction after ten was lost. Find specific source references and destination states to bound the investigation instead of treating a broad time range as established fact.

Have the operational owner approve a temporary working method. If staff enter records manually, retain evidence of what was performed outside the integration. Otherwise resumed automation can repeat those actions. Decide whether to pause all processing or only a consequential step. A complete stop may cause unnecessary disruption, while continuing without understanding the effects can also be inappropriate. Document the reason and responsible decision-maker.

Preserve the current evidence

Collect relevant error categories, timestamps, source references, destination identifiers and the configuration version in use. Where necessary, preserve a controlled copy of state before changing configuration or data. Do not immediately remove failed executions to produce a clean dashboard. The sequence of attempts may be the best evidence of which operation already affected the destination and where uncertainty began.

Keep credentials and unnecessary personal information out of broad incident channels. An operator often needs only a safe reference and error category. Identify evidence that is missing because logging is limited or a supplier exposes no history. This prevents false certainty based on a fragment from one application. A preserved factual record supports targeted recovery and later improvements without requiring unrestricted copies of production payloads or secrets.

Separate access, data and availability problems

Expired or revoked authorisation needs different handling from an unreachable service. Inspect the actual response and current supplier documentation. Do not automatically classify every rejection as temporary. Repeated authentication or resubmission can create unnecessary load and make diagnosis harder. Restore access through the agreed administrative process and limit permissions to those genuinely required for the integration’s task.

For invalid content, investigate the smallest concrete example. Has a mandatory field changed, does a destination reference no longer exist or does the payload violate the current contract? Correct the cause at the appropriate source. Adding a broad exception so invalid records continue regardless can hide a technical error while creating business-data problems. Explain which previous assumption changed and update the mapping or validation accordingly.

Recovery may involve replay. Prepare the duplicate-operation controls first: Prevent duplicate orders and invoices when API messages repeat.

Evaluate temporary failures deliberately

A temporary connection problem may disappear after a suitable delay. Retrying needs limits and should account for a service that may still be overloaded. Microsoft’s Retry pattern discusses stopping, immediate repetition and delayed repetition. Appropriate configuration depends on the actual operation and supplier requirements rather than a universal retry count applied to every error and every endpoint.

Consider the business consequence of repeating the request. Reading a record differs from creating a document. Reuse the existing recognition of previous attempts and investigate uncertain outcomes. Waiting longer cannot supply a missing customer reference. Classify failures so an employee can understand when waiting may help and when a correction or explicit decision is required before the item can safely move forward.

Microsoft Azure Architecture: Retry pattern ↗

Investigate uncertain outcomes separately

A lost response can leave the source showing failure while the destination already holds the result. Gather the source reference, any external key and the period in which the action was attempted. Use the destination’s supported facilities to find the intended operation. Check identity and contents; a similar amount or the same customer name alone is weak evidence of a reliable match.

If no dependable conclusion is possible, keep the item uncertain until an authorised owner decides. Record that decision, the remaining risk and the checks performed. Do not bypass an earlier attempt by inventing a new random operation key. Recovery must retain the same duplicate protection as normal processing and distinguish a genuine new instruction from repetition of work whose outcome has not yet been established.

Trial recovery on a limited selection

Choose a representative failed item that can be investigated safely. Correct the established cause and run the agreed trial. Inspect the business result in both applications before releasing a large backlog. Then test a more difficult case, such as a revised order or a partly completed route. One simple success does not prove every historic failure had the same cause or can use the same recovery action.

Keep failure categories separate. Work blocked by expired access may resume after authorisation is repaired, while invalid product codes need substantive correction first. Record the groups intentionally left waiting. This staged approach avoids using the resolution of one technical issue as permission to send every old record through the same chain again, regardless of its previous effects, missing information or outstanding business decision.

Restore the backlog in a controlled order

Choose an order that respects dependencies. An order may require a confirmed customer mapping before it can be handled safely. During recovery monitor both progress and new incoming work, not just error counts. Respect the supplier’s current limits and destination capacity. Releasing everything simultaneously can produce another incident even when the original problem has been corrected and ordinary traffic would now succeed.

Show which items are confirmed, skipped, corrected or still uncertain. Reconcile source and destination references for the agreed period. Count business operations rather than successful execution attempts, because one operation can have several technical tries. Close the backlog only when the operational owner can assess the relevant final state and temporary manual actions are demonstrably not being performed again by resumed automation.

Improve the controls after the incident

Record the cause, the impact actually demonstrated and remaining uncertainty. Distinguish an external supplier outage from an internal design weakness that made restoration harder. The trigger may be external while the integration lacks a usable destination-reference history. Those findings need different owners and improvements. A blame-oriented summary obscures practical tasks that could make the next interruption easier to understand.

Select a small number of concrete actions, such as an expired-access test, a useful recovery queue or a missing-order reconciliation. Assign an owner and observable outcome to each. Reconstruct the incident as a test for later releases without copying credentials or unnecessary personal information. The objective is earlier detection and safer recovery, rather than an unsupportable promise that no connected service or business process will ever fail again.

Turn the explanation into a project decision

Record your assumptions, unknowns and acceptance checks in the matching tool. Use the service guides to discuss delivery and scope.

• API integration ↗
• CRM integration ↗

Questions about this topic

Continue with a related question