Plan how your automation fails and recovers

Describe failure in business terms

Start with the intended outcome: one accepted inquiry creates one owned follow-up task, or one approved order creates one dispatch request. Then list how that outcome can become missing, late, duplicated, incomplete or attached to the wrong record. A technical message such as “HTTP 500” is evidence, but it does not say whether the external system performed the action before its response was lost.

Distinguish a definite rejection from an uncertain result. Invalid data may be rejected before any action occurs. A timeout may leave you unable to tell whether the action completed. Retrying both cases in the same way can create duplicates. Preserve a stable identifier for the intended business action and use it to check existing results or supply an idempotency key when the receiving service supports one.

Create a failure-mode register

Use one row for each important step and failure. Prioritize cases by business impact and recoverability rather than filling a register with every possible technical error.

• Step and intended outcome: Which record should change, once, and what evidence proves it?
• Failure mode: Can the action be rejected, time out, repeat, arrive out of order or partially complete?
• Detection: Which status, reconciliation count, age threshold or operator report reveals the problem?
• Retry rule: Is retry safe, how many attempts are allowed and what delay or provider instruction governs them?
• Duplicate protection: Which stable business identifier prevents or exposes a repeated action?
• Alert and pause owner: Who investigates, who may disable the workflow and what information do they receive?
• Manual queue and evidence: How does work continue, and how will an operator prove that recovery restored the intended state?

Retry only when another attempt can help

Microsoft’s retry pattern distinguishes transient faults from failures that require another response. A brief connectivity problem or a documented rate limit may justify delayed retry. Invalid credentials, rejected content or a missing required field needs correction or escalation. Use the service’s documented response codes and retry guidance where available. Cap attempts, add delay and jitter where appropriate, and stop retrying when the action’s useful time window has passed.

Google’s SRE guidance warns that retries can amplify overload. If every waiting job retries immediately, the recovering service receives more work than before the failure. Set concurrency and retry budgets, honor provider guidance and give operators a pause control. A circuit breaker or queued release can be useful at larger volumes, but even a small workflow needs a written answer to “When do attempts stop, and who decides to resume?”

Hypothetical example: task creation times out

Consider a hypothetical facilities company whose workflow creates one inspection task after a request is approved. Each intended task uses the approval record identifier as its external reference. One afternoon, the task system times out after receiving a request. The automation records the attempt as uncertain rather than failed and does not immediately create a second task.

The recovery worker queries for the external reference. If the task exists, it links that task and marks the outcome reconciled. If it does not exist, the worker retries after the documented delay. After the retry limit, the item moves to a visible manual queue and alerts the duty owner. A manual operator searches the same reference before creating anything and records the resulting task identifier. This is a hypothetical planning example, not a Kinetivy client result.

Write the operator runbook around decisions

A useful alert identifies the workflow, affected business reference, last confirmed state, safe next check and owner. Keep personal or confidential field contents out of broad alert channels and logs. Link to an access-controlled record when detail is needed. Group repeated symptoms into an incident where possible so operators do not perform conflicting recovery steps on hundreds of individual notifications.

The runbook should explain how to pause new work without losing accepted inputs, inspect pending and uncertain items, choose retry or manual completion, and reconcile after service returns. State what an operator must never do, such as blindly replaying all timed-out payments or deleting failed records before evidence is captured. Test that a backup owner can follow it without knowledge held only by the original builder.

Use this recovery and reconciliation checklist

Exercise failures with isolated sample records before relying on the workflow. Recovery has finished when source and destination agree on every intended action and unresolved items have an owner. A green automation status alone does not establish that result.

• Trigger a definite rejection, transient failure, timeout with uncertain outcome and repeated delivery.
• Confirm retry follows documented limits and does not retry permanent validation or authorization errors.
• Verify the stable identifier prevents or exposes duplicate business actions.
• Pause processing, continue the critical work manually and resume without losing or repeating queued items.
• Check alert ownership, protected diagnostic detail and escalation when the primary owner is unavailable.
• Reconcile counts and identifiers across the source, queue and destination; retain evidence of manual changes.
• Record the cause, recovery decision and change needed before closing the incident.