Direct answer
Reliable production systems assume dependencies can fail and outcomes can be ambiguous. Use bounded timeouts and retries, idempotency where possible, observable queues, reconciliation and explicit recovery paths.
In production, a failed response does not always mean a failed operation
A service may process a request but lose the response before it reaches the caller. The caller sees a timeout and retries. If the operation is not designed to handle duplicates, the retry may create a second payment, order or task.
This is a basic distributed-systems problem: the caller may not know whether the operation happened. Reliability depends on explicit state, safe retry behaviour and a way to reconcile uncertain outcomes.
Four mechanisms, four different purposes
| Mechanism | Purpose | Common mistake |
|---|---|---|
| Timeout | Bounds how long a caller waits | Setting it without considering downstream work and user expectations |
| Retry | Reattempts a transient failure | Retrying every error or retrying without a limit |
| Queue | Decouples producers and consumers and absorbs bursts | Treating accepted work as completed work |
| Idempotency | Makes repeated requests safe for a defined operation | Assuming a unique request ID alone guarantees correct behaviour |
| Circuit breaker | Limits repeated calls to an unhealthy dependency | Using it without a recovery and fallback plan |
| Dead-letter handling | Preserves work that cannot be processed normally | Allowing failed messages to accumulate without ownership |
| Reconciliation | Resolves uncertain or inconsistent outcomes | Assuming transport-level success proves business-level correctness |
Design the workflow around explicit states
- Start
Receive request with operation ID
Validate request and authority
Record operation state
Call dependency with timeout
- Decision
Outcome known?
- Success→ step 10: Persist confirmed result
- Transient failure→ step 6: Retry within bounded policy
- Ambiguous outcome→ step 7: Query or reconcile operation status
- Permanent failure→ step 8: Record failure and notify owner
Retry within bounded policy
↺ repeat from step 4
Query or reconcile operation status
- Outcome
Record failure and notify owner
- Decision
Outcome resolved?
- Yes→ step 10: Persist confirmed result
- No→ step 11: Escalate or retain for investigation
Persist confirmed result
- Outcome
Escalate or retain for investigation
- Outcome
Return verified status
This is a conceptual pattern. The correct implementation depends on whether the downstream service supports idempotency keys, status queries, transactions or compensating actions.
Retries need budgets and back-off
A retry policy should distinguish transient failures from permanent ones, cap the number of attempts and avoid synchronised retry storms. Exponential back-off with jitter is a common technique, but the exact policy depends on the service and workload. Retries should fit within the end-to-end latency budget rather than extending indefinitely.
Queues also require operational controls: message age, processing rate, retry count, poison-message handling and clear ownership of dead-letter queues. Otherwise, a queue can hide a growing backlog until users notice delayed work.
Measure recovery, not just uptime
Useful indicators include request success rate, latency percentiles, queue age, retry rate, dependency error rate, unresolved operations and time to recover. A system may appear available while critical business transactions are stuck.
Define recovery objectives appropriate to the business, test restoration and failover procedures, and record how operators can determine whether a transaction completed. For important operations, a reconciliation mechanism is often as valuable as the retry mechanism itself.
The Neuromotion perspective: resilient software assumes that dependencies will fail, responses will be ambiguous and recovery will be necessary. The engineering task is to make those conditions bounded, observable and recoverable.
Next step: Select one important workflow that crosses a system boundary. Document its timeout, retry, duplicate-prevention, reconciliation and escalation behaviour. If any answer is “we will handle it manually”, define who does so and how the unresolved state is detected.