Integrations That Don't Fail: Retries and Idempotency

Integrations are like promises: easy to make, hard to keep. 70% of production incidents involve external integrations. The difference between a robust system and chaos lies in how you handle failures.

The Integration Problem

// ❌ Naive code that will fail
async function createUserInCRM(userData: User) {
  try {
    await crmApi.create(userData);
    await emailService.sendWelcome(userData.email);
  } catch (error) {
    // User lost in CRM, welcome email never sent
    console.error("Failed to create user in CRM", error);
    throw error;
  }
}

Pattern 1: Retry with Exponential Backoff

Never trust that a request will succeed on the first attempt.

class RetryableHttpClient {
  async request<T>(
    operation: () => Promise<T>,
    options: { maxRetries?: number; baseDelay?: number; maxDelay?: number } = {}
  ): Promise<T> {
    const { maxRetries = 3, baseDelay = 1000, maxDelay = 10000 } = options;

for (let attempt = 0; attempt <= maxRetries; attempt++) { try { return await operation(); } catch (error) { lastError = error as Error; if (attempt === maxRetries) throw lastError; const delay = Math.min(baseDelay * Math.pow(2, attempt), maxDelay); await new Promise(r => setTimeout(r, delay + Math.random() * 1000)); } } throw lastError!; } } `

Pattern 2: Idempotency

Idempotency means: running the same operation multiple times produces the same result as running it once.

// ✅ Idempotent with idempotency key
async function processPayment(paymentData: Payment & { idempotencyKey: string }) {
  const existing = await paymentRepository.findByIdempotencyKey(paymentData.idempotencyKey);

return await db.transaction(async (tx) => { const payment = await tx.payment.create({ data: { ...paymentData, status: 'processing' } });

try { await paymentGateway.charge(payment.amount); return await tx.payment.update({ where: { id: payment.id }, data: { status: 'completed' } }); } catch (error) { await tx.payment.update({ where: { id: payment.id }, data: { status: 'failed' } }); throw error; } }); } `

Pattern 3: Circuit Breaker

Prevent failure cascades when external services are unavailable.

class CircuitBreaker {
  private failures = 0;
  private lastFailureTime = 0;

async execute(operation: () => Promise): Promise { if (this.state === 'OPEN') { if (Date.now() - this.lastFailureTime > this.options.recoveryTimeout) { this.state = 'HALF_OPEN'; } else { throw new Error('Circuit breaker is OPEN'); } } try { const result = await operation(); this.failures = 0; this.state = 'CLOSED'; return result; } catch (error) { this.failures++; this.lastFailureTime = Date.now(); if (this.failures >= this.options.failureThreshold) this.state = 'OPEN'; throw error; } } } `

Pattern 4: Queue for Async Processing

Webhooks should be processed asynchronously to avoid timeouts.

Pattern 5: Dead Letter Queue

For failures that cannot be automatically recovered.

Conclusion

Robust integrations aren't about avoiding failures — they're about designing for failure. With retries, idempotency, circuit breakers, and async processing, you build systems that survive external chaos.

Remember: the network is unreliable. External APIs fail. Your system needs to be ready.

---

Need help implementing robust integrations? Talk to us about an integrations consultation.