Designing hybrid cloud connectivity for failure—not only for day one

A diagram showing two connections can create false confidence. Both paths may depend on the same carrier, building entry, router, firewall, cloud location, DNS service or person authorised to make a change. Resilience must be assessed as an end-to-end behaviour.

Define the service outcome first

Establish how much disruption the business can tolerate and which services require continuity. Not every workload needs the same recovery behaviour. The target may range from manual recovery within hours to automatic convergence within minutes.

The design should distinguish availability of the network path from availability of the application and all services it depends on.

Map real failure domains

Trace each path through customer edge, cabling, facility, carrier, exchange, cloud attachment, gateway, firewall and region. Confirm which elements are genuinely independent. Provider product names do not prove physical or organisational diversity.

Where full diversity is not practical, record the shared dependency and its business impact.

Specify routing convergence and policy

Document route advertisements, filters, preference, timers, graceful-restart behaviour and fallback conditions. Validate what each side will do when a session remains up but the service beyond it is unavailable.

A routing protocol can detect a peer failure, but it may not detect every upstream or application failure. Health signals and operational intervention may still be required.

Check stateful inspection and symmetry

When firewalls or other stateful services are present, a failover path can create asymmetric traffic or invalidate session state. The architecture should show how both directions traverse the intended inspection layer and what session impact is expected during convergence.

Treat DNS and traffic management as part of failover

Confirm how names resolve during site, region and connectivity failure. Review caching, TTLs, health checks, authoritative services and split-horizon behaviour. A network path can recover while clients continue using an unavailable endpoint.

Include providers and operational dependencies

List lead times, support contracts, escalation routes, access permissions and change processes. A technically redundant design can still fail operationally when no one owns the carrier ticket, cloud configuration or firewall change.

Test scenarios, not only components

Create a small scenario matrix covering circuit loss, customer-edge failure, routing-session loss, cloud attachment failure, firewall failure, DNS dependency loss and region unavailability where relevant.

For each scenario, define the trigger, expected path, expected impact, detection evidence, validation steps, rollback or recovery action, and accountable owner.

Turn design intent into operational readiness

Provide runbooks, monitoring, dashboards, alarms and escalation contacts. Test during a controlled window before a critical migration. Re-test after material changes to routing, security or provider connectivity.

Resilience is an observable behaviour

The strongest design is not the one with the most redundant icons. It is the one whose important failure paths are understood, monitored, tested and owned. The evidence should exist before the first real outage supplies it.