Affected
Degraded performance from 9:27 PM to 9:39 PM, Operational from 9:39 PM to 9:39 PM
- PostmortemUTCPostmortemUTC
Incident Postmortem: US Region 3 Performance Degradation
Incident Date: July 13th 2026
Severity: SEV-2 (Regional degradation, mitigated via failover — no full outage)
Status: Resolved via temporary failover; permanent fix pending upstream carrier repair (ETA 24h)
Author: [Your Name / Team]Summary
At approximately 04:20 CST, automated monitoring detected degraded performance on US Region 3 infrastructure. Root cause was traced to an upstream fiber cable break at our backend hosting provider's facility, which forced their systems onto a secondary circuit and introduced significant latency. Our team isolated Region 3 traffic, failed over remaining regions to a backup server, and coordinated with the hosting provider to move to a temporary secondary facility while the fiber break is repaired.
Timeline (all times CST)
Time
Event
04:20
Automated uptime/monitoring bot fires initial alert for degraded performance on US Region 3 servers.
04:20 – 04:25
On-call team begins monitoring the alert to confirm whether it's transient noise or a sustained issue.
04:25
Monitoring confirms sustained, significant degradation — latency spikes to ~1500ms (baseline is presumably sub-100ms; confirm your normal baseline for the report). Incident formally declared.
04:25 – 04:32 (est.)
Team isolates Region 3 from active traffic routing to prevent degraded connections from affecting dependent services.
04:32 – 04:40 (est.)
Remaining healthy regions failed over/re-pointed to backup server infrastructure as a precaution / load redistribution.
04:40 – 04:50 (est.)
Team engages primary hosting provider to report the anomaly and request root cause.
04:50 – 05:05 (est.)
Hosting provider identifies the issue as backend API latency, traces it to their own backend hosting dependency.
05:05 – 05:20 (est.)
Team escalates directly to the backend hosting provider. Provider confirms a fiber cable break upstream, currently running on a secondary (backup) circuit, which is the source of the added latency.
05:20 (est.)
Team requests temporary migration to a secondary facility to restore normal performance while the fiber repair is completed.
05:20 – Present
Operating on secondary facility / backup routing. Region 3 remains isolated from primary path pending fiber repair.
T+24h (est.)
Fiber repair expected to complete; provider will cut back from secondary circuit to primary, and we will fail Region 3 back from backup infrastructure to normal production routing.
Note: Timestamps marked (est.) are inferred from your sequence of events. Recommend pulling exact timestamps from your monitoring system (Datadog/PagerDuty/etc.) and hosting provider's incident ticket to replace these before this doc is finalized/shared externally.
Root Cause
A physical fiber cable break occurred at (or upstream of) our backend hosting provider's facility. This forced their infrastructure onto a secondary/backup fiber circuit, which introduced substantial additional round-trip latency. This latency propagated through their backend API layer, which our Region 3 servers depend on, resulting in end-to-end ping/latency of approximately 1500ms for affected traffic — well outside acceptable thresholds.
This was not a compute, application, or configuration failure on our side — it was a Tier 2 (hosting provider) → Tier 3 (backend hosting/carrier) dependency failure that cascaded upward.
Impact
Affected: US Region 3 traffic experienced degraded performance (~1500ms latency) for approximately [X minutes/hours — fill in from alert-to-isolation time, ~12 min based on the timeline above] before isolation.
Not affected: Other regions remained stable and were proactively failed over to backup infrastructure as a precautionary measure, not due to direct impact.
User-facing impact: [Fill in — e.g., "Elevated latency on API calls routed through Region 3; no reported downtime or data loss."]
Resolution / Current Mitigation
Region 3 isolated from primary routing.
Remaining regions failed over to backup server infrastructure.
Backend hosting provider temporarily relocating operations to a secondary facility.
Full resolution (fiber repair + cutback to primary circuit/facility) expected within 24 hours.
- ResolvedUTCResolvedUTCThis incident has been resolved.
- MonitoringUTCMonitoringUTC
We have reached out to our Hosting Provider and implemented a temporary fix. Service should be restored shortly. We are monitoring this situation to ensure resolvement.
- IdentifiedUTCIdentifiedUTC
Our team has discovered the root cause of the slower loading speed. We have reached out to our hosting provider for more information.
- InvestigatingUTCInvestigatingUTCWe are currently investigating this incident.