This incident has been resolved. All affected clusters have resumed normal processing.
Posted Mar 21, 2024 - 13:54 EDT
Monitoring
Messaging latency has cleared across US clusters, minus US-01. Engineers continue working to clear messaging backlogs for US-01.
Posted Mar 21, 2024 - 13:44 EDT
Update
REST API errors on US-01 have subsided to normal thresholds and are stable. Engineers continue working to clear all message backlogs on all on US clusters, some latency may be experienced as clusters continue to recover.
Posted Mar 21, 2024 - 13:23 EDT
Update
It has been observed that REST API on US-01 has begun experiencing elevated error rates. Engineers continue working to restore service.
Posted Mar 21, 2024 - 13:02 EDT
Identified
Engineers have identified the issue causing elevated latency and are working to restore service. We are seeing improvement in message sending and backlogs have begun clearing across the impacted US clusters.
Posted Mar 21, 2024 - 12:14 EDT
Investigating
We have identified an issue leading to increased messaging latency across our US clusters, with the exception of US-08. Our team of engineers is currently investigating the issue.
Posted Mar 21, 2024 - 12:01 EDT
This incident affected: US 01 Cluster (REST APIs, Outbound Messaging), US 02 Cluster (Outbound Messaging), US 03 Cluster (Outbound Messaging), US 06 Cluster (Outbound Messaging), US 05 Cluster (Outbound Messaging), US 07 Cluster (Outbound Messaging), and US 04 Cluster (Outbound Messaging).