Resolved -
After more work over the past several hours, it appears that search for all carriers have recovered to around previous levels and other auxiliary functions such as callback delivery have resumed normal flow and delivery latency. As always, we will run an internal post-mortem process to harden and improve system stability going forward.
Feb 11, 16:21 UTC
Update -
Search success has stabilized, but has not fully reached previous levels in some cases. We will monitor over the next 24 hours and work on any specific carrier-system combinations that need may additional focus as we continue to mitigate the infrastructure provider issue.
Feb 11, 03:55 UTC
Update -
After monitoring, it appears that most traffic has been restored to normal. Callbacks may be impacted for customers who have whitelisted our IPs, as we have to migrate traffic through different exit nodes than usual; we continue to find a mitigation approach to ensure customers receive callbacks.
Feb 11, 01:28 UTC
Monitoring -
A networking provider is fully down, preventing typical traffic flows. We've deployed a workaround to the issue, and will continue to monitor. Normal traffic flow should resume, although there may be a knock-on effect with success rates for specific carriers.
Feb 11, 00:42 UTC
Update -
There appears to be an issue with an infrastructure provider, and will deploy a workaround shortly that should mitigate the issue.
Feb 10, 23:17 UTC
Investigating -
We are investigating an issue that is impacting our ability to deliver search results in a timely manner. We are currently determining the scope and source of the issue.
Feb 10, 21:05 UTC