Phaseo - Increased Error Rates – Incident details

Experiencing partially degraded performance

Increased Error Rates

Resolved
Operational
Started 17 days agoLasted about 8 hours

Affected

Phaseo Backend

Operational from 1:35 PM to 2:44 PM, Degraded performance from 2:44 PM to 12:00 AM

Cloudflare (API Provider)

Operational from 1:35 PM to 2:44 PM, Degraded performance from 2:44 PM to 12:00 AM

Workers KV

Operational from 1:35 PM to 2:44 PM, Degraded performance from 2:44 PM to 12:00 AM

Updates
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    Traffic engineering efforts have successfully mitigated the majority of congestion and packet drops.

  • Update
    Update

    Traffic engineering efforts have successfully mitigated the majority of congestion and packet drops. Services are largely stable, with only minor residual impact remaining. A small number of intermittent errors may still be observed for services with origins in North America as we work to fully clear the remaining load.

  • Update
    Update

    Our Engineers are continuing our traffic engineering work to mitigate the impact.

  • Update
    Update

    Our network teams are continuing active traffic engineering efforts to reroute traffic away from congested links and restore full service stability. We are closely monitoring dashboards and customer reports to confirm recovery and will continue to provide status updates as we manage the remaining load.

  • Update
    Update

    We are continuing our traffic engineering work to mitigate the impact.

  • Update
    Update

    We are investigating a fiber cut experienced by a third party network provider in Eastern North America. Customers connecting through North America or accessing services in Europe may see increased latencies and timeouts as Cloudflare engineers look to mitigate.

  • Update
    Update

    We are continuing to work on a fix for this issue.

  • Update
    Update

    We are continuing to work on a fix for this issue.

  • Identified
    Identified

    The issue has been identified and a fix is being implemented.

  • Investigating
    Investigating

    Cloudflare is investigating increased error rates and latency in multiple services beginning 13:35 UTC.