Traffic Impacting Event
Incident Report for F5 Distributed Cloud
Postmortem
Incident Flow Timeline
Incident Started November 8th, 15:20 UTC
Incident Stopped November 8th, 16:05 UTC
Incident Started November 8th, 18:09 UTC
Incident Stopped November 8th, 18:014 UTC

Summary:

On November 8th, 15:20 UTC and November 8th, 18:09 UTC we identified a large denial of service attack on F5 Distributed Cloud Platform.  This caused a reduction in performance for some of the core services ultimately impacting a small number of customers. DDoS mitigation functions were immediately deployed.

Root cause:

The root cause was denial of service attack on the platform focused on layers 3, 4 and 7 resulted degraded performance for small number of customers.

A number F5 Distributed Cloud DDoS protection were used to mitigate the problem. Tools like ACLs, Rate Limiting, Geo Blocks and TLS Fingerprints.

‌The degraded performance was restored, and the incident was mitigated on November 8th, 18:014 UTC.

Incident flow:

Incident Time Description
November 8th, 15:20 UTC Our monitoring system detected large denial of service attack
November 8th, 15:25 UTC Our monitoring system detected high resource consumption on core platform services.
November 8th, 15:28 UTC F5XC Team identified the root cause as DDoS attacked and started mitigating it
November 8th, 16:05 UTC The attack was mitigated
November 8th, 18:09 UTC We observed another peak in the attack
November 8th, 18:014 UTC The attack was mitigated

Corrective measures:

  • Making our observability stack more equipped to ensure we have traces to allow issues to be pinpointed faster.
Posted Nov 16, 2023 - 00:35 UTC

Resolved
The F5 Distributed Cloud network experienced two traffic loss periods due a volumetric event on our network. These events occurred between 15:20 to 16:05 UTC and 18:09 to 18:14 UTC. These events have concluded and we have put mitigations around the source of the traffic to prevent further impact.

If you continue to experience any traffic impact please open a ticket with the F5 Distributed Cloud Support team from your console.
Posted Nov 08, 2023 - 15:30 UTC