Incident started at:  September 12th, 9:52 AM PST
Work around applied: September 12th, 10:20 AM PST

Total Outage Duration: 0h 28m

Summary:

On September 12th, 9:52 AM PST we identified that some of F5 Distributed Cloud customers were facing login issues. We started investigating and observed that one of the micro-service responsible for decrypting the client secret information was crashing due to over memory utilization.

Root cause:

The root cause was traced back to the backend job, during the schedule maintenance window, to tune backward compatibility. The job caused over memory utilization on one of the micro-service responsible for decrypting the client secret information and forced it to crash.

The incident was detected on September 12th, 9:52 AM PST, prompting an immediate investigation by the F5XC Team. By 10:20 AM PST, corrective measures were applied by increasing the memory resource and the login service was restored.

Incident flow:

Corrective measures:

• The identified backend job has been tuned and tested for causing over memory utilization.
• Process change has been put in place to perform stress test on these micro-service post bulk operations
• We are also making changes in our observability stack to ensure we have traces to allow issues to be pinpointed faster.