Postmortem -
Read details
Jul 3, 01:59 UTC
Resolved -
The F5 Distributed Cloud team has successfully concluded the monitoring stage, and the Distributed Cloud Console Core services, along with traffic processing, remained in steady state. We appreciate your patience and understanding during this time. Our team is advocating for continuously improving our services for you. Thanks again for your trust in F5.
Jul 3, 01:06 UTC
Update -
Our service remains fully operational, and we are actively monitoring the environment to ensure its ongoing reliability. For any concerns or issues, please contact F5 Support for assistance
Thank you for your patience and understanding
F5 Distributed Cloud
Jul 2, 19:00 UTC
Update -
Our service remains stable and we will continue to monitor the environment to ensure service stability. If you experience any issue, please contact F5 Support for further assistance. Thank you.
Jul 2, 07:03 UTC
Update -
Dear customer,
F5 knows that this event impacted the operational and reputational commitments made for services reliant on our platform. We are committed to correcting the controllable elements to avoid similar lapses moving forward. F5 strives to provide the highest quality of service for our customers, and we are thankful for the opportunity to partner with you to improve the services you have entrusted to us to operate on your behalf. As part of service continuity, the F5 Distributed Cloud team will extend our proactive monitoring to ensure all services remain fully operational. In the meantime, we would like to share important information associated with the preliminary findings of the incident.
WHAT HAPPENED?
EVENT DESCRIPTION
On 2025-06-30 at 06:08 UTC, the F5 Distributed Cloud Support team started responding to multiple monitoring alerts and customer reports regarding difficulties accessing various websites and applications. Upon investigation, the team identified issues with endpoint configurations within the Regional Edge (RE) network, which resulted in a disruption of traffic processing functionality. Customers were provided with a manual solution (https://my.f5.com/manage/s/article/K000152246) to serve as an immediate solution while we continued troubleshooting.
Preliminary investigation revealed that the issue was caused after the execution of a scheduled maintenance to upgrade F5 Distributed Cloud Global Controller software to the latest version.
During troubleshooting efforts, the F5 Distributed Cloud team initiated a reboot of the configuration propagator service. Following the reboot, synchronization for the affected tenants resumed successfully, restoring services by 19:10 UTC on June 30, 2025. F5 will maintain ongoing monitoring of the environment to ensure the incident is fully resolved and to promptly identify and address any potential impacts or unrelated issues that may arise.
IS THE SERVICE EVENT FULLY RESOLVED?
Yes, Distributed Cloud Traffic processing has been restored for all impacted customers. All Distributed Cloud Console core services are operational. The F5 Distributed Cloud team continues to monitor the situation to ensure service stability.
WHAT HAPPENS NEXT?
ONGOING INVESTIGATION
Preliminary investigation ascertains the issue triggered after implementation of a scheduled maintenance to upgrade F5 Distributed Cloud Global Controller to the latest software version (https://www.f5cloudstatus.com/incidents/97pw5jnc6w7s). We will have a detailed Root Cause Analysis (RCA) once the root cause has been identified.
As of now, upcoming updates towards the monitoring outcome will be shared twice a day to provide relevant information on the stability of our XC Console services. Feel free to reach out to our XC support team if you’re facing any service issues or have questions. We’re here to help and support you.
We appreciate your patience and understanding during this time. Your satisfaction is our top priority, and we are dedicated to continuously improving our services for you. We are implementing additional measures to ensure this does not happen again.
Best regards,
The F5 Distributed Cloud Team
Jul 1, 21:25 UTC
Update -
Our services remain operational as we continue to monitor the situation in order to ensure ongoing service stability. If you experience any issue, please contact F5 Support for further assistance.
Jul 1, 05:23 UTC
Monitoring -
Dear customers:
The F5 Distributed Cloud support team has identified and fixed the issue affecting XC vips.
We will maintain proactive monitoring to ensure ongoing service stability. If you continue to experience any issues, please contact F5 Support for further assistance.
Jun 30, 20:45 UTC
Identified -
Dear customers:
We strongly recommend implementing the fix outlined in the following article: https://my.f5.com/manage/s/article/K000152246.
Updating the description on the load balancer, as described in the article, will trigger the config synchronization, effectively addressing the issue.
Please note that this incident may be impacting other customers as well, which could result in delays in support responses from our SOC team. To help us prioritize urgent cases, we kindly ask that you reach out to support only if the recommended fix does not resolve the problem.
For customers seeking further reassurance about routing traffic back, we can confirm that other customers who have applied the fix have experienced stable outcomes. As an additional measure of precaution, we recommend waiting until the incident is declared fully service restored before routing traffic back through the load balancer to ensure service stability to your operation.
Jun 30, 17:55 UTC
Update -
Please note that the workaround recommended in our previous updates is designed to trigger a configuration sync, which will help in resolving the issue. Kindly proceed with implementing the change as outlined in the link below:
https://my.f5.com/manage/s/article/K000152246
Updating the description on the load balancer will initiate the synchronisation of the configuration database with our central database.
Please contact support if you require any assistance in implementing the workaround.
Jun 30, 14:07 UTC
Update -
Please note that the workaround has been reinstated and customers can proceed with applying the change as per the link below:
https://my.f5.com/manage/s/article/K000152246
Please contact support if you require any assistance to implement the workaround.
We are committed to keeping you informed with updates as they become available. Thank you for your patience and understanding.
Jun 30, 11:41 UTC
Update -
Please note that we encountered a temporary issue, and the current workaround is not performing as expected. We continue to work tirelessly in resolving the issue and will share further updates once the workaround has been reinstated.
Jun 30, 10:00 UTC
Update -
As mentioned in our last update, customers can perform a no-impact configuration change on the load balancer and the origin object to restore the services as a workaround.
Please refer to this link below for more information regarding the implementation of the workaround:
https://my.f5.com/manage/s/article/K000152246
Jun 30, 09:28 UTC
Update -
The F5 Distributed Cloud support team is actively working to resolve this issue. In the meantime, as a workaround, customers can perform a no-impact configuration change on the load balancer and the origin object to restore the services.
We are committed to keeping you informed with updates as they become available. Thank you for your patience and understanding.
Jun 30, 08:38 UTC
Update -
The F5 Distributed Cloud support team has identified an issue with some of the distributed cloud endpoint configurations. Our team is actively working to resolve this incident, we are committed to keeping you informed with updates as they become available. Thank you for your patience and understanding.
Jun 30, 07:53 UTC
Investigating -
This advisory is to inform you that we are currently investigating reports of service degradation affecting some of our services.
We understand this may be affecting your operations and we are committed to providing transparent and timely updates as more information becomes available. While we gather more information, we recommend monitoring the progress on this site (https://www.f5cloudstatus.com/) for the latest update.
Investigation Status:
• We have identified potential service impact at 07:00 UTC.
• Our incident response team has been fully mobilized;
• Initial investigation and impact assessment efforts are underway.
Next Steps:
• A detailed incident notification will be provided within 30 minutes;
• Our teams are working to determine the root cause of the incident;
• We will share mitigation steps as soon as they become available.
We appreciate your patience as we work to resolve this situation. If you are experiencing a critical business impact, please contact our support team through your established channels.
Thank you for your continued support and trust in F5.
Jun 30, 07:18 UTC