Date and Time: July 24, 2024Duration: 22 minutes Affected Services: Authentication and core servicesImpact: Customers in the EU region were hanging and returned as 504 timeoutsReported By: Internal monitoring systems and customers
__________________________________________________________________________
Executive summary:
On Wednesday, July 24th, at 08:43 GMT, Frontegg's internal monitoring systems indicated that the API Gateway encountered an issue following the deployment of a new OpenTelemetry propagator (OTEL instrumentation), causing service disruptions in the EU. As a result, some of our customers were experiencing timeout errors (HTTP status 504) returned by Frontegg.
During the upgrade of our API Gateway, Frontegg also updated the OpenTelemetry library. This update inadvertently caused the system to send data one piece at a time instead of using efficient batches due to a misconfiguration in the data handling settings.
OTEL transmitted millions of traces individually rather than in aggregated batches. Although our system was rigorously tested under various conditions, the high load in the EU environment caused our auto-scaling mechanism to lag behind the incoming traffic. This led to the API gateway being overwhelmed by the volume of client requests.
Cause Analysis:
The primary cause of the incident was the deployment of a new OTEL instrumentation in the API Gateway, which led to a significant increase in trace data volume. Contributing factors included:
Customer Impact
During the incident, customers in the European region experienced significant service degradation. Specific issues included failures in hosted login monitors and general service instability.
Mitigation and resolution:
Upon receiving the initial alerts, the Frontegg team began investigating the issue promptly. After identifying the problem with the OTEL propagator and collector, we increased the allocated resources and reverted to the latest working version. Following the implementation of this change, the systems returned to normal operations.
Mitigation:
Resolution:
Prevention and Future steps: Enhance OTEL Propagator: Implement batch processing, asynchronous handling, and strict timeouts.
Communication:
Enhance Status Page Communication: Ensure the status page provides clear and timely updates during incidents. Develop and maintain standardized templates for incident communication to facilitate prompt and consistent information, even if the root cause is not immediately identified.