Api are having Performance degradation
Incident Report for Frontegg
Postmortem

Executive summary:

On August 15th, 2022 at 02:01 IST (UTC +2) Frontegg underwent a sophisticated DDOS subdomain organized attack. The attackers used multiple servers spread across a variety of Digital Ocean IPs. Each Server executed a low number of requests per second so our WAF did not trigger rate-limiting rules, yet it was recognized that many of the paths were related to WordPress engine's known weakness. 

By 03:21 the attack had been successfully mitigated. At 04:46 a second organized attack began. The restrictions put in place by the previous attack were helpful in mitigating the second attack. By 05:30 all traffic returned to normal

Affect:

The incident caused a degraded performance to our API gateway. As a result, our API returned 504 and 524 errors to partial traffic over the course of the incident. The majority of these errors were returned between 02:01 IST and 02:30 IST, when our mitigation efforts began to have an effect. A majority of traffic was still able to go through without error during this time.

Mitigation and resolution:

Our initial response to the attack was to increase our rate limiting and WAF constraints. This initial step was implemented at 02:30 IST. Once we understood the level of sophistication and distribution of the attack, we implemented changes on the application level, including a different routing mechanism and added more specific WAF constraints based on origins of the attacking traffic, which took effect by 03:21 IST.

Preventive steps:

In order to prevent attacks like this in the future, we are implementing a more sophisticated route blocking mechanism to our API-gateway. Additionally we have reported the incident to the cloud provider which hosted a majority of the attacking traffic, and we are consulting with our WAF provider for further guidance on preventing such attacks.

Posted Aug 16, 2022 - 15:05 IDT

Resolved
This incident has been resolved.
Posted Aug 15, 2022 - 03:39 IDT
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Aug 15, 2022 - 03:24 IDT
Update
We are continuing to investigate this issue.
Posted Aug 15, 2022 - 03:23 IDT
Update
We are continuing to investigate this issue.
Posted Aug 15, 2022 - 03:00 IDT
Investigating
We are currently investigating this issue.
Posted Aug 15, 2022 - 02:54 IDT
This incident affected: User authentication, Machine to machine authentication, SSO & SAML authentication, Audit logs, and Management portal.