On Monday, March 11th Semgrep Cloud Platform experienced a large, unexpected increase in traffic that uncovered a misconfiguration in one of our web services. This misconfiguration meant that we were not caching static data, resulting in our web service handling 10-15x its normal traffic. As traffic was spiking, the web service hosts were unable to handle the dramatically increased load, and began to crash loop. After configuring caching on our static data, our services quickly exited their crash loops and began to operate as normal.The Semgrep Engineering Team has performed a postmortem since the incident and have determined a number of action items that will help prevent and mitigate issues like this in the future.

1. Configure all static data to be cached through our CDN
2. Implement improved metrics and alerting at multiple levels of our infrastructure
3. Revaluate resource allocation for our critical web services