Exploring Solutions for Signoz Crashing Due to High Request Volume

TLDR Thomas is experiencing crashing of Signoz with high request volumes (~1 million traces/day), seeing an empty monitoring UI and occasionally failing pods. Srikanth suggested providing sufficient resources.

Photo of Thomas
Sun, 02 Jul 2023 18:27:41 UTC

Hey! My platform saw increased number of requests in the past weeks (~1 million traces/day, I think) and signoz keeps crashing. I log in to the monitoring UI, and everything is simply empty. Sometimes all services are up (according to the k8s dashboard), and sometimes some pods fail (otel-collector, otel-collector-metrics). Looking through logs, I see `The node was low on resource: memory. Container my-release-signoz-otel-collector was using 3589960Ki, which exceeds its request of 200Mi.` - is there a simple way to fix this if I installed signoz via helm?

Photo of Srikanth
Sun, 02 Jul 2023 23:54:47 UTC

The simple way to fix it is to give enough resources.