Exploring Solutions for Signoz Crashing Due to High Request Volume
TLDR Thomas is experiencing crashing of Signoz with high request volumes (~1 million traces/day), seeing an empty monitoring UI and occasionally failing pods. Srikanth suggested providing sufficient resources.
Srikanth
Sun, 02 Jul 2023 23:54:47 UTCThe simple way to fix it is to give enough resources.
SigNoz Community
Indexed 1060 threads
Built with ClickHouse as datastore, SigNoz is an open-source APM to help you find issues in your deployed applications & solve them quickly | Knowledge Base powered by Struct.AI
View in App
Thomas
Sun, 02 Jul 2023 18:27:41 UTCHey! My platform saw increased number of requests in the past weeks (~1 million traces/day, I think) and signoz keeps crashing. I log in to the monitoring UI, and everything is simply empty. Sometimes all services are up (according to the k8s dashboard), and sometimes some pods fail (otel-collector, otel-collector-metrics). Looking through logs, I see `The node was low on resource: memory. Container my-release-signoz-otel-collector was using 3589960Ki, which exceeds its request of 200Mi.` - is there a simple way to fix this if I installed signoz via helm?