Issues with SigNoz Deployment on EC2 Instance

TLDR Praveen reported issues with SigNoz freezing on an EC2 instance and using all available disk space. Yash rV and Alexei provided insights on how to configure resources appropriately for it to run smoothly. Praveen decided to modify the setup and monitor the system.

Powered by Struct AI
Join the chat
Jun 28, 2023 (5 months ago)
Photo of md5-f51c0e60e6cc9a45b4e3ffbbbc2adc49
12:31 PM
I've setup signoz on an EC2 instance - t2.medium, 40-50 GB EBS. My issue stems from the fact that the system enters into a frozen state and I'm required to reboot the instance in order to ssh and restart Signoz. Signoz is the only "heavyweight" process on that instance. Any body else encounter this issue?
12:52 PM
After restarting, I'm also seeing that docker has used all 50GB on the device, how much space does signoz need?
Yash rV
Photo of md5-cf77c5ac54f90e38660b2ee9331d5457
Yash rV
02:22 PM
Hello Praveen,
I encountered similar issues when running it for the first time. However, I found a helpful article that provides information on determining the required memory for it to run. I recommend referring to that article for more details.


Photo of md5-d44b8ad79c1dd6595985e83066c4506b
02:28 PM
depends on your workloads and how many traces/metrics you send to it.

The minimum recommended specs for a clickhouse node is 16GB of RAM and 8CPUs I find.

For PROD I run a m6i.2xlarge with a 1.8TB GP3 EBS mount.
02:29 PM
I run all of the SigNoz services in ECS to not collocate where Clickhouse is running (the one which needs the most resources)
Photo of md5-f51c0e60e6cc9a45b4e3ffbbbc2adc49
02:32 PM
thanks Yash rV and Alexei! Your suggestions and links seem to refer to Kubernetes deployment. I'm deploying via a standalone docker instance and not leveraging Kubernetes at this time. I've just now turned off the default app and only keeping logs on the system for 1 day - I rely on ELK for logging so no use for SigNoz logs. I'll monitor and check if that helps