SigNoz Production Feedback and Improvement Suggestions

@here those using SigNoz in production, can you please comment with your experience in a reddit thread going on? I think some folks need feedback. Also, feel free to share both friction points and useful things about SigNoz. We shall take the feedback seriously 🙂 https://www.reddit.com/r/devops/comments/11n272q/who_uses_signoz_in_production/

thanks for the feedback Apoorva. I feel them to be very correct. Will take measures to enable them in coming quarter

I do not have a Reddit account, do you want my feedback here?

yes Guillaume it will be helpful anyhow

Appreciated:
• Open source software
• Deploy method is easy to set up for Kubernetes
• Configuration is pretty smooth for the whole environment
• Community available on Slack and helping you on many situations
• Nice UI
• When we found what we need, graph is nice
Cons:
• Had some breaking changes with Helm charts upgrades (need to check the diff to be sure to avoid templating differences)
• UI/UX could be improved to find services or traces, because many users are lost (they are used to Kibana)
• Templating the other services like clickhouse is not perfect, some configurations do not work as expected (even though documentation says so), some options are missing (requests and limits, env variables).
• Not being developper, it is hard to understand what is wrong when you can't find your spans or it's missing some and the documentation is not helping much, you would need help from a developper to be able to test it properly.
• Service map is not really usable right now.
• Usage explorer is probably missing filters, else we can stick to OpenSearch.
• Exceptions is a nice view but we already have it with OpenSearch.
It's a fine software already, it's gonna be good no doubts.

Just got done doing a configuration and deployment for a client who is shifting to OTel and decided on SigNoz for displaying the data.

Going to be doing a more thorough analysis this week, but 3 things come to mind right now:

• We used a separate observability k8s cluster, and shipped all data from Dev, QA, and Prod to it. We added a source_env label at the gateway otel-collector, which allowed us to do visualizations per-environment, but there doesn’t appear to be a way to filter on the Services / Service Map screens. Ideally it would be nice to set up default filters for source_env: prod
• A major focus of our consulting is DevOps and automation. We automated the deployment of Signoz as well as the gateway/agent setups in observed clusters, but I didn’t see a way to automate the creation of users, visualizations, or alerts.
• Our client has a mix of Windows + Linux machines in k8s. Getting container metrics from Windows required replacing the extract_metadata_from_filepath regex, which should have been easy but per https://github.com/helm/helm/issues/8320 is actually very broken and required cloning all of presets.logCollection.operators in a values file.
◦ For reference, the regex we used was ^.*[\\\/]{1,2}(?P<namespace>[^_]+)_(?P<pod_name>[^_]+)_(?P<uid>[a-f0-9\-]+)[\\\/]{1,2}(?P<container_name>[^\._\\\/]+)[\\\/]{1,2}(?P<restart_count>\d+)\.log$ which might be OS-agnostic. For some reason the Windows file-paths were of the format \\var\\log\\… with 2 slashes as separators.

Ankit i am using signoz in production, but i have one issue that signoz container take to much memory so i can deployed separately on different server but i can't connect signoz(which is running on different server)with my app(which is running on different server). so can you please directed to me how i can connect?