Scale and Optimization Problem in the Production Move of Signoz
TLDR Simran had problems with scaling Signoz and tried a distributed shards setup with multiple replicas, but faced issues. Prashant confirmed that multiple replicas are not supported and suggested multiple shards of ClickHouse and multiple replicas of SigNoz Otel-Collector as a solution.
Sep 09, 2023 (2 months ago)
Wanted to check if this is same with the current version of Signoz as well ?
Plus any leads towards optimizing Signoz's clickhouse for arounds 600k - 1millions span per minute setup will be helpfull. Based on what we are seeing, signoz collector is facing throttle to write to clickhouse.
Indexed 1023 threads (61% resolved)
Issues with SigNoz Setup and Data Persistence in AKS
Vaibhavi experienced issues setting up SigNoz in AKS, and faced data persistence issues after installation. Srikanth provided guidance on ClickHouse version compatibility and resource requirements, helping Vaibhavi troubleshoot and resolve the issue.
Dashboard Load Issues and Possible Solutions
Al experiences dashboard loading issues since updating to `0.18.1`. Srikanth believes the issue is not version related and suggests examining queries, memory resources, and server distribution for improvements.
Configuring Signoz Query Service for Clickhouse
An asked how to configure Signoz for multiple shards or replicas in Clickhouse. Varaprasad confirmed that it's possible by providing Clickhouse host and IP for the query service.