#support

Scale and Optimization Problem in the Production Move of Signoz

TLDR Simran had problems with scaling Signoz and tried a distributed shards setup with multiple replicas, but faced issues. Prashant confirmed that multiple replicas are not supported and suggested multiple shards of ClickHouse and multiple replicas of SigNoz Otel-Collector as a solution.

2mo

Join the chat

Sep 09, 2023 (2 months ago)

Photo of md5-75fcf9dbcf7e4bda919dd9b8c66c8256

Simran

10:40 AM

#support Recently we moved Signoz to production but faced issue with scale. Hence we tried to moved to distributed shards setup with 2 replicas and 3 shards. But I can see multiple replicas with shards is not supported as mention here - https://signoz.io/docs/operate/clickhouse/distributed-clickhouse/

Wanted to check if this is same with the current version of Signoz as well ?

Plus any leads towards optimizing Signoz's clickhouse for arounds 600k - 1millions span per minute setup will be helpfull. Based on what we are seeing, signoz collector is facing throttle to write to clickhouse.

Photo of md5-1899629483c7ab1dccfbee6cc2f637b9

Prashant

06:36 PM

As mentioned in the shared docs, multiple replicas of ClickHouse cluster are not supported. But only multiple shards.

06:37

Prashant

06:37 PM

Adding multiple shard of ClickHouse (spread across multiple nodes) and multiple replicas of SigNoz Otel-Collector should help to handle high load.

SigNoz Community

Built with ClickHouse as datastore, SigNoz is an open-source APM to help you find issues in your deployed applications & solve them quickly | Knowledge Base powered by Struct.AI

Indexed 1023 threads (61% resolved)

Join Our Community

Similar Threads

Issues with SigNoz Setup and Data Persistence in AKS

Vaibhavi experienced issues setting up SigNoz in AKS, and faced data persistence issues after installation. Srikanth provided guidance on ClickHouse version compatibility and resource requirements, helping Vaibhavi troubleshoot and resolve the issue.

8mo

Dashboard Load Issues and Possible Solutions

Al experiences dashboard loading issues since updating to `0.18.1`. Srikanth believes the issue is not version related and suggests examining queries, memory resources, and server distribution for improvements.

7mo

Configuring Signoz Query Service for Clickhouse

An asked how to configure Signoz for multiple shards or replicas in Clickhouse. Varaprasad confirmed that it's possible by providing Clickhouse host and IP for the query service.

1mo