Issue with Clickhouse Cluster after Installing SigNoz with Helm
TLDR Saleem had issues connecting to the pods after installing SigNoz using Helm. Prashant suggested upgrading to the latest SigNoz chart '0.19.0', which resolved the issue.
Saleem
Fri, 14 Jul 2023 04:49:35 UTCclickhouse cluster pod is created and up but still not able to connect operator
`1 cluster.go:84] Run query on: chi-uc-care-monitor-clickhouse-cluster-0-0.uc-tools.svc.cluster.local of [chi-uc-care-monitor-clickhouse-cluster-0-0.uc-tools.svc.cluster.local]`
E0714 03:02:24.036026 1 connection.go:98] connect():FAILED Ping(
Saleem
Fri, 14 Jul 2023 04:49:41 UTCplease guid me
Prashant
Fri, 14 Jul 2023 11:39:42 UTCHi Saleem :wave: Could you please upgrade to the latest SigNoz chart `0.19.0` ? And see how it goes.
Saleem
Fri, 14 Jul 2023 15:23:08 UTCits working now. May I know the issue caused for it?
Prashant
Fri, 14 Jul 2023 15:23:54 UTCglad to know that.
Prashant
Fri, 14 Jul 2023 15:23:59 UTCIt was likely caused by this:
Saleem
Fri, 14 Jul 2023 15:25:49 UTCohh ok..Thanks alot Prashant :slightly_smiling_face:
Prashant
Fri, 14 Jul 2023 15:27:35 UTCYou are welcome :v:
SigNoz Community
Indexed 1060 threads
Built with ClickHouse as datastore, SigNoz is an open-source APM to help you find issues in your deployed applications & solve them quickly | Knowledge Base powered by Struct.AI
View in App
Saleem
Thu, 13 Jul 2023 18:36:41 UTCHI Team I have installed signoz with helm. Below pods are stuck in init:0/1 state signoz-alertmanager otel-collector otel-collector-metrics signoz-query-service current pod status is as shown below screenshot it seems clickhouse cluster is not created only two statefulsets signoz-alertmanager and signoz-query-service are created and are in init:0/1 state Below are logs of clickhouse-operator `I0713 18:24:35.520431 1 cluster.go:84] Run query on: chi-uc-care-monitor-clickhouse-cluster-0-0.uc-tools.svc.cluster.local of [chi-uc-care-monitor-clickhouse-cluster-0-0.uc-tools.svc.cluster.local]` E0713 18:24:35.531029 1 connection.go:98] connect():FAILED Ping(:***@chi-uc-care-monitor-clickhouse-cluster-0-0.uc-tools.svc.cluster.local:8123/). Err: dial tcp: lookup chi-uc-care-monitor-clickhouse-cluster-0-0.uc-tools.svc.cluster.local on 10.100.0.10:53: no such host
E0713 18:24:35.531214 1 connection.go:126] QueryContext():FAILED connect(:***@chi-uc-care-monitor-clickhouse-cluster-0-0.uc-tools.svc.cluster.local:8123/) for SQL: SELECT
DISTINCT name,
concat('SYSTEM SYNC REPLICA "', database, '"."', name, '"') AS sync_table_query
FROM
system.tables
WHERE
engine LIKE 'Replicated%'
W0713 18:24:35.531319 1 cluster.go:91] QueryAny():FAILED to run query on: chi-uc-care-monitor-clickhouse-cluster-0-0.uc-tools.svc.cluster.local of [chi-uc-care-monitor-clickhouse-cluster-0-0.uc-tools.svc.cluster.local] skip to next. err: FAILED connect(:***@chi-uc-care-monitor-clickhouse-cluster-0-0.uc-tools.svc.cluster.local:8123/) for SQL: SELECT
DISTINCT name,
concat('SYSTEM SYNC REPLICA "', database, '"."', name, '"') AS sync_table_query
FROM
system.tables
WHERE
engine LIKE 'Replicated%!'(MISSING)
E0713 18:24:35.531344 1 cluster.go:95] QueryAny():FAILED to run query on all hosts [chi-uc-care-monitor-clickhouse-cluster-0-0.uc-tools.svc.cluster.local]
I0713 18:24:35.531472 1 cluster_schemer.go:188] HostSyncTables():Sync tables: [] as []
I0713 18:24:35.531631 1 worker-deleter.go:521] deleteCluster():Delete cluster uc-tools/cluster - started
I0713 18:24:35.610996 1 deleter.go:292] deleteServiceCluster():uc-tools/cluster-uc-care-monitor-clickhouse-cluster
I0713 18:24:35.638329 1 worker-deleter.go:487] deleteShard():Delete shard uc-tools/0 - started
I0713 18:24:35.688622 1 deleter.go:279] deleteServiceShard():uc-tools/shard-uc-care-monitor-clickhouse-cluster-0
I0713 18:24:35.694098 1 worker-deleter.go:425] deleteHost():Delete host cluster/0-0 - started
I0713 18:24:36.030438 1 cluster.go:84] Run query on: chi-uc-care-monitor-clickhouse-cluster-0-0.uc-tools.svc.cluster.local of [chi-uc-care-monitor-clickhouse-cluster-0-0.uc-tools.svc.cluster.local]
E0713 18:24:36.035212 1 connection.go:98] connect():FAILED Ping(:***@chi-uc-care-monitor-clickhouse-cluster-0-0.uc-tools.svc.cluster.local:8123/). Err: dial tcp: lookup chi-uc-care-monitor-clickhouse-cluster-0-0.uc-tools.svc.cluster.local on 10.100.0.10:53: no such host
E0713 18:24:36.035254 1 connection.go:126] QueryContext():FAILED connect(:***@chi-uc-care-monitor-clickhouse-cluster-0-0.uc-tools.svc.cluster.local:8123/) for SQL: SELECT
DISTINCT name,