Issues Running SigNoz on Azure AKS with External ClickHouse DB

TLDR Nicolai faced an error while trying to run SigNoz with an external ClickHouse database on Azure AKS. Prashant suggested additional steps and the need for detailed documentation for using SigNoz with an external ClickHouse database.

Photo of Nicolai
Tue, 14 Mar 2023 13:28:44 UTC

Hi all :slightly_smiling_face: I'm in need of help, I'm trying to run Signoz on an Azure AKS cluster with an external clickhouse database. Everything seems to startup, but otel-collector and otel-collector-metrics throws an error `error connecting to primary db: code: 170, message: Requested cluster 'cluster' not found` I've set the `clusterName` and the `global.clusterName` . Most other settings are default. ```CLICKHOUSE_CLUSTER : signoz CLICKHOUSE_DATABASE : signoz_metrics CLICKHOUSE_HOST : *redacted* CLICKHOUSE_HTTP_PORT : 8123 CLICKHOUSE_PASSWORD : *redacted* CLICKHOUSE_PORT : 9000 CLICKHOUSE_SECURE : false CLICKHOUSE_TRACE_DATABASE : signoz_traces CLICKHOUSE_USER : admin CLICKHOUSE_VERIFY : false K8S_CLUSTER_NAME : dev-monitoring-aks K8S_NAMESPACE : fieldRef(v1:metadata.namespace) K8S_NODE_NAME : fieldRef(v1:spec.nodeName) K8S_POD_IP : fieldRef(v1:status.podIP) K8S_POD_NAME : fieldRef( K8S_POD_UID : fieldRef(v1:metadata.uid) LOW_CARDINAL_EXCEPTION_GROUPING : false OTEL_RESOURCE_ATTRIBUTES :$(K8S_NODE_NAME),signoz.component=$(SIGNOZ_COMPONENT),$(K8S_CLUSTER_NAME),k8s.pod.uid=$(K8S_POD_UID),k8s.pod.ip=$(K8S_POD_IP) SIGNOZ_COMPONENT : otel-collector``` Above are the env vars for the pod. The init container connects fine to the database and reports `clickhouse ready, starting otel collector now` . I'm using latest helm chart. Am I doing something wrong or missing some configuration?

Photo of Prashant
Tue, 14 Mar 2023 21:26:13 UTC

In case you would like to use external ClickHouse DB, you would have to follow additional steps. • at least one instance of zookeeper • create cluster configuration along with zookeeper remote server • copy `custom-scripts.xml` as well as binary inside `user-scripts` folder • make sure user credentials passed has enough privileges to create/access databases ^ For all these, you can refer here: cc <@4K143e>

Photo of Prashant
Tue, 14 Mar 2023 21:28:01 UTC

<@4K1456> we need detailed docs on how to use SigNoz with external clickhouse. Also, we should build both arm/amd binaries for the user-script.