SigNoz Deployment Issue with External ClickHouse and PVC

Ragulraj

Hi team i was deploying signoz v0.18.1 throught k8 , all service were running except otel-collector and otel-collector metrics . i got logs as Error: cannot build pipelines: failed to create "clickhousetraces" exporter, in pipeline "traces": error connecting to primary db: code: 170, message: Requested cluster 'cluster' not found 2023/04/19 10:15:30 application run finished with error: cannot build pipelines: failed to create "clickhousetraces" exporter, in pipeline "traces": error connecting to primary db: code: 170, message: Requested cluster 'cluster' not found . as i checked in clickhouse DB for clusters , there is some cluster list , how to get the cluster name whick otel-collector is looking

Srikanth

Are you using external ClickHouse with SigNoz helm?

Syed

in case of external clickhouse, please set your clustername as *cluster*

Syed

I faced similar kind of issue.

Ragulraj

i am running all the service in k8 cluster as individual pod with this otel docker image signoz/signoz-otel-collector:0.66.7

Ragulraj

clustername in which file i should configure , in clickhouse ? can you share me the file name which i need to change plz ?

Syed

Let me share the repo link Ragulraj

Syed

Just clone this repo and do docker-compose up -d, This will create 6 nodes based clickhouse server on a single node. I am still testing thats why I am using single node. Cluster name is set in macros.xml and remote_server.xml

Srikanth

What did you use to deploy to Kubernetes? If you use SigNoz helm charts, it automatically sets it up. If not, you need to make sure there is a cluster named `cluster` in CickHouse.

Ragulraj

No i am not using signoz helm charts , this is k8 deployment file i prepared for clickhouse

Ragulraj

f not, you need to make sure there is a cluster named `cluster` in CickHouse. this in which file i need to configure ?

Ragulraj

in this below files i couldnt get clusters config

Srikanth

If you hand-roll your own manifests, it will not be possible for me to help since I don’t have any idea what your config will look like.

Ragulraj

based on the docker-compose.yaml from signoz docs , i created a k8 deployment for each service

Ragulraj

my k8 deployment will be almost similar like docker-compose.yaml

Ragulraj

in clickhouse DB there is no cluster named "cluster"

Ragulraj

Hi Srikanth i was trying now to setup signoz with SigNoz helm charts provided in docs

Ragulraj

few pods were running some were in init , seems there is issue in pvc connecting with pv

Ragulraj

the pvc are in pending state

Srikanth

Prashant can help with the helm deployment

Prashant

It case of external ClickHouse, there are some pre-requisites for external clickhouse: • UDF script and histogram quantile binary needed in clickhouse: and • must have at least one zookeeper instance • distributed cluster named `cluster` set up required: • user credentaials passed must have enough privileges to create/manage databases in CH • secure connection not supported between clickhouse and signoz components yet

Prashant

Ragulraj It is likely that you are missing zookeeper as well as including a distributed cluster named `cluster` in clickhouse config.

Ragulraj

Hi Prashant as we are facing many difficulties in external clickhouse , i started a new installation of signoz through signoz helm

Syed

Ragulraj I am using external clickhouse, I also shared my repo to install, I am not facing any issue with external clickhouse

Ragulraj

the repo you shared is for docker-compose , i am using k8

Syed

docker-compose to install clickhouse server. My signoz is running on AWS EKS

Syed

clickhouse server is currently running on ec2 and all other signoz resources are deployed on eks cluster

Ragulraj

Ok acutally i am moving with signoz helm now , for easy understand

Ragulraj

Prashant Syed here pods were still in pending

Syed

These are my pods, Ragulraj check your pv and pvc

Ragulraj

my pvc is still in pending

Syed

I think this is not related to signoz issue, check how you are provisioning pv. In my case we are using dynamic storage class for provisioning PV.

Ragulraj

do i need to create any pv manually

Syed

Try to create PV and bind it pvc manually

Syed

Yes if you do not have storage class

Syed

for dynamic provisioning

Ragulraj

Ok, what type of storage class i should give in my PV

Ragulraj

if you have any sample can you share me please ?

Syed

where your k8s cluster is running

Ragulraj

it is running in OCI server

Syed

Syed

check this documentation

Ragulraj

Ok i will check with this , thank you so much

Ragulraj

and you created individual pv for the pvc created from signoz helm am i right ?

Syed

no, my pv got automatically created when I defing volumeclaimtemplate

Ragulraj

where you defined volume claim template , pv created automatically means by helm ?