TLDR surya was having issues with zookeeper and other services. Prashant suggested `nodeAffinity` settings adjustment and PVC recreation to resolve the volume node affinity conflict.
zookeeper is not working
Can't find the reason can anyone help me with this issue
ClickHouse in cluster mode requires zookeeper pod to be ready.
Can you share `kubectl pod describe` for the zookeeper pod?
Also, make sure you have sufficient resources in the cluster/machine.
kubectl describe my-release-zookeeper-0 pod
error: the server doesn't have a resource type "my-release-zookeeper-0"
```kubectl -n platform describe pod/my-release-zookeeper-0```
Warning FailedScheduling 4m40s (x228 over 4h12m) default-scheduler 0/4 nodes are available: 4 node(s) had volume node affinity conflict.
but it works properly till last week
why suddenly get this error
`volume node affinity conflict` error happens when the PVC used by the pod is scheduled on different zones/region.
It is an issue seen in kubernetes cluster with nodes from multiple zones/regions.
I had encountered this before as well. I resolved it by setting `nodeAffinity` for those components to match the same zone of the PVC.
if you do not care about data loss, you can go about deleting the PVC and restart statefulset pod(s).
It should spawn the new PVCs in same zone as that of the pods.
No, i need data
Prashant, Since zookeeper, alert manager and query service pvc where facing this issue, i won't face any metric data loss after deleting the affected pvc.
> Since zookeeper, alert manager and query service pvc where facing this issue, i won't face any metric data loss after deleting the affected pvc.
actually SQLite would be removed since it is attached to `query-service` statefulset.
That would mean data related to user credentials, alerts, and dashboards would be affected.
Yes i was supposed to sign up again
And recreated the dashboard using config json
yes, that is correct
btw, which cloud vendor are you using for managing K8s cluster?
In what frequency do you collect metrics and persist to db
> In what frequency do you collect metrics and persist to db
depends on your collection interval. By default it should be 30s for Hostmetrics and K8s Metrics.
Indexed 1060 threads
Built with ClickHouse as datastore, SigNoz is an open-source APM to help you find issues in your deployed applications & solve them quickly | Knowledge Base powered by Struct.AI