Resolving PodDisruptionBudgets Conflicts in Clickhouse Restart
TLDR Al encountered PodDisruptionBudgets conflicts on ClickHouse restart, which interfered with Kubernetes upgrade. Prashant assured that only one PDB should remain in the new chart. Al later confirmed successful remediation.
Jul 18, 2023 (4 months ago)
I notice when restarting clickhouse, the following events occur:
Pod "chi-signoz-clickhouse-cluster-0-0-0" matches multiple PodDisruptionBudgets. Chose "signoz-clickhouse-cluster" arbitrarily. Pod "chi-signoz-clickhouse-cluster-0-0-0" matches multiple PodDisruptionBudgets. Chose "signoz-clickhouse" arbitrarily.
This also seemed to interfere with upgrading kubernetes:
(UpgradeFailed) Drain of aks-node did not complete: Evict blocked by conflicting disruption budgets on pod chi-signoz-clickhouse-cluster-0-0-0 on node aks-node: chi-signoz-clickho use-cluster-0-0-0. See http://aka.ms/aks/debugdrainfailures Code: UpgradeFailed Message: Drain of aks-node did not complete: Evict blocked by conflicting disruption budgets on pod chi-signoz-clickhouse-cluster-0-0-0 on node aks-node: chi-signoz-clickhouse-cluster-0-0-0.
Jul 19, 2023 (4 months ago)
There should be only one PDB
Have you followed all steps from v0.23 migration docs before upgrading?
Indexed 1023 threads (61% resolved)
Increasing Persistent Volume Size in Test Environment
surya tried to increase the disk size in their test environment and encountered an issue. Prashant provided guidance and the user successfully increased the Persistent Volume (PV) for ClickHouse to 50Gi.
SigNoz crashing in k8s due to ClickHouse OOM
Travis reported SigNoz crashing in k8s due to ClickHouse OOM. The team suggested increasing resources for ClickHouse, and other troubleshooting steps, but the issue remains unresolved.
Issues with SigNoz Setup and Data Persistence in AKS
Vaibhavi experienced issues setting up SigNoz in AKS, and faced data persistence issues after installation. Srikanth provided guidance on ClickHouse version compatibility and resource requirements, helping Vaibhavi troubleshoot and resolve the issue.