#support

Resolving PodDisruptionBudgets Conflicts in Clickhouse Restart

TLDR Al encountered PodDisruptionBudgets conflicts on ClickHouse restart, which interfered with Kubernetes upgrade. Prashant assured that only one PDB should remain in the new chart. Al later confirmed successful remediation.

4mo

Join the chat

Jul 18, 2023 (4 months ago)

Photo of md5-4a36c3bc25964c64330a5747ba443346

Al

08:14 PM

Hi team!!

I notice when restarting clickhouse, the following events occur:

Pod "chi-signoz-clickhouse-cluster-0-0-0" matches multiple PodDisruptionBudgets. Chose "signoz-clickhouse-cluster" arbitrarily.

Pod "chi-signoz-clickhouse-cluster-0-0-0" matches multiple PodDisruptionBudgets. Chose "signoz-clickhouse" arbitrarily.

This also seemed to interfere with upgrading kubernetes:

(UpgradeFailed) Drain of aks-node did not complete: Evict blocked by conflicting disruption budgets on pod chi-signoz-clickhouse-cluster-0-0-0 on node aks-node: chi-signoz-clickho
use-cluster-0-0-0. See http://aka.ms/aks/debugdrainfailures                                                                                                                                                                                   
Code: UpgradeFailed                                                                                                                                                                                                                           

Message: Drain of aks-node did not complete: Evict blocked by conflicting disruption budgets on pod chi-signoz-clickhouse-cluster-0-0-0 on node aks-node: chi-signoz-clickhouse-cluster-0-0-0.

Jul 19, 2023 (4 months ago)

Photo of md5-1899629483c7ab1dccfbee6cc2f637b9

Prashant

06:32 PM

Hi Al,

There should be only one PDB signoz-clickhouse-cluster in signoz-0.19.x chart.

Ref: https://github.com/Altinity/clickhouse-operator/blob/release-0.21.2/pkg/model/creator.go#L838

Have you followed all steps from v0.23 migration docs before upgrading?

Al

06:35 PM

The PDB errors we experienced were before upgrading to signoz-0.19.1. Glad to hear this should be resolved. Will monitor.

06:35

Al

06:35 PM

Thanks Prashant

Prashant

06:48 PM

I see. Do let us know how it goes with the K8s version upgrade as well.

Al

06:51 PM

We managed to work through the k8s upgrade, just required additional effort to work through it.

Thanks!

SigNoz Community

Built with ClickHouse as datastore, SigNoz is an open-source APM to help you find issues in your deployed applications & solve them quickly | Knowledge Base powered by Struct.AI

Indexed 1023 threads (61% resolved)

Join Our Community

Similar Threads

Increasing Persistent Volume Size in Test Environment

surya tried to increase the disk size in their test environment and encountered an issue. Prashant provided guidance and the user successfully increased the Persistent Volume (PV) for ClickHouse to 50Gi.

yesterday

SigNoz crashing in k8s due to ClickHouse OOM

Travis reported SigNoz crashing in k8s due to ClickHouse OOM. The team suggested increasing resources for ClickHouse, and other troubleshooting steps, but the issue remains unresolved.

8mo

Issues with SigNoz Setup and Data Persistence in AKS

Vaibhavi experienced issues setting up SigNoz in AKS, and faced data persistence issues after installation. Srikanth provided guidance on ClickHouse version compatibility and resource requirements, helping Vaibhavi troubleshoot and resolve the issue.

8mo