#support

Resolving PodDisruptionBudgets Conflicts in Clickhouse Restart

TLDR Al encountered PodDisruptionBudgets conflicts on ClickHouse restart, which interfered with Kubernetes upgrade. Prashant assured that only one PDB should remain in the new chart. Al later confirmed successful remediation.

2mo

Join the chat

Jul 18, 2023 (2 months ago)

Photo of md5-001a6b8b05601dc8ac56c5f364768cc1

Al

08:14 PM

Hi team!!

I notice when restarting clickhouse, the following events occur:

Pod "chi-signoz-clickhouse-cluster-0-0-0" matches multiple PodDisruptionBudgets. Chose "signoz-clickhouse-cluster" arbitrarily.

Pod "chi-signoz-clickhouse-cluster-0-0-0" matches multiple PodDisruptionBudgets. Chose "signoz-clickhouse" arbitrarily.

This also seemed to interfere with upgrading kubernetes:

(UpgradeFailed) Drain of aks-node did not complete: Evict blocked by conflicting disruption budgets on pod chi-signoz-clickhouse-cluster-0-0-0 on node aks-node: chi-signoz-clickho
use-cluster-0-0-0. See http://aka.ms/aks/debugdrainfailures                                                                                                                                                                                   
Code: UpgradeFailed                                                                                                                                                                                                                           

Message: Drain of aks-node did not complete: Evict blocked by conflicting disruption budgets on pod chi-signoz-clickhouse-cluster-0-0-0 on node aks-node: chi-signoz-clickhouse-cluster-0-0-0.

Jul 19, 2023 (2 months ago)

Photo of md5-1899629483c7ab1dccfbee6cc2f637b9

Prashant

06:32 PM

Hi Al,

There should be only one PDB signoz-clickhouse-cluster in signoz-0.19.x chart.

Ref: https://github.com/Altinity/clickhouse-operator/blob/release-0.21.2/pkg/model/creator.go#L838

Have you followed all steps from v0.23 migration docs before upgrading?

Al

06:35 PM

The PDB errors we experienced were before upgrading to signoz-0.19.1. Glad to hear this should be resolved. Will monitor.

06:35

Al

06:35 PM

Thanks Prashant

Prashant

06:48 PM

I see. Do let us know how it goes with the K8s version upgrade as well.

Al

06:51 PM

We managed to work through the k8s upgrade, just required additional effort to work through it.

Thanks!

SigNoz Community

Built with ClickHouse as datastore, SigNoz is an open-source APM to help you find issues in your deployed applications & solve them quickly | Knowledge Base powered by Struct.AI

Indexed 826 threads (62% resolved)

Join Our Community Manage subscription

Similar Threads

SigNoz crashing in k8s due to ClickHouse OOM

Travis reported SigNoz crashing in k8s due to ClickHouse OOM. The team suggested increasing resources for ClickHouse, and other troubleshooting steps, but the issue remains unresolved.

6mo

Issues with SigNoz Setup and Data Persistence in AKS

Vaibhavi experienced issues setting up SigNoz in AKS, and faced data persistence issues after installation. Srikanth provided guidance on ClickHouse version compatibility and resource requirements, helping Vaibhavi troubleshoot and resolve the issue.

5mo

Error Upgrading Chart Version in Kubernetes Environment

Manikandan encountered an error when upgrading to a different chart version. Srikanth helps correct the mistake by suggesting to pull the new releases and upgrade.

5mo