#general

Otel-collector Pods Stuck in Running State

TLDR Guillaume experienced an issue where pods remained in 'Running' state but not 'Ready' status. Srikanth suggested potential connection issues with ClickHouse. The issue resolved itself after some time, with no changes made.

5mo

Join the chat

Mar 28, 2023 (6 months ago)

Photo of md5-acabeb270eee485e29143a208813eecc

Guillaume

03:49 PM

Hi (again), I'm having another issue, when I tried to change the settings of the collector, I did a reset to the previously working env, but the new pod of the otel-collector are in Running state but never get into Ready status. The logs of the pods :

2023-03-28T15:46:48.226Z    info    service/telemetry.go:111    Setting up own telemetry...
2023-03-28T15:46:48.226Z    info    service/telemetry.go:141    Serving Prometheus metrics    {"address": "0.0.0.0:8888", "level": "Basic"}
2023-03-28T15:46:48.226Z    info    components/components.go:30    Stability level of component is undefined    {"kind": "exporter", "data_type": "traces", "name": "clickhousetraces", "stability": "Undefined"}

It's not really clear, what can I do to check this up?

Photo of md5-ce04a9988e2fd758a659dc55be6f2543

Srikanth

04:05 PM

> new pod of the otel-collector are in Running state but never get into Ready status?
What do you mean by this? What’s the issue if they are running?

04:05

Srikanth

04:05 PM

is it the full logs of pod?

Mar 29, 2023 (6 months ago)

Guillaume

09:01 AM

I mean that in Kubernetes, the column STATUS shows Running but the column READY show 0/1 and after some time the pod is restarted (because of the readiness probe I assume), and it keeps going.

Yes it's the full logs until the pod restarts

Srikanth

12:29 PM

If this is all it means it hasn’t started yet, it could be related to not having connection issues with ClickHouse or ClickHouse isn’t processing the migration request quickly.

Apr 03, 2023 (5 months ago)

Guillaume

08:37 AM

You're right, I don't know why it was in this state though. But a night passed, I made no changes, and it came back to be working. So I'll come back if it happens again. 🙂 Thanks for assisting!

SigNoz Community

Built with ClickHouse as datastore, SigNoz is an open-source APM to help you find issues in your deployed applications & solve them quickly | Knowledge Base powered by Struct.AI

Indexed 825 threads (61% resolved)

Join Our Community Manage subscription

Similar Threads

Setting Up Dual Exporters for Otel-collector and Resolving Payload Error

Guillaume was facing an issue setting up dual exporters for otel-collector and had a pod not being "Ready". Srikanth suggested checking error logs and dealing with large payloads, which led to a successful setup.

5mo

Troubleshooting SigNoz Auto-Instrumentation Configuration

igor is having trouble configuring auto-instrumentation for Java applications using SigNoz, with traces not appearing in the SigNoz UI. Prashant advises to check logs of the otel sidecar, use service name for endpoint, verify supported libraries, and test with telemetrygen. However, the issue still persists.

5mo

SigNoz crashing in k8s due to ClickHouse OOM

Travis reported SigNoz crashing in k8s due to ClickHouse OOM. The team suggested increasing resources for ClickHouse, and other troubleshooting steps, but the issue remains unresolved.

6mo