Signoz Kubernetes Setup Issue with Trace Logs

TLDR Jatin faced an issue with trace logs in otel-collector pod. Srikanth helped troubleshoot and identified storage issue, which was resolved by increasing pv.

Photo of Jatin
Jatin
Mon, 17 Apr 2023 09:30:30 UTC

Hi everyone we have setup signoz on kubernetes it was using helm as per the official documentation. It was working fine all of sudden i connot see traces and logs now when checked found this in otel-collector pod logs. `clickhousetracesexporter/writer.go:130 Could not write a batch of spans to model table: {"kind": "exporter", "data_type": "traces", "name": "clickhousetraces", "batch": [{"traceId":"f9fee5443ec20685d028c11d3a437eb3","spanId":"e8465a702eaa144c","parentSpanId":"8d63de4de998b172",`

Photo of Srikanth
Srikanth
Mon, 17 Apr 2023 09:49:49 UTC

There will be an error at the end; please share the full log.

Photo of Jatin
Jatin
Mon, 17 Apr 2023 10:07:30 UTC

Hi Srikanth please check

Photo of Srikanth
Srikanth
Tue, 18 Apr 2023 03:27:34 UTC

You ran out of storage.

Photo of Jatin
Jatin
Tue, 18 Apr 2023 06:09:37 UTC

Thanks Srikanth after increasing pv everything is working fine