Issues with SigNoz on DigitalOcean k8s After Upgrade
TLDR Thomas experienced issues with SigNoz on DigitalOcean k8s after upgrading to v0.19. Prashant identified a PR to address the issue and provided a solution.
May 23, 2023 (6 days ago)
Thomas
01:10 AMnot enough memory - 1181 mb used out of 100 mb
) and decided that I needed to update from v0.18 to v0.19 anyway, so I did an upgrade. Some services are pending, and my-release-signoz-query-service
is failing with Readiness probe failed: Get "<http://10.244.1.34:8080/api/v1/version>": dial tcp 10.244.1.34:8080: connect: connection refused
. 1) how can I fix it? (if you need more details, just ask 🙂) 2) how can I ensure the infra doesn't crash in the future? Thank you.Prashant
02:52 PMpods states as well as logs of failing pods would be helpful.
Prashant
02:52 PMhttps://signoz.io/docs/operate/migration/upgrade-0.19/
Thomas
07:52 PM[email protected]:/tmp$ kubectl -n platform logs pod/my-release-signoz-query-service-0
2023-05-23T19:50:24.790Z INFO version/version.go:43
SigNoz version : v0.19.0
Commit SHA-1 : 6e8be3f
Commit timestamp : 2023-05-20T18:20:50Z
Branch : HEAD
Go version : go1.18.10
For SigNoz Official Documentation, visit https://signoz.io/docs
For SigNoz Community Slack, visit http://signoz.io/slack
For discussions about SigNoz, visit https://community.signoz.io
Check SigNoz Github repo for license details.
Copyright 2022 SigNoz
2023-05-23T19:50:24.790Z WARN query-service/main.go:61 No JWT secret key is specified.
main.main
/go/src/github.com/signoz/signoz/ee/query-service/main.go:61
runtime.main
/usr/local/go/src/runtime/proc.go:250
2023-05-23T19:50:26.023Z INFO license/manager.go:127 No active license found, defaulting to basic plan
2023-05-23T19:50:26.025Z INFO app/server.go:117 Using ClickHouse as datastore ...
ts=2023-05-23T19:50:26.032Z caller=query_logger.go:113 level=error component=activeQueryTracker msg="Failed to create directory for logging active queries"
ts=2023-05-23T19:50:26.034Z caller=engine.go:349 level=debug component="query engine" msg="Lookback delta is zero, setting to default value" value=5m0s
ts=2023-05-23T19:50:26.035Z caller=reader.go:364 level=info msg="Loading configuration file" filename=/root/config/prometheus.yml
ts=2023-05-23T19:50:26.036Z caller=query_logger.go:113 level=error msg="Failed to create directory for logging active queries"
ts=2023-05-23T19:50:26.039Z caller=engine.go:349 level=debug component="promql evaluator" msg="Lookback delta is zero, setting to default value" value=5m0s
2023-05-23T19:50:26.052Z INFO alertManager/notifier.go:94 Starting notifier with alert manager:[ ]
2023-05-23T19:50:26.052Z INFO app/server.go:587 rules manager is ready
ts=2023-05-23T19:50:26.053Z caller=reader.go:381 level=info msg="Completed loading of configuration file" filename=/root/config/prometheus.yml
2023-05-23T19:50:26.075Z DEBUG rules/apiParams.go:86 postable rule(parsed):%!(EXTRA *rules.PostableRule=&{Exception EXCEPTIONS_BASED_ALERT threshold_rule 300000000000 0 {"op":"1","target":5,"matchType":"4"} map[details:https://monitoring.fireacademy.io/exceptions severity:warning] map[description:This alert is fired when the defined metric (current value: {{$value}}) crosses the threshold ({{$threshold}}) summary:The rule threshold is set to {{$threshold}}, and the observed metric value is {{$value}}] false https://monitoring.fireacademy.io/alerts/new [] })
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x1046d20]
goroutine 1 [running]:
, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, 0x0, 0x0, ...}, ...)
/go/src/github.com/signoz/signoz/pkg/query-service/rules/apiParams.go:114 +0x4e0
/go/src/github.com/signoz/signoz/pkg/query-service/rules/apiParams.go:63
?, 0xc000209100?, 0x6f1?})
/go/src/github.com/signoz/signoz/pkg/query-service/rules/apiParams.go:59 +0x65
/go/src/github.com/signoz/signoz/pkg/query-service/rules/manager.go:156 +0x16a
/go/src/github.com/signoz/signoz/pkg/query-service/rules/manager.go:130 +0x25
/go/src/github.com/signoz/signoz/ee/query-service/app/server.go:454 +0x93
main.main()
/go/src/github.com/signoz/signoz/ee/query-service/main.go:71 +0x574
Prashant
07:58 PMPrashant
07:59 PMThomas
08:17 PMPrashant
11:29 PMPrashant
11:31 PMThomas
11:42 PMMay 24, 2023 (5 days ago)
Prashant
10:41 AMSigNoz Community
Similar Threads
Issues with SigNoz Setup and Data Persistence in AKS
Vaibhavi experienced issues setting up SigNoz in AKS, and faced data persistence issues after installation. Srikanth provided guidance on ClickHouse version compatibility and resource requirements, helping Vaibhavi troubleshoot and resolve the issue.
SigNoz crashing in k8s due to ClickHouse OOM
Travis reported SigNoz crashing in k8s due to ClickHouse OOM. The team suggested increasing resources for ClickHouse, and other troubleshooting steps, but the issue remains unresolved.
Issue with Installing Latest SigNoz Version
Goutham faced issues installing latest SigNoz version with pods in initiation state. Syed suggested checking ClickHouse server. Prashant recommended running `helm upgrade` to resolve the issue.
Upgrading to SigNoz 0.19.0 with Helm Chart
Daniel asked about running a migration script for upgrading to SigNoz 0.19.0 using Helm chart. Prashant provided updated migration documentation link.

SigNoz Setup Issue with External ClickHouse in Kubernetes
Ragulraj faced issues with SigNoz setup on Kubernetes using an external ClickHouse. Prashant and Srikanth provided pre-requisites and configuration steps. Ashu agreed to update the documentation.