Issues with SigNoz on DigitalOcean k8s After Upgrade
TLDR Thomas experienced issues with SigNoz on DigitalOcean k8s after upgrading to v0.19. Prashant identified a PR to address the issue and provided a solution.
May 23, 2023 (4 months ago)
Thomas
01:10 AMnot enough memory - 1181 mb used out of 100 mb
) and decided that I needed to update from v0.18 to v0.19 anyway, so I did an upgrade. Some services are pending, and my-release-signoz-query-service
is failing with Readiness probe failed: Get "<http://10.244.1.34:8080/api/v1/version>": dial tcp 10.244.1.34:8080: connect: connection refused
. 1) how can I fix it? (if you need more details, just ask ๐) 2) how can I ensure the infra doesn't crash in the future? Thank you.Prashant
02:52 PMpods states as well as logs of failing pods would be helpful.
Prashant
02:52 PMhttps://signoz.io/docs/operate/migration/upgrade-0.19/
Thomas
07:52 PMyakuhito@catstation:/tmp$ kubectl -n platform logs pod/my-release-signoz-query-service-0
2023-05-23T19:50:24.790Z INFO version/version.go:43
SigNoz version : v0.19.0
Commit SHA-1 : 6e8be3f
Commit timestamp : 2023-05-20T18:20:50Z
Branch : HEAD
Go version : go1.18.10
For SigNoz Official Documentation, visit https://signoz.io/docs
For SigNoz Community Slack, visit http://signoz.io/slack
For discussions about SigNoz, visit https://community.signoz.io
Check SigNoz Github repo for license details.
Copyright 2022 SigNoz
2023-05-23T19:50:24.790Z WARN query-service/main.go:61 No JWT secret key is specified.
main.main
/go/src/github.com/signoz/signoz/ee/query-service/main.go:61
runtime.main
/usr/local/go/src/runtime/proc.go:250
2023-05-23T19:50:26.023Z INFO license/manager.go:127 No active license found, defaulting to basic plan
2023-05-23T19:50:26.025Z INFO app/server.go:117 Using ClickHouse as datastore ...
ts=2023-05-23T19:50:26.032Z caller=query_logger.go:113 level=error component=activeQueryTracker msg="Failed to create directory for logging active queries"
ts=2023-05-23T19:50:26.034Z caller=engine.go:349 level=debug component="query engine" msg="Lookback delta is zero, setting to default value" value=5m0s
ts=2023-05-23T19:50:26.035Z caller=reader.go:364 level=info msg="Loading configuration file" filename=/root/config/prometheus.yml
ts=2023-05-23T19:50:26.036Z caller=query_logger.go:113 level=error msg="Failed to create directory for logging active queries"
ts=2023-05-23T19:50:26.039Z caller=engine.go:349 level=debug component="promql evaluator" msg="Lookback delta is zero, setting to default value" value=5m0s
2023-05-23T19:50:26.052Z INFO alertManager/notifier.go:94 Starting notifier with alert manager:[ ]
2023-05-23T19:50:26.052Z INFO app/server.go:587 rules manager is ready
ts=2023-05-23T19:50:26.053Z caller=reader.go:381 level=info msg="Completed loading of configuration file" filename=/root/config/prometheus.yml
2023-05-23T19:50:26.075Z DEBUG rules/apiParams.go:86 postable rule(parsed):%!(EXTRA *rules.PostableRule=&{Exception EXCEPTIONS_BASED_ALERT threshold_rule 300000000000 0 {"op":"1","target":5,"matchType":"4"} map[details:https://monitoring.fireacademy.io/exceptions severity:warning] map[description:This alert is fired when the defined metric (current value: {{$value}}) crosses the threshold ({{$threshold}}) summary:The rule threshold is set to {{$threshold}}, and the observed metric value is {{$value}}] false https://monitoring.fireacademy.io/alerts/new [] })
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x1046d20]
goroutine 1 [running]:
, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, 0x0, 0x0, ...}, ...)
/go/src/github.com/signoz/signoz/pkg/query-service/rules/apiParams.go:114 +0x4e0
/go/src/github.com/signoz/signoz/pkg/query-service/rules/apiParams.go:63
?, 0xc000209100?, 0x6f1?})
/go/src/github.com/signoz/signoz/pkg/query-service/rules/apiParams.go:59 +0x65
/go/src/github.com/signoz/signoz/pkg/query-service/rules/manager.go:156 +0x16a
/go/src/github.com/signoz/signoz/pkg/query-service/rules/manager.go:130 +0x25
/go/src/github.com/signoz/signoz/ee/query-service/app/server.go:454 +0x93
main.main()
/go/src/github.com/signoz/signoz/ee/query-service/main.go:71 +0x574
Prashant
07:58 PMPrashant
07:59 PMThomas
08:17 PMPrashant
11:18 PMPrashant
11:29 PMPrashant
11:31 PMThomas
11:42 PMMay 24, 2023 (4 months ago)
Prashant
10:41 AMSigNoz Community
Indexed 826 threads (62% resolved)
Similar Threads
Issue with Restarting Containers After Signoz Update
Div had issues with restarting containers after updating Signoz. Srikanth suggested checking the migration guide, which resolved the issue.
Alert Issues After Upgrading SigNoz
James experienced issues with alert rules after upgrading SigNoz. Srikanth suggested using sqlite tool to remove the alerts.
SigNoz Production Feedback and Improvement Suggestions
Users shared their SigNoz experiences and feedback. Issues mentioned include memory consumption, UI/UX improvements, connection difficulties, and automation limitations.
Extracting Custom Fields as Attributes from Log in SigNoz
Harald wants to have custom fields from their body parsed as attributes in the SigNoz logger. They have tried implementing suggestions from nitya-signoz and Prashant, but the issue remains unsolved due to a potential race condition encountered while executing the code. They have provided a full example using Kind for further assistance.
SigNoz Install Script Errors
Nishant encountered errors with the install.sh script. Prashant suggested using `bash install.sh` or `./install.sh` instead of `sh`.