Issue with Restarting Containers After Signoz Update
TLDR Div had issues with restarting containers after updating Signoz. Srikanth suggested checking the migration guide, which resolved the issue.
Powered by Struct AI
7
2mo
Solved
Jul 18, 2023 (2 months ago)
Div
Div
04:09 PMHi Team, I just upgraded Signoz to the latest version and cannot restart the containers since then.
04:10
Div
04:10 PMThe query-service seems to be restarting
04:13
Div
04:13 PM{"log":"Commit timestamp : 2023-07-17T20:01:32Z\n","stream":"stdout","time":"2023-07-18T16:11:35.896880744Z"}
{"log":"Branch : HEAD\n","stream":"stdout","time":"2023-07-18T16:11:35.896883694Z"}
{"log":"Go version : go1.18.10\n","stream":"stdout","time":"2023-07-18T16:11:35.896893464Z"}
{"log":"\n","stream":"stdout","time":"2023-07-18T16:11:35.896896484Z"}
{"log":"For SigNoz Official Documentation, visit https://signoz.io/docs\n","stream":"stdout","time":"2023-07-18T16:11:35.896899174Z"}
{"log":"For SigNoz Community Slack, visit http://signoz.io/slack\n","stream":"stdout","time":"2023-07-18T16:11:35.896901934Z"}
{"log":"For discussions about SigNoz, visit https://community.signoz.io\n","stream":"stdout","time":"2023-07-18T16:11:35.896904694Z"}
{"log":"\n","stream":"stdout","time":"2023-07-18T16:11:35.896907434Z"}
{"log":"Check SigNoz Github repo for license details.\n","stream":"stdout","time":"2023-07-18T16:11:35.896910024Z"}
{"log":"Copyright 2022 SigNoz\n","stream":"stdout","time":"2023-07-18T16:11:35.896912685Z"}
{"log":"\n","stream":"stdout","time":"2023-07-18T16:11:35.896915335Z"}
{"log":"\n","stream":"stdout","time":"2023-07-18T16:11:35.896917865Z"}
{"log":"2023-07-18T16:11:35.896Z\u0009WARN\u0009query-service/main.go:120\u0009No JWT secret key is specified.\n","stream":"stdout","time":"2023-07-18T16:11:35.896920475Z"}
{"log":"2023-07-18T16:11:45.552Z\u0009INFO\u0009license/manager.go:127\u0009No active license found, defaulting to basic plan\n","stream":"stdout","time":"2023-07-18T16:11:45.553026927Z"}
{"log":"2023-07-18T16:11:45.554Z\u0009INFO\u0009app/server.go:120\u0009Using ClickHouse as datastore ...\n","stream":"stdout","time":"2023-07-18T16:11:45.554237486Z"}
{"log":"ts=2023-07-18T16:11:45.556Z caller=query_logger.go:113 level=error component=activeQueryTracker msg=\"Failed to create directory for logging active queries\"\n","stream":"stderr","time":"2023-07-18T16:$
{"log":"ts=2023-07-18T16:11:45.557Z caller=engine.go:349 level=debug component=\"query engine\" msg=\"Lookback delta is zero, setting to default value\" value=5m0s\n","stream":"stderr","time":"2023-07-18T16:11$
{"log":"ts=2023-07-18T16:11:45.557Z caller=reader.go:364 level=info msg=\"Loading configuration file\" filename=/root/config/prometheus.yml\n","stream":"stderr","time":"2023-07-18T16:11:45.557665109Z"}
{"log":"ts=2023-07-18T16:11:45.558Z caller=query_logger.go:113 level=error msg=\"Failed to create directory for logging active queries\"\n","stream":"stderr","time":"2023-07-18T16:11:45.558250584Z"}
{"log":"ts=2023-07-18T16:11:45.560Z caller=reader.go:381 level=info msg=\"Completed loading of configuration file\" filename=/root/config/prometheus.yml\n","stream":"stderr","time":"2023-07-18T16:11:45.5608573$
{"log":"ts=2023-07-18T16:11:45.562Z caller=engine.go:349 level=debug component=\"promql evaluator\" msg=\"Lookback delta is zero, setting to default value\" value=5m0s\n","stream":"stderr","time":"2023-07-18T1$
{"log":"2023-07-18T16:11:45.564Z\u0009INFO\u0009alertManager/notifier.go:94\u0009Starting notifier with alert manager:[ ]\n","stream":"stdout","time":"2023-07-18T16:11:45.565613083Z$
{"log":"2023-07-18T16:11:45.564Z\u0009INFO\u0009app/server.go:601\u0009rules manager is ready\n","stream":"stdout","time":"2023-07-18T16:11:45.565641083Z"}
{"log":"2023-07-18T16:11:45.569Z\u0009DEBUG\u0009rules/apiParams.go:86\u0009postable rule(parsed):%!(EXTRA *rules.PostableRule=\u0026{Chat Service High Operations METRIC_BASED_ALERT threshold_rule 30000000000$
{"log":"panic: runtime error: invalid memory address or nil pointer dereference\n","stream":"stderr","time":"2023-07-18T16:11:45.571944576Z"}
{"log":"[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x1089ee0]\n","stream":"stderr","time":"2023-07-18T16:11:45.571957467Z"}
{"log":"\n","stream":"stderr","time":"2023-07-18T16:11:45.571967287Z"}
{"log":"goroutine 1 [running]:\n","stream":"stderr","time":"2023-07-18T16:11:45.571971397Z"}
{"log":"{{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, 0x0, 0x0, ...}, ...)\n","stream":"stderr","time":"2023-07-18T16:11:45.571990087Z"}
{"log":"\u0009/go/src/github.com/signoz/signoz/pkg/query-service/rules/apiParams.go:114 +0x4e0\n","stream":"stderr","time":"2023-07-18T16:11:45.571996327Z"}
{"log":"\n","stream":"stderr","time":"2023-07-18T16:11:45.572000748Z"}
{"log":"\u0009/go/src/github.com/signoz/signoz/pkg/query-service/rules/apiParams.go:63\n","stream":"stderr","time":"2023-07-18T16:11:45.572004678Z"}
{"log":"{0xc0006c4c00?, 0xc0006c4000?, 0x391?})\n","stream":"stderr","time":"2023-07-18T16:11:45.572021168Z"}
{"log":"\u0009/go/src/github.com/signoz/signoz/pkg/query-service/rules/apiParams.go:59 +0x65\n","stream":"stderr","time":"2023-07-18T16:11:45.572025588Z"}
{"log":"\n","stream":"stderr","time":"2023-07-18T16:11:45.572028518Z"}
{"log":"\u0009/go/src/github.com/signoz/signoz/pkg/query-service/rules/manager.go:156 +0x16a\n","stream":"stderr","time":"2023-07-18T16:11:45.572037428Z"}
{"log":"\n","stream":"stderr","time":"2023-07-18T16:11:45.572040458Z"}
{"log":"\u0009/go/src/github.com/signoz/signoz/pkg/query-service/rules/manager.go:130 +0x25\n","stream":"stderr","time":"2023-07-18T16:11:45.572048039Z"}
{"log":"\n","stream":"stderr","time":"2023-07-18T16:11:45.572051189Z"}
{"log":"\u0009/go/src/github.com/signoz/signoz/ee/query-service/app/server.go:468 +0x93\n","stream":"stderr","time":"2023-07-18T16:11:45.572059819Z"}
{"log":"main.main()\n","stream":"stderr","time":"2023-07-18T16:11:45.572067189Z"}
{"log":"\u0009/go/src/github.com/signoz/signoz/ee/query-service/main.go:130 +0x725\n","stream":"stderr","time":"2023-07-18T16:11:45.57207816Z"}
Srikanth
Srikanth
04:48 PMWhat version did you update from? You might not have followed migration steps https://signoz.io/docs/operate/migration/
Jul 19, 2023 (2 months ago)
Div
Div
02:45 AM Srikanth Migrated from 0.18.1 to 0.23 (latest)
Srikanth
Srikanth
02:55 AMDiv
Div
04:46 AMDone, fixed now 🙏
SigNoz Community
Built with ClickHouse as datastore, SigNoz is an open-source APM to help you find issues in your deployed applications & solve them quickly | Knowledge Base powered by Struct.AI
Indexed 826 threads (62% resolved)
Similar Threads
Alert Issues After Upgrading SigNoz
James experienced issues with alert rules after upgrading SigNoz. Srikanth suggested using sqlite tool to remove the alerts.
4
3mo
Solved
Troubleshooting Interactive Message Errors on Query-Service
Javier asked how to check query-service logs and shared an error message. Srikanth helped Javier identify the problem and suggested restarting the `clickhouse-setup_otel-collector-1` container. Javier confirmed the solution worked.
19
3mo
Solved
Issues with SigNoz on DigitalOcean k8s After Upgrade
Thomas experienced issues with SigNoz on DigitalOcean k8s after upgrading to v0.19. Prashant identified a PR to address the issue and provided a solution.
12
4mo
Solved