Issue with Restarting Containers After Signoz Update

TLDR Div had issues with restarting containers after updating Signoz. Srikanth suggested checking the migration guide, which resolved the issue.

Photo of Div
Div
Tue, 18 Jul 2023 16:09:13 UTC

Hi Team, I just upgraded Signoz to the latest version and cannot restart the containers since then.

Photo of Div
Div
Tue, 18 Jul 2023 16:10:15 UTC

The query-service seems to be restarting

Photo of Div
Div
Tue, 18 Jul 2023 16:13:50 UTC

```{"log":"Commit timestamp : 2023-07-17T20:01:32Z\n","stream":"stdout","time":"2023-07-18T16:11:35.896880744Z"} {"log":"Branch : HEAD\n","stream":"stdout","time":"2023-07-18T16:11:35.896883694Z"} {"log":"Go version : go1.18.10\n","stream":"stdout","time":"2023-07-18T16:11:35.896893464Z"} {"log":"\n","stream":"stdout","time":"2023-07-18T16:11:35.896896484Z"} {"log":"For SigNoz Official Documentation, visit \n","stream":"stdout","time":"2023-07-18T16:11:35.896899174Z"} {"log":"For SigNoz Community Slack, visit \n","stream":"stdout","time":"2023-07-18T16:11:35.896901934Z"} {"log":"For discussions about SigNoz, visit \n","stream":"stdout","time":"2023-07-18T16:11:35.896904694Z"} {"log":"\n","stream":"stdout","time":"2023-07-18T16:11:35.896907434Z"} {"log":"Check SigNoz Github repo for license details.\n","stream":"stdout","time":"2023-07-18T16:11:35.896910024Z"} {"log":"Copyright 2022 SigNoz\n","stream":"stdout","time":"2023-07-18T16:11:35.896912685Z"} {"log":"\n","stream":"stdout","time":"2023-07-18T16:11:35.896915335Z"} {"log":"\n","stream":"stdout","time":"2023-07-18T16:11:35.896917865Z"} {"log":"2023-07-18T16:11:35.896Z\u0009WARN\u0009query-service/main.go:120\u0009No JWT secret key is specified.\n","stream":"stdout","time":"2023-07-18T16:11:35.896920475Z"} {"log":"2023-07-18T16:11:45.552Z\u0009INFO\u0009license/manager.go:127\u0009No active license found, defaulting to basic plan\n","stream":"stdout","time":"2023-07-18T16:11:45.553026927Z"} {"log":"2023-07-18T16:11:45.554Z\u0009INFO\u0009app/server.go:120\u0009Using ClickHouse as datastore ...\n","stream":"stdout","time":"2023-07-18T16:11:45.554237486Z"} {"log":"ts=2023-07-18T16:11:45.556Z caller=query_logger.go:113 level=error component=activeQueryTracker msg=\"Failed to create directory for logging active queries\"\n","stream":"stderr","time":"2023-07-18T16:$ {"log":"ts=2023-07-18T16:11:45.557Z caller=engine.go:349 level=debug component=\"query engine\" msg=\"Lookback delta is zero, setting to default value\" value=5m0s\n","stream":"stderr","time":"2023-07-18T16:11$ {"log":"ts=2023-07-18T16:11:45.557Z caller=reader.go:364 level=info msg=\"Loading configuration file\" filename=/root/config/prometheus.yml\n","stream":"stderr","time":"2023-07-18T16:11:45.557665109Z"} {"log":"ts=2023-07-18T16:11:45.558Z caller=query_logger.go:113 level=error msg=\"Failed to create directory for logging active queries\"\n","stream":"stderr","time":"2023-07-18T16:11:45.558250584Z"} {"log":"ts=2023-07-18T16:11:45.560Z caller=reader.go:381 level=info msg=\"Completed loading of configuration file\" filename=/root/config/prometheus.yml\n","stream":"stderr","time":"2023-07-18T16:11:45.5608573$ {"log":"ts=2023-07-18T16:11:45.562Z caller=engine.go:349 level=debug component=\"promql evaluator\" msg=\"Lookback delta is zero, setting to default value\" value=5m0s\n","stream":"stderr","time":"2023-07-18T1$ {"log":"2023-07-18T16:11:45.564Z\u0009INFO\u0009alertManager/notifier.go:94\u0009Starting notifier with alert manager:[]\n","stream":"stdout","time":"2023-07-18T16:11:45.565613083Z$ {"log":"2023-07-18T16:11:45.564Z\u0009INFO\u0009app/server.go:601\u0009rules manager is ready\n","stream":"stdout","time":"2023-07-18T16:11:45.565641083Z"} {"log":"2023-07-18T16:11:45.569Z\u0009DEBUG\u0009rules/apiParams.go:86\u0009postable rule(parsed):%!(EXTRA *rules.PostableRule=\u0026{Chat Service High Operations METRIC_BASED_ALERT threshold_rule 30000000000$ {"log":"panic: runtime error: invalid memory address or nil pointer dereference\n","stream":"stderr","time":"2023-07-18T16:11:45.571944576Z"} {"log":"[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x1089ee0]\n","stream":"stderr","time":"2023-07-18T16:11:45.571957467Z"} {"log":"\n","stream":"stderr","time":"2023-07-18T16:11:45.571967287Z"} {"log":"goroutine 1 [running]:\n","stream":"stderr","time":"2023-07-18T16:11:45.571971397Z"} {"log":"{{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, 0x0, 0x0, ...}, ...)\n","stream":"stderr","time":"2023-07-18T16:11:45.571990087Z"} {"log":"\u0009/go/src/github.com/signoz/signoz/pkg/query-service/rules/apiParams.go:114 +0x4e0\n","stream":"stderr","time":"2023-07-18T16:11:45.571996327Z"} {"log":"\n","stream":"stderr","time":"2023-07-18T16:11:45.572000748Z"} {"log":"\u0009/go/src/github.com/signoz/signoz/pkg/query-service/rules/apiParams.go:63\n","stream":"stderr","time":"2023-07-18T16:11:45.572004678Z"} {"log":"{0xc0006c4c00?, 0xc0006c4000?, 0x391?})\n","stream":"stderr","time":"2023-07-18T16:11:45.572021168Z"} {"log":"\u0009/go/src/github.com/signoz/signoz/pkg/query-service/rules/apiParams.go:59 +0x65\n","stream":"stderr","time":"2023-07-18T16:11:45.572025588Z"} {"log":"\n","stream":"stderr","time":"2023-07-18T16:11:45.572028518Z"} {"log":"\u0009/go/src/github.com/signoz/signoz/pkg/query-service/rules/manager.go:156 +0x16a\n","stream":"stderr","time":"2023-07-18T16:11:45.572037428Z"} {"log":"\n","stream":"stderr","time":"2023-07-18T16:11:45.572040458Z"} {"log":"\u0009/go/src/github.com/signoz/signoz/pkg/query-service/rules/manager.go:130 +0x25\n","stream":"stderr","time":"2023-07-18T16:11:45.572048039Z"} {"log":"\n","stream":"stderr","time":"2023-07-18T16:11:45.572051189Z"} {"log":"\u0009/go/src/github.com/signoz/signoz/ee/query-service/app/server.go:468 +0x93\n","stream":"stderr","time":"2023-07-18T16:11:45.572059819Z"} {"log":"main.main()\n","stream":"stderr","time":"2023-07-18T16:11:45.572067189Z"} {"log":"\u0009/go/src/github.com/signoz/signoz/ee/query-service/main.go:130 +0x725\n","stream":"stderr","time":"2023-07-18T16:11:45.57207816Z"}```

Photo of Srikanth
Srikanth
Tue, 18 Jul 2023 16:48:51 UTC

What version did you update from? You might not have followed migration steps

Photo of Div
Div
Wed, 19 Jul 2023 02:45:37 UTC

Srikanth Migrated from 0.18.1 to 0.23 (latest)

Photo of Srikanth
Srikanth
Wed, 19 Jul 2023 02:55:24 UTC

Please check

Photo of Div
Div
Wed, 19 Jul 2023 04:46:57 UTC

Done, fixed now :pray:

SigNoz Community Logo

SigNoz Community

Indexed 1060 threads

Built with ClickHouse as datastore, SigNoz is an open-source APM to help you find issues in your deployed applications & solve them quickly | Knowledge Base powered by Struct.AI

View in App

Similar Threads

Alert Issues After Upgrading SigNoz
Troubleshooting Interactive Message Errors on Query-Service
Issues with SigNoz on DigitalOcean k8s After Upgrade
Resolving Signoz Query Service Error
Gauge Metrics Not Publishing in Kotlin App with Signoz