Migrating SigNoz installation Issues and Dashboard Unavailability
TLDR Mario had trouble migrating SigNoz installation from 0.26.0 to 0.28.0 and issues with dashboard page. Srikanth advised checking migration success. Eventually, Mario reported the dashboard working again.
Mario
Tue, 24 Oct 2023 10:18:46 UTCAnother issue, which started happening after the attempted migrations (see my message above): I can't see my dashboards anymore:
• dashboard page returns `422 / Unprocessable entity`, and visually it appears as if there are no dashboards;
• as I don't mind recreating them (I use some among the ones available
Srikanth
Fri, 27 Oct 2023 11:54:45 UTCIs this still an issue or did you manager to get the migrations work? Ideally, The dashboard should work if the migrations are successful.
Mario
Wed, 01 Nov 2023 09:37:41 UTC(I've been out of work a few days) today it worked and dashboard are up again. I attempted several times last week always targeting v0.28.0, was it edited in the meantime?
SigNoz Community
Indexed 1060 threads
Built with ClickHouse as datastore, SigNoz is an open-source APM to help you find issues in your deployed applications & solve them quickly | Knowledge Base powered by Struct.AI
View in App
Mario
Tue, 24 Oct 2023 10:11:01 UTCHi there. I was migrating a SigNoz installation (k8s / eks via helm) from 0.26.0 to 0.28.0 In a development environment everything went smooth at first attempt running the update via helm. On a production environment (where there's way more data) I get the error below on the migration pod. Being a timeout, I allowed the migration pod to restart and continue but I have the impression it loops and won't get to the bottom of it. I also tried with the instructions as described in the, that seems to work, but at the subsequent `helm update` I'm back to the issue below.
Is there a way to increase that `distributed_ddl_task_timeout`? It's not obvious to me where and how should be set.
```{"level":"error","timestamp":"2023-10-24T09:53:26.669Z","caller":"migrationmanager/manager.go:79","msg":"Failed to run migrations for migrator","component":"migrationmanager","migrator":"logs","error":"failed to create database, err: code: 159, message: Watching task /clickhouse/signoz-clickhouse/task_queue/ddl/query-0000000251 is executing longer than distributed_ddl_task_timeout (=180) seconds. There are 1 unfinished hosts (0 of them are currently active), they are going to execute the query in background","stacktrace":"\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/migrationmanager/manager.go:79\nmain.main\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/cmd/signozschemamigrator/migrate.go:83\nruntime.main\n\t/opt/hostedtoolcache/go/1.20.10/x64/src/runtime/proc.go:250"}
{"level":"fatal","timestamp":"2023-10-24T09:53:26.972Z","caller":"signozschemamigrator/migrate.go:85","msg":"Failed to run migrations","component":"migrate cli","error":"failed to create database, err: code: 159, message: Watching task /clickhouse/signoz-clickhouse/task_queue/ddl/query-0000000251 is executing longer than distributed_ddl_task_timeout (=180) seconds. There are 1 unfinished hosts (0 of them are currently active), they are going to execute the query in background","stacktrace":"main.main\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/cmd/signozschemamigrator/migrate.go:85\nruntime.main\n\t/opt/hostedtoolcache/go/1.20.10/x64/src/runtime/proc.go:250"}```