Troubleshooting Data Migration from Version 10.2 to 21.0
TLDR Anil seeks assistance on data migration from version 10.2 to 21.0. They faced a series of problems, from renaming tables to pods getting stuck in init. Srikanth was unable to conclusively resolve these issues.
Powered by Struct AI
42
2mo
Jul 26, 2023 (2 months ago)
Anil
Anil
02:15 PM Srikanth, i have one question regarding migration.
we have version 10.2 running in our environments, we need to upgrade to 21.0 . how the migration of data takes places ? could you please help me on this
we have version 10.2 running in our environments, we need to upgrade to 21.0 . how the migration of data takes places ? could you please help me on this
Jul 27, 2023 (2 months ago)
Srikanth
Anil
Anil
05:26 AMoh , so its like series of steps we need to consider
09:24
Anil
09:24 AM Srikanth, we have seen that migration to 0.12 , need to change the table names prefixed with distributed_
from 0.12 to 0.19 version we need to run the migration script which provided in the document right
from 0.12 to 0.19 version we need to run the migration script which provided in the document right
09:24
Anil
09:24 AMplease correct me if my understanding is wrong
Srikanth
Srikanth
01:13 PMYes
Anil
Anil
01:17 PM Srikanth, i executed migration script and upgraded from 10.2 to 0.12.0.
here is my script
RENAME TABLE samples_v2 TO distributed_samples_v2, time_series_v2 TO distributed_time_series_v2, usage TO distributed_usage;
RENAME TABLE dependency_graph_minutes TO distributed_dependency_graph_minutes, dependency_graph_minutes_db_calls_mv TO distributed_dependency_graph_minutes_db_calls_mv, dependency_graph_minutes_messaging_calls_mv TO distributed_dependency_graph_minutes_messaging_calls_mv, durationSort TO distributed_durationSort, signoz_error_index_v2 TO distributed_signoz_error_index_v2, signoz_spans TO distributed_signoz_spans, top_level_operations TO distributed_top_level_operations, signoz_index_v2 TO distributed_signoz_index_v2, usage TO distributed_usage, usage_explorer TO distributed_usage_explorer;
here is my script
RENAME TABLE samples_v2 TO distributed_samples_v2, time_series_v2 TO distributed_time_series_v2, usage TO distributed_usage;
RENAME TABLE dependency_graph_minutes TO distributed_dependency_graph_minutes, dependency_graph_minutes_db_calls_mv TO distributed_dependency_graph_minutes_db_calls_mv, dependency_graph_minutes_messaging_calls_mv TO distributed_dependency_graph_minutes_messaging_calls_mv, durationSort TO distributed_durationSort, signoz_error_index_v2 TO distributed_signoz_error_index_v2, signoz_spans TO distributed_signoz_spans, top_level_operations TO distributed_top_level_operations, signoz_index_v2 TO distributed_signoz_index_v2, usage TO distributed_usage, usage_explorer TO distributed_usage_explorer;
01:17
Anil
01:17 PMbut getting
my-release-signoz-otel-collector-6f47d74d65-nklqp 0/1 Init:0/1 0 8m15s
my-release-signoz-otel-collector-ddb76f957-dgjm6 0/1 Init:0/1 0 6m20s
my-release-signoz-otel-collector-6f47d74d65-nklqp 0/1 Init:0/1 0 8m15s
my-release-signoz-otel-collector-ddb76f957-dgjm6 0/1 Init:0/1 0 6m20s
Srikanth
Srikanth
01:17 PMYou don’t have to change the table names your self
Anil
Anil
01:18 PMthen how data will be migrated ?
Srikanth
Srikanth
01:20 PMIt will be taken care during the upgrade. The
distributed_ tables are virtual tables which will use the existing tables. The migration only says if you use ClickHouse queries in dashboards you need to change queries.Anil
Anil
01:20 PMoh thanks buddy
01:21
Anil
01:21 PMhow to take back up of data
01:21
Anil
01:21 PM?
01:21
Anil
01:21 PMjust in case i am asking
Srikanth
Srikanth
01:26 PMThat’s a non-trivial process.
Anil
Anil
01:37 PM Srikanth, i just followed what you said
but still we are seeing this
my-release-signoz-alertmanager-0 0/1 Init:0/1 0 116s
my-release-signoz-frontend-57bc68fb7d-946tf 0/1 Init:0/1 0 117s
my-release-signoz-frontend-cf9b8d8c9-dg9b6 1/1 Running 0 10m
my-release-signoz-otel-collector-6f47d74d65-5kj5j 0/1 Init:0/1 0 117s
my-release-signoz-otel-collector-ddb76f957-6nr52 1/1 Running 0 10m
my-release-signoz-otel-collector-metrics-5dd74686c-5cpwb 1/1 Running 0 10m
my-release-signoz-otel-collector-metrics-84bbbddc6c-jfgkq 0/1 Init:0/1 0 117s
my-release-signoz-query-service-0 0/1 Init:0/1 0 116s
but still we are seeing this
my-release-signoz-alertmanager-0 0/1 Init:0/1 0 116s
my-release-signoz-frontend-57bc68fb7d-946tf 0/1 Init:0/1 0 117s
my-release-signoz-frontend-cf9b8d8c9-dg9b6 1/1 Running 0 10m
my-release-signoz-otel-collector-6f47d74d65-5kj5j 0/1 Init:0/1 0 117s
my-release-signoz-otel-collector-ddb76f957-6nr52 1/1 Running 0 10m
my-release-signoz-otel-collector-metrics-5dd74686c-5cpwb 1/1 Running 0 10m
my-release-signoz-otel-collector-metrics-84bbbddc6c-jfgkq 0/1 Init:0/1 0 117s
my-release-signoz-query-service-0 0/1 Init:0/1 0 116s
01:37
Anil
01:37 PMpods are in init stage only
01:38
Anil
01:38 PMchat version is 0.2.5 to 0.6.0
Srikanth
Srikanth
01:38 PMWhere are ClickHouse pods?
Anil
Anil
01:39 PMchi-my-release-clickhouse-cluster-0-0-0 1/1 Running 0 100s
my-release-clickhouse-operator-b4c9cb776-jqxrm 2/2 Running 0 117s
my-release-k8s-infra-otel-agent-2gt5n 1/1 Running 0 117s
my-release-k8s-infra-otel-agent-4244z 1/1 Running 0 117s
my-release-k8s-infra-otel-agent-4989f 1/1 Running 0 117s
my-release-k8s-infra-otel-agent-4g59w 1/1 Running 0 117s
my-release-k8s-infra-otel-agent-948gs 1/1 Running 0 117s
my-release-k8s-infra-otel-agent-bjbtp 1/1 Running 0 117s
my-release-k8s-infra-otel-agent-jl4bt 1/1 Running 0 117s
my-release-k8s-infra-otel-agent-jql8z 1/1 Running 0 117s
my-release-k8s-infra-otel-agent-kz86h 1/1 Running 0 117s
my-release-k8s-infra-otel-agent-lxrg7 1/1 Running 0 117s
my-release-k8s-infra-otel-agent-m2wzw 1/1 Running 0 117s
my-release-k8s-infra-otel-agent-p2ht4 1/1 Running 0 117s
my-release-k8s-infra-otel-agent-qpqcp 1/1 Running 0 117s
my-release-k8s-infra-otel-agent-rr6j6 1/1 Running 0 117s
my-release-k8s-infra-otel-agent-xt67c 1/1 Running 0 117s
my-release-k8s-infra-otel-agent-z28nm 1/1 Running 0 117s
my-release-k8s-infra-otel-deployment-5b8566bc46-bfzcf 1/1 Running 0 117s
my-release-signoz-alertmanager-0 0/1 Init:0/1 0 116s
my-release-signoz-frontend-57bc68fb7d-946tf 0/1 Init:0/1 0 117s
my-release-signoz-frontend-cf9b8d8c9-dg9b6 1/1 Running 0 10m
my-release-signoz-otel-collector-6f47d74d65-5kj5j 0/1 Init:0/1 0 117s
my-release-signoz-otel-collector-ddb76f957-6nr52 1/1 Running 0 10m
my-release-signoz-otel-collector-metrics-5dd74686c-5cpwb 1/1 Running 0 10m
my-release-signoz-otel-collector-metrics-84bbbddc6c-jfgkq 0/1 Init:0/1 0 117s
my-release-signoz-query-service-0 0/1 Init:0/1 0 116s
my-release-zookeeper-0 1/1 Running 0 10m
my-release-clickhouse-operator-b4c9cb776-jqxrm 2/2 Running 0 117s
my-release-k8s-infra-otel-agent-2gt5n 1/1 Running 0 117s
my-release-k8s-infra-otel-agent-4244z 1/1 Running 0 117s
my-release-k8s-infra-otel-agent-4989f 1/1 Running 0 117s
my-release-k8s-infra-otel-agent-4g59w 1/1 Running 0 117s
my-release-k8s-infra-otel-agent-948gs 1/1 Running 0 117s
my-release-k8s-infra-otel-agent-bjbtp 1/1 Running 0 117s
my-release-k8s-infra-otel-agent-jl4bt 1/1 Running 0 117s
my-release-k8s-infra-otel-agent-jql8z 1/1 Running 0 117s
my-release-k8s-infra-otel-agent-kz86h 1/1 Running 0 117s
my-release-k8s-infra-otel-agent-lxrg7 1/1 Running 0 117s
my-release-k8s-infra-otel-agent-m2wzw 1/1 Running 0 117s
my-release-k8s-infra-otel-agent-p2ht4 1/1 Running 0 117s
my-release-k8s-infra-otel-agent-qpqcp 1/1 Running 0 117s
my-release-k8s-infra-otel-agent-rr6j6 1/1 Running 0 117s
my-release-k8s-infra-otel-agent-xt67c 1/1 Running 0 117s
my-release-k8s-infra-otel-agent-z28nm 1/1 Running 0 117s
my-release-k8s-infra-otel-deployment-5b8566bc46-bfzcf 1/1 Running 0 117s
my-release-signoz-alertmanager-0 0/1 Init:0/1 0 116s
my-release-signoz-frontend-57bc68fb7d-946tf 0/1 Init:0/1 0 117s
my-release-signoz-frontend-cf9b8d8c9-dg9b6 1/1 Running 0 10m
my-release-signoz-otel-collector-6f47d74d65-5kj5j 0/1 Init:0/1 0 117s
my-release-signoz-otel-collector-ddb76f957-6nr52 1/1 Running 0 10m
my-release-signoz-otel-collector-metrics-5dd74686c-5cpwb 1/1 Running 0 10m
my-release-signoz-otel-collector-metrics-84bbbddc6c-jfgkq 0/1 Init:0/1 0 117s
my-release-signoz-query-service-0 0/1 Init:0/1 0 116s
my-release-zookeeper-0 1/1 Running 0 10m
Srikanth
Srikanth
01:39 PMCheck the logs of ClickHouse if there are any issues
01:40
Srikanth
01:40 PMOr it may be resources issue.
Anil
Anil
01:41 PMhelm --namespace platform install my-release signoz/signoz --set clickhouse.persistence.size=125Gi --version 0.2.5
helm -n platform upgrade my-release signoz/signoz --version 0.6.0
helm -n platform upgrade my-release signoz/signoz --version 0.6.0
01:41
Anil
01:41 PMso first we installed 10.2 and then upgraded
01:44
Anil
01:44 PMlogs
2023.07.27 13:34:47.755403 [ 7 ] {} <Warning> Application: Listen [::]:8123 failed: Poco::Exception. Code: 1000, e.code() = 0, DNS error: EAI: Address family for hostname not supported (version 22.8.8.3 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
2023.07.27 13:34:47.755642 [ 7 ] {} <Warning> Application: Listen [::]:9000 failed: Poco::Exception. Code: 1000, e.code() = 0, DNS error: EAI: Address family for hostname not supported (version 22.8.8.3 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
2023.07.27 13:34:47.755952 [ 7 ] {} <Warning> Application: Listen [::]:9004 failed: Poco::Exception. Code: 1000, e.code() = 0, DNS error: EAI: Address family for hostname not supported (version 22.8.8.3 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
2023.07.27 13:34:47.756158 [ 7 ] {} <Warning> Application: Listen [::]:9005 failed: Poco::Exception. Code: 1000, e.code() = 0, DNS error: EAI: Address family for hostname not supported (version 22.8.8.3 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
2023.07.27 13:34:47.756370 [ 7 ] {} <Warning> Application: Listen [::]:9363 failed: Poco::Exception. Code: 1000, e.code() = 0, DNS error: EAI: Address family for hostname not supported (version 22.8.8.3 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
2023.07.27 13:34:47.845312 [ 7 ] {} <Warning> Application: Listen [::]:9009 failed: Poco::Exception. Code: 1000, e.code() = 0, DNS error: EAI: Address family for hostname not supported (version 22.8.8.3 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
2023.07.27 13:34:47.755403 [ 7 ] {} <Warning> Application: Listen [::]:8123 failed: Poco::Exception. Code: 1000, e.code() = 0, DNS error: EAI: Address family for hostname not supported (version 22.8.8.3 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
2023.07.27 13:34:47.755642 [ 7 ] {} <Warning> Application: Listen [::]:9000 failed: Poco::Exception. Code: 1000, e.code() = 0, DNS error: EAI: Address family for hostname not supported (version 22.8.8.3 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
2023.07.27 13:34:47.755952 [ 7 ] {} <Warning> Application: Listen [::]:9004 failed: Poco::Exception. Code: 1000, e.code() = 0, DNS error: EAI: Address family for hostname not supported (version 22.8.8.3 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
2023.07.27 13:34:47.756158 [ 7 ] {} <Warning> Application: Listen [::]:9005 failed: Poco::Exception. Code: 1000, e.code() = 0, DNS error: EAI: Address family for hostname not supported (version 22.8.8.3 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
2023.07.27 13:34:47.756370 [ 7 ] {} <Warning> Application: Listen [::]:9363 failed: Poco::Exception. Code: 1000, e.code() = 0, DNS error: EAI: Address family for hostname not supported (version 22.8.8.3 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
2023.07.27 13:34:47.845312 [ 7 ] {} <Warning> Application: Listen [::]:9009 failed: Poco::Exception. Code: 1000, e.code() = 0, DNS error: EAI: Address family for hostname not supported (version 22.8.8.3 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
Srikanth
Srikanth
01:45 PMWhat does pod describe show for pending pods?
Anil
Anil
01:46 PMEvents:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 11m default-scheduler Successfully assigned platform/my-release-signoz-alertmanager-0 to ip-10-0-1-36.us-west-2.compute.internal
Normal Pulled 11m kubelet Container image "docker.io/busybox:1.35" already present on machine
Normal Created 11m kubelet Created container my-release-signoz-alertmanager-init
Normal Started 11m kubelet Started container my-release-signoz-alertmanager-init
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 11m default-scheduler Successfully assigned platform/my-release-signoz-alertmanager-0 to ip-10-0-1-36.us-west-2.compute.internal
Normal Pulled 11m kubelet Container image "docker.io/busybox:1.35" already present on machine
Normal Created 11m kubelet Created container my-release-signoz-alertmanager-init
Normal Started 11m kubelet Started container my-release-signoz-alertmanager-init
01:47
Anil
01:47 PMthere is no error in describe pods, 5 pods are in init stage only
01:47
Anil
01:47 PMif you want we can huddle
Srikanth
Srikanth
01:48 PMIf there is no error I am not sure how I can help. This is probably generic k8s pending pods issue.
Anil
Anil
01:51 PM2023.07.27 13:34:47.756158 [ 7 ] {} <Warning> Application: Listen [::]:9005 failed: Poco::Exception. Code: 1000, e.code() = 0, DNS error: EAI: Address family for hostname not supported (version 22.8.8.3 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
Srikanth
Srikanth
01:55 PMThat is a warning.
Anil
Anil
01:58 PMi could observe that previous pods like front end, otel collector and otel metric pods of previous version are still running only. they are not deleting
01:58
Anil
01:58 PMwhen upgrading
Srikanth
Srikanth
02:00 PMThat’s because your new pods are stuck in init. You should resolve why they are pending init. There is no error logs from them and nor the pod describe show any errors.
Anil
Anil
02:01 PMwhen we are installing version 10.2 , we are setting pvc to that. is that causing issue ?
02:03
Anil
02:03 PMthis is the install command we used
helm --namespace platform install my-release signoz/signoz --set clickhouse.persistence.size=125Gi --version 0.2.5
helm --namespace platform install my-release signoz/signoz --set clickhouse.persistence.size=125Gi --version 0.2.5
02:06
Anil
02:06 PM?
Srikanth
Srikanth
02:10 PMI don’t think that is related to why pods are getting stuck.
Anil
Anil
04:48 PMError: cannot build pipelines: failed to create "clickhouselogsexporter" exporter
Srikanth
Srikanth
04:52 PMShare the full error when you share any error
SigNoz Community
Built with ClickHouse as datastore, SigNoz is an open-source APM to help you find issues in your deployed applications & solve them quickly | Knowledge Base powered by Struct.AI
Indexed 825 threads (61% resolved)
Similar Threads
SigNoz crashing in k8s due to ClickHouse OOM
Travis reported SigNoz crashing in k8s due to ClickHouse OOM. The team suggested increasing resources for ClickHouse, and other troubleshooting steps, but the issue remains unresolved.
77
6mo
Kubernetes Signoz-otel-collector Issue and Clickhouse Cold Storage
Pruthvi faced an issue with Kubernetes signoz-otel-collector. nitya-signoz suggested deleting the `signoz_logs` database and restarting collectors. Pruthvi then asked about Clickhouse cold storage on S3 and observed a spike in cost, which Ankit agreed to investigate further.
2 31
6mo
Solved
Issues with SigNoz Setup and Data Persistence in AKS
Vaibhavi experienced issues setting up SigNoz in AKS, and faced data persistence issues after installation. Srikanth provided guidance on ClickHouse version compatibility and resource requirements, helping Vaibhavi troubleshoot and resolve the issue.
59
5mo
Solved
Troubleshooting SigNoz Auto-Instrumentation Configuration
igor is having trouble configuring auto-instrumentation for Java applications using SigNoz, with traces not appearing in the SigNoz UI. Prashant advises to check logs of the otel sidecar, use service name for endpoint, verify supported libraries, and test with telemetrygen. However, the issue still persists.
34
5mo
Resolving Signoz Query Service Error
Einav encountered an error related to a missing table in the Signoz service which was preventing data visibility in the UI. Srikanth guided them to restart specific components and drop a database table, which resolved the issue.
17
1mo
Solved