#support

Troubleshooting Data Migration from Version 10.2 to 21.0

TLDR Anil seeks assistance on data migration from version 10.2 to 21.0. They faced a series of problems, from renaming tables to pods getting stuck in init. Srikanth was unable to conclusively resolve these issues.

Powered by Struct AI
Jul 26, 2023 (4 months ago)
Anil
Photo of md5-0dab8f2f962f63d6c9665dc6d172db0c
Anil
02:15 PM
Srikanth, i have one question regarding migration.
we have version 10.2 running in our environments, we need to upgrade to 21.0 . how the migration of data takes places ? could you please help me on this
Jul 27, 2023 (4 months ago)
Anil
Photo of md5-0dab8f2f962f63d6c9665dc6d172db0c
Anil
05:26 AM
oh , so its like series of steps we need to consider
09:24
Anil
09:24 AM
Srikanth, we have seen that migration to 0.12 , need to change the table names prefixed with distributed_
from 0.12 to 0.19 version we need to run the migration script which provided in the document right
09:24
Anil
09:24 AM
please correct me if my understanding is wrong
Srikanth
Photo of md5-ce04a9988e2fd758a659dc55be6f2543
Srikanth
01:13 PM
Yes
Anil
Photo of md5-0dab8f2f962f63d6c9665dc6d172db0c
Anil
01:17 PM
Srikanth, i executed migration script and upgraded from 10.2 to 0.12.0.
here is my script

RENAME TABLE samples_v2 TO distributed_samples_v2, time_series_v2 TO distributed_time_series_v2, usage TO distributed_usage;

RENAME TABLE dependency_graph_minutes TO distributed_dependency_graph_minutes, dependency_graph_minutes_db_calls_mv TO distributed_dependency_graph_minutes_db_calls_mv, dependency_graph_minutes_messaging_calls_mv TO distributed_dependency_graph_minutes_messaging_calls_mv, durationSort TO distributed_durationSort, signoz_error_index_v2 TO distributed_signoz_error_index_v2, signoz_spans TO distributed_signoz_spans, top_level_operations TO distributed_top_level_operations, signoz_index_v2 TO distributed_signoz_index_v2, usage TO distributed_usage, usage_explorer TO distributed_usage_explorer;
01:17
Anil
01:17 PM
but getting

my-release-signoz-otel-collector-6f47d74d65-nklqp 0/1 Init:0/1 0 8m15s
my-release-signoz-otel-collector-ddb76f957-dgjm6 0/1 Init:0/1 0 6m20s
Srikanth
Photo of md5-ce04a9988e2fd758a659dc55be6f2543
Srikanth
01:17 PM
You don’t have to change the table names your self
Anil
Photo of md5-0dab8f2f962f63d6c9665dc6d172db0c
Anil
01:18 PM
then how data will be migrated ?
Srikanth
Photo of md5-ce04a9988e2fd758a659dc55be6f2543
Srikanth
01:20 PM
It will be taken care during the upgrade. The distributed_ tables are virtual tables which will use the existing tables. The migration only says if you use ClickHouse queries in dashboards you need to change queries.
Anil
Photo of md5-0dab8f2f962f63d6c9665dc6d172db0c
Anil
01:20 PM
oh thanks buddy
01:21
Anil
01:21 PM
how to take back up of data
01:21
Anil
01:21 PM
?
01:21
Anil
01:21 PM
just in case i am asking
Srikanth
Photo of md5-ce04a9988e2fd758a659dc55be6f2543
Srikanth
01:26 PM
That’s a non-trivial process.
Anil
Photo of md5-0dab8f2f962f63d6c9665dc6d172db0c
Anil
01:37 PM
Srikanth, i just followed what you said

but still we are seeing this

my-release-signoz-alertmanager-0 0/1 Init:0/1 0 116s
my-release-signoz-frontend-57bc68fb7d-946tf 0/1 Init:0/1 0 117s
my-release-signoz-frontend-cf9b8d8c9-dg9b6 1/1 Running 0 10m
my-release-signoz-otel-collector-6f47d74d65-5kj5j 0/1 Init:0/1 0 117s
my-release-signoz-otel-collector-ddb76f957-6nr52 1/1 Running 0 10m
my-release-signoz-otel-collector-metrics-5dd74686c-5cpwb 1/1 Running 0 10m
my-release-signoz-otel-collector-metrics-84bbbddc6c-jfgkq 0/1 Init:0/1 0 117s
my-release-signoz-query-service-0 0/1 Init:0/1 0 116s
01:37
Anil
01:37 PM
pods are in init stage only
01:38
Anil
01:38 PM
chat version is 0.2.5 to 0.6.0
Srikanth
Photo of md5-ce04a9988e2fd758a659dc55be6f2543
Srikanth
01:38 PM
Where are ClickHouse pods?
Anil
Photo of md5-0dab8f2f962f63d6c9665dc6d172db0c
Anil
01:39 PM
chi-my-release-clickhouse-cluster-0-0-0 1/1 Running 0 100s
my-release-clickhouse-operator-b4c9cb776-jqxrm 2/2 Running 0 117s
my-release-k8s-infra-otel-agent-2gt5n 1/1 Running 0 117s
my-release-k8s-infra-otel-agent-4244z 1/1 Running 0 117s
my-release-k8s-infra-otel-agent-4989f 1/1 Running 0 117s
my-release-k8s-infra-otel-agent-4g59w 1/1 Running 0 117s
my-release-k8s-infra-otel-agent-948gs 1/1 Running 0 117s
my-release-k8s-infra-otel-agent-bjbtp 1/1 Running 0 117s
my-release-k8s-infra-otel-agent-jl4bt 1/1 Running 0 117s
my-release-k8s-infra-otel-agent-jql8z 1/1 Running 0 117s
my-release-k8s-infra-otel-agent-kz86h 1/1 Running 0 117s
my-release-k8s-infra-otel-agent-lxrg7 1/1 Running 0 117s
my-release-k8s-infra-otel-agent-m2wzw 1/1 Running 0 117s
my-release-k8s-infra-otel-agent-p2ht4 1/1 Running 0 117s
my-release-k8s-infra-otel-agent-qpqcp 1/1 Running 0 117s
my-release-k8s-infra-otel-agent-rr6j6 1/1 Running 0 117s
my-release-k8s-infra-otel-agent-xt67c 1/1 Running 0 117s
my-release-k8s-infra-otel-agent-z28nm 1/1 Running 0 117s
my-release-k8s-infra-otel-deployment-5b8566bc46-bfzcf 1/1 Running 0 117s
my-release-signoz-alertmanager-0 0/1 Init:0/1 0 116s
my-release-signoz-frontend-57bc68fb7d-946tf 0/1 Init:0/1 0 117s
my-release-signoz-frontend-cf9b8d8c9-dg9b6 1/1 Running 0 10m
my-release-signoz-otel-collector-6f47d74d65-5kj5j 0/1 Init:0/1 0 117s
my-release-signoz-otel-collector-ddb76f957-6nr52 1/1 Running 0 10m
my-release-signoz-otel-collector-metrics-5dd74686c-5cpwb 1/1 Running 0 10m
my-release-signoz-otel-collector-metrics-84bbbddc6c-jfgkq 0/1 Init:0/1 0 117s
my-release-signoz-query-service-0 0/1 Init:0/1 0 116s
my-release-zookeeper-0 1/1 Running 0 10m
Srikanth
Photo of md5-ce04a9988e2fd758a659dc55be6f2543
Srikanth
01:39 PM
Check the logs of ClickHouse if there are any issues
01:40
Srikanth
01:40 PM
Or it may be resources issue.
Anil
Photo of md5-0dab8f2f962f63d6c9665dc6d172db0c
Anil
01:41 PM
helm --namespace platform install my-release signoz/signoz --set clickhouse.persistence.size=125Gi --version 0.2.5

helm -n platform upgrade my-release signoz/signoz --version 0.6.0
01:41
Anil
01:41 PM
so first we installed 10.2 and then upgraded
01:44
Anil
01:44 PM
logs


2023.07.27 13:34:47.755403 [ 7 ] {} <Warning> Application: Listen [::]:8123 failed: Poco::Exception. Code: 1000, e.code() = 0, DNS error: EAI: Address family for hostname not supported (version 22.8.8.3 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
2023.07.27 13:34:47.755642 [ 7 ] {} <Warning> Application: Listen [::]:9000 failed: Poco::Exception. Code: 1000, e.code() = 0, DNS error: EAI: Address family for hostname not supported (version 22.8.8.3 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
2023.07.27 13:34:47.755952 [ 7 ] {} <Warning> Application: Listen [::]:9004 failed: Poco::Exception. Code: 1000, e.code() = 0, DNS error: EAI: Address family for hostname not supported (version 22.8.8.3 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
2023.07.27 13:34:47.756158 [ 7 ] {} <Warning> Application: Listen [::]:9005 failed: Poco::Exception. Code: 1000, e.code() = 0, DNS error: EAI: Address family for hostname not supported (version 22.8.8.3 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
2023.07.27 13:34:47.756370 [ 7 ] {} <Warning> Application: Listen [::]:9363 failed: Poco::Exception. Code: 1000, e.code() = 0, DNS error: EAI: Address family for hostname not supported (version 22.8.8.3 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
2023.07.27 13:34:47.845312 [ 7 ] {} <Warning> Application: Listen [::]:9009 failed: Poco::Exception. Code: 1000, e.code() = 0, DNS error: EAI: Address family for hostname not supported (version 22.8.8.3 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
Srikanth
Photo of md5-ce04a9988e2fd758a659dc55be6f2543
Srikanth
01:45 PM
What does pod describe show for pending pods?
Anil
Photo of md5-0dab8f2f962f63d6c9665dc6d172db0c
Anil
01:46 PM
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 11m default-scheduler Successfully assigned platform/my-release-signoz-alertmanager-0 to ip-10-0-1-36.us-west-2.compute.internal
Normal Pulled 11m kubelet Container image "docker.io/busybox:1.35" already present on machine
Normal Created 11m kubelet Created container my-release-signoz-alertmanager-init
Normal Started 11m kubelet Started container my-release-signoz-alertmanager-init
01:47
Anil
01:47 PM
there is no error in describe pods, 5 pods are in init stage only
01:47
Anil
01:47 PM
if you want we can huddle
Srikanth
Photo of md5-ce04a9988e2fd758a659dc55be6f2543
Srikanth
01:48 PM
If there is no error I am not sure how I can help. This is probably generic k8s pending pods issue.
Anil
Photo of md5-0dab8f2f962f63d6c9665dc6d172db0c
Anil
01:51 PM
2023.07.27 13:34:47.756158 [ 7 ] {} <Warning> Application: Listen [::]:9005 failed: Poco::Exception. Code: 1000, e.code() = 0, DNS error: EAI: Address family for hostname not supported (version 22.8.8.3 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
Srikanth
Photo of md5-ce04a9988e2fd758a659dc55be6f2543
Srikanth
01:55 PM
That is a warning.
Anil
Photo of md5-0dab8f2f962f63d6c9665dc6d172db0c
Anil
01:58 PM
i could observe that previous pods like front end, otel collector and otel metric pods of previous version are still running only. they are not deleting
01:58
Anil
01:58 PM
when upgrading
Srikanth
Photo of md5-ce04a9988e2fd758a659dc55be6f2543
Srikanth
02:00 PM
That’s because your new pods are stuck in init. You should resolve why they are pending init. There is no error logs from them and nor the pod describe show any errors.
Anil
Photo of md5-0dab8f2f962f63d6c9665dc6d172db0c
Anil
02:01 PM
when we are installing version 10.2 , we are setting pvc to that. is that causing issue ?
02:03
Anil
02:03 PM
this is the install command we used

helm --namespace platform install my-release signoz/signoz --set clickhouse.persistence.size=125Gi --version 0.2.5
02:06
Anil
02:06 PM
?
Srikanth
Photo of md5-ce04a9988e2fd758a659dc55be6f2543
Srikanth
02:10 PM
I don’t think that is related to why pods are getting stuck.
Anil
Photo of md5-0dab8f2f962f63d6c9665dc6d172db0c
Anil
04:48 PM
Error: cannot build pipelines: failed to create "clickhouselogsexporter" exporter
Srikanth
Photo of md5-ce04a9988e2fd758a659dc55be6f2543
Srikanth
04:52 PM
Share the full error when you share any error

SigNoz Community

Built with ClickHouse as datastore, SigNoz is an open-source APM to help you find issues in your deployed applications & solve them quickly | Knowledge Base powered by Struct.AI

Indexed 1023 threads (61% resolved)

Join Our Community

Similar Threads

Issue with Helm Installation in GKE Autopilot Cluster

Kalman faced issues with helm installation with pods stuck in init state, and some crashing in a GKE autopilot cluster. Mayur provided suggestions to diagnose the issue, including checking IAM permissions and storage classes, and adjusting resource limits in the helm values. The thread is unresolved.

4

38
1mo

SigNoz crashing in k8s due to ClickHouse OOM

Travis reported SigNoz crashing in k8s due to ClickHouse OOM. The team suggested increasing resources for ClickHouse, and other troubleshooting steps, but the issue remains unresolved.

77
8mo

Kubernetes Signoz-otel-collector Issue and Clickhouse Cold Storage

Pruthvi faced an issue with Kubernetes signoz-otel-collector. nitya-signoz suggested deleting the `signoz_logs` database and restarting collectors. Pruthvi then asked about Clickhouse cold storage on S3 and observed a spike in cost, which Ankit agreed to investigate further.

ack

2

31
8mo

Issues with SigNoz Setup and Data Persistence in AKS

Vaibhavi experienced issues setting up SigNoz in AKS, and faced data persistence issues after installation. Srikanth provided guidance on ClickHouse version compatibility and resource requirements, helping Vaibhavi troubleshoot and resolve the issue.

59
8mo

Troubleshooting SigNoz Auto-Instrumentation Configuration

igor is having trouble configuring auto-instrumentation for Java applications using SigNoz, with traces not appearing in the SigNoz UI. Prashant advises to check logs of the otel sidecar, use service name for endpoint, verify supported libraries, and test with telemetrygen. However, the issue still persists.

34
8mo