#support

Troubleshooting K8s Deployment and Suggested Fields

TLDR saajid had issues with k8s deployment suggested fields. nitya-signoz suggested redeploying and shared troubleshooting steps.

6mo

Join the chat

Mar 08, 2023 (6 months ago)

Photo of md5-6291de702a56e957b4242d67fc5d3458

saajid

06:21 PM

Hi,
The docs state that the k8s deployment automatically parses container_name and other relevant k8 fields, and I can see them within the log record. Do I have to do another step to make it a "suggested field" or should I be able to query it as is.

Side Question: Are there any k8 helm examples for modifying the collector for json parsing specifically?

Thanks in advance.

Image 1 for Hi,<br> The docs state that the k8s deployment automatically parses container_name and other relevant k8 fields, and I can see them within the log record. Do I have to do another step to make it a "suggested field" or should I be able to query it as is.<br><br>Side Question: Are there any k8 helm examples for modifying the collector for json parsing specifically?<br><br>Thanks in advance.

Mar 09, 2023 (6 months ago)

Photo of md5-a52b9d6c34f193d9a1ff940024f36f77

nitya-signoz

07:18 AM

I don’t see k8s_container_name in selected or interesting fields, if it’s not present you won’t be able to query it.

Are your logs getting ingested? also where is your k8s cluster running can you provide some more info.

saajid

02:31 PM

Yep, all the logs are coming through as expected, and I can see the resources_string.k8s_container_name in the log record. I'm running on eks on aws.

Image 1 for Yep, all the logs are coming through as expected, and I can see the <code>resources_string.k8s_container_name</code> in the log record. I'm running on eks on aws.

nitya-signoz

03:37 PM

If thats the case then these should also appear as fields on the left side of the logs page.

can you share the output of the following commands.

kubectl exec -n platform -it chi-my-release-clickhouse-cluster-0-0-0 -- sh

clickhouse client

use signoz_logs;

select distinct name from distributed_logs_resource_keys;

saajid

03:40 PM

SELECT DISTINCT name
FROM distributed_logs_resource_keys

Query id: 02a3aa7e-bcc4-4b13-a527-0223530f3dd4

┌─name────────────────────────┐
│ host_name                   │
│ os_type                     │
│ signoz_component            │
└─────────────────────────────┘

03:42

saajid

03:42 PM

I've deployed the helm to another environment with the same setup and it show and is queriable as expected... is it possible a redeployment should solve it?

nitya-signoz

03:42 PM

One more question how many shards of clickhouse do you have ?

saajid

03:46 PM

It should just be the one.

nitya-signoz

03:47 PM

Got it then if your logs data is not important you can stop the otel collector pod, delete the signoz_logs database and restart the collectors.

Else we can schedule a call to fix this.

Commands will be same as this https://signoz-community.slack.com/archives/C01HWQ1R0BC/p1678376272927989?thread_ts=1678299713.440809&cid=C01HWQ1R0BC

drop database signoz_logs

saajid

03:48 PM

Awesome, thank you. I'll give it a go and report back

07:00

saajid

07:00 PM

Reporting back, deleting the database broke things and signoz was unable to recover gracefully, no logs and was unable to get new logs even after rebooting all pods. I did however took everything down and redeployed and things seem to be working as the other environment now. Thanks for your time.

Mar 10, 2023 (6 months ago)

nitya-signoz

04:39 AM

That’s strange, did you stop the collectors before deleting the database?

And by broke can you share what happened exactly ?

saajid

02:31 PM

yep, stop everything but the clickhouse pod.

And by broke I mean no logs after starting them back up and no errors in either the agents or the collectors.

SigNoz Community

Built with ClickHouse as datastore, SigNoz is an open-source APM to help you find issues in your deployed applications & solve them quickly | Knowledge Base powered by Struct.AI

Indexed 825 threads (61% resolved)

Join Our Community Manage subscription

Similar Threads

Issues with SigNoz Setup and Data Persistence in AKS

Vaibhavi experienced issues setting up SigNoz in AKS, and faced data persistence issues after installation. Srikanth provided guidance on ClickHouse version compatibility and resource requirements, helping Vaibhavi troubleshoot and resolve the issue.

5mo

SigNoz crashing in k8s due to ClickHouse OOM

Travis reported SigNoz crashing in k8s due to ClickHouse OOM. The team suggested increasing resources for ClickHouse, and other troubleshooting steps, but the issue remains unresolved.

6mo

Issue with Filters on Logs and Duplicate Fields

Patrick reported issues with filter functionality and duplicate fields in logs. Srikanth looked into the problem. nitya-signoz offered a solution involving an update and guide.

6mo

Dashboard Load Issues and Possible Solutions

Al experiences dashboard loading issues since updating to `0.18.1`. Srikanth believes the issue is not version related and suggests examining queries, memory resources, and server distribution for improvements.

4mo

Troubleshooting and Adding Log Files to SigNoz POC

Noor has requested help with incorporating log files into their SigNoz POC. In collaboration with vishal-signoz and nitya-signoz, they managed to successfully setup and resolve their issues.

115

1mo