#support

Trouble with EKS Nodegroup Taints Blocking data logging

TLDR Saad had problems managing the logging of tainted nodes in his EKS cluster by his otel-collector. Viljar suggested adjusting tolerations on the singoz helm chart.

3mo

Join the chat

Aug 31, 2023 (3 months ago)

Photo of md5-0bd9a0371f7ef9304146ba0a1709ceb1

Saad

09:30 AM

Hi Team, I have deployed signoz on my EKS cluster - I have a managed nodegroup which have on-demand nodes and there is another group managed by karpenter which are on spot and they are tainted nodes.

When i deploy signoz - it is deployed on to the on-demand nodes (cause the other nodes are tainted) but the otel-collector which is a deamon set is not collecting logs from the other ndoes which are tainted - so i have added the tolerations for all the otel daemon set and the deployment - but still the pod is not schedule on the tainted nodes - and my applications are running on those tainted nodes and since the otel-collector is not running on these nodes - the logs are not getting exported

Need URGENT help here. Thanks

Sep 05, 2023 (3 months ago)

Photo of md5-c8bbdeca90864acb83624f1a951e337c

Viljar

05:47 AM

check your affinity and taints. i had same issue . otelAgent tolerations on singoz helm chart is what needs to be configured.

otelAgent:
  # NB: if adding node group and wanting it to be monitred add toleration here
  tolerations:
    - key: ""
      operator: "Exists"
      effect: "NoSchedule"

05:47

Viljar

05:47 AM

something like that

SigNoz Community

Built with ClickHouse as datastore, SigNoz is an open-source APM to help you find issues in your deployed applications & solve them quickly | Knowledge Base powered by Struct.AI

Indexed 1023 threads (61% resolved)

Join Our Community

Similar Threads

Troubleshooting Signoz Deployment for Log Collection on OCI k8s Cluster

sudhanshu faces issues with log collection despite successful application performance monitoring after deploying Signoz on OCI k8s cluster. Prashant suggests updating paths used for log collection and verifying container logs on the nodes. Log files' paths seem correct, but issues persist. Further solutions are needed.

1mo

Issues with SigNoz Setup and Data Persistence in AKS

Vaibhavi experienced issues setting up SigNoz in AKS, and faced data persistence issues after installation. Srikanth provided guidance on ClickHouse version compatibility and resource requirements, helping Vaibhavi troubleshoot and resolve the issue.

8mo

Issue with Pending States in AWS Cluster

Jatin reported that Kubernetes pods get stuck in pending state when nodes go down in their AWS cluster. Despite providing `kubectl describe` results and logs, Prashant couldn't specify the cause, citing deeper investigation into the cluster and k8s resources would be required.

4mo