Trouble with EKS Nodegroup Taints Blocking data logging

TLDR Saad had problems managing the logging of tainted nodes in his EKS cluster by his otel-collector. Viljar suggested adjusting tolerations on the singoz helm chart.

Powered by Struct AI
Aug 31, 2023 (3 months ago)
Photo of md5-0bd9a0371f7ef9304146ba0a1709ceb1
09:30 AM
Hi Team, I have deployed signoz on my EKS cluster - I have a managed nodegroup which have on-demand nodes and there is another group managed by karpenter which are on spot and they are tainted nodes.

When i deploy signoz - it is deployed on to the on-demand nodes (cause the other nodes are tainted) but the otel-collector which is a deamon set is not collecting logs from the other ndoes which are tainted - so i have added the tolerations for all the otel daemon set and the deployment - but still the pod is not schedule on the tainted nodes - and my applications are running on those tainted nodes and since the otel-collector is not running on these nodes - the logs are not getting exported

Need URGENT help here. Thanks
Sep 05, 2023 (3 months ago)
Photo of md5-c8bbdeca90864acb83624f1a951e337c
05:47 AM
check your affinity and taints. i had same issue . otelAgent tolerations on singoz helm chart is what needs to be configured.
  # NB: if adding node group and wanting it to be monitred add toleration here
    - key: ""
      operator: "Exists"
      effect: "NoSchedule"
05:47 AM
something like that