Incorporating Namespace Metrics in a JVM Application

TLDR Soumyadeep sought advice on displaying namespace metrics in their JVM application, specifically using the java agent. Shivanshu suggested using a kubernetes namespace informer and the `pod_association` function. Ultimately, Soumyadeep resolved the issue by adding a kube manifest, and posed a new query about the "Resolved" alerts on signoz UI.

Photo of Soumyadeep
Soumyadeep
Fri, 18 Aug 2023 15:18:50 UTC

Hey I have auto instrumented my jvm application using the the java agent, My application uses dropwizard but I dont have dropwizard metrics collection enabled for OTEL so the metrics it is collecting I assume is the jetty metrics like "http_request_duration" etc. My service is running in kubernetes so I would like namespace to be part of these metrics but thats not coming. I tried to add `k8sattributes` processor to all pipelines but I dont think the metric source itself has the namespace then it cant process it. Also, I noticed that all spans get collected within "signoz_calls_total" so even that metric works for me, but the namespace in that is always the namespace where the collector is running instead of the service whose spans are being collected. Are there any suggestions here that I can do with minimal effort to get namespace?

Photo of Shivanshu
Shivanshu
Fri, 18 Aug 2023 15:53:42 UTC

Hey, the way it works is, there's an kubernetes namespace informer which fetches the namespace metadata and then embed the relevant info using `k8sattributes` processor. The catch here is, we need a way to correlate the pod metadata somehow to the attributes collected, for that purpose `pod_association` is used, you'd need a service account configured with `get`, `watch` and `list` permissions to be able to fetch the data from API server. Kindly refer to the documentation here for configuration . I had tried this in past and it works, let us know if you still face the issue.

Photo of Soumyadeep
Soumyadeep
Sat, 19 Aug 2023 09:37:10 UTC

I had done the above but that didnt work.

Photo of Soumyadeep
Soumyadeep
Sat, 19 Aug 2023 09:38:26 UTC

But I figured a diff way to add them to resource attribute.

Photo of Soumyadeep
Soumyadeep
Sat, 19 Aug 2023 09:38:46 UTC

I just added it as part of my kube manifest that add namespace to resource attribute and that worked.

Photo of Soumyadeep
Soumyadeep
Sat, 19 Aug 2023 09:38:49 UTC

Thanks for the help.

Photo of Soumyadeep
Soumyadeep
Sat, 19 Aug 2023 09:40:05 UTC

Another question I had was I dont see a way to switch off "Resolved" alerts on signoz UI. Is there a way?

Photo of Pranay
Pranay
Mon, 21 Aug 2023 04:10:29 UTC

Can you share what do you mean by "Resolved" alerts? Do you mean you are getting alert notification even if the issues is resolved?