Unable to Monitor jdbc_connections_active Metrics in SigNoz

TLDR Ragulraj is unable to monitor jdbc_connections_active metrics with SigNoz despite modifying configurations and adding dependencies. Srikanth provided troubleshooting steps and info on alternatives, but the issue remains unresolved.

Photo of Ragulraj
Ragulraj
Thu, 13 Apr 2023 04:01:40 UTC

Hello team i was trying to monitor jdbc_connections_active metrics for my service in signoz , but not able to get data when i give PromQL in dashboard and even i tried by enabling Dotel.instrumentation.jdbc-datasource.enabled=true , is there any other equivalent metrics for jdbc

Photo of Srikanth
Srikanth
Thu, 13 Apr 2023 06:20:28 UTC

> jdbc-datasource.enabled=true This is not related to the metric. _`jdbc-datasource` which creates spans whenever the `java.sql.DataSource#getConnection` method is called._ Where is the source of metrics data? like micrometer or OTEL native or something else?

Photo of Srikanth
Srikanth
Thu, 13 Apr 2023 06:23:27 UTC

OpenTeleme has `db_client_connections_usage` with state either `idle` /`used` and there are several other which can be found here

Photo of Ragulraj
Ragulraj
Thu, 13 Apr 2023 07:15:25 UTC

```#!/bin/bash java -Dlog-path=/logs -Dspring.config.location=file:/properties/fincluez-l2t-ms.yml,file:/properties/fincluez-t-ms-loan.yml -Dlogging.config=file:/properties/logback-spring-loan.xml -javaagent:/opentelemetry/opentelemetry-javaagent.jar -Dotel.metrics.exporter=none -Dotel.exporter.otlp.endpoint=http://${signoz_ip} -Dotel.instrumentation.jdbc-datasource.enabled=true -Dotel.resource.attributes="service.name=fincluez-l2t-loans" -jar loans-l2t-22.2.0-0.1.war``` i gave those while running service , do i need to change Dotel.instrumentation.jdbc-datasource.enabled=true to Dotel.instrumentation.db.client.connections.usage=idle like this ?

Photo of Srikanth
Srikanth
Thu, 13 Apr 2023 07:18:32 UTC

> do i need to change Dotel.instrumentation.jdbc-datasource.enabled=true to Dotel.instrumentation.db.client.connections.usage=idle like this ? The `db_client_connections_usage` is a metric name, not a configuration option. > ```Dotel.metrics.exporter=none``` Remove this if you want metrics. If this is from our docs/blogs we will get it updated.

Photo of Ragulraj
Ragulraj
Thu, 13 Apr 2023 09:49:33 UTC

if i remove this Dotel.metrics.exporter=none can i monitor other metrics which mentioned in docs , or it only specific to jdbc metric

Photo of Srikanth
Srikanth
Thu, 13 Apr 2023 10:01:47 UTC

You can monitor all of the mentioned metrics.

Photo of Ragulraj
Ragulraj
Thu, 13 Apr 2023 11:33:33 UTC

still i am not able to monitor db_client_connections_usage , after removing the above configuration

Photo of Srikanth
Srikanth
Thu, 13 Apr 2023 11:35:39 UTC

Where did the “instances” come from? Did you add the label yourself? Does it even exist? When you work with any metric data, the easy and the best thing is to run as simple as query with metric name and see what data exists.

Photo of Ragulraj
Ragulraj
Thu, 13 Apr 2023 11:37:28 UTC

even if i run query only with metrics name i am not getting

Photo of Srikanth
Srikanth
Thu, 13 Apr 2023 11:39:30 UTC

Then there is likely high chance the data is not being received. Check your setup properly. Check the logs of the collector if it has any issues.

Photo of Ragulraj
Ragulraj
Thu, 13 Apr 2023 11:41:58 UTC

Ok i ll check

Photo of Ragulraj
Ragulraj
Thu, 13 Apr 2023 13:04:59 UTC

, from this list of metrics , we able to montior all except jdbc and hikari metric . as we are using hikari connection we need those metrics . is there any alternative metrics for these two . as i checked the logs i got "failed to scrape prometheus endpoint" as warning from otel-collector other than this no other info from logs . please help in monitoring these 2 metrics

Photo of Srikanth
Srikanth
Thu, 13 Apr 2023 13:06:27 UTC

> failed to scrape prometheus endpoint You need to fix this. Make sure the endpoint you are configuring is correct and reachable by collector.

Photo of Ragulraj
Ragulraj
Fri, 14 Apr 2023 07:24:24 UTC

i crosschecked with some query in clickhouse DB -> SELECT name FROM ( SELECT DISTINCT JSONExtractString(labels, '__name__') AS name FROM signoz_metrics.time_series) AS tmp1 WHERE name ILIKE '%jvm%' ,its giving some result as some list jvm metrics .same i tried with (jdbc) SELECT name FROM ( SELECT DISTINCT JSONExtractString(labels, '__name__') AS name FROM signoz_metrics.time_series) AS tmp1 WHERE name ILIKE '%jdbc%' the response result is 0 . any changes i need to do from clickhouse DB side ?

Photo of Srikanth
Srikanth
Fri, 14 Apr 2023 07:43:20 UTC

Is your failed to scrape fixed? I was requesting you to fix the failed to scrape. How does this cross checking or anything about the ClickHouse help if there is no data collected?

Photo of Ragulraj
Ragulraj
Fri, 14 Apr 2023 09:34:19 UTC

i was trying to fix those scrape i have a doubt in this prometheus: config: scrape_configs: - job_name: "otel-collector" scrape_interval: 60s static_configs: - targets: ["otel-collector:8889"] - job_name: "jvm-metrics" scrape_interval: 10s metrics_path: "/actuator/prometheus" static_configs: - targets: ["<IP of the machine:8090>"] here i should provide our service details or it should prometheus details

Photo of Ragulraj
Ragulraj
Fri, 14 Apr 2023 11:37:28 UTC

please help in me fixing failed to scrape , any hint like where to check

Photo of Srikanth
Srikanth
Fri, 14 Apr 2023 11:53:10 UTC

It should be the endpoint of your service which is emitting the metrics. For example, if your application is running on some host `10.53.34.50` and port `8090` you can verify if it emits metrics by curling with `curl `. And since there is already `metric_path: "/actuator/prometheus"` covered in config, in the target you just need to provide `host:port` where port is `8090` and `host` is the host address where application service is running.

Photo of Ragulraj
Ragulraj
Mon, 17 Apr 2023 10:04:42 UTC

eventhough it is showing failing to scrape , i am able to get other metrics except jdbc and hikari

Photo of Ragulraj
Ragulraj
Mon, 17 Apr 2023 10:05:11 UTC

can u give me hint , that were i need to check failed to scrape

Photo of Srikanth
Srikanth
Mon, 17 Apr 2023 11:17:28 UTC

> eventhough it is showing failing to scrape , i am able to get other metrics except jdbc and hikari The jdbc and hikari metrics are coming from the micrometer and need to be scraped. The rest of the metrics could be coming from OTLP. > can u give me hint , that were i need to check failed to scrape This is mostly like a network issue. Please make sure the host and port are correct and are reachable to the SigNoz collector.

Photo of Ragulraj
Ragulraj
Tue, 18 Apr 2023 10:11:06 UTC

host and port are correct i am able to reach to the service from otel collector . whether I need to any dependency regarding micrometer in my pom.xml

Photo of Ragulraj
Ragulraj
Tue, 18 Apr 2023 10:33:03 UTC

and we are using signoz 0.8.2 version is jdbc and hikari metrics available for that version

Photo of Ragulraj
Ragulraj
Tue, 18 Apr 2023 10:50:43 UTC

i tried by adding some dependency in my pom.xml ```<dependency> <groupId>io.opentelemetry</groupId> <artifactId>opentelemetry-api</artifactId> </dependency> <dependency> <groupId>io.opentelemetry</groupId> <artifactId>opentelemetry-api-metrics</artifactId> <version>1.9.1-alpha</version> </dependency>```

Photo of Ragulraj
Ragulraj
Tue, 18 Apr 2023 10:56:29 UTC

curl -X GET 10.53.34.50:3774/alpha-transformation-ms/actuator/prometheus and i am able to run this for checking whether service is reachable or not

Photo of Srikanth
Srikanth
Tue, 18 Apr 2023 12:25:49 UTC

The `metrics_path` in the configuration then should be `/alpha-transformation-ms/actuator/prometheus`

Photo of Ragulraj
Ragulraj
Tue, 18 Apr 2023 12:36:28 UTC

yes same i given in config,yml also

Photo of Srikanth
Srikanth
Tue, 18 Apr 2023 12:51:26 UTC

If you run `curl -X GET 10.53.34.50:3774/alpha-transformation-ms/actuator/prometheus` from the collector host, does it resolve and return any data? Did you have the micrometer setup

Photo of Ragulraj
Ragulraj
Wed, 19 Apr 2023 03:43:11 UTC

yes i am able to get data from the curl , regarding micrometer we have added only dependency in out spring application based on the docs

Photo of Ragulraj
Ragulraj
Wed, 19 Apr 2023 03:43:58 UTC

any other things i need to configure regarding micrometer apart from this ?

Photo of Ragulraj
Ragulraj
Wed, 19 Apr 2023 04:00:51 UTC

is htrod and load-htrod service is mandatory one ?