Integrating TimescaleDB Prometheus Data with SigNoz

Hi all, I need some help

I have enabled Prometheus integration form TimeseriesDB, as stated here:
https://docs.timescale.com/mst/latest/integrations/prometheus-mst/
then I have edited file: deploy/docker/clickhouse-setup/otel-collector-metrics-config.yaml
and added this section:

        # Timescale Dev
        - job_name: timescale-dev-collector
          scrape_interval: 10s
          evaluation_interval: 10s
          scheme: https
          static_configs:
                  - targets: ['my-url:9273']
          tls_config:
            insecure_skip_verify: true
          basic_auth:
            username: milan
            password: password

But I don’t see anything in SigNoz. I have tried to add new dashboard with system_filesystem_usage_total, system_cpu_time_total, as PromQL, but nothing is displayed.
How I can check Prometheus data from my TimeseriesDB on SigNoz instance?

Thanks

> But I don’t see anything in SigNoz. I have tried to add new dashboard with system_filesystem_usage_total, system_cpu_time_total, as PromQL, but nothing is displayed.
How are these related to the Timescale Prometheus metric data?

On Timescale TB I have Enabled Prometheus service integration
On SigNoz I have set up otel-collector-metrics-config.yaml, as stated above.
Now I am not sure how check are data flowing into SigNoz?

Milan can you verify that the my-url:9273 is accessible from VM/Cluster where signoz is deployed?

After that, you can use suggestions from Query Builder from dashboard page to verify if any timescale related metrics have been collected.

1. I am able to see my-url:9273 from my signoz instance
2. I did stop, removed containers clickhouse-setup-otel-collector-1 and clickhouse-setup-otel-collector-metrics-1 and re-run install script
3. however, in dashboard when I try to save panel with e.g system_filesystem_usage_total I get response: “No Data”
I am not sure anything is coming to my SigNoz instace from TimeseriesDB?

>however, in dashboard when I try to save panel with e.g system_filesystem_usage_total I get response: “No Data”

Is the endpoint my-url:9273 expose this metric?

when I use GET my-url:9273/metrics, I am able to see huge response
like
cpu_usage_guest, diskio_merged_reads,…
and when I try to use them in dashboard I get response: No Data

Are you sure the system_filesystem_usage_total exists in response from my-url:9273/metrics

you are right, that does not exist. but I can’t display any of metrics that do exists

Did you mean you can’t display or you can display?

can’t

Can you open the network tab and share the response when you try to run query?

What is the metric type and temporality for them?

# HELP disk_used_percent Telegraf collected metric
# TYPE disk_used_percent gauge

Can you confirm if it was really scraped by otel-collector-metrics? Do logs show any errors? Make sure it is reachable.

That is the problem, I did prepare it, as stated above, but not sure how to see on Signoz side :-(\

> Do logs show any errors?

Here is output

when I do this:


docker logs clickhouse-setup-otel-collector-metrics-1 | grep error

* error decoding 'receivers': error reading receivers configuration for "prometheus": prometheus receiver failed to unmarshal yaml to prometheus config: yaml: unmarshal errors:
  line 17: field evaluation_interval not found in type config.ScrapeConfig
  line 29: field evaluation_interval not found in type config.ScrapeConfig
2023/06/07 20:09:50 application run finished with error: failed to get config: cannot unmarshal the configuration: 1 error(s) decoding:

* error decoding 'receivers': error reading receivers configuration for "prometheus": prometheus receiver failed to unmarshal yaml to prometheus config: yaml: unmarshal errors:
  line 17: field evaluation_interval not found in type config.ScrapeConfig
  line 29: field evaluation_interval not found in type config.ScrapeConfig
Error: failed to get config: cannot unmarshal the configuration: 1 error(s) decoding:

* error decoding 'receivers': error reading receivers configuration for "prometheus": prometheus receiver failed to unmarshal yaml to prometheus config: yaml: unmarshal errors:
  line 17: field evaluation_interval not found in type config.ScrapeConfig
  line 29: field evaluation_interval not found in type config.ScrapeConfig
2023/06/07 20:09:50 application run finished with error: failed to get config: cannot unmarshal the configuration: 1 error(s) decoding:

and this

docker logs clickhouse-setup-otel-collector-metrics-1 | grep error

7->10.33.105.29:63788: read: connection reset by peer    {"grpc_log": true}
2023-06-07T20:14:54.465Z    warn    zapgrpc/zapgrpc.go:191    [transport] transport: http2Server.HandleStreams failed to read frame: read tcp 173.19.0.7:4317->10.33.105.29:63822: read: connection reset by peer    {"grpc_log": true}
2023-06-07T20:14:55.280Z    warn    zapgrpc/zapgrpc.go:191    [transport] transport: http2Server.HandleStreams failed to read frame: read tcp 173.19.0.7:4317->10.33.144.245:57520: read: connection reset by peer    {"grpc_log": true}
2023-06-07T20:14:59.327Z    warn    zapgrpc/zapgrpc.go:191    [transport] transport: http2Server.HandleStreams failed to read frame: read tcp 173.19.0.7:4317->10.33.144.245:57546: read: connection reset by peer    {"grpc_log": true}
2023-06-07T20:15:06.169Z    warn    zapgrpc/zapgrpc.go:191    [transport] transport: http2Server.HandleStreams failed to read frame: read tcp 173.19.0.7:4317->10.33.105.29:29906: read: connection reset by peer    {"grpc_log": true}
2023-06-07T20:15:06.821Z    warn    zapgrpc/zapgrpc.go:191    [transport] transport: http2Server.HandleStreams failed to read frame: read tcp 173.19.0.7:4317->10.33.144.245:57492: read: connection reset by peer    {"grpc_log": true}

It says config is invalid.
>

field evaluation_interval not found in type config.ScrapeConfig

>

I managed to set all up. Issue was that env variable evaluation_interval: 10s was not needed.