Empty HostMetrics Dashboard in SigNoz v0.19.0
TLDR Riccardo explained facing issues with empty dashboard panels after installing SigNoz v0.19.0. Prashant and Srikanth offered troubleshooting suggestions, including upgrading to otel-collector 0.76.1, but the issue remains unresolved.
Prashant
Wed, 07 Jun 2023 04:55:40 UTCcan you share screenshot of the empty dashboard?
Can you open a widget and try plotting any of the metrics below?
Prashant
Wed, 07 Jun 2023 05:10:57 UTCAlso, do check logs of otel-collectors if there were any errors.
Riccardo
Wed, 07 Jun 2023 07:59:50 UTCHi Prashant, thank you for your help.
Here is the screenshot of empty dashboard:
And the screenshot of Traces section.
I also attached the log of otel-collector on source VM; it show a "connection refused" that seems replied by application, because there is no firewall between VMs and if from source I try to telnet the destination I obtain the following:
```Trying 213.152.204.19...
Connected to
Riccardo
Wed, 07 Jun 2023 08:01:23 UTC
Prashant
Wed, 07 Jun 2023 08:12:53 UTCRiccardo which version of signoz are you using? also, did you select hostname from top selection bar?
Riccardo
Wed, 07 Jun 2023 08:15:27 UTCSignoz v0.19.0
Riccardo
Wed, 07 Jun 2023 08:15:38 UTCI can't see top selection bar
Prashant
Wed, 07 Jun 2023 08:15:57 UTCFrom logs, it seems like SigNoz-Otel-Collector is either not accessible or not healthy.
Prashant
Wed, 07 Jun 2023 08:16:08 UTClogs of signoz-otel-collector would be helpful.
Prashant
Wed, 07 Jun 2023 08:16:22 UTCalso, look for errors in clickhouse container.
Riccardo
Wed, 07 Jun 2023 08:16:51 UTCI already have attached the log from otel collector on source VM
Riccardo
Wed, 07 Jun 2023 08:17:07 UTCdo you need some other log?
Prashant
Wed, 07 Jun 2023 08:17:28 UTCThe one you shared would be logs from otel-collector binary.
Riccardo
Wed, 07 Jun 2023 08:17:36 UTCyes
Prashant
Wed, 07 Jun 2023 08:17:55 UTCBut the signoz cluster contains of two collector components: signoz-otel-collector and signoz-otel-collector-metrics.
Prashant
Wed, 07 Jun 2023 08:18:34 UTCwhich has the OTLP receiver and responsible for writing data to clickhouse db.
Prashant
Wed, 07 Jun 2023 08:18:59 UTClogs of signoz-otel-collector container would be helpful
Riccardo
Wed, 07 Jun 2023 08:19:33 UTCcould you please tell me where to find those logs?
Prashant
Wed, 07 Jun 2023 08:20:16 UTChow did you install signoz? where is it deployed?
Riccardo
Wed, 07 Jun 2023 08:21:08 UTCI've installed it as docker standalone, following tutorial on
Riccardo
Wed, 07 Jun 2023 08:22:07 UTC
Riccardo
Wed, 07 Jun 2023 08:23:00 UTC
Riccardo
Wed, 07 Jun 2023 08:23:57 UTCwith ```docker ps``` all containers seem to be working: ```CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 095aa079cd0c signoz/frontend:0.19.0 "nginx -g 'daemon of…" 23 hours ago Up 21 hours 80/tcp, 0.0.0.0:3301->3301/tcp, :::3301->3301/tcp frontend e0d39167b1b5 signoz/alertmanager:0.23.1 "/bin/alertmanager -…" 23 hours ago Up 21 hours 9093/tcp clickhouse-setup_alertmanager_1 f07b23de2200 signoz/signoz-otel-collector:0.76.1 "/signoz-collector -…" 23 hours ago Up 21 hours 0.0.0.0:4317-4318->4317-4318/tcp, :::4317-4318->4317-4318/tcp clickhouse-setup_otel-collector_1 e254a8502327 signoz/signoz-otel-collector:0.76.1 "/signoz-collector -…" 23 hours ago Up 21 hours 4317-4318/tcp clickhouse-setup_otel-collector-metrics_1 80ed2daaa665 signoz/query-service:0.19.0 "./query-service -co…" 23 hours ago Up 21 hours (healthy) 8080/tcp query-service 9ac491ef7bee clickhouse/clickhouse-server:22.8.8-alpine "/entrypoint.sh" 23 hours ago Up 21 hours (healthy) 0.0.0.0:8123->8123/tcp, :::8123->8123/tcp, 0.0.0.0:9000->9000/tcp, :::9000->9000/tcp, 0.0.0.0:9181->9181/tcp, :::9181->9181/tcp, 9009/tcp clickhouse ac8aed417d9f bitnami/zookeeper:3.7.1 "/opt/bitnami/script…" 23 hours ago Up 21 hours 0.0.0.0:2181->2181/tcp, :::2181->2181/tcp, 0.0.0.0:2888->2888/tcp, :::2888->2888/tcp, 0.0.0.0:3888->3888/tcp, :::3888->3888/tcp, 8080/tcp zookeeper-1 79a1737c0e82 jaegertracing/example-hotrod:1.30 "/go/bin/hotrod-linu…" 23 hours ago Up 21 hours 8080-8083/tcp hotrod 07ab59b1e368 grubykarol/locust:1.2.3-python3.9-alpine3.12 "/docker-entrypoint.…" 23 hours ago Up 21 hours 5557-5558/tcp, 8089/tcp ```
Prashant
Wed, 07 Jun 2023 08:25:04 UTCLogs of signoz otel-collector: ```docker logs clickhouse-setup_otel-collector_1```
Prashant
Wed, 07 Jun 2023 08:25:46 UTCLogs of `clickhouse`: ```docker logs --tail 200 clickhouse```
Riccardo
Wed, 07 Jun 2023 08:30:27 UTC
Riccardo
Wed, 07 Jun 2023 08:30:33 UTC
Prashant
Wed, 07 Jun 2023 08:35:30 UTC> 2023-06-06T12:52:26.041Z error clickhousetracesexporter/writer.go:416 Could not append span to batch: > "name": "TestingSpan", "startTimeUnixNano": 11651379494838206464, "serviceName": "<nil-service-name>", > ... > error": "clickhouse: dateTime overflow. timestamp must be between 1925-01-01 00:00:00 and 2283-11-11 00:00:00"} I see dummy trace data with `2339` year as start time. Can you make sure you have correct time set on the instance or the environment? cc <@4K143e> Srikanth
Riccardo
Wed, 07 Jun 2023 08:36:36 UTCthe date should be correct: ```root@signoz:~# date Wed 07 Jun 2023 10:35:53 AM CEST```
Prashant
Wed, 07 Jun 2023 08:37:19 UTCSrikanth could you please look into this?
Prashant
Wed, 07 Jun 2023 08:38:21 UTCon side note, can you upgrade the binary from `0.66.0` to `0.76.x`?
Prashant
Wed, 07 Jun 2023 08:39:01 UTC`0.76.1` to be exact
Riccardo
Wed, 07 Jun 2023 08:40:09 UTCyes, I could, but I have to tell you that I tried with 0.78.0 binary and it was not working (a lot of errors on startup) So I had to fall back to the version named in the tutorial, because I'm not expert in signoz
Riccardo
Wed, 07 Jun 2023 08:40:53 UTCcould you please paste the exact link to download the right binary version?
Riccardo
Wed, 07 Jun 2023 08:41:17 UTCThe source VM is a Debian 10 x64
Prashant
Wed, 07 Jun 2023 10:05:33 UTCYou can find all the release assets here:
Riccardo
Wed, 07 Jun 2023 10:16:39 UTCI have installed 0.76.1 binary, but now the binary log show the following:
```Jun 07 12:12:00
Srikanth
Wed, 07 Jun 2023 15:54:43 UTCWhat’s the SDK you are using?
Riccardo
Wed, 07 Jun 2023 16:13:23 UTCCurrently no SDK, just otel-collector binary installed on a Debian 10 Next step will be to evaluate wordpress (php) instrumenting
Srikanth
Thu, 08 Jun 2023 01:43:36 UTCCan you share what is the main problem here?
Riccardo
Thu, 08 Jun 2023 07:09:57 UTCThe main problem: the dashboard is empty:
Srikanth
Thu, 08 Jun 2023 08:12:08 UTCShare the signoz version and dashboard json you are using.
Riccardo
Thu, 08 Jun 2023 09:05:40 UTCsignoz v0.19.0
Riccardo
Thu, 08 Jun 2023 09:06:14 UTC
SigNoz Community
Indexed 1061 threads
Built with ClickHouse as datastore, SigNoz is an open-source APM to help you find issues in your deployed applications & solve them quickly | Knowledge Base powered by Struct.AI
View in App
Riccardo
Tue, 06 Jun 2023 13:01:59 UTCHi all, I'm new to SigNoz software, I just installed an instance of Docker Standalone on a VM on Debian 11 (dest). The purpose is to collect data from another Debian 10 VM (source) on the same subnet. On the source VM I've installed otelcol-contrib_0.66.0_linux_amd64.deb package, then latest docker/standalone/config.yaml where the otlp endpoint has been set to the dest VM. The setup seems working correctly, if I run on source the command: ```telemetrygen traces --traces 1 --otlp-endpoint localhost:4317 --otlp-insecure``` then I can see on the Traces section the telemetrygen Service data (lets-go and okey-dokey). So I've generated HostMetrics dashboards for the source VM and imported the resulting json to the Dashboards section on the dest VM. The HostMetrics Dashboard - <source-VM-Name> appeared, but all the panels are empty. I've also tried troubleshooting using: ```troubleshoot checkEndpoint --endpoint=dest-IP:4317``` and the answer was: ```INFO workspace/main.go:28 STARTING! INFO checkEndpoint/checkEndpoint.go:41 checking reachability of SigNoz endpoint INFO workspace/main.go:46 Successfully sent sample data to signoz ...``` There is no firewall between the two VM and tcpdump show traffic packets flowing from source to dest and back. Where am I wrong? Any help is very appreciated, tks.