Configuring SigNoz with Istio Traces and Service Monitors
TLDR Zakaria asked about integrating SigNoz with istio traces and Prometheus service monitors. Srikanth suggested using OpenTelemetry operator and provided guidance for troubleshooting and solving configuration issues in the process.
Srikanth
Tue, 04 Apr 2023 19:23:49 UTC> but we’re missing some metrics of Prometheus service monitors and pod monitors These are the k8s objects watched by (Prometheus) operator. You could use the OpenTelemetry operator + Prometheus receiver with `kubernetes_sd_configs`.
Zakaria
Wed, 05 Apr 2023 09:10:12 UTCHey thanks for the information, I configured the above using target allocator and open telemetry operator. Though I see the metrics are being scrapped but I cannot see the metrics inside clickhouse db signoz.metrics.v2 table Is there some sort of format matching required?
Srikanth
Thu, 06 Apr 2023 07:41:49 UTCNo, there is no such format requirement. How did you confirm they are being scraped but not appearing in DB?
Zakaria
Thu, 06 Apr 2023 07:42:26 UTCI figured it out
Zakaria
Thu, 06 Apr 2023 07:43:28 UTCDo you guys have any logs or mechanism where we can see what clickhouse query was generated by the promql we specified in signoz ui?
Srikanth
Thu, 06 Apr 2023 07:44:23 UTCWe don’t convert the promql to ClickHouse SQL. SigNoz works as a remote read system.
Srikanth
Thu, 06 Apr 2023 07:44:47 UTC> I figured it out What was the issue?
Zakaria
Thu, 06 Apr 2023 07:46:54 UTCIt was a config issue, the otl exporter should be from the open telemetry operator to the signoz otel collector, I was trying to make it work from the otelDeploy
Zakaria
Mon, 10 Apr 2023 07:34:18 UTCSrikanth do you know if we can send istio traces to clickhouse and explore the traces using signoz? What approach should we take like use zipkin or Jaeger or send directly using istio otel extension?
Srikanth
Mon, 10 Apr 2023 07:36:13 UTCYou could use either jaeger or istio otel extension.
Zakaria
Mon, 10 Apr 2023 09:54:39 UTCI configured the otel extension but cannot see any columns in clickhouse db I'm using your signoz custom collector, hitting the otlp endpoint from istio mesh config.
Srikanth
Mon, 10 Apr 2023 10:04:12 UTCDo you see any data in Services/Traces tab of the SigNoz UI? > but cannot see any columns in clickhouse db Where are you searching for this info? i.e. table and DB.
Zakaria
Mon, 10 Apr 2023 10:04:37 UTCNo I don't see any data in signoz ui as well
Zakaria
Mon, 10 Apr 2023 10:05:06 UTCsignoz_traces database, I see no rows are being added but from istiod I see it's pushing data to otel
Srikanth
Mon, 10 Apr 2023 10:05:16 UTCCan you share the logs of the Collector?
Zakaria
Mon, 10 Apr 2023 10:05:48 UTCI tried to enable logging exporter but I think the collector implementation of signoz collector doesn't support that
Zakaria
Mon, 10 Apr 2023 10:06:31 UTCI am directly hitting the signoz collector on 4317, I'm not using the otel contrib.
Srikanth
Mon, 10 Apr 2023 10:06:49 UTC> I tried to enable logging exporter but I think the collector implementation of signoz collector doesn’t support that We support the logging exporter.
Zakaria
Mon, 10 Apr 2023 10:07:08 UTCOh okay ill enable and check.
Srikanth
Mon, 10 Apr 2023 10:07:09 UTCDoes istiod show the successful message?
Zakaria
Mon, 10 Apr 2023 11:10:08 UTCI'm not seeing any logs from traces after enabling log exporter We don't see any errors on istiod side too, could be like the istio is not sending traces to otel? Figuring out issue is with collector or the istio itself.
Srikanth
Mon, 10 Apr 2023 11:17:36 UTCThis is high likely an issue with the istio not able to send the traces to the collector since you don’t see any traces from the log exporter.
Zakaria
Tue, 11 Apr 2023 07:12:06 UTCThanks alot, we solved the issue, the sampling rate was low on istio and the collector port for otlp should be named as grpc-otlp, only then it works. I want to ask a generic question, what is the difference between the open telemetry contrib collector and the signoz collector? Is everything which is support by the contrib collector is support by the signoz one?
Srikanth
Tue, 11 Apr 2023 07:38:38 UTCMostly yes, except we remove some vendor speicific components and components which we believe make the collector bloated. In addition to that, there is some SigNoz specific work we do, such as our own processor components to keep backward compatibility and exporters of our own with ever evolving changes. Overall The SigNoz distro makes it easier for us to build on top of the opentelemetry.
SigNoz Community
Indexed 1060 threads
Built with ClickHouse as datastore, SigNoz is an open-source APM to help you find issues in your deployed applications & solve them quickly | Knowledge Base powered by Struct.AI
View in App
Zakaria
Tue, 04 Apr 2023 18:13:49 UTCHi, We're trying this stack to have logs, traces, metrics in one ui. Currently we use kube prometheus stack, I see the defaults are good, shipping metrics and logs is very easy from otel but we're missing some metrics of Prometheus service monitors and pod monitors. Is there any way to push service monitors to signoz? Or is there any alternative which signoz provides?