Configuring OTel Collector to Export Data to SigNoz
TLDR James asked how to configure the otel collector to send data to SigNoz. Ankit suggested using the otlp exporter in the collector configuration. James found that SigNoz uses its own otel collector and adjusted the port bindings accordingly.
Jun 22, 2023 (5 months ago)
otelcollocally on my host, SigNoz is running via Docker, and I have applications running locally on the host as well. My applications are able to collect and send data to otelcol successfully, I can see the data being written when a file exporter is configured. Now I'd like otelcol to send data to Signoz. However, I've seen a lot of conflicting guides, and most produces an error. One example I've been trying to work with is here:
However, attempting to configure these
clickhousetracesexporters always results in an error from otelcol when attempting to start of:
error decoding 'exporters': unknown type: "clickhousetraces" for id: "clickhousetraces" (valid values: [opencensus zipkin file jaeger otlphttp kafka prometheus prometheusremotewrite logging otlp])
Interestingly, with that SAME URL, we see two other exporters mentioned that are not included later on,
kafka/metrics. When I attempt to configure these I receive the error that effectively nothing is running on port 9092, and I can confirm the local docker configuration doesn't expose anything on port 9092 and I'm not entirely sure if that configuration even includes kafka anything: https://github.com/SigNoz/signoz/blob/develop/deploy/docker/clickhouse-setup/docker-compose.yaml
I feel like this situation should be fairly straightforward - what should my exporter(s) be configured as when using otelcol locally and the Signoz local docker instance?
clickhousetracesis part of signoz's distribution. You need to use a different image
this is true...Maybe the blog just shows an example config ( Ashu please note).
James the kafkaexporter in the blog needs a kafka already running. The broker of kafka has a port address of 9092 which would make the above config work then.
deply/docker/clickhouse-setup/docker-compose.yamlconfiguration. Having trouble knowing how to export what it receives to SigNoz atm
exporters: otlp: endpoint: "<IP of machine hosting SigNoz>:4317" tls: insecure: true
Started looking back at this and I'm a bit confused. I configured my otelcol to send data to Signoz on port 4317, but a connection was refused. Nothing was listening on that port. Looking to the Docker configuration, we have this:
This appears to indicate the service which I could potentially route traffic from port 4317 to is...another otel collector? Is that the correct way to get data ingested into SigNoz? Basically:
App -> OTel Collector -> SigNoz OTel Collector -> SigNoz?
Jun 23, 2023 (5 months ago)
No, it makes the port accessible to the host machine, which otherwise could only be accessed by other containers.
The flow will be
App -> OTel Collector (optional) -> SigNoz
signoz-otel-collectoris based on in this repo, and specifically where it creates its own instance of an otel collector here: https://github.com/SigNoz/signoz-otel-collector/blob/main/cmd/signozcollector/main.go. I assumed this was the case as the configuration passed to this collector was, in fact, the identical shape of the configuration for otel.
So, in other words, if I wanted to have my own generic otel collector that I can use to send data to both Signoz AND other endpoints, such as my own company's database, the process would look like:
App > Otel > SigNoz Otel > Signoz
This is a bit challenging since, at least locally for a PoC, my own Otel is binding on port 4318 to receive telemetry, but of course the SigNoz Otel Collector wants to bind on the same port.
Indexed 1023 threads (61% resolved)
SigNoz Instrumentation Error in GCP/GKE with Python and FastAPI
Andre faced an error while instrumenting their app service with SigNoz. Srikanth identified the incorrect config and suggested the correct endpoint, resolving the issue.
Issues with Opentelemetry Maven Plugin and Kafka Exporter
sarthak is facing issues implementing Kafka with Maven for Opentelemetry while trying to decouple the source application performance from the Signoz cluster. Pranay and nitya-signoz ask clarifying questions about the desired flow.
Troubleshooting SigNoz Auto-Instrumentation Configuration
igor is having trouble configuring auto-instrumentation for Java applications using SigNoz, with traces not appearing in the SigNoz UI. Prashant advises to check logs of the otel sidecar, use service name for endpoint, verify supported libraries, and test with telemetrygen. However, the issue still persists.
Troubleshooting SigNoz Dashboard Visibility Issue
Naman had an issue with SigNoz not displaying services even though they've enabled it and sent data successfully. Srikanth engaged in investigation and troubleshooting, but the issue remained unresolved.
Parsing JSON Logs in Kubernetes for Different Applications
Nick sought advice on parsing JSON logs in Kubernetes for apps without OTEL support. nitya-signoz suggested using additional operators, creating pipelines, and mapping according to the business logic.