#support

Troubleshooting SigNoz Auto-Instrumentation Configuration

TLDR igor is having trouble configuring auto-instrumentation for Java applications using SigNoz, with traces not appearing in the SigNoz UI. Prashant advises to check logs of the otel sidecar, use service name for endpoint, verify supported libraries, and test with telemetrygen. However, the issue still persists.

Powered by Struct AI
Mar 15, 2023 (9 months ago)
igor
Photo of md5-73ce99d63b3f8c60fe84a17cd16490f2
igor
02:22 PM
Hi Team, i'm trying to use signoz for the first time and but i'm trying to config the auto-instrumentation for my java applications, but the unfortunaly when I add the annotations, the services or traces don't show at signoz page, I try to test my collector using the troubleshoot tool, and appears to be working fine, but never work with my pods. Any tip?

PS: I already use the side car and instrumentation instance options and using k8s and trying to trace k8s pods
Pranay
Photo of md5-8df7ce0274b2473ec07403336e48b574
Pranay
04:47 PM
Prashant may have more insights here
04:48
Pranay
04:48 PM
igor Have you checked this grid on providing otel collector addresses - https://signoz.io/docs/install/troubleshooting/#signoz-otel-collector-address-grid

Might be helpful!
Prashant
Photo of md5-1899629483c7ab1dccfbee6cc2f637b9
Prashant
04:58 PM
igor are you following the OpenTelemetry Operator guide for the auto-instrumentation?
igor
Photo of md5-73ce99d63b3f8c60fe84a17cd16490f2
igor
05:33 PM
yes, actually I tried the troubleshooting tools to test inside the cluster that my application are configured and reached the otel-collector, which is configured in another cluster:
Image 1 for yes, actually I tried the troubleshooting tools to test inside the cluster that my application are configured and reached the otel-collector, which is configured in another cluster:
10:25
igor
10:25 PM
Prashant probably i'm doing something wrong during the auto instrumentation phase, i'll share my yaml files maybe you can help me
10:28
igor
10:28 PM
sidecar yaml:
apiVersion: <http://opentelemetry.io/v1alpha1|opentelemetry.io/v1alpha1>
kind: OpenTelemetryCollector
metadata:
annotations:
<http://kubectl.kubernetes.io/last-applied-configuration|kubectl.kubernetes.io/last-applied-configuration>: |
{"apiVersion":"<http://opentelemetry.io/v1alpha1|opentelemetry.io/v1alpha1>","kind":"OpenTelemetryCollector","metadata":{"annotations":{},"name":"signoz-otel-collector-sidecar","namespace":"default"},"spec":{"config":"receivers:\n otlp:\n protocols:\n http:\n grpc:\nprocessors:\n batch:\nexporters:\n logging:\n otlp:\n endpoint: 100.94.35.234:4317\n tls:\n insecure: true\nservice:\n pipelines:\n traces:\n receivers: [otlp]\n processors: [batch]\n exporters: [logging, otlp]\n metrics:\n receivers: [otlp]\n processors: [batch]\n exporters: [logging, otlp]\n","mode":"sidecar"}}
creationTimestamp: "2023-03-15T12:50:40Z"
generation: 3
name: signoz-otel-collector-sidecar
namespace: default
resourceVersion: "850306767"
uid: c6a30f0e-1a88-4628-a6ac-c57400ff20c9
spec:
config: |
receivers:
otlp:
protocols:
http:
grpc:
processors:
batch:
exporters:
logging:
otlp:
endpoint: 100.94.35.234:4317
tls:
insecure: true
service:
pipelines:
traces:
receivers: [otlp]
processors: [batch]
exporters: [logging, otlp]
metrics:
receivers: [otlp]
processors: [batch]
exporters: [logging, otlp]
mode: sidecar
~
~
~
~
10:28
igor
10:28 PM
instrumentation yaml:
apiVersion: <http://opentelemetry.io/v1alpha1|opentelemetry.io/v1alpha1>
kind: Instrumentation
metadata:
annotations:
<http://kubectl.kubernetes.io/last-applied-configuration|kubectl.kubernetes.io/last-applied-configuration>: |
{"apiVersion":"<http://opentelemetry.io/v1alpha1|opentelemetry.io/v1alpha1>","kind":"Instrumentation","metadata":{"annotations":{},"name":"signoz-otel-collector-instrumentation","namespace":"default"},"spec":{"exporter":{"endpoint":"<https://signoz-otel-collector.dev.sicredi.cloud:4317>"},"java":{"image":"<http://ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:latest|ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:latest>"},"nodejs":{"image":"<http://ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-nodejs:latest|ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-nodejs:latest>"},"propagators":["tracecontext","baggage","b3"],"python":{"image":"<http://ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-python:latest|ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-python:latest>"},"sampler":{"argument":"0.25","type":"parentbased_traceidratio"}}}
creationTimestamp: "2023-03-15T01:54:20Z"
generation: 4
name: signoz-otel-collector-instrumentation
namespace: default
resourceVersion: "850306883"
uid: 2ca19a50-dcea-4c99-97fb-4f60cef3ba03
spec:
exporter:
endpoint: 100.94.35.234:4317
java:
image: <http://ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:latest|ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:latest>
nodejs:
image: <http://ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-nodejs:latest|ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-nodejs:latest>
propagators:
- tracecontext
- baggage
- b3
python:
image: <http://ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-python:latest|ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-python:latest>
sampler:
argument: "0.25"
type: parentbased_traceidratio
~
10:29
igor
10:29 PM
application pod deloymenty yaml:
10:29
igor
10:29 PM
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
<http://deployment.kubernetes.io/revision|deployment.kubernetes.io/revision>: "23"
creationTimestamp: "2022-07-04T15:57:14Z"
generation: 40
labels:
app: plataforma-monitoramento-infraphone-dialer
devconsole: managed
devconsole_application: plataforma-monitoramento
devconsole_component: plataforma-monitoramento-infraphone-dialer
name: plataforma-monitoramento-infraphone-dialer-deployment
namespace: default
resourceVersion: "850309601"
uid: 77e93b40-e95e-4e60-b065-641f516f664a
spec:
progressDeadlineSeconds: 220
replicas: 1
revisionHistoryLimit: 1
selector:
matchLabels:
app: plataforma-monitoramento-infraphone-dialer
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
annotations:
<http://instrumentation.opentelemetry.io/inject-java|instrumentation.opentelemetry.io/inject-java>: "true"
<http://sidecar.opentelemetry.io/inject|sidecar.opentelemetry.io/inject>: "true"
creationTimestamp: null
labels:
app: plataforma-monitoramento-infraphone-dialer
date: "1668455468726"
Mar 16, 2023 (9 months ago)
Prashant
Photo of md5-1899629483c7ab1dccfbee6cc2f637b9
Prashant
06:53 AM
can you check the logs of the otel sidecar?
06:54
Prashant
06:54 AM
&gt; endpoint: 100.94.35.234:4317
I don't think the endpoint IP should point to private internal IP directly like that.
06:55
Prashant
06:55 AM
if the internal IP of the Pod/Svc changes, it would stop working.
06:56
Prashant
06:56 AM
use service name instead, if signoz in same cluster.
12:01
Prashant
12:01 PM
igor also make sure libraries/framework used by your application is supported: https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/docs/supported-libraries.md
Mar 17, 2023 (9 months ago)
igor
Photo of md5-73ce99d63b3f8c60fe84a17cd16490f2
igor
06:13 PM
I will check the libraries, but i'm using a static ip to connect, and using the troubleshoot tools shows that the collector is working, any other tip about how setup the sidecar and also the annotations?

ps:
i have the signoz installed in 1 cluster and my pods with the applications that i wanna monitor are configured in another cluster
06:38
igor
06:38 PM
And looking at my otel-collector pod the only log thats i foung is this, but nothing related with my application pods
06:39
igor
06:39 PM
Image 1 for
Mar 20, 2023 (9 months ago)
Prashant
Photo of md5-1899629483c7ab1dccfbee6cc2f637b9
Prashant
05:47 AM
&gt; i have the signoz installed in 1 cluster and my pods with the applications that i wanna monitor are configured in another cluster
I do not think that internal static private IP could be used across clusters unless you can configured some internal solutions to enable it.

In any case, make sure that the passed endpoint (static IP or public loadbalancer endpoint) can be used across clusters by running troubleshoot or telemetrygen from the application cluster.
igor
Photo of md5-73ce99d63b3f8c60fe84a17cd16490f2
igor
10:58 AM
Prashant That ip is an external ip from the loadbalancer for the otel-collector service and I already used the troubleshoot tool to test from the application cluster and is reaching the otel-collector.
Image 1 for <@4K165d> That ip is an external ip from the loadbalancer for the otel-collector service and I already used the troubleshoot tool to test from the application cluster and is reaching the otel-collector.
11:03
igor
11:03 AM
I think for me that problem is during the sidecar/instrumentation configuration or the pod annotation part.
Prashant
Photo of md5-1899629483c7ab1dccfbee6cc2f637b9
Prashant
11:20 AM
I see. Let me verify with the configuration shared by you.
igor
Photo of md5-73ce99d63b3f8c60fe84a17cd16490f2
igor
12:57 PM
thank you
Mar 21, 2023 (9 months ago)
igor
Photo of md5-73ce99d63b3f8c60fe84a17cd16490f2
igor
02:15 PM
Prashant I also try to create the petclinic test application, but same error
Mar 23, 2023 (9 months ago)
igor
Photo of md5-73ce99d63b3f8c60fe84a17cd16490f2
igor
01:00 PM
Prashant If you have and tip or any way to test, will be very helpful because signoz looks amazing
Apr 03, 2023 (8 months ago)
igor
Photo of md5-73ce99d63b3f8c60fe84a17cd16490f2
igor
12:32 PM
Prashant Were you able to take a look at the configuration that i sent to you ?
Prashant
Photo of md5-1899629483c7ab1dccfbee6cc2f637b9
Prashant
04:30 PM
Hey igor!
I was able to see the pet clinic app in signoz without any issue using OpenTelemetry Operator
Image 1 for Hey <@4K1d53>!
I was able to see the pet clinic app in signoz without any issue using OpenTelemetry Operator
04:34
Prashant
04:34 PM
also, check logs of SigNoz otel-collector.
04:35
Prashant
04:35 PM
as well as that of the otc-container container inside the intrumented pod, you should see logs like this:
2023-04-03T16:25:14.770Z    info    MetricsExporter    {"kind": "exporter", "data_type": "metrics", "name": "logging", "#metrics": 78}
2023-04-03T16:26:42.474Z    info    TracesExporter    {"kind": "exporter", "data_type": "traces", "name": "logging", "#spans": 5}
2023-04-03T16:26:47.529Z    info    TracesExporter    {"kind": "exporter", "data_type": "traces", "name": "logging", "#spans": 20}
2023-04-03T16:29:14.866Z    info    MetricsExporter    {"kind": "exporter", "data_type": "metrics", "name": "logging", "#metrics": 83}
04:40
Prashant
04:40 PM
&gt; endpoint: 100.94.35.234:4317
can you try to use this public endpoint with telemetrygen instead? After which verify it in SigNoz UI.
telemetrygen traces --traces 1 --otlp-endpoint 100.94.35.234:4317 --otlp-insecure
Image 1 for &gt; endpoint: 100.94.35.234:4317
can you try to use this public endpoint with `telemetrygen` instead? After which verify it in SigNoz UI.
```telemetrygen traces --traces 1 --otlp-endpoint 100.94.35.234:4317 --otlp-insecure```
igor
Photo of md5-73ce99d63b3f8c60fe84a17cd16490f2
igor
05:49 PM
Follow a couple of logs from otel-collector
/go/pkg/mod/go.opentelemetry.io/collector/processor/[email protected]/batch_processor.go:144
2023-04-03T17:40:06.880Z    warn    [email protected]/batch_processor.go:178    Sender failed    {"kind": "processor", "name": "batch", "pipeline": "logs", "error": "sending_queue is full"}
2023-04-03T17:40:07.881Z    warn    [email protected]/batch_processor.go:178    Sender failed    {"kind": "processor", "name": "batch", "pipeline": "logs", "error": "sending_queue is full"}
2023-04-03T17:40:08.178Z    info    exporterhelper/queued_retry.go:426    Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "metrics", "name": "clickhousemetricswrite", "error": "code: 243, message: Cannot reserve 1.00 MiB, not enough space", "interval": "137.169576ms"}
2023-04-03T17:40:08.882Z    warn    [email protected]/batch_processor.go:178    Sender failed    {"kind": "processor", "name": "batch", "pipeline": "logs", "error": "sending_queue is full"}
2023-04-03T17:40:09.883Z    warn    [email protected]/batch_processor.go:178    Sender failed    {"kind": "processor", "name": "batch", "pipeline": "logs", "error": "sending_queue is full"}
2023-04-03T17:40:10.883Z    warn    [email protected]/batch_processor.go:178    Sender failed    {"kind": "processor", "name": "batch", "pipeline": "logs", "error": "sending_queue is full"}
2023-04-03T17:40:11.884Z    warn    [email protected]/batch_processor.go:178    Sender failed    {"kind": "processor", "name": "batch", "pipeline": "logs", "error": "sending_queue is full"}
2023-04-03T17:40:12.886Z    warn    [email protected]/batch_processor.go:178    Sender failed    {"kind": "processor", "name": "batch", "pipeline": "logs", "error": "sending_queue is full"}
2023-04-03T17:40:13.887Z    warn    [email protected]/batch_processor.go:178    Sender failed    {"kind": "processor", "name": "batch", "pipeline": "logs", "error": "sending_queue is full"}
2023-04-03T17:40:14.887Z    warn    [email protected]/batch_processor.go:178    Sender failed    {"kind": "processor", "name": "batch", "pipeline": "logs", "error": "sending_queue is full"}
2023-04-03T17:40:15.889Z    warn    [email protected]/batch_processor.go:178    Sender failed    {"kind": "processor", "name": "batch", "pipeline": "logs", "error": "sending_queue is full"}
2023-04-03T17:40:16.890Z    error    exporterhelper/queued_retry.go:310    Dropping data because sending_queue is full. Try increasing queue_size.    {"kind": "exporter", "data_type": "logs", "name": "clickhouselogsexporter", "dropped_items": 1144}

    /go/pkg/mod/go.opentelemetry.io/[email protected]/exporter/exporterhelper/queued_retry.go:310

    /go/pkg/mod/go.opentelemetry.io/[email protected]/exporter/exporterhelper/logs.go:114

    /go/pkg/mod/go.opentelemetry.io/collector/[email protected]/logs.go:36

    /go/pkg/mod/go.opentelemetry.io/collector/processor/[email protected]/batch_processor.go:339

    /go/pkg/mod/go.opentelemetry.io/collector/processor/[email protected]/batch_processor.go:176

    /go/pkg/mod/go.opentelemetry.io/collector/processor/[email protected]/batch_processor.go:144
2023-04-03T17:40:16.890Z    warn    [email protected]/batch_processor.go:178    Sender failed    {"kind": "processor", "name": "batch", "pipeline": "logs", "error": "sending_queue is full"}
2023-04-03T17:40:17.891Z    warn    [email protected]/batch_processor.go:178    Sender failed    {"kind": "processor", "name": "batch", "pipeline": "logs", "error": "sending_queue is full"}
2023-04-03T17:40:18.226Z    info    exporterhelper/queued_retry.go:426    Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "metrics", "name": "clickhousemetricswrite", "error": "code: 243, message: Cannot reserve 1.00 MiB, not enough space", "interval": "125.471225ms"}
2023-04-03T17:40:18.892Z    warn    [email protected]/batch_processor.go:178    Sender failed    {"kind": "processor", "name": "batch", "pipeline": "logs", "error": "sending_queue is full"}
2023-04-03T17:40:19.892Z    warn    [email protected]/batch_processor.go:178    Sender failed    {"kind": "processor", "name": "batch", "pipeline": "logs", "error": "sending_queue is full"}
2023-04-03T17:40:20.893Z    warn    [email protected]/batch_processor.go:178    Sender failed    {"kind": "processor", "name": "batch", "pipeline": "logs", "error": "sending_queue is full"}
2023-04-03T17:40:21.894Z    warn    [email protected]/batch_processor.go:178    Sender failed    {"kind": "processor", "name": "batch", "pipeline": "logs", "error": "sending_queue is full"}
2023-04-03T17:40:22.895Z    warn    [email protected]/batch_processor.go:178    Sender failed    {"kind": "processor", "name": "batch", "pipeline": "logs", "error": "sending_queue is full"}
2023-04-03T17:40:23.896Z    warn    [email protected]/batch_processor.go:178    Sender failed    {"kind": "processor", "name": "batch", "pipeline": "logs", "error": "sending_queue is full"}
2023-04-03T17:40:24.445Z    error    exporterhelper/queued_retry.go:310    Dropping data because sending_queue is full. Try increasing queue_size.    {"kind": "exporter", "data_type": "metrics", "name": "clickhousemetricswrite", "dropped_items": 442}
Apr 04, 2023 (8 months ago)
Prashant
Photo of md5-1899629483c7ab1dccfbee6cc2f637b9
Prashant
05:39 AM
&gt;
2023-04-03T17:40:18.226Z    info    exporterhelper/queued_retry.go:426    Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "metrics", "name": "clickhousemetricswrite", "error": "code: 243, message: Cannot reserve 1.00 MiB, not enough space", "interval": "125.471225ms"}

igor You might want to increase ClickHouse PVC size. It seems to disk full issue.
07:29
Prashant
07:29 AM
Later, you can reduce TTL from SigNoz UI setting page to appropriate durations which would not let this happen in future.

SigNoz Community

Built with ClickHouse as datastore, SigNoz is an open-source APM to help you find issues in your deployed applications & solve them quickly | Knowledge Base powered by Struct.AI

Indexed 1023 threads (61% resolved)

Join Our Community