#support

Troubleshooting SigNoz Auto-Instrumentation Configuration

TLDR igor is having trouble configuring auto-instrumentation for Java applications using SigNoz, with traces not appearing in the SigNoz UI. Prashant advises to check logs of the otel sidecar, use service name for endpoint, verify supported libraries, and test with telemetrygen. However, the issue still persists.

8mo

Join the chat

Mar 15, 2023 (9 months ago)

Photo of md5-73ce99d63b3f8c60fe84a17cd16490f2

igor

02:22 PM

Hi Team, i'm trying to use signoz for the first time and but i'm trying to config the auto-instrumentation for my java applications, but the unfortunaly when I add the annotations, the services or traces don't show at signoz page, I try to test my collector using the troubleshoot tool, and appears to be working fine, but never work with my pods. Any tip?

PS: I already use the side car and instrumentation instance options and using k8s and trying to trace k8s pods

Photo of md5-8df7ce0274b2473ec07403336e48b574

Pranay

04:47 PM

Prashant may have more insights here

04:48

Pranay

04:48 PM

igor Have you checked this grid on providing otel collector addresses - https://signoz.io/docs/install/troubleshooting/#signoz-otel-collector-address-grid

Might be helpful!

Photo of md5-1899629483c7ab1dccfbee6cc2f637b9

Prashant

04:58 PM

igor are you following the OpenTelemetry Operator guide for the auto-instrumentation?

igor

05:33 PM

yes, actually I tried the troubleshooting tools to test inside the cluster that my application are configured and reached the otel-collector, which is configured in another cluster:

Image 1 for yes, actually I tried the troubleshooting tools to test inside the cluster that my application are configured and reached the otel-collector, which is configured in another cluster:

05:33

igor

05:33 PM

i followed this one:
https://signoz.io/docs/tutorial/opentelemetry-operator-usage/#opentelemetry-auto-instrumentation-injection

10:25

igor

10:25 PM

Prashant probably i'm doing something wrong during the auto instrumentation phase, i'll share my yaml files maybe you can help me

10:28

igor

10:28 PM

sidecar yaml:
apiVersion: <http://opentelemetry.io/v1alpha1|opentelemetry.io/v1alpha1>
kind: OpenTelemetryCollector
metadata:
annotations:
<http://kubectl.kubernetes.io/last-applied-configuration|kubectl.kubernetes.io/last-applied-configuration>: |

{"apiVersion":"<http://opentelemetry.io/v1alpha1|opentelemetry.io/v1alpha1>","kind":"OpenTelemetryCollector","metadata":{"annotations":{},"name":"signoz-otel-collector-sidecar","namespace":"default"},"spec":{"config":"receivers:\n  otlp:\n    protocols:\n      http:\n      grpc:\nprocessors:\n  batch:\nexporters:\n  logging:\n  otlp:\n    endpoint: 100.94.35.234:4317\n    tls:\n      insecure: true\nservice:\n  pipelines:\n    traces:\n      receivers: [otlp]\n      processors: [batch]\n      exporters: [logging, otlp]\n    metrics:\n      receivers: [otlp]\n      processors: [batch]\n      exporters: [logging, otlp]\n","mode":"sidecar"}}

creationTimestamp: "2023-03-15T12:50:40Z"
generation: 3
name: signoz-otel-collector-sidecar
namespace: default
resourceVersion: "850306767"
uid: c6a30f0e-1a88-4628-a6ac-c57400ff20c9
spec:
config: |
receivers:
otlp:
protocols:
http:
grpc:
processors:
batch:
exporters:
logging:
otlp:
endpoint: 100.94.35.234:4317
tls:
insecure: true
service:
pipelines:
traces:
receivers: [otlp]
processors: [batch]
exporters: [logging, otlp]
metrics:
receivers: [otlp]
processors: [batch]
exporters: [logging, otlp]
mode: sidecar
~
~
~
~

10:28

igor

10:28 PM

instrumentation yaml:
apiVersion: <http://opentelemetry.io/v1alpha1|opentelemetry.io/v1alpha1>
kind: Instrumentation
metadata:
annotations:
<http://kubectl.kubernetes.io/last-applied-configuration|kubectl.kubernetes.io/last-applied-configuration>: |

{"apiVersion":"<http://opentelemetry.io/v1alpha1|opentelemetry.io/v1alpha1>","kind":"Instrumentation","metadata":{"annotations":{},"name":"signoz-otel-collector-instrumentation","namespace":"default"},"spec":{"exporter":{"endpoint":"<https://signoz-otel-collector.dev.sicredi.cloud:4317>"},"java":{"image":"<http://ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:latest|ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:latest>"},"nodejs":{"image":"<http://ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-nodejs:latest|ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-nodejs:latest>"},"propagators":["tracecontext","baggage","b3"],"python":{"image":"<http://ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-python:latest|ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-python:latest>"},"sampler":{"argument":"0.25","type":"parentbased_traceidratio"}}}

creationTimestamp: "2023-03-15T01:54:20Z"
generation: 4
name: signoz-otel-collector-instrumentation
namespace: default
resourceVersion: "850306883"
uid: 2ca19a50-dcea-4c99-97fb-4f60cef3ba03
spec:
exporter:
endpoint: 100.94.35.234:4317
java:

image: <http://ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:latest|ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:latest>

nodejs:

image: <http://ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-nodejs:latest|ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-nodejs:latest>

propagators:
- tracecontext
- baggage
- b3
python:

image: <http://ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-python:latest|ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-python:latest>

sampler:
argument: "0.25"
type: parentbased_traceidratio
~

10:29

igor

10:29 PM

application pod deloymenty yaml:

10:29

igor

10:29 PM

apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
<http://deployment.kubernetes.io/revision|deployment.kubernetes.io/revision>: "23"
creationTimestamp: "2022-07-04T15:57:14Z"
generation: 40
labels:
app: plataforma-monitoramento-infraphone-dialer
devconsole: managed
devconsole_application: plataforma-monitoramento
devconsole_component: plataforma-monitoramento-infraphone-dialer
name: plataforma-monitoramento-infraphone-dialer-deployment
namespace: default
resourceVersion: "850309601"
uid: 77e93b40-e95e-4e60-b065-641f516f664a
spec:
progressDeadlineSeconds: 220
replicas: 1
revisionHistoryLimit: 1
selector:
matchLabels:
app: plataforma-monitoramento-infraphone-dialer
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
annotations:
<http://instrumentation.opentelemetry.io/inject-java|instrumentation.opentelemetry.io/inject-java>: "true"
<http://sidecar.opentelemetry.io/inject|sidecar.opentelemetry.io/inject>: "true"
creationTimestamp: null
labels:
app: plataforma-monitoramento-infraphone-dialer
date: "1668455468726"

Mar 16, 2023 (9 months ago)

Prashant

06:53 AM

can you check the logs of the otel sidecar?

06:54

Prashant

06:54 AM

> endpoint: 100.94.35.234:4317
I don't think the endpoint IP should point to private internal IP directly like that.

06:55

Prashant

06:55 AM

if the internal IP of the Pod/Svc changes, it would stop working.

06:56

Prashant

06:56 AM

use service name instead, if signoz in same cluster.

12:01

Prashant

12:01 PM

igor also make sure libraries/framework used by your application is supported: https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/docs/supported-libraries.md

Mar 17, 2023 (9 months ago)

igor

06:13 PM

I will check the libraries, but i'm using a static ip to connect, and using the troubleshoot tools shows that the collector is working, any other tip about how setup the sidecar and also the annotations?

ps:
i have the signoz installed in 1 cluster and my pods with the applications that i wanna monitor are configured in another cluster

06:38

igor

06:38 PM

And looking at my otel-collector pod the only log thats i foung is this, but nothing related with my application pods

06:39

igor

06:39 PM

Mar 20, 2023 (9 months ago)

Prashant

05:47 AM

> i have the signoz installed in 1 cluster and my pods with the applications that i wanna monitor are configured in another cluster
I do not think that internal static private IP could be used across clusters unless you can configured some internal solutions to enable it.

In any case, make sure that the passed endpoint (static IP or public loadbalancer endpoint) can be used across clusters by running troubleshoot or telemetrygen from the application cluster.

igor

10:58 AM

Prashant That ip is an external ip from the loadbalancer for the otel-collector service and I already used the troubleshoot tool to test from the application cluster and is reaching the otel-collector.

Image 1 for <@4K165d> That ip is an external ip from the loadbalancer for the otel-collector service and I already used the troubleshoot tool to test from the application cluster and is reaching the otel-collector.

11:03

igor

11:03 AM

I think for me that problem is during the sidecar/instrumentation configuration or the pod annotation part.

Prashant

11:20 AM

I see. Let me verify with the configuration shared by you.

igor

12:57 PM

thank you

Mar 21, 2023 (9 months ago)

igor

02:15 PM

Prashant I also try to create the petclinic test application, but same error

Mar 23, 2023 (9 months ago)

igor

01:00 PM

Prashant If you have and tip or any way to test, will be very helpful because signoz looks amazing

Apr 03, 2023 (8 months ago)

igor

12:32 PM

Prashant Were you able to take a look at the configuration that i sent to you ?

Prashant

04:30 PM

Hey igor!
I was able to see the pet clinic app in signoz without any issue using OpenTelemetry Operator

Image 1 for Hey <@4K1d53>!
I was able to see the pet clinic app in signoz without any issue using OpenTelemetry Operator

04:34

Prashant

04:34 PM

also, check logs of SigNoz otel-collector.

04:35

Prashant

04:35 PM

as well as that of the otc-container container inside the intrumented pod, you should see logs like this:

2023-04-03T16:25:14.770Z    info    MetricsExporter    {"kind": "exporter", "data_type": "metrics", "name": "logging", "#metrics": 78}
2023-04-03T16:26:42.474Z    info    TracesExporter    {"kind": "exporter", "data_type": "traces", "name": "logging", "#spans": 5}
2023-04-03T16:26:47.529Z    info    TracesExporter    {"kind": "exporter", "data_type": "traces", "name": "logging", "#spans": 20}
2023-04-03T16:29:14.866Z    info    MetricsExporter    {"kind": "exporter", "data_type": "metrics", "name": "logging", "#metrics": 83}

04:40

Prashant

04:40 PM

> endpoint: 100.94.35.234:4317
can you try to use this public endpoint with telemetrygen instead? After which verify it in SigNoz UI.

telemetrygen traces --traces 1 --otlp-endpoint 100.94.35.234:4317 --otlp-insecure

igor

05:49 PM

Follow a couple of logs from otel-collector

/go/pkg/mod/go.opentelemetry.io/collector/processor/[email protected]/batch_processor.go:144
2023-04-03T17:40:06.880Z    warn    [email protected]/batch_processor.go:178    Sender failed    {"kind": "processor", "name": "batch", "pipeline": "logs", "error": "sending_queue is full"}
2023-04-03T17:40:07.881Z    warn    [email protected]/batch_processor.go:178    Sender failed    {"kind": "processor", "name": "batch", "pipeline": "logs", "error": "sending_queue is full"}
2023-04-03T17:40:08.178Z    info    exporterhelper/queued_retry.go:426    Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "metrics", "name": "clickhousemetricswrite", "error": "code: 243, message: Cannot reserve 1.00 MiB, not enough space", "interval": "137.169576ms"}
2023-04-03T17:40:08.882Z    warn    [email protected]/batch_processor.go:178    Sender failed    {"kind": "processor", "name": "batch", "pipeline": "logs", "error": "sending_queue is full"}
2023-04-03T17:40:09.883Z    warn    [email protected]/batch_processor.go:178    Sender failed    {"kind": "processor", "name": "batch", "pipeline": "logs", "error": "sending_queue is full"}
2023-04-03T17:40:10.883Z    warn    [email protected]/batch_processor.go:178    Sender failed    {"kind": "processor", "name": "batch", "pipeline": "logs", "error": "sending_queue is full"}
2023-04-03T17:40:11.884Z    warn    [email protected]/batch_processor.go:178    Sender failed    {"kind": "processor", "name": "batch", "pipeline": "logs", "error": "sending_queue is full"}
2023-04-03T17:40:12.886Z    warn    [email protected]/batch_processor.go:178    Sender failed    {"kind": "processor", "name": "batch", "pipeline": "logs", "error": "sending_queue is full"}
2023-04-03T17:40:13.887Z    warn    [email protected]/batch_processor.go:178    Sender failed    {"kind": "processor", "name": "batch", "pipeline": "logs", "error": "sending_queue is full"}
2023-04-03T17:40:14.887Z    warn    [email protected]/batch_processor.go:178    Sender failed    {"kind": "processor", "name": "batch", "pipeline": "logs", "error": "sending_queue is full"}
2023-04-03T17:40:15.889Z    warn    [email protected]/batch_processor.go:178    Sender failed    {"kind": "processor", "name": "batch", "pipeline": "logs", "error": "sending_queue is full"}
2023-04-03T17:40:16.890Z    error    exporterhelper/queued_retry.go:310    Dropping data because sending_queue is full. Try increasing queue_size.    {"kind": "exporter", "data_type": "logs", "name": "clickhouselogsexporter", "dropped_items": 1144}

    /go/pkg/mod/go.opentelemetry.io/[email protected]/exporter/exporterhelper/queued_retry.go:310

    /go/pkg/mod/go.opentelemetry.io/[email protected]/exporter/exporterhelper/logs.go:114

    /go/pkg/mod/go.opentelemetry.io/collector/[email protected]/logs.go:36

    /go/pkg/mod/go.opentelemetry.io/collector/processor/[email protected]/batch_processor.go:339

    /go/pkg/mod/go.opentelemetry.io/collector/processor/[email protected]/batch_processor.go:176

    /go/pkg/mod/go.opentelemetry.io/collector/processor/[email protected]/batch_processor.go:144
2023-04-03T17:40:16.890Z    warn    [email protected]/batch_processor.go:178    Sender failed    {"kind": "processor", "name": "batch", "pipeline": "logs", "error": "sending_queue is full"}
2023-04-03T17:40:17.891Z    warn    [email protected]/batch_processor.go:178    Sender failed    {"kind": "processor", "name": "batch", "pipeline": "logs", "error": "sending_queue is full"}
2023-04-03T17:40:18.226Z    info    exporterhelper/queued_retry.go:426    Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "metrics", "name": "clickhousemetricswrite", "error": "code: 243, message: Cannot reserve 1.00 MiB, not enough space", "interval": "125.471225ms"}
2023-04-03T17:40:18.892Z    warn    [email protected]/batch_processor.go:178    Sender failed    {"kind": "processor", "name": "batch", "pipeline": "logs", "error": "sending_queue is full"}
2023-04-03T17:40:19.892Z    warn    [email protected]/batch_processor.go:178    Sender failed    {"kind": "processor", "name": "batch", "pipeline": "logs", "error": "sending_queue is full"}
2023-04-03T17:40:20.893Z    warn    [email protected]/batch_processor.go:178    Sender failed    {"kind": "processor", "name": "batch", "pipeline": "logs", "error": "sending_queue is full"}
2023-04-03T17:40:21.894Z    warn    [email protected]/batch_processor.go:178    Sender failed    {"kind": "processor", "name": "batch", "pipeline": "logs", "error": "sending_queue is full"}
2023-04-03T17:40:22.895Z    warn    [email protected]/batch_processor.go:178    Sender failed    {"kind": "processor", "name": "batch", "pipeline": "logs", "error": "sending_queue is full"}
2023-04-03T17:40:23.896Z    warn    [email protected]/batch_processor.go:178    Sender failed    {"kind": "processor", "name": "batch", "pipeline": "logs", "error": "sending_queue is full"}
2023-04-03T17:40:24.445Z    error    exporterhelper/queued_retry.go:310    Dropping data because sending_queue is full. Try increasing queue_size.    {"kind": "exporter", "data_type": "metrics", "name": "clickhousemetricswrite", "dropped_items": 442}

Apr 04, 2023 (8 months ago)

Prashant

05:39 AM

2023-04-03T17:40:18.226Z    info    exporterhelper/queued_retry.go:426    Exporting failed. Will retry the request after interval.    {"kind": "exporter", "data_type": "metrics", "name": "clickhousemetricswrite", "error": "code: 243, message: Cannot reserve 1.00 MiB, not enough space", "interval": "125.471225ms"}

igor You might want to increase ClickHouse PVC size. It seems to disk full issue.

07:29

Prashant

07:29 AM

Later, you can reduce TTL from SigNoz UI setting page to appropriate durations which would not let this happen in future.

SigNoz Community

Built with ClickHouse as datastore, SigNoz is an open-source APM to help you find issues in your deployed applications & solve them quickly | Knowledge Base powered by Struct.AI

Indexed 1023 threads (61% resolved)

Join Our Community

Similar Threads

Issue Accessing Pod Logs in SigNoz UI on AKS

prashant is facing an issue accessing pod logs of their application in SigNoz UI on AKS. nitya-signoz and Prashant provide suggestions related to log file paths and potential issues, but the problem remains unresolved.

6mo

Resolving Signoz Query Service Error

Einav encountered an error related to a missing table in the Signoz service which was preventing data visibility in the UI. Srikanth guided them to restart specific components and drop a database table, which resolved the issue.

3mo

Troubleshooting Memory Space Issue in Kubernetes with Signoz

Abel had trouble running signoz on Kubernetes due to 'not enough space'. Pranay provided steps to increase PV. Eventually, Abel confirmed solution after changing PV size to '50Gi'.

Slow Log Messages in SigNoz with Logrus and OpenTelemetry

Harald reported slow log messages in SigNoz with logrus and open telemetry. They narrowed down the problem to focusing on relevant k8s namespaces, which improved performance to near real-time.

5mo

Headers in Monitoring HTTP Endpoints & Otel-Collector Error

Rohit asks about providing headers in monitoring HTTP endpoints and reports errors in otel-collector logs. Srikanth mentions headers are unsupported and requires more investigation for errors.

9mo