Troubleshooting SigNoz Auto-Instrumentation Configuration
TLDR igor is having trouble configuring auto-instrumentation for Java applications using SigNoz, with traces not appearing in the SigNoz UI. Prashant advises to check logs of the otel sidecar, use service name for endpoint, verify supported libraries, and test with telemetrygen. However, the issue still persists.
Mar 15, 2023 (9 months ago)
igor
02:22 PMPS: I already use the side car and instrumentation instance options and using k8s and trying to trace k8s pods
Pranay
04:47 PMPranay
04:48 PMMight be helpful!
Prashant
04:58 PMigor
05:33 PMigor
05:33 PMigor
10:25 PMigor
10:28 PMapiVersion: <http://opentelemetry.io/v1alpha1|opentelemetry.io/v1alpha1>
kind: OpenTelemetryCollector
metadata:
annotations:
<http://kubectl.kubernetes.io/last-applied-configuration|kubectl.kubernetes.io/last-applied-configuration>: |
{"apiVersion":"<http://opentelemetry.io/v1alpha1|opentelemetry.io/v1alpha1>","kind":"OpenTelemetryCollector","metadata":{"annotations":{},"name":"signoz-otel-collector-sidecar","namespace":"default"},"spec":{"config":"receivers:\n otlp:\n protocols:\n http:\n grpc:\nprocessors:\n batch:\nexporters:\n logging:\n otlp:\n endpoint: 100.94.35.234:4317\n tls:\n insecure: true\nservice:\n pipelines:\n traces:\n receivers: [otlp]\n processors: [batch]\n exporters: [logging, otlp]\n metrics:\n receivers: [otlp]\n processors: [batch]\n exporters: [logging, otlp]\n","mode":"sidecar"}}
creationTimestamp: "2023-03-15T12:50:40Z"
generation: 3
name: signoz-otel-collector-sidecar
namespace: default
resourceVersion: "850306767"
uid: c6a30f0e-1a88-4628-a6ac-c57400ff20c9
spec:
config: |
receivers:
otlp:
protocols:
http:
grpc:
processors:
batch:
exporters:
logging:
otlp:
endpoint: 100.94.35.234:4317
tls:
insecure: true
service:
pipelines:
traces:
receivers: [otlp]
processors: [batch]
exporters: [logging, otlp]
metrics:
receivers: [otlp]
processors: [batch]
exporters: [logging, otlp]
mode: sidecar
~
~
~
~
igor
10:28 PMapiVersion: <http://opentelemetry.io/v1alpha1|opentelemetry.io/v1alpha1>
kind: Instrumentation
metadata:
annotations:
<http://kubectl.kubernetes.io/last-applied-configuration|kubectl.kubernetes.io/last-applied-configuration>: |
{"apiVersion":"<http://opentelemetry.io/v1alpha1|opentelemetry.io/v1alpha1>","kind":"Instrumentation","metadata":{"annotations":{},"name":"signoz-otel-collector-instrumentation","namespace":"default"},"spec":{"exporter":{"endpoint":"<https://signoz-otel-collector.dev.sicredi.cloud:4317>"},"java":{"image":"<http://ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:latest|ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:latest>"},"nodejs":{"image":"<http://ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-nodejs:latest|ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-nodejs:latest>"},"propagators":["tracecontext","baggage","b3"],"python":{"image":"<http://ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-python:latest|ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-python:latest>"},"sampler":{"argument":"0.25","type":"parentbased_traceidratio"}}}
creationTimestamp: "2023-03-15T01:54:20Z"
generation: 4
name: signoz-otel-collector-instrumentation
namespace: default
resourceVersion: "850306883"
uid: 2ca19a50-dcea-4c99-97fb-4f60cef3ba03
spec:
exporter:
endpoint: 100.94.35.234:4317
java:
image: <http://ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:latest|ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:latest>
nodejs:
image: <http://ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-nodejs:latest|ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-nodejs:latest>
propagators:
- tracecontext
- baggage
- b3
python:
image: <http://ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-python:latest|ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-python:latest>
sampler:
argument: "0.25"
type: parentbased_traceidratio
~
igor
10:29 PMigor
10:29 PMapiVersion: apps/v1
kind: Deployment
metadata:
annotations:
<http://deployment.kubernetes.io/revision|deployment.kubernetes.io/revision>: "23"
creationTimestamp: "2022-07-04T15:57:14Z"
generation: 40
labels:
app: plataforma-monitoramento-infraphone-dialer
devconsole: managed
devconsole_application: plataforma-monitoramento
devconsole_component: plataforma-monitoramento-infraphone-dialer
name: plataforma-monitoramento-infraphone-dialer-deployment
namespace: default
resourceVersion: "850309601"
uid: 77e93b40-e95e-4e60-b065-641f516f664a
spec:
progressDeadlineSeconds: 220
replicas: 1
revisionHistoryLimit: 1
selector:
matchLabels:
app: plataforma-monitoramento-infraphone-dialer
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
annotations:
<http://instrumentation.opentelemetry.io/inject-java|instrumentation.opentelemetry.io/inject-java>: "true"
<http://sidecar.opentelemetry.io/inject|sidecar.opentelemetry.io/inject>: "true"
creationTimestamp: null
labels:
app: plataforma-monitoramento-infraphone-dialer
date: "1668455468726"
Mar 16, 2023 (9 months ago)
Prashant
06:53 AMPrashant
06:54 AMI don't think the endpoint IP should point to private internal IP directly like that.
Prashant
06:55 AMPrashant
06:56 AMPrashant
12:01 PMMar 17, 2023 (9 months ago)
igor
06:13 PMps:
i have the signoz installed in 1 cluster and my pods with the applications that i wanna monitor are configured in another cluster
igor
06:38 PMigor
06:39 PMMar 20, 2023 (9 months ago)
Prashant
05:47 AMI do not think that internal static private IP could be used across clusters unless you can configured some internal solutions to enable it.
In any case, make sure that the passed endpoint (static IP or public loadbalancer endpoint) can be used across clusters by running
troubleshoot
or telemetrygen
from the application cluster.igor
10:58 AMigor
11:03 AMPrashant
11:20 AMigor
12:57 PMMar 21, 2023 (9 months ago)
igor
02:15 PMMar 23, 2023 (9 months ago)
igor
01:00 PMApr 03, 2023 (8 months ago)
igor
12:32 PMPrashant
04:30 PMI was able to see the pet clinic app in signoz without any issue using OpenTelemetry Operator
Prashant
04:34 PMPrashant
04:35 PMotc-container
container inside the intrumented pod, you should see logs like this:2023-04-03T16:25:14.770Z info MetricsExporter {"kind": "exporter", "data_type": "metrics", "name": "logging", "#metrics": 78}
2023-04-03T16:26:42.474Z info TracesExporter {"kind": "exporter", "data_type": "traces", "name": "logging", "#spans": 5}
2023-04-03T16:26:47.529Z info TracesExporter {"kind": "exporter", "data_type": "traces", "name": "logging", "#spans": 20}
2023-04-03T16:29:14.866Z info MetricsExporter {"kind": "exporter", "data_type": "metrics", "name": "logging", "#metrics": 83}
Prashant
04:40 PMcan you try to use this public endpoint with
telemetrygen
instead? After which verify it in SigNoz UI.telemetrygen traces --traces 1 --otlp-endpoint 100.94.35.234:4317 --otlp-insecure
igor
05:49 PM/go/pkg/mod/go.opentelemetry.io/collector/processor/[email protected]/batch_processor.go:144
2023-04-03T17:40:06.880Z warn [email protected]/batch_processor.go:178 Sender failed {"kind": "processor", "name": "batch", "pipeline": "logs", "error": "sending_queue is full"}
2023-04-03T17:40:07.881Z warn [email protected]/batch_processor.go:178 Sender failed {"kind": "processor", "name": "batch", "pipeline": "logs", "error": "sending_queue is full"}
2023-04-03T17:40:08.178Z info exporterhelper/queued_retry.go:426 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "clickhousemetricswrite", "error": "code: 243, message: Cannot reserve 1.00 MiB, not enough space", "interval": "137.169576ms"}
2023-04-03T17:40:08.882Z warn [email protected]/batch_processor.go:178 Sender failed {"kind": "processor", "name": "batch", "pipeline": "logs", "error": "sending_queue is full"}
2023-04-03T17:40:09.883Z warn [email protected]/batch_processor.go:178 Sender failed {"kind": "processor", "name": "batch", "pipeline": "logs", "error": "sending_queue is full"}
2023-04-03T17:40:10.883Z warn [email protected]/batch_processor.go:178 Sender failed {"kind": "processor", "name": "batch", "pipeline": "logs", "error": "sending_queue is full"}
2023-04-03T17:40:11.884Z warn [email protected]/batch_processor.go:178 Sender failed {"kind": "processor", "name": "batch", "pipeline": "logs", "error": "sending_queue is full"}
2023-04-03T17:40:12.886Z warn [email protected]/batch_processor.go:178 Sender failed {"kind": "processor", "name": "batch", "pipeline": "logs", "error": "sending_queue is full"}
2023-04-03T17:40:13.887Z warn [email protected]/batch_processor.go:178 Sender failed {"kind": "processor", "name": "batch", "pipeline": "logs", "error": "sending_queue is full"}
2023-04-03T17:40:14.887Z warn [email protected]/batch_processor.go:178 Sender failed {"kind": "processor", "name": "batch", "pipeline": "logs", "error": "sending_queue is full"}
2023-04-03T17:40:15.889Z warn [email protected]/batch_processor.go:178 Sender failed {"kind": "processor", "name": "batch", "pipeline": "logs", "error": "sending_queue is full"}
2023-04-03T17:40:16.890Z error exporterhelper/queued_retry.go:310 Dropping data because sending_queue is full. Try increasing queue_size. {"kind": "exporter", "data_type": "logs", "name": "clickhouselogsexporter", "dropped_items": 1144}
/go/pkg/mod/go.opentelemetry.io/[email protected]/exporter/exporterhelper/queued_retry.go:310
/go/pkg/mod/go.opentelemetry.io/[email protected]/exporter/exporterhelper/logs.go:114
/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/logs.go:36
/go/pkg/mod/go.opentelemetry.io/collector/processor/[email protected]/batch_processor.go:339
/go/pkg/mod/go.opentelemetry.io/collector/processor/[email protected]/batch_processor.go:176
/go/pkg/mod/go.opentelemetry.io/collector/processor/[email protected]/batch_processor.go:144
2023-04-03T17:40:16.890Z warn [email protected]/batch_processor.go:178 Sender failed {"kind": "processor", "name": "batch", "pipeline": "logs", "error": "sending_queue is full"}
2023-04-03T17:40:17.891Z warn [email protected]/batch_processor.go:178 Sender failed {"kind": "processor", "name": "batch", "pipeline": "logs", "error": "sending_queue is full"}
2023-04-03T17:40:18.226Z info exporterhelper/queued_retry.go:426 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "clickhousemetricswrite", "error": "code: 243, message: Cannot reserve 1.00 MiB, not enough space", "interval": "125.471225ms"}
2023-04-03T17:40:18.892Z warn [email protected]/batch_processor.go:178 Sender failed {"kind": "processor", "name": "batch", "pipeline": "logs", "error": "sending_queue is full"}
2023-04-03T17:40:19.892Z warn [email protected]/batch_processor.go:178 Sender failed {"kind": "processor", "name": "batch", "pipeline": "logs", "error": "sending_queue is full"}
2023-04-03T17:40:20.893Z warn [email protected]/batch_processor.go:178 Sender failed {"kind": "processor", "name": "batch", "pipeline": "logs", "error": "sending_queue is full"}
2023-04-03T17:40:21.894Z warn [email protected]/batch_processor.go:178 Sender failed {"kind": "processor", "name": "batch", "pipeline": "logs", "error": "sending_queue is full"}
2023-04-03T17:40:22.895Z warn [email protected]/batch_processor.go:178 Sender failed {"kind": "processor", "name": "batch", "pipeline": "logs", "error": "sending_queue is full"}
2023-04-03T17:40:23.896Z warn [email protected]/batch_processor.go:178 Sender failed {"kind": "processor", "name": "batch", "pipeline": "logs", "error": "sending_queue is full"}
2023-04-03T17:40:24.445Z error exporterhelper/queued_retry.go:310 Dropping data because sending_queue is full. Try increasing queue_size. {"kind": "exporter", "data_type": "metrics", "name": "clickhousemetricswrite", "dropped_items": 442}
Apr 04, 2023 (8 months ago)
Prashant
05:39 AM2023-04-03T17:40:18.226Z info exporterhelper/queued_retry.go:426 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "clickhousemetricswrite", "error": "code: 243, message: Cannot reserve 1.00 MiB, not enough space", "interval": "125.471225ms"}
igor You might want to increase ClickHouse PVC size. It seems to disk full issue.
Prashant
07:29 AMSigNoz Community
Indexed 1023 threads (61% resolved)
Similar Threads
Issue Accessing Pod Logs in SigNoz UI on AKS
prashant is facing an issue accessing pod logs of their application in SigNoz UI on AKS. nitya-signoz and Prashant provide suggestions related to log file paths and potential issues, but the problem remains unresolved.
Resolving Signoz Query Service Error
Einav encountered an error related to a missing table in the Signoz service which was preventing data visibility in the UI. Srikanth guided them to restart specific components and drop a database table, which resolved the issue.
Troubleshooting Memory Space Issue in Kubernetes with Signoz
Abel had trouble running signoz on Kubernetes due to 'not enough space'. Pranay provided steps to increase PV. Eventually, Abel confirmed solution after changing PV size to '50Gi'.
Slow Log Messages in SigNoz with Logrus and OpenTelemetry
Harald reported slow log messages in SigNoz with logrus and open telemetry. They narrowed down the problem to focusing on relevant k8s namespaces, which improved performance to near real-time.
Headers in Monitoring HTTP Endpoints & Otel-Collector Error
Rohit asks about providing headers in monitoring HTTP endpoints and reports errors in otel-collector logs. Srikanth mentions headers are unsupported and requires more investigation for errors.