Headers in Monitoring HTTP Endpoints & Otel-Collector Error

TLDR Rohit asks about providing headers in monitoring HTTP endpoints and reports errors in otel-collector logs. Srikanth mentions headers are unsupported and requires more investigation for errors.

Photo of Rohit
Rohit
Wed, 01 Mar 2023 17:24:52 UTC

Hi Team, I’ve been experimenting with Signoz for quite a while, and hats-off to an amazing product. I have 2 things that i needed help with - 1. Is there a way to provide headers in monitoring http endpoints? One of the endpoints that I want to monitor requires an authorisation header. 2. I am constantly seeing these errors in my otel-collector logs. Is this concerning? Can these be disabled somehow so I don’t miss the important logs? ```error exporterhelper/queued_retry.go:394 Exporting failed. The error is not retryable. Dropping data. {"kind": "exporter", "data_type": "metrics", "name": "clickhousemetricswrite", "error": "Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination; Permanent error: invalid temporality and type combination", "errorCauses": [{"error": "Permanent error: invalid temporality and type combination"}, {"error": "Permanent error: invalid temporality and type combination"}, {"error": "Permanent error: invalid temporality and type combination"}, {"error": "Permanent error: invalid temporality and type combination"}, {"error": "Permanent error: invalid temporality and type combination"}, {"error": "Permanent error: invalid temporality and type combination"}, {"error": "Permanent error: invalid temporality and type combination"}, {"error": "Permanent error: invalid temporality and type combination"}, {"error": "Permanent error: invalid temporality and type combination"}, {"error": "Permanent error: invalid temporality and type combination"}], "dropped_items": 143} /go/pkg/mod/go.opentelemetry.io/[email protected]/exporter/exporterhelper/queued_retry.go:394 /go/pkg/mod/go.opentelemetry.io/[email protected]/exporter/exporterhelper/metrics.go:135 /go/pkg/mod/go.opentelemetry.io/[email protected]/exporter/exporterhelper/queued_retry.go:205 /go/pkg/mod/go.opentelemetry.io/[email protected]/exporter/exporterhelper/internal/bounded_memory_queue.go:61```

Photo of Srikanth
Srikanth
Thu, 02 Mar 2023 00:28:06 UTC

> 1. Is there a way to provide headers in monitoring http endpoints? One of the endpoints that I want to monitor requires an authorisation header. Not currently supported now. > 1. I am constantly seeing these errors in my otel-collector logs. Is this concerning? Can these be disabled somehow so I don’t miss the important logs? What language SDKs are you using? This happens when the metric temporality is delta. This shouldn’t happen unless you explicitly configure the SDKs to prefer delta as the spec recommended default is cumulative. If you didn’t change any SDK configuration let me know what SDK are you using otherwise you can use the cumulative temporality to not see this error.

Photo of Rohit
Rohit
Thu, 02 Mar 2023 09:42:39 UTC

Srikanth 1. Okay, thank you. I’ll explore other solutions. 2. I am using FastAPI (Python) - To be more specific -> opentelemetry-instrumentation-fastapi = “^0.36b0” I wanted to understand if this error is coming from metrics or from traces? Since I’m seeing these errors in my otel-collector logs and not otel-collector-metrics logs. For metrics, I have implemented , and using prometheus scrape configs in signoz to import metrics.

Photo of Srikanth
Srikanth
Thu, 02 Mar 2023 09:57:47 UTC

Did you configure the temporality through env or constructor params?

Photo of Rohit
Rohit
Thu, 02 Mar 2023 10:44:51 UTC

Sorry for being a bit dumb here. But I didn’t really understand what you mean. This is my configuration -> In my start application code -> ```FastAPIInstrumentor.instrument_app(app) prometheus_instrumentor = Instrumentator( should_group_status_codes=True, should_ignore_untemplated=True, should_respect_env_var=True, should_instrument_requests_inprogress=True, excluded_handlers=["/health", "/metrics"], env_var_name="ENABLE_METRICS", inprogress_name="http_requests_inprogress", inprogress_labels=True, ) prometheus_instrumentor.instrument(app).expose(app, include_in_schema=False)``` In my environment variables -> ```OTEL_RESOURCE_ATTRIBUTES=service.name=inventory,deployment.environment=stage,version=1.0.0 OTEL_EXPORTER_OTLP_ENDPOINT= OTEL_METRICS_EXPORTER=none OTEL_PYTHON_FASTAPI_EXCLUDED_URLS=health,metrics ENABLE_METRICS=true SERVICE_NAME=inventory ENVIRONMENT=stage``` In my Dockerfile CMD -> ```ENTRYPOINT ["opentelemetry-instrument", "--traces_exporter", "otlp_proto_http", "--metrics_exporter" ,"otlp_proto_http", "gunicorn", "app.main:app", "--workers", "2", "--worker-class", "uvicorn.workers.UvicornWorker","--bind", "0.0.0.0:8000"]``` My opentelemetry http exporter is placed behind a load balancer, which handles the SSL configuration

Photo of Rohit
Rohit
Fri, 03 Mar 2023 13:01:54 UTC

Srikanth anything that you could maybe help me with here please?

Photo of Srikanth
Srikanth
Fri, 03 Mar 2023 17:26:22 UTC

I am not sure what is the issue here. I will have to spend some time digging this deeper.