Setting Up Dual Exporters for Otel-collector and Resolving Payload Error

Hi, I'm trying to set up a dual exporters for the otel-collector, I'm wondering if I got this right. Here are the values for the chart :

otelCollector:
  config:
    exporters:
      otlp/new:
        endpoint: new.ns.svc.cluster.local:4317
service:
  pipelines:
    traces:
      exporters:
      - clickhousetraces
      - otlp/new

I read that's kinda the idea right? Tho it doesn't seem to be working. My collectors are going up, no errors showing:

2023-03-28T15:09:54.556Z    info    service/telemetry.go:111    Setting up own telemetry...
2023-03-28T15:09:54.557Z    info    service/telemetry.go:141    Serving Prometheus metrics    {"address": "0.0.0.0:8888", "level": "Basic"}
2023-03-28T15:09:54.557Z    info    components/components.go:30    Stability level of component is undefined    {"kind": "exporter", "data_type": "metrics", "name": "clickhousemetricswrite", "stability": "Undefined"}
time="2023-03-28T15:09:54Z" level=info msg="Executing:\nCREATE DATABASE IF NOT EXISTS signoz_metrics ON CLUSTER cluster\n" component=clickhouse

And I still receive data in clickhouse as I can fetch it and read it through signoz frontend.

If there are no error logs from the otlp/new: then it’s working but the log you shared doesn’t give full picture since it doesn’t show collector started to accept data yet.

Ok it was linked to the issue of pod never being Ready. I succeeded to deploy other pods and have a configuration Ready, I can see an error message:

2023-03-29T09:21:25.733Z    warn    zapgrpc/zapgrpc.go:191    [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {
  "Addr": "apm.monitoring.svc.cluster.local:4318",
  "ServerName": "apm.monitoring.svc.cluster.local:4318",
  "Attributes": null,
  "BalancerAttributes": null,
  "Type": 0,
  "Metadata": null
}. Err: connection error: desc = "error reading server preface: http2: frame too large"    {"grpc_log": true}

This means the request payload is too large. Either configure the grpc server to receiver bigger streams or batch process with a size that doesn’t lead to very large payloads from exporter.

Yes thanks, I found it quickly how to change this, it worked after the changes.