#support

Using Kafka Broker for Telem Data to Ensure Fault Tolerance

TLDR sarthak inquires about using kafka broker with ClickHouse for fault tolerance. Srikanth and Ankit discuss scale and data flow feasibility. sarthak also requests a circuit-breaking mechanism, which Ankit explains is already in place.

4mo

Join the chat

May 17, 2023 (4 months ago)

Photo of md5-bc67a182dac7a9be9e74164eebcd86ee

sarthak

04:02 AM

hello everyone , is it recommended to use kafka broker as receiver of telem data to export in clickhouse instead of standard grpc in case we need to handle high scale and keep fault tolerance so as to prevent possible data loss in case storage/clickhouse failure ?

May 18, 2023 (4 months ago)

Photo of md5-ce04a9988e2fd758a659dc55be6f2543

Srikanth

01:50 AM

How much scale are we talking about? It might be overkill for regular users. Just putting the queue alone doesn’t guarantee the prevention of data loss since exporter will eventually drop the data when ClickHouse is not reachable.

Photo of md5-dbe7088320fe1d922707613e02f3420d

Ankit

04:07 AM

> It might be overkill for regular users.
correct. I think the data flow would look like otel-collector => kafka => clickhouse so we expect Kafka to handle bursts in traffic and downtime of clickhouse

Srikanth

04:55 AM

They mentioned they want to use it as a receiver as a substitute for the gRPC OTLP receiver and then export it to ClickHouse.

sarthak

08:52 AM

ok , so is there a way to implement some circuit breaking mechanism at microservice level keeping transport mechanism to as it is (gRPC) which can be pass with other env variables so that source service does not become down in case signoz backend is completely down as it will be continuously sending telemetry event to signoz , just faced this on my basic setup and testing

Ankit

02:05 PM

source service starts dropping telemetry data if signoz is down. It should not affect application other than it will need more memory to keep a batch after which it starts dropping data. The service will also print logs about being unable to send data to signoz but application should work just fine

SigNoz Community

Built with ClickHouse as datastore, SigNoz is an open-source APM to help you find issues in your deployed applications & solve them quickly | Knowledge Base powered by Struct.AI

Indexed 825 threads (61% resolved)

Join Our Community Manage subscription

Similar Threads

Issues with Opentelemetry Maven Plugin and Kafka Exporter

sarthak is facing issues implementing Kafka with Maven for Opentelemetry while trying to decouple the source application performance from the Signoz cluster. Pranay and nitya-signoz ask clarifying questions about the desired flow.

4mo

Configuring OTel Collector to Export Data to SigNoz

James asked how to configure the otel collector to send data to SigNoz. Ankit suggested using the otlp exporter in the collector configuration. James found that SigNoz uses its own otel collector and adjusted the port bindings accordingly.

3mo

Setting Up Dual Exporters for Otel-collector and Resolving Payload Error

Guillaume was facing an issue setting up dual exporters for otel-collector and had a pod not being "Ready". Srikanth suggested checking error logs and dealing with large payloads, which led to a successful setup.

5mo