Using Signoz for Data Pipeline Observability and Correlation

TLDR AjeethKumar asks if Signoz can be used for data pipeline observability and log correlation in Apache Airflow and Spark. Srikanth questions the annotation method for correlation, as OTEL integration may not be in place.

Photo of AjeethKumar
Tue, 14 Mar 2023 13:57:37 UTC

Hi Team, Can I use Signoz for Data pipeline observability use case to ingest logs from Apache Airflow, Spark and then correlate logs to find root cause of pipeline failure?

Photo of Srikanth
Tue, 14 Mar 2023 15:57:56 UTC

How do you annotate the logs of systems in pipeline to correlate? I don’t think either of the mentioned systems have OTEL integration (yet).