Using Signoz for Data Pipeline Observability and Correlation
TLDR AjeethKumar asks if Signoz can be used for data pipeline observability and log correlation in Apache Airflow and Spark. Srikanth questions the annotation method for correlation, as OTEL integration may not be in place.
Srikanth
Tue, 14 Mar 2023 15:57:56 UTCHow do you annotate the logs of systems in pipeline to correlate? I don’t think either of the mentioned systems have OTEL integration (yet).
SigNoz Community
Indexed 1061 threads
Built with ClickHouse as datastore, SigNoz is an open-source APM to help you find issues in your deployed applications & solve them quickly | Knowledge Base powered by Struct.AI
View in App
AjeethKumar
Tue, 14 Mar 2023 13:57:37 UTCHi Team, Can I use Signoz for Data pipeline observability use case to ingest logs from Apache Airflow, Spark and then correlate logs to find root cause of pipeline failure?