Signoz Server Data Transfer Issue After Restart
TLDR Ulisses experienced data transfer issues to signoz server after a restart. Srikanth suggested tweaking agent batch size and span settings but no clear solution was reached.
Aug 28, 2023 (3 months ago)
I started to test signoz and have setup 3 java applications that are using the otel java agent on tomcat to send data directly to signoz collector on port 4317.
Sometimes i need to restart the signoz server and one of the applications just can't send data until i do a a restart of this application.
[otel.javaagent 2023-08-28 08:09:36:463 -0400] [OkHttp <http://signoz.server:4317/...>] ERROR io.opentelemetry.exporter.internal.grpc.OkHttpGrpcExporter - Failed to export spans. The request could not be executed. Full error message: FRAME_SIZE_ERROR: 4740180
[otel.javaagent 2023-08-28 08:09:41:467 -0400] [OkHttp <http://signoz.server:4317/...>] ERROR io.opentelemetry.exporter.internal.grpc.OkHttpGrpcExporter - Failed to export spans. The request could not be executed. Full error message: FRAME_SIZE_ERROR: 4740180
[otel.javaagent 2023-08-28 08:09:42:582 -0400] [OkHttp <http://signoz.server:4317/...>] ERROR io.opentelemetry.exporter.internal.grpc.OkHttpGrpcExporter - Failed to export logs. The request could not be executed. Full error message: FRAME_SIZE_ERROR: 4740180
There is any option on otel agent or signoz i'm missing?
max_recv_msg_size_mibof otlp receiver https://github.com/open-telemetry/opentelemetry-collector/blob/main/config/configgrpc/README.md#server-configuration
receivers.otlp.protocols.grpc.max_recv_msg_size_mib: 100, restarted the containers with docker compose down/up, but the problem continues.
Aug 29, 2023 (3 months ago)
Aug 30, 2023 (3 months ago)
Indexed 1023 threads (61% resolved)
Issue with Otel Java Agent and Tomcat Server Logs
Ulisses is experiencing issues with a specific application that stops sending logs and metrics when the otel collector container is restarted. Srikanth suggested increasing the maximum received message size.
Troubleshooting SigNoz Auto-Instrumentation Configuration
igor is having trouble configuring auto-instrumentation for Java applications using SigNoz, with traces not appearing in the SigNoz UI. Prashant advises to check logs of the otel sidecar, use service name for endpoint, verify supported libraries, and test with telemetrygen. However, the issue still persists.
Troubleshooting Missing Otel Data in SigNoz
Brian sought help with missing Otel data in SigNoz. Srikanth asked about data sending method. Brian shared code and considered it user error.
Issue Accessing Pod Logs in SigNoz UI on AKS
prashant is facing an issue accessing pod logs of their application in SigNoz UI on AKS. nitya-signoz and Prashant provide suggestions related to log file paths and potential issues, but the problem remains unresolved.
Troubleshooting Memory Space Issue in Kubernetes with Signoz
Abel had trouble running signoz on Kubernetes due to 'not enough space'. Pranay provided steps to increase PV. Eventually, Abel confirmed solution after changing PV size to '50Gi'.