#support

Signoz Server Data Transfer Issue After Restart

TLDR Ulisses experienced data transfer issues to signoz server after a restart. Srikanth suggested tweaking agent batch size and span settings but no clear solution was reached.

3mo

Join the chat

Aug 28, 2023 (3 months ago)

Photo of md5-dde71d32676a10e2ef9d125fe3650e2e

Ulisses

01:02 PM

Hello, i'm having the following problem:
I started to test signoz and have setup 3 java applications that are using the otel java agent on tomcat to send data directly to signoz collector on port 4317.
Sometimes i need to restart the signoz server and one of the applications just can't send data until i do a a restart of this application.

[otel.javaagent 2023-08-28 08:09:36:463 -0400] [OkHttp <http://signoz.server:4317/...>] ERROR io.opentelemetry.exporter.internal.grpc.OkHttpGrpcExporter - Failed to export spans. The request could not be executed. Full error message: FRAME_SIZE_ERROR: 4740180

[otel.javaagent 2023-08-28 08:09:41:467 -0400] [OkHttp <http://signoz.server:4317/...>] ERROR io.opentelemetry.exporter.internal.grpc.OkHttpGrpcExporter - Failed to export spans. The request could not be executed. Full error message: FRAME_SIZE_ERROR: 4740180

[otel.javaagent 2023-08-28 08:09:42:582 -0400] [OkHttp <http://signoz.server:4317/...>] ERROR io.opentelemetry.exporter.internal.grpc.OkHttpGrpcExporter - Failed to export logs. The request could not be executed. Full error message: FRAME_SIZE_ERROR: 4740180

There is any option on otel agent or signoz i'm missing?

Photo of md5-ce04a9988e2fd758a659dc55be6f2543

Srikanth

01:06 PM

max_recv_msg_size_mib of otlp receiver https://github.com/open-telemetry/opentelemetry-collector/blob/main/config/configgrpc/README.md#server-configuration

Ulisses

01:46 PM

I have updated the otel-collector-config.yaml with
receivers.otlp.protocols.grpc.max_recv_msg_size_mib: 100, restarted the containers with docker compose down/up, but the problem continues.

02:37

Ulisses

02:37 PM

To add some context, all 3 applications are running in the same VM server, but on different tomcat instances.

Aug 29, 2023 (3 months ago)

Srikanth

03:21 AM

Lower the agent batch size to export most frequently and check.

Ulisses

06:52 PM

otel.bsp.max.export.batch.size or otel.blrp.max.export.batch.size? or both?

Aug 30, 2023 (3 months ago)

Srikanth

12:11 PM

You might have to tweak both. How many spans does your app roughly produce? You need to adjust the settings, or you could drop the spans with head-based sampling if the volume and size are big.

Ulisses

12:42 PM

This app is generating like 2220 spans per minute. There are some operations that generates 28000+ spans. I can't even open the Trace Detail page because the browser can't handle that number of elements and will freeze.

SigNoz Community

Built with ClickHouse as datastore, SigNoz is an open-source APM to help you find issues in your deployed applications & solve them quickly | Knowledge Base powered by Struct.AI

Indexed 1023 threads (61% resolved)

Join Our Community

Similar Threads

Issue with Otel Java Agent and Tomcat Server Logs

Ulisses is experiencing issues with a specific application that stops sending logs and metrics when the otel collector container is restarted. Srikanth suggested increasing the maximum received message size.

Troubleshooting SigNoz Auto-Instrumentation Configuration

igor is having trouble configuring auto-instrumentation for Java applications using SigNoz, with traces not appearing in the SigNoz UI. Prashant advises to check logs of the otel sidecar, use service name for endpoint, verify supported libraries, and test with telemetrygen. However, the issue still persists.

8mo

Troubleshooting Missing Otel Data in SigNoz

Brian sought help with missing Otel data in SigNoz. Srikanth asked about data sending method. Brian shared code and considered it user error.

5mo

Issue Accessing Pod Logs in SigNoz UI on AKS

prashant is facing an issue accessing pod logs of their application in SigNoz UI on AKS. nitya-signoz and Prashant provide suggestions related to log file paths and potential issues, but the problem remains unresolved.

6mo

Troubleshooting Memory Space Issue in Kubernetes with Signoz

Abel had trouble running signoz on Kubernetes due to 'not enough space'. Pranay provided steps to increase PV. Eventually, Abel confirmed solution after changing PV size to '50Gi'.

yesterday