#support

Debugging and Optimizing Slow Signoz Dashboard

TLDR Div reported slow Signoz dashboard and errors. Srikanth suggested creating a materialized column and adding filters to speed up queries. vishal-signoz confirmed the suggested approach.

7mo

Join the chat

May 10, 2023 (7 months ago)

Photo of md5-69a12cc92bad5b259938b070788a11aa

Div

03:19 AM

Hi Team, recently Signoz started to be much slower in our case to query the data for last 6 hours and sometimes the dashboard failing like with this error

• is there any documentation to debug slow issues or optimise the signoz dashboard ?

Image 1 for Hi Team, recently Signoz started to be much slower in our case to query the data for last 6 hours and sometimes the dashboard failing like with this error

• is there any documentation to debug slow issues or optimise the signoz dashboard ?

03:20

Div

03:20 AM

we are in the latest version 0.18.3

Photo of md5-ce04a9988e2fd758a659dc55be6f2543

Srikanth

03:24 AM

What are the resources (CPU and Memory) given to ClickHouse?

Div

03:29 AM

03:34

Div

03:34 AM

I have noticed when doing any search on signoz the clickhouse container CPU usage is going beyond 200%

03:34

Div

03:34 AM

using docker stats | grep container_id

Srikanth

04:05 AM

How many CPU cores are available?

Div

04:09 AM

with nproc it shows 2, and no specific cpu core setting is done on the docker container, so it has access to the 2 CPU cores from host

Photo of md5-dbe7088320fe1d922707613e02f3420d

Ankit

04:09 AM

Srikanth can you please get on a call with Div and check the query, the amount of data fetched, the processing speed of clickhouse, etc. We can discuss more based on the data. You if you have a set of commands, that can be run and will collect these data would also be good

04:10

Ankit

04:10 AM

2CPUs are less... how much data are you handling?

Div

04:14 AM

In a 1 day this is traces count

Srikanth

04:17 AM

2 CPU == 2 concurrent queries. What does the chart you mentioned above query?

Div

04:18 AM

the chart above if you mean the clickhouse one, is the stats I got from docker stats | grep container_id and while filtering traces for 1 Day

Srikanth

04:19 AM

No, I meant the original chat service chart

Div

04:20 AM

ah ok, it has 4 panels
1- network latency
2 - GC metrics
3- error rates
4- memory usage

Srikanth

04:24 AM

You gc queries the stringTagMap in traces which will get really slow and would lead to timeout for other panels since the gc query is not yet finished. Create a materialized column for it and check again.

Div

04:28 AM

as a quick solution, increasing the CPU cores would be good enough ?

Srikanth

04:38 AM

That might solve your issue temporarily, but I would recommend creating materialized column so you will not face the same issue.

Div

04:42 AM

is there any doc ref I can look at on how to add this materialized column for our usecase ?

we are sending the GC events via collector tracing service to signoz.

Srikanth

05:03 AM

I don’t think we have docs for that yet. We will look into it since it might also effect the exporter. If you can exec into clickhouse -> clieckhouse-client and run the query and share the end of the output that would be helpful.

SELECT toStartOfInterval(timestamp, INTERVAL 60 SECOND) AS ts, 
avg(durationNano) as value 
from signoz_traces.distributed_signoz_index_v2 where (timestamp >= (now() - toIntervalHour(6))) 
AND stringTagMap['gc_type']='GC' group by ts order by ts

Ankit

09:06 AM

vishal-signoz adding serviceName in the filters should speed up things, right?

09:08

Ankit

09:08 AM

vishal-signoz also, adding mapContains(stringTagMap, 'gc_type') will make the query faster?

Photo of md5-f936d3e5743d23344d6c60813189716f

vishal-signoz

09:09 AM

> adding serviceName in the filters should speed up things, right?
Yes, more filters means less DB rows scanned and it will be faster

09:10

vishal-signoz

09:10 AM

> also, adding mapContains(stringTagMap, 'gc_type') will make the query faster?
Yes if usecase is just to check if gc_type exists then this way it should be faster.

Ankit

09:36 AM

> Yes if usecase is just to check if gc_type exists then this way it should be faster.
I meant adding mapContains along with AND stringTagMap['gc_type']='GC'

vishal-signoz

10:05 AM

> I meant adding mapContains along with AND stringTagMap['gc_type']='GC'
Tested this on a test environment. This doesn’t reduce number of rows scanned and no significant performance changes with mapContains filter

SigNoz Community

Built with ClickHouse as datastore, SigNoz is an open-source APM to help you find issues in your deployed applications & solve them quickly | Knowledge Base powered by Struct.AI

Indexed 1023 threads (61% resolved)

Join Our Community

Similar Threads

Issues with SigNoz Setup and Data Persistence in AKS

Vaibhavi experienced issues setting up SigNoz in AKS, and faced data persistence issues after installation. Srikanth provided guidance on ClickHouse version compatibility and resource requirements, helping Vaibhavi troubleshoot and resolve the issue.

8mo

Dashboard Load Issues and Possible Solutions

Al experiences dashboard loading issues since updating to `0.18.1`. Srikanth believes the issue is not version related and suggests examining queries, memory resources, and server distribution for improvements.

7mo

Creating Signoz Dashboard for GC Metrics

Div wanted to create a dashboard for GC events and metrics. Ankit and vishal-signoz provided help with Clickhouse queries and template variables.

7mo

Services UI Goes Blank Due to Retention Issue with S3 Bucket

oluchi reports issue with services UI going blank after a while. Conversation explores possible reasons, such as S3 connection problems and disk space, but no resolution is reached.

7mo

Clickhouse Query Timeout in SigNoz Dashboard

Daniel experienced query timeouts in SigNoz dashboard. vishal-signoz confirmed it's a non-configurable 60s limit, but created a GitHub issue for adding the feature.

8mo