#support

Issues with Loading Charts after Increasing ClickHouse Size

TLDR Muhammed reported issues with charts after increasing ClickHouse size. Srikanth asked for logs and provided possible reasons but needs to reproduce the problem to debug further.

Powered by Struct AI
white_check_mark1
Feb 19, 2023 (7 months ago)
Muhammed
Photo of md5-c6b4211e54e9d9c5de7b081d0bcda56a
Muhammed
08:22 AM
Hi,
I have increased the size of ClickHouse, but some of the charts are not loading correctly, is there any extra step todo?
08:26
Muhammed
08:26 AM
These are the possible states, Clickhouse layout is 1 replica and 4 shards
Image 1 for These are the possible states, Clickhouse layout is 1 replica and 4 shardsImage 2 for These are the possible states, Clickhouse layout is 1 replica and 4 shards
08:27
Muhammed
08:27 AM
Also for other charts I'm getting diffrent response for each refresh
03:04
Muhammed
03:04 PM
Pranay
Feb 20, 2023 (7 months ago)
vishal-signoz
Photo of md5-f936d3e5743d23344d6c60813189716f
vishal-signoz
04:11 AM
Muhammed On refresh time period changes so that might be reason.
Srikanth are we aware of any metrics issues after increasing shards?
Muhammed
Photo of md5-c6b4211e54e9d9c5de7b081d0bcda56a
Muhammed
02:29 PM
there is no increase in shards, but shards are restarted in a bad order
Srikanth
Photo of md5-ce04a9988e2fd758a659dc55be6f2543
Srikanth
03:54 PM
How many shards were up and able to reachable when you saw this behaviour?
Muhammed
Photo of md5-c6b4211e54e9d9c5de7b081d0bcda56a
Muhammed
05:13 PM
I have the total of 4 shards, all of them were reachable
05:13
Muhammed
05:13 PM
I'm having this problem right now
Srikanth
Photo of md5-ce04a9988e2fd758a659dc55be6f2543
Srikanth
05:13 PM
Can you share the logs of each shard process?
Muhammed
Photo of md5-c6b4211e54e9d9c5de7b081d0bcda56a
Muhammed
05:15 PM
yes
05:16
Muhammed
05:16 PM
05:17
Muhammed
05:17 PM
there is no error log as I see
Srikanth
Photo of md5-ce04a9988e2fd758a659dc55be6f2543
Srikanth
05:18 PM
What’s the output logs for other shards?
Muhammed
Photo of md5-c6b4211e54e9d9c5de7b081d0bcda56a
Muhammed
05:19 PM
I put all of them in that file
05:19
Muhammed
05:19 PM
search for "k logs"
Srikanth
Photo of md5-ce04a9988e2fd758a659dc55be6f2543
Srikanth
05:22 PM
Each shard runs as a separate pod?
Muhammed
Photo of md5-c6b4211e54e9d9c5de7b081d0bcda56a
Muhammed
05:22 PM
yes
05:23
Muhammed
05:23 PM
using signoz chart (with altinity operator)
Srikanth
Photo of md5-ce04a9988e2fd758a659dc55be6f2543
Srikanth
05:23 PM
And the shared file is only for one pod chi-signoz-clickhouse-cluster-0-0-0 right?
Muhammed
Photo of md5-c6b4211e54e9d9c5de7b081d0bcda56a
Muhammed
05:24 PM
no
05:24
Muhammed
05:24 PM
search for "k logs" and you see there is other pods
Srikanth
Photo of md5-ce04a9988e2fd758a659dc55be6f2543
Srikanth
05:29 PM
Ah I see
white_check_mark1
05:36
Srikanth
05:36 PM
Is this a frequent issue or happening occasionally? I am not sure what might be the reason here.
Muhammed
Photo of md5-c6b4211e54e9d9c5de7b081d0bcda56a
Muhammed
05:37 PM
Let me checkout signoz right now
05:38
Muhammed
05:38 PM
that's heppening right now
05:39
Muhammed
05:39 PM
Image 1 for Image 2 for Image 3 for
Srikanth
Photo of md5-ce04a9988e2fd758a659dc55be6f2543
Srikanth
05:41 PM
Hmm, interesting which version are you on? We might have to debug this carefully.
Muhammed
Photo of md5-c6b4211e54e9d9c5de7b081d0bcda56a
Muhammed
05:41 PM
v0.16.0
05:42
Muhammed
05:42 PM
I think It will be fixed by just a rebalance
but I don't know how to do it in ClickHouse
Srikanth
Photo of md5-ce04a9988e2fd758a659dc55be6f2543
Srikanth
05:44 PM
That’s a somewhat manual process, as it doesn’t automatically take care of it. We have some logic to avoid writing the duplicate time series, so that could also be the reason.
Muhammed
Photo of md5-c6b4211e54e9d9c5de7b081d0bcda56a
Muhammed
05:44 PM
Oh
Srikanth
Photo of md5-ce04a9988e2fd758a659dc55be6f2543
Srikanth
05:46 PM
I will try to reproduce and see what we can do about this.
Muhammed
Photo of md5-c6b4211e54e9d9c5de7b081d0bcda56a
Muhammed
05:54 PM
Thanks, Let me know If I can help about with reproducing problem