Memory Limit Exceeded Error in Production

Getting error as per below

2023.05.29 14:07:00.725018 [ 28 ] {3878bf9e-aad5-42f8-827c-eb6b46e61409::20230528_1646_3017_14_13054} <Error> MutatePlainMergeTreeTask: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 3.60 GiB (attempt to allocate chunk of 4220276 bytes), maximum: 3.60 GiB. OvercommitTracker decision: Memory overcommit isn't used. Waiting time or overcommit denominator are set to zero. (MEMORY_LIMIT_EXCEEDED) (version 22.8.8.3 (official build))
2023.05.29 14:07:00.726425 [ 28 ] {3878bf9e-aad5-42f8-827c-eb6b46e61409::20230528_1646_3017_14_13054} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 241. DB::Exception: Memory limit (total) exceeded: would use 3.60 GiB (attempt to allocate chunk of 4220276 bytes), maximum: 3.60 GiB. OvercommitTracker decision: Memory overcommit isn't used. Waiting time or overcommit denominator are set to zero. (MEMORY_LIMIT_EXCEEDED), Stack trace

Any idea?

In case you missed it - https://signoz-community.slack.com/archives/C01HWUTP4HH/p1685391280904899

Pranay Prashant I have been experiencing the same issues from last 2 days on production. The allocated memory is 14GB. I dont see any traffic spikes/ span counts. I will try to increase the size and try. But from the kube pod memory graphs i can see it is requesting as much as >25GB sometimes.