Increase Memory Limits to Fix Data Absence in Events Section
TLDR Muhammed is facing a problem after adding a JSON parser, where there's no data in events section. nitya-signoz suggests increasing memory limits. Issue is solved after increasing memory limits.
Feb 28, 2023 (9 months ago)
Muhammed
09:17 AMAfter adding json parser to my logs in
k8s-infra log counter in logs section works but there is no data in events section, also fields are loaded correctly
Ankit
09:25 AM
nitya-signoz
09:39 AMAlso if your fields are loaded but not visible in the UI there might be issues with timestamp.
Can you run the commands and share output
kubectl exec -n platform -it chi-my-release-clickhouse-cluster-0-0-0 -- sh
clickhouse client
use signoz_logs;
select timestamp,observed_timestamp from distributed_logs order by timestamp desc limit 10;Muhammed
09:41 AM┌───────────timestamp─┬──observed_timestamp─┐
│ 1677577033101391005 │ 1677577033114197754 │
│ 1677577033087210293 │ 1677577033114149167 │
│ 1677577033081940586 │ 1677577033113942963 │
│ 1677577033079232086 │ 1677577033114117714 │
│ 1677577033074985781 │ 1677577033114088974 │
│ 1677577033071399878 │ 1677577033113746096 │
│ 1677577033068867909 │ 1677577033114495821 │
│ 1677577033058288277 │ 1677577033114040042 │
│ 1677577033054256687 │ 1677577033114446222 │
│ 1677577033036741379 │ 1677577033114394174 │
└─────────────────────┴─────────────────────┘Muhammed
09:41 AM
Muhammed
09:41 AMnitya-signoz
09:43 AM/logs endpoint from the network tab?nitya-signoz
09:47 AMMuhammed
09:49 AM/logs only aggregatenitya-signoz
09:50 AMnitya-signoz
09:50 AMMuhammed
09:51 AMMuhammed
09:52 AM{"status":"error","data":"Failed to fetch logs from the DB","errorType":"internal","error":"code: 241, message: Memory limit (total) exceeded: would use 3.62 GiB (attempt to allocate chunk of 5064994 bytes), maximum: 3.60 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker.: (while reading column attributes_string_value): While executing MergeTreeReverse"}Muhammed
09:53 AM resources:
requests:
cpu: 100m
memory: 200Mi
limits:
cpu: 2000m
memory: 4GiMuhammed
09:54 AMor there are other settings?
nitya-signoz
09:56 AM
Palash
10:34 AMI am not aware of such case... Muhammed can you please share the loom video for the same
Muhammed
10:45 AMPalash
10:48 AM
SigNoz Community
Indexed 1023 threads (61% resolved)
Similar Threads
Issues with SigNoz Setup and Data Persistence in AKS
Vaibhavi experienced issues setting up SigNoz in AKS, and faced data persistence issues after installation. Srikanth provided guidance on ClickHouse version compatibility and resource requirements, helping Vaibhavi troubleshoot and resolve the issue.
SigNoz crashing in k8s due to ClickHouse OOM
Travis reported SigNoz crashing in k8s due to ClickHouse OOM. The team suggested increasing resources for ClickHouse, and other troubleshooting steps, but the issue remains unresolved.
Dashboard Load Issues and Possible Solutions
Al experiences dashboard loading issues since updating to `0.18.1`. Srikanth believes the issue is not version related and suggests examining queries, memory resources, and server distribution for improvements.
Services UI Goes Blank Due to Retention Issue with S3 Bucket
oluchi reports issue with services UI going blank after a while. Conversation explores possible reasons, such as S3 connection problems and disk space, but no resolution is reached.
Kubernetes Signoz-otel-collector Issue and Clickhouse Cold Storage
Pruthvi faced an issue with Kubernetes signoz-otel-collector. nitya-signoz suggested deleting the `signoz_logs` database and restarting collectors. Pruthvi then asked about Clickhouse cold storage on S3 and observed a spike in cost, which Ankit agreed to investigate further.
2