#support

Self-hosted Signoz Retains All Traces Beyond Limit

TLDR Prashant has a problem with Signoz retaining traces beyond the default limit, causing storage to be full. Srikanth suggests using "OPTIMIZE TABLE ..." command in Clickhouse to trigger cleanup.

4mo

Join the chat

Jul 23, 2023 (4 months ago)

Photo of md5-9666afc0ebaac245df53cf43559d3630

Prashant

01:49 PM

Hey in my self hosted signoz setup. Signoz is retaining all traces it should clear up metrics that are older than a than a week, which is the default behavior. This is leading to my storage being full. I installed signoz using the default chart on my kubernetes cluster.

Jul 24, 2023 (4 months ago)

Photo of md5-ce04a9988e2fd758a659dc55be6f2543

Srikanth

04:19 AM

That shouldn’t be the case, How long did it retain beyond the TTL limit?

Prashant

06:02 AM

The trace time limit is 7 days. I can see traces from 27th of June. Is there a way to force a cleanup?

Srikanth

06:47 AM

Can you share one such example trace and confirm if you haven’t updated the retention from settings?

Prashant

07:26 AM

Here’s the trace details screenshot and the settings for traces.

Jul 25, 2023 (4 months ago)

Srikanth

02:47 AM

That’s unusual. You need to exec into clickhouse and run OPTIMIZE TABLE ... FINAL to force trigger the cleanup. Run this on the signoz_index_v2 table.

SigNoz Community

Built with ClickHouse as datastore, SigNoz is an open-source APM to help you find issues in your deployed applications & solve them quickly | Knowledge Base powered by Struct.AI

Indexed 1023 threads (61% resolved)

Join Our Community

Similar Threads

Issues with SigNoz Setup and Data Persistence in AKS

Vaibhavi experienced issues setting up SigNoz in AKS, and faced data persistence issues after installation. Srikanth provided guidance on ClickHouse version compatibility and resource requirements, helping Vaibhavi troubleshoot and resolve the issue.

8mo

Trouble Setting Trace Retention Days in SigNoz

Jose faced issues with trace retention settings. vishal-signoz advised connecting to SQLite DB and clearing TTL status table to update the retention setting.

8mo

Dashboard Load Issues and Possible Solutions

Al experiences dashboard loading issues since updating to `0.18.1`. Srikanth believes the issue is not version related and suggests examining queries, memory resources, and server distribution for improvements.

7mo