#general

Managing Data Lifecycles in Clickhouse

TLDR Alejandro needed help with data management in Clickhouse. Srikanth and vishal-signoz provided links and examples for implementing tag level TTL and data deletion.

4mo

Join the chat

May 11, 2023 (4 months ago)

Photo of md5-a8b7de0a2b191db78968c382f3454e26

Alejandro

08:36 PM

Looks I have not space at clickhouse:

Cannot reserve 1.00 MiB, not enough space

how can i drop my data from clickhouse since x days?

May 12, 2023 (4 months ago)

Photo of md5-ce04a9988e2fd758a659dc55be6f2543

Srikanth

02:15 AM

See https://clickhouse.com/docs/en/sql-reference/statements/alter/delete

Alejandro

11:28 AM

Ahaa ok. I thought I have an option via SigNoz UI to clean data from determined date

11:29

Alejandro

11:29 AM

On the other hand, is a way to say for example application with tag service name helloworld keep traces from last 15 days ?

Srikanth

03:35 PM

Yes, there is some form of tag level TTL, but there is no way to do it from UI.

May 14, 2023 (4 months ago)

Alejandro

05:06 PM

How can I do it ? Without the UI ?

May 15, 2023 (4 months ago)

Srikanth

04:49 AM

through clickhouse command line

Alejandro

02:06 PM

can you share that command ?

Srikanth

02:29 PM

I know it exists but haven’t tried it ourselves.

Photo of md5-f936d3e5743d23344d6c60813189716f

vishal-signoz

02:36 PM

Here’s an example Alejandro for the your case

CREATE TABLE events
(
    `event` String,
    `time` DateTime,
    `value` UInt64
)
ENGINE = MergeTree
ORDER BY (event, time)
TTL time + INTERVAL 1 MONTH DELETE WHERE event != 'error',
    time + INTERVAL 6 MONTH DELETE WHERE event = 'error'

02:36

vishal-signoz

02:36 PM

Refer: https://clickhouse.com/blog/using-ttl-to-manage-data-lifecycles-in-clickhouse

02:37

vishal-signoz

02:37 PM

Follow this: https://stackoverflow.com/questions/72037815/clickhouse-ttl-settings-with-where-condition

SigNoz Community

Built with ClickHouse as datastore, SigNoz is an open-source APM to help you find issues in your deployed applications & solve them quickly | Knowledge Base powered by Struct.AI

Indexed 825 threads (61% resolved)

Join Our Community Manage subscription

Similar Threads

Services UI Goes Blank Due to Retention Issue with S3 Bucket

oluchi reports issue with services UI going blank after a while. Conversation explores possible reasons, such as S3 connection problems and disk space, but no resolution is reached.

4mo

Issues with SigNoz Setup and Data Persistence in AKS

Vaibhavi experienced issues setting up SigNoz in AKS, and faced data persistence issues after installation. Srikanth provided guidance on ClickHouse version compatibility and resource requirements, helping Vaibhavi troubleshoot and resolve the issue.

5mo

SigNoz crashing in k8s due to ClickHouse OOM

Travis reported SigNoz crashing in k8s due to ClickHouse OOM. The team suggested increasing resources for ClickHouse, and other troubleshooting steps, but the issue remains unresolved.

6mo

Troubleshooting Full Disk Space and Removing Old Data

Romario faced full disk space issue and wanted to remove old data. Ankit guided them through installing ClickHouse client, truncating tables, and using `docker system prune` to clear space.

4mo

Dashboard Load Issues and Possible Solutions

Al experiences dashboard loading issues since updating to `0.18.1`. Srikanth believes the issue is not version related and suggests examining queries, memory resources, and server distribution for improvements.

4mo