#support

Resolving Log Retention and Login Issues in Clickhouse

TLDR duuZ faced issues with log retention and login in Clickhouse. nitya-signoz provided solutions for log cleanup, PVC increase, and running one replica of the query service.

7mo

Join the chat

Mar 01, 2023 (7 months ago)

Photo of md5-aece7b06eda2a85a5bcfb5fd39332e2c

duuZ

04:30 AM

Hi team, One quick question is there a way to implement retention of logs data via helm chart and not fromt the UI, or do we have a way to clear that vial clickhouse command?

Photo of md5-a52b9d6c34f193d9a1ff940024f36f77

nitya-signoz

04:40 AM

As of now, it’s not possible via the helm chart.

What do you mean by or do we have a way to clear that vial clickhouse command ? do you want to remove ttl?

duuZ

04:59 AM

actually, my pvc is full and it cannot be increased as its limit is reached. Whole signoz system is stuck because DB is not responding because of the disk issue. So if i clear the logs i think i can login from the UI

nitya-signoz

05:01 AM

If you are able to exec into clickhouse then try deleting the data

kubectl exec -n platform -it chi-my-release-clickhouse-cluster-0-0-0 -- sh

clickhouse client

use signoz_logs;

truncate table signoz_logs.logs;

duuZ

05:02 AM

okay thanks, logs will be created automatically in the next run

nitya-signoz

05:02 AM

yeah, once you are done cleaning up, do set up retention period from the UI.

05:03

nitya-signoz

05:03 AM

You can also push your data to s3 for cold storage https://signoz.io/docs/userguide/retention-period/#configuring-cold-storage---amazon-s3

duuZ

06:21 AM

nitya-signoz Actually it didnt help, im not able to login to signoz with admin account it says user does not exists. Can you help

06:22

duuZ

06:22 AM

i enable the s3 cold storage and in the middle of everything i believe because my data is huge it stuck again, Now im not able to login from admin account. Do you know in which table we are storing these user details

08:25

duuZ

08:25 AM

and anlso is there any way to refresh all the data like logs traces and metrices

01:39

duuZ

01:39 PM

hen i try with admin user it say account doesn't exists, but i can see all the user in the sqlite DB. When enabling s3 cold storage is there any migration scripts that we can run manually, just to make sure everything moved correctly, As i believe because the data is huge when i enable the s3 cold storage and somehow process stopped, it stopped the migration script. Any help guys..

nitya-signoz

02:37 PM

vishal-signoz any idea about the admin user issue?

02:38

nitya-signoz

02:38 PM

You can’t move things manually to s3, it’s done by clickhouse internally. You can just check the current running process in clickhouse.

you can check them by
SELECT query_id, query FROM system.processes;

It will show you if the ttl is applied or not

02:40

nitya-signoz

02:40 PM

How much storage have you allocated to clickhouse and how much data are you ingesting ?

duuZ

02:40 PM

initially 200 gb

02:41

duuZ

02:41 PM

and the ingestion is around 300 GB i gues

nitya-signoz

02:41 PM

Is it 300 GB per day?

duuZ

02:42 PM

no iniatilly we are dumping some data to test, daily 10-15 GB i guess

nitya-signoz

02:43 PM

Okay just to clear out you are facing two issue
• Not able to login
• ClickHouse disk full
right ?

duuZ

02:45 PM

yes not able to login, for clickhouse ive incresed the pvc to 500 with AWS support.

nitya-signoz

02:53 PM

can do a huddle for the login issue

duuZ

02:55 PM

i think its working now, im able to login and able to see few items after increasing pvc

nitya-signoz

02:55 PM

Cool that’s great.

duuZ

02:56 PM

also i truncate a table in signoz_traces, will it recreated in next run

nitya-signoz

02:56 PM

To stop ingestion for somtime, you can stop the otel collector and the otel-collector metrics.

02:56

nitya-signoz

02:56 PM

Then run truncate, also you can change TTL at this point of time and configure S3 as it will be fast.

duuZ

02:57 PM

yes that is what i did yesterday. It is stopped for now.

03:27

duuZ

03:27 PM

As of now it look like something happened when i enabled s3 cold storage. sometimes im able to see things and sometime i'm getting authorization error while accessing the api's from frontend.

Mar 02, 2023 (7 months ago)

nitya-signoz

03:35 AM

Can you check if any of the pods are restarting ?

duuZ

06:05 AM

no its not restarting, but i deployed the new setup and sometime im able to see thing and sometimes it logging me out

06:06

duuZ

06:06 AM

aby idea

06:21

duuZ

06:21 AM

looks like when we are running the setup with replica count 2, with some calls it gives 403

nitya-signoz

06:34 AM

ahh as of now you will have to run one replica of query service

duuZ

06:34 AM

yes did the same and it works, thanks

SigNoz Community

Built with ClickHouse as datastore, SigNoz is an open-source APM to help you find issues in your deployed applications & solve them quickly | Knowledge Base powered by Struct.AI

Indexed 825 threads (61% resolved)

Join Our Community Manage subscription

Similar Threads

Services UI Goes Blank Due to Retention Issue with S3 Bucket

oluchi reports issue with services UI going blank after a while. Conversation explores possible reasons, such as S3 connection problems and disk space, but no resolution is reached.

4mo

Issues with SigNoz Setup and Data Persistence in AKS

Vaibhavi experienced issues setting up SigNoz in AKS, and faced data persistence issues after installation. Srikanth provided guidance on ClickHouse version compatibility and resource requirements, helping Vaibhavi troubleshoot and resolve the issue.

5mo

Dashboard Load Issues and Possible Solutions

Al experiences dashboard loading issues since updating to `0.18.1`. Srikanth believes the issue is not version related and suggests examining queries, memory resources, and server distribution for improvements.

4mo

Configuring ClickHouse for S3 Retention and EC2 Deployment

sarthak asks about configuring S3 data retention and if ClickHouse documentation is enough. vishal-signoz shares guide links and confirms ClickHouse will query S3 data, SigNoz handles everything, and using EC2 with vertical scaling is fine for their use case.

4mo

SigNoz crashing in k8s due to ClickHouse OOM

Travis reported SigNoz crashing in k8s due to ClickHouse OOM. The team suggested increasing resources for ClickHouse, and other troubleshooting steps, but the issue remains unresolved.

6mo