Resolving Log Retention and Login Issues in Clickhouse
TLDR duuZ faced issues with log retention and login in Clickhouse. nitya-signoz provided solutions for log cleanup, PVC increase, and running one replica of the query service.
nitya-signoz
Wed, 01 Mar 2023 04:40:16 UTCAs of now, it’s not possible via the helm chart. What do you mean by `or do we have a way to clear that vial clickhouse command` ? do you want to remove ttl?
duuZ
Wed, 01 Mar 2023 04:59:19 UTCactually, my pvc is full and it cannot be increased as its limit is reached. Whole signoz system is stuck because DB is not responding because of the disk issue. So if i clear the logs i think i can login from the UI
nitya-signoz
Wed, 01 Mar 2023 05:01:04 UTCIf you are able to exec into clickhouse then try deleting the data ```kubectl exec -n platform -it chi-my-release-clickhouse-cluster-0-0-0 -- sh clickhouse client use signoz_logs; truncate table signoz_logs.logs;```
duuZ
Wed, 01 Mar 2023 05:02:12 UTCokay thanks, logs will be created automatically in the next run
nitya-signoz
Wed, 01 Mar 2023 05:02:43 UTCyeah, once you are done cleaning up, do set up retention period from the UI.
nitya-signoz
Wed, 01 Mar 2023 05:03:39 UTCYou can also push your data to s3 for cold storage
duuZ
Wed, 01 Mar 2023 06:21:05 UTCnitya-signoz Actually it didnt help, im not able to login to signoz with admin account it says user does not exists. Can you help
duuZ
Wed, 01 Mar 2023 06:22:28 UTCi enable the s3 cold storage and in the middle of everything i believe because my data is huge it stuck again, Now im not able to login from admin account. Do you know in which table we are storing these user details
duuZ
Wed, 01 Mar 2023 08:25:45 UTCand anlso is there any way to refresh all the data like logs traces and metrices
duuZ
Wed, 01 Mar 2023 13:39:29 UTChen i try with admin user it say account doesn't exists, but i can see all the user in the sqlite DB. When enabling s3 cold storage is there any migration scripts that we can run manually, just to make sure everything moved correctly, As i believe because the data is huge when i enable the s3 cold storage and somehow process stopped, it stopped the migration script. Any help guys..
nitya-signoz
Wed, 01 Mar 2023 14:37:21 UTC<@4K15aa> any idea about the admin user issue?
nitya-signoz
Wed, 01 Mar 2023 14:38:32 UTCYou can’t move things manually to s3, it’s done by clickhouse internally. You can just check the current running process in clickhouse. you can check them by `SELECT query_id, query FROM system.processes;` It will show you if the ttl is applied or not
nitya-signoz
Wed, 01 Mar 2023 14:40:13 UTCHow much storage have you allocated to clickhouse and how much data are you ingesting ?
duuZ
Wed, 01 Mar 2023 14:40:45 UTCinitially 200 gb
duuZ
Wed, 01 Mar 2023 14:41:02 UTCand the ingestion is around 300 GB i gues
nitya-signoz
Wed, 01 Mar 2023 14:41:22 UTCIs it 300 GB per day?
duuZ
Wed, 01 Mar 2023 14:42:06 UTCno iniatilly we are dumping some data to test, daily 10-15 GB i guess
nitya-signoz
Wed, 01 Mar 2023 14:43:54 UTCOkay just to clear out you are facing two issue • Not able to login • ClickHouse disk full right ?
duuZ
Wed, 01 Mar 2023 14:45:07 UTCyes not able to login, for clickhouse ive incresed the pvc to 500 with AWS support.
nitya-signoz
Wed, 01 Mar 2023 14:53:12 UTCcan do a huddle for the login issue
duuZ
Wed, 01 Mar 2023 14:55:18 UTCi think its working now, im able to login and able to see few items after increasing pvc
nitya-signoz
Wed, 01 Mar 2023 14:55:33 UTCCool that’s great.
duuZ
Wed, 01 Mar 2023 14:56:00 UTCalso i truncate a table in signoz_traces, will it recreated in next run
nitya-signoz
Wed, 01 Mar 2023 14:56:28 UTCTo stop ingestion for somtime, you can stop the otel collector and the otel-collector metrics.
nitya-signoz
Wed, 01 Mar 2023 14:56:38 UTCThen run truncate, also you can change TTL at this point of time and configure S3 as it will be fast.
duuZ
Wed, 01 Mar 2023 14:57:09 UTCyes that is what i did yesterday. It is stopped for now.
duuZ
Wed, 01 Mar 2023 15:27:33 UTCAs of now it look like something happened when i enabled s3 cold storage. sometimes im able to see things and sometime i'm getting authorization error while accessing the api's from frontend.
nitya-signoz
Thu, 02 Mar 2023 03:35:11 UTCCan you check if any of the pods are restarting ?
duuZ
Thu, 02 Mar 2023 06:05:58 UTCno its not restarting, but i deployed the new setup and sometime im able to see thing and sometimes it logging me out
duuZ
Thu, 02 Mar 2023 06:06:02 UTCaby idea
duuZ
Thu, 02 Mar 2023 06:21:21 UTClooks like when we are running the setup with replica count 2, with some calls it gives 403
nitya-signoz
Thu, 02 Mar 2023 06:34:17 UTCahh as of now you will have to run one replica of query service
duuZ
Thu, 02 Mar 2023 06:34:47 UTCyes did the same and it works, thanks
SigNoz Community
Indexed 1061 threads
Built with ClickHouse as datastore, SigNoz is an open-source APM to help you find issues in your deployed applications & solve them quickly | Knowledge Base powered by Struct.AI
View in App
duuZ
Wed, 01 Mar 2023 04:30:49 UTCHi team, One quick question is there a way to implement retention of logs data via helm chart and not fromt the UI, or do we have a way to clear that vial clickhouse command?