Configuring ClickHouse for S3 Retention and EC2 Deployment
TLDR sarthak asks about configuring S3 data retention and if ClickHouse documentation is enough. vishal-signoz shares guide links and confirms ClickHouse will query S3 data, SigNoz handles everything, and using EC2 with vertical scaling is fine for their use case.
vishal-signoz
Wed, 03 May 2023 11:26:39 UTCYou can set retention based on time period like move to s3 by following this doc:
vishal-signoz
Wed, 03 May 2023 11:27:42 UTCEven in s3 the performance drop is around 30% so you can access s3 data like normal data (with some performance drop)
sarthak
Wed, 03 May 2023 11:54:31 UTCthanks vishal for the response , so you are saying , suppose if i retain after 7 days data into s3 by configuring then , after that clickhouse will automatically query old data from s3 , actually i also had that doubt that if i need to write some code/query using s3 table engine to get visuals from data stored in s3 .
vishal-signoz
Wed, 03 May 2023 11:56:25 UTC> so you are saying , suppose if i retain after 7 days data into s3 by configuring then , after that clickhouse will automatically query old data from s3 Yes
sarthak
Wed, 03 May 2023 11:56:51 UTCthanks bro for clearification
vishal-signoz
Wed, 03 May 2023 11:57:03 UTC> actually i also had that doubt that if i need to write some code/query using s3 table engine to get visuals from data stored in s3 No, SigNoz handles everything for you
sarthak
Wed, 03 May 2023 11:58:24 UTCactually i want to deeply understand about clickhouse working and storage pipeline , that's why i m exploring so that i can handle all issues whatever comes while going into production experiment .
sarthak
Wed, 03 May 2023 11:58:44 UTCso acc to you , clickhouse official documentation is enough right ?
vishal-signoz
Wed, 03 May 2023 11:59:16 UTCYes, you can follow this guide to connect to clickhouse:
sarthak
Wed, 03 May 2023 11:59:39 UTCokay
sarthak
Wed, 03 May 2023 12:02:57 UTCone more thing , i think i should discuss is that , i m thinking to use ec2 over elastic load balancing for replication but will be keeping vertically scalable rather than sharding after estimating the current scale i need to handle, will it be the good option in terms of best practices .
vishal-signoz
Wed, 03 May 2023 12:30:52 UTCyes it should be fine
sarthak
Wed, 03 May 2023 12:57:38 UTCyeah , actually i m deploying it into production as a experiment for application performance analysis
sarthak
Wed, 03 May 2023 12:58:18 UTCso i was exploring some open source apm , i found this suitable in terms of features , usecases coverage , technical architecture wise
sarthak
Wed, 03 May 2023 13:00:38 UTCso i started reading more about observebility , opentelem and signoz in detail so that i can setup completely infrastructure in scalable way so as to remove dependency on saas based apm's
sarthak
Wed, 03 May 2023 13:01:57 UTCnow i understood its working , deployed on stage and now trying to get proper grip on all components including best deployment stretagy
SigNoz Community
Indexed 1060 threads
Built with ClickHouse as datastore, SigNoz is an open-source APM to help you find issues in your deployed applications & solve them quickly | Knowledge Base powered by Struct.AI
View in App
sarthak
Wed, 03 May 2023 09:46:04 UTChi , is there a way or some configuration tag in clickhouse-storage.xml to write a rule to move data on s3 based on time period eg : 1 month and empty the clickhouse tables so that it can ingest data periodically without any space issues .