Issues with Log Retention Configuration on Self-Hosted SigNoz Installation

TLDR Cameron was having trouble configuring log retention for a self-hosted SigNoz installation. Prashant helped troubleshoot, asking for specific logs, but issue is unresolved.

Photo of Cameron
Cameron
Wed, 13 Sep 2023 18:20:35 UTC

Hello. I'm trying to configure log retention on a self-hosted SigNoz installation. I'm following this faq for retention, but nothing is pushing to the bucket on gcp I have setup for this. Where would I need to look for errors to see if something is not correct or troubleshooting? Thank you.

Photo of Prashant
Prashant
Wed, 13 Sep 2023 19:41:00 UTC

Hi Cameron :wave: Can you share your `clickhouse.coldStorage` section of your override config with redacted sensitive informations? In case of Docker, `storage_configuration` section.

Photo of Cameron
Cameron
Wed, 13 Sep 2023 20:38:09 UTC

Hi Prashant. Here is a redacted config. We are using a google service account with an ssh key.

Photo of Cameron
Cameron
Wed, 13 Sep 2023 20:39:23 UTC

Photo of Cameron
Cameron
Wed, 13 Sep 2023 20:40:15 UTC

This is the clickhouse-storage.xml from docker

Photo of Prashant
Prashant
Wed, 13 Sep 2023 21:24:24 UTC

> <endpoint></endpoint> You will need to include a folder at the end for example `/data/` . Is that already set?

Photo of Prashant
Prashant
Wed, 13 Sep 2023 21:25:07 UTC

apart from that, everything looks good. can you check and share ClickHouse ~pod~ container logs related to cold storage (S3-protocol)?

Photo of Cameron
Cameron
Wed, 13 Sep 2023 23:30:57 UTC

Yes, folder name is data/

Photo of Cameron
Cameron
Wed, 13 Sep 2023 23:31:15 UTC

where would I find the clickHouse pod logs?

Photo of Prashant
Prashant
Thu, 14 Sep 2023 00:08:25 UTC

Cameron In case of Docker, that would be clickhouse container logs.

Photo of Cameron
Cameron
Mon, 18 Sep 2023 22:10:49 UTC

When going through the clickhouse logs what should I be looking for? Thanks.

Photo of Prashant
Prashant
Tue, 19 Sep 2023 05:06:58 UTC

any error/warning level logs related to S3 or including `` in lgos

Photo of Cameron
Cameron
Wed, 20 Sep 2023 15:48:10 UTC

Hi. We are not able to see logs related to s3 or googleapis on clickhouse container logs. What else do you suggest?

Photo of Prashant
Prashant
Wed, 20 Sep 2023 18:04:58 UTC

Cameron are you seeing any error or warnings in the clickhouse pod?