Reducing Log Output from ClickHouse with Helm Chart

Hello, I'm trying to lower the log output from ClickHouse, through the Helm chart you provide. I tried to use the help in the values.yaml to do so, but it seems not to be working. Here my values.yaml: (I know I duplicated default/log_queries=0 but it's on purpose to try to make it work).

    clickhouse:
      defaultProfiles:
        default/allow_experimental_window_functions: "1"
        default/allow_nondeterministic_mutations: "1"
        default/log_queries: "0"
        logger:
          level: warning
      enable: true
      profiles:
        default/log_queries: "0"
        logger:
          level: warning
##this block doesn't work, clickhouse get error about unknown settings:
##      logger/level: "warning"
##end of test
      resources:
        limits:
          cpu: 2000m
          memory: 4Gi
        requests:
          cpu: 50m
          memory: 4Gi

But then, the pod gets deployed, and still shows information logs such as:

clickhouse 2023.03.06 16:32:04.571158 [ 234 ] {4483843e-d883-4d01-af4d-2872f07b4545} <Information> executeQuery: Read 170 rows, 674.05 KiB in 0.015092185 sec., 11264 rows/sec., 43.62 MiB/sec.
clickhouse 2023.03.06 16:32:04.598816 [ 234 ] {bd514de5-4a53-4cbe-a959-d8ccb2a02ddf} <Information> executeQuery: Read 1190 rows, 2.22 MiB in 0.027129067 sec., 43864 rows/sec., 81.98 MiB/sec.
clickhouse 2023.03.06 16:32:04.774338 [ 235 ] {7dfafbe7-0766-49c4-a97f-cc583c6a5e54} <Information> executeQuery: Read 1169664 rows, 8.92 MiB in 0.15812709 sec., 7396986 rows/sec., 56.43 MiB/sec.
clickhouse 2023.03.06 16:32:05.360248 [ 224 ] {7556b233-89b2-4a20-8b11-b0c82da94f3d} <Information> executeQuery: Read 1169664 rows, 8.92 MiB in 0.129886739 sec., 9005261 rows/sec., 68.70 MiB/sec.
clickhouse 2023.03.06 16:32:05.456336 [ 218 ] {776364b3-20cb-41f4-8736-acc3b165b0b3} <Information> executeQuery: Read 452 rows, 11.19 KiB in 0.002209947 sec., 204529 rows/sec., 4.95 MiB/sec.
clickhouse 2023.03.06 16:32:05.485650 [ 228 ] {ddbec803-a941-4f7b-9a19-0bac720bfb82} <Information> executeQuery: Read 3618 rows, 88.59 KiB in 0.006745317 sec., 536372 rows/sec., 12.83 MiB/sec.
clickhouse 2023.03.06 16:32:05.564685 [ 240 ] {8bf6ccc3-6663-41f3-a444-54803c07f2d7} <Information> executeQuery: Read 3617 rows, 88.59 KiB in 0.01204406 sec., 300314 rows/sec., 7.18 MiB/sec.

and on the pod, I can check some config files:

bash-5.1$ cat etc/clickhouse-server/config.d/01-clickhouse-02-logger.xml 
<yandex>
    <logger>
        <!-- Possible levels: https://github.com/pocoproject/poco/blob/develop/Foundation/include/Poco/Logger.h#L105 -->
        <level>information</level>
        <log>/var/log/clickhouse-server/clickhouse-server.log</log>
        <errorlog>/var/log/clickhouse-server/clickhouse-server.err.log</errorlog>
        <size>1000M</size>
        <count>10</count>
        <!-- Default behavior is autodetection (log to console if not daemon mode and is tty) -->
        <console>true</console>
    </logger>
</yandex>

Could you tell me what I'm missing here please? Thanks a lot

Can you can try out the changes suggested by this article https://theorangeone.net/posts/calming-down-clickhouse/

This is what I'm trying to do actually. According to your chart https://github.com/SigNoz/charts/blob/main/charts/signoz/values.yaml#L225-L230 :

profiles: {}

  # -- Default user profile configuration for Clickhouse. !!! Please DO NOT override this !!!
  defaultProfiles:
    default/allow_experimental_window_functions: "1"
    default/allow_nondeterministic_mutations: "1"

so I added the following to the list:

default/log_queries: "0"

and when I restart the pod, I don't see anywhere the changes, so it seems what is changed in the values.yaml are not properly applied maybe.

I can see this file /etc/clickhouse-server/users.d/chop-generated-profiles.xml:

yandex>
    <profiles>
        <default>
            <allow_experimental_window_functions>1</allow_experimental_window_functions>
            <allow_nondeterministic_mutations>1</allow_nondeterministic_mutations>
        </default>
    </profiles>
</yandex>

and this /etc/clickhouse-server/users.d/02-clickhouse-default-profile.xml:

<yandex>
  <profiles>
    <default>
      <log_queries>1</log_queries>
      <connect_timeout_with_failover_ms>1000</connect_timeout_with_failover_ms>
      <distributed_aggregation_memory_efficient>1</distributed_aggregation_memory_efficient>
      <parallel_view_processing>1</parallel_view_processing>
    </default>
  </profiles>
</yandex>

without my settings of log_queries=0 anywhere 😕

Any heads up about this please? [SOLVED]

Prashant can you help here?

Hello Guillaume 👋

You can lower the log output of chi pods to warning by setting the following configuration in override-values.yaml :

clickhouse:
  clickhouseOperator:
    logger:
      level: warning

Followed by helm upgrade or helm install command.

Seems to be working this way, thanks!

That’s great 🎉