Troubleshooting Signoz Deployment for Log Collection on OCI k8s Cluster
TLDR sudhanshu faces issues with log collection despite successful application performance monitoring after deploying Signoz on OCI k8s cluster. Prashant suggests updating paths used for log collection and verifying container logs on the nodes. Log files' paths seem correct, but issues persist. Further solutions are needed.
Prashant
Wed, 11 Oct 2023 20:50:29 UTCHi sudhanshu :wave:
OCI does not use docker as CRI hence the issue with log collection. It uses CRI-O.
> log_dir="/var/log/crio/pods" The default log directory where all logs will go unless directly specified by the kubelet.
Ref:
sudhanshu
Thu, 12 Oct 2023 03:53:23 UTCTried but still did not work. I even tried with path /var/log/containers/*.log
sudhanshu
Thu, 12 Oct 2023 03:54:15 UTCif fluentbit and otelcollector running on the same machine and tailing the same path. Does it create any issue?
sudhanshu
Thu, 12 Oct 2023 04:54:30 UTCI tried to send logs using python sdk to otel collector and it is working fine means issues most likely to logs path or k8s logging config
sudhanshu
Thu, 12 Oct 2023 07:03:09 UTCPrashant Any idea in debugging?
sudhanshu
Thu, 12 Oct 2023 07:18:41 UTC<#C01HWQ1R0BC|support>
sudhanshu
Thu, 12 Oct 2023 07:53:25 UTCIs compulsory to deploy signoz in platform namespace Prashant
sudhanshu
Thu, 12 Oct 2023 07:53:28 UTC?
Prashant
Thu, 12 Oct 2023 08:54:27 UTCnot mandatory, you can deploy in any namespace.
sudhanshu
Thu, 12 Oct 2023 08:54:43 UTCAny other way to debug
Prashant
Thu, 12 Oct 2023 08:54:59 UTC> Any idea in debugging? can you try to ssh into the nodes and find the path for container logs?
sudhanshu
Thu, 12 Oct 2023 08:55:06 UTCI tried
sudhanshu
Thu, 12 Oct 2023 08:55:25 UTCit is /var/log/pods/
sudhanshu
Thu, 12 Oct 2023 08:56:10 UTCPath wise ok
sudhanshu
Thu, 12 Oct 2023 08:56:30 UTCI checked
Prashant
Thu, 12 Oct 2023 09:00:23 UTCcan you share complete path of the container log file?
sudhanshu
Thu, 12 Oct 2023 09:07:45 UTC```/var/log/pods/poc-dev_game-centre-9c6668c99-rrksr_b65c2fe1-d866-42ed-9251-2400eef9a13f/game-centre/559.log```
sudhanshu
Thu, 12 Oct 2023 09:07:50 UTC`/var/log/pods/default_game-centre-77c96bc74-4vwsj_210c748d-8b69-462d-acc6-fd8cbdced8aa/game-centre/0.log`
sudhanshu
Thu, 12 Oct 2023 09:08:02 UTClike these two pods path
sudhanshu
Thu, 12 Oct 2023 09:42:21 UTCI used signoz multiple times for logging. It is first time when I am facing this
sudhanshu
Thu, 12 Oct 2023 09:42:39 UTCAPM and monitoring working as expected.
SigNoz Community
Indexed 1061 threads
Built with ClickHouse as datastore, SigNoz is an open-source APM to help you find issues in your deployed applications & solve them quickly | Knowledge Base powered by Struct.AI
View in App
sudhanshu
Wed, 11 Oct 2023 11:56:08 UTCI deployed signoz in oci k8s cluster . APM working fine but logs are not coming . Below is my k8s-infra yaml , ```k8s-infra: # -- Whether to enable K8s infra monitoring enabled: true # -- Endpoint/IP Address of the SigNoz or any other OpenTelemetry backend. # Set it to `` for SigNoz SaaS.
#
# If set to null and the chart is installed as dependency, it will attempt
# to autogenerate the endpoint of SigNoz OtelCollector.
otelCollectorEndpoint: null
# -- Whether the OTLP endpoint is insecure.
# Set this to false, in case of secure OTLP endpoint.
otelInsecure: true
# -- Whether to skip verifying the certificate.
insecureSkipVerify: false
# -- API key of SigNoz SaaS
signozApiKey: ""
# OTLP receivers TLS
otelTlsSecrets:
# -- Whether to enable OpenTelemetry OTLP secrets
enabled: false
# -- Path for the secrets volume
path: /secrets
# -- Name of the existing secret with TLS certificate, key and CA to be used.
# Files in the secret must be named `cert.pem`, `key.pem` and `ca.pem`.
existingSecretName:
# -- TLS certificate to be included in the secret
certificate: |
<INCLUDE_CERTIFICATE_HERE>
# -- TLS private key to be included in the secret
key: |
<INCLUDE_PRIVATE_KEY_HERE>
# -- TLS certificate authority (CA) certificate to be included in the secret
ca: ""
# -- Which namespace to install k8s-infra components.
# By default installed to the namespace same as the chart.
namespace: ""
# -- Presets to easily set up OtelCollector configurations.
presets:
loggingExporter:
enabled: false
# Verbosity of the logging export: basic, normal, detailed
verbosity: basic
# Number of messages initially logged each second
samplingInitial: 2
# Sampling rate after the initial messages are logged
samplingThereafter: 500
logsCollection:
enabled: true
include:
- /var/log/pods/*/*/*.log
startAt: beginning
includeFilePath: true
includeFileName: false
blacklist:
enabled: true
signozLogs: false
namespaces:
- kube-system
pods:
- hotrod
- locust
containers: []
additionalExclude: []```