Implementing SigNoz Self-Hosted on K8s cluster
TLDR Nicolas was having issues setting up SigNoz on a K8s cluster and forwarding logs through OTel Agent. They were advised by Prashant to use crio log path instead of Docker in Oracle Cloud's managed K8s offering.
Nov 17, 2023 (1 week ago)
Nicolas
01:34 AMfilelog receiver on the OTel Agent to forward my pod logs to SigNoz, but none of my logs are making it through.Some things that work:
• Traces: the services I've auto-instrumented are getting their traces into SigNoz
• Manually instrumented logs (via Python OTel SDK): just for debugging, I used the SDK to configure my Python
logging with an OTel handler, which successfully forwarded logs to SigNozHowever, I specifically want to leverage an OTel Agent (via K8s Daemonset) to capture logs, since that should provide the highest coverage. This approach is not yielding any success, even though it's claimed as the default out-of-the-box behavior :thinking_face: Any help would be much appreciated!
Nicolas
01:40 AMk8s-infra preset to try and hone in on the problem. I've hard-coded an expected log file path in the include attribute, and when I tail the logs of the otel agent pod, I do see it detect the log file I care about:Nicolas
01:40 AM2023-11-17T01:18:40.789Z info fileconsumer/file.go:194 Started watching file from end. To read preexisting logs, configure the argument 'start_at' to 'beginning' {"kind": "receiver", "name": "filelog/k8s", "data_type": "logs", "component": "fileconsumer", "path": "/var/log/pods/solver_service-00030-deployment-7bc9bb776d-5nfwl_87c60b29-9487-44c3-a209-767ad8410eb9/retrieval-pod/0.log"}Nicolas
01:40 AMNicolas
01:40 AM
nitya-signoz
05:49 AMNicolas
05:40 PMNicolas
10:41 PMNov 18, 2023 (1 week ago)
nitya-signoz
04:15 AMNov 21, 2023 (1 week ago)
Nicolas
11:48 PMNov 26, 2023 (3 days ago)
Prashant
09:52 AMpresets.logsCollection
SigNoz Community
Indexed 1023 threads (61% resolved)
Similar Threads
Troubleshooting Signoz Deployment for Log Collection on OCI k8s Cluster
sudhanshu faces issues with log collection despite successful application performance monitoring after deploying Signoz on OCI k8s cluster. Prashant suggests updating paths used for log collection and verifying container logs on the nodes. Log files' paths seem correct, but issues persist. Further solutions are needed.
Troubleshooting Signoz Log Collection in OCI
Harsh had problems collecting logs with Signoz after moving to OCI. Prashant helped pinpoint the problem to OCI's CRI and provided a solution using a specific crio logs path.
Troubleshooting Signoz Deployment Issues
Juan is having issues running signoz data, only logs are working. The issues were identified by Srikanth as non-working network and harmless table issues in a k8s cluster. However, the issues are not yet resolved.
Issues with Installing Signoz on Standalone VM with Two Additional Disks
vvpreo is having difficulty installing Signoz on a standalone VM due to an issue with OTEL collector not starting because of migration problems. Ankit and vishal-signoz proposed various solutions, but the issue remained unresolved. They planned a screen-sharing session to resolve it.
Configuring Logs in Signoz and Clickhouse
Varaprasad queries on pushing logs to ClickHouse and analyzing from SigNoz. Srikanth advises using `signoz-otel-collector` and shares a guide. Varaprasad explains their plan of action. Ankit confirms vector parsing and sharding setup in ClickHouse.