SigNoz Community

SigNoz Community

#support

Issues with Transmitting Logs from Logstash to Signoz

TLDR Krishnadas struggled to transmit logs from Logstash to Signoz. After a rich dialogue with Prashant and nitya-signoz, the issue was resolved when Krishnadas removed the existing docker container logs receiver and added the filelog receiver for their logs folder.

Powered by Struct AI

17

4mo

Jul 26, 2023 (4 months ago)

Photo of md5-12063f62d3a94a65cd24766e394864e6

Krishnadas

08:16 AM

I am trying to get logs from logstash to Signoz. Logstash is installed directly in the ubuntu VM and signoz is installed using docker in the same VM. But I am unable to get any logs in Signoz via open telemetry collector.

This is my logstash config

input {
file {
path => "/home/iislogs.log"
start_position => "beginning"
}
}
output {
tcp {
port => 2255
host => "localhost"
}
}

Signoz setup is done as per this doc https://signoz.io/docs/userguide/logstash_to_signoz/

What must I be doing wrong here ? Please help

Photo of md5-1899629483c7ab1dccfbee6cc2f637b9

Prashant

08:40 AM

Did you restart otel-collector properly after config change and exposed ports?

Photo of md5-12063f62d3a94a65cd24766e394864e6

Krishnadas

08:42 AM

Yes. I had restarted logstash and also docker compose down and up all containers

08:46

Krishnadas

08:46 AM

Are we supposed to see any logs in otel collector when it is receiving data from a receiver ? Because I am not seeing any logs for that yet

09:18

Krishnadas

09:18 AM

Prashant I have set logstash to send data to localhost:2255 and set signoz to receive data at 0.0.0.0:2255.. Is this correct ? I can see in logstash logs that it is opening connection correctly. But no movement in signoz otel collector logs

09:56

Krishnadas

09:56 AM

Prashant I have also tried to test logs from mounted volume as per https://signoz.io/docs/userguide/collect_logs_from_file/

This is not showing up in Signoz UI either.

I can see the following log in otel collector container. But nothing more.

2023-07-26T09:48:23.883Z info fileconsumer/file.go:192 Started watching file {"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer", "path": "/tmp/python.log"}

10:57

Krishnadas

10:57 AM

Curiously enough the docker container logs are working fine and getting exported to clickhouse logs and visualized in signoz UI. Both logstash and filelog receivers that I added arent working

01:15

Krishnadas

01:15 PM

Prashant, nitya-signoz Any idea on this please ?

Photo of md5-a52b9d6c34f193d9a1ff940024f36f77

nitya-signoz

03:29 PM

• For file log receiver, are new logs getting written to the file, if not then you will have to use the start_at : beginning configuration.https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/filelogreceiver
• For logstash can you share what are the logs of logstash ? also have you exposed the port of otel-collector ?

Photo of md5-12063f62d3a94a65cd24766e394864e6

Krishnadas

03:50 PM

Thanks for the reply nitya-signoz.

1. Yes. New logs are getting written. I am using the same tutorial of python logging as given in Signoz docs
2. Yes. I have exposed the 2255 port on otel collector. Logstash log snippet is attached.

Photo of md5-a52b9d6c34f193d9a1ff940024f36f77

nitya-signoz

03:52 PM

• Can you check if you have mounted your file properly ?
• Also please share your otel-collector config

Photo of md5-12063f62d3a94a65cd24766e394864e6

Krishnadas

03:57 PM

1. Yes I can see the file after exec ing into the docker container.
2. Attached.

Photo of md5-a52b9d6c34f193d9a1ff940024f36f77

nitya-signoz

03:59 PM

This seems correct, what environment are you running this ? linux ?

Photo of md5-12063f62d3a94a65cd24766e394864e6

Krishnadas

04:00 PM

Yes. In Ubuntu 20.04.6 LTS (Focal Fossa)

04:48

Krishnadas

04:48 PM

Any idea nitya-signoz ? I would have thought the filelog receiver would have been fairly simple to work with :(

Jul 27, 2023 (4 months ago)

Photo of md5-1899629483c7ab1dccfbee6cc2f637b9

Prashant

01:28 AM

Krishnadas updating content of already mounted otelcol-config file is not often updated in the restarted container.

hard remove of the otel-collector container is required for that.

docker stop <otelcollector-container>
docker rm <otelcollector-container>


docker-compose -f ...

OR

./install.sh

Photo of md5-12063f62d3a94a65cd24766e394864e6

Krishnadas

01:40 AM

I was always doing docker-compose down and then docker-compose up -d.

Funnily enough, I removed the existing docker container logs receiver from the otel-collector-config and just added the filelog receiver for my logs folder and it started working. I will play around a bit and confirm what changed

SigNoz Community

Built with ClickHouse as datastore, SigNoz is an open-source APM to help you find issues in your deployed applications & solve them quickly | Knowledge Base powered by Struct.AI

Indexed 1023 threads (61% resolved)

Join Our Community

Similar Threads

Removing Default Logs and Collecting Application Logs

Srikanth wanted to remove default logs and collect their application logs. nitya-signoz provided two solutions, and Srikanth chose the second one, which involves editing the ote-collector-config.yaml file.

2

Logs View Filter Issue and Troubleshooting Logging Configuration

Alex experienced filtering issues in logs view. After troubleshooting with Ankit, they discovered a bug in the otelcollector helm chart and intends to file a PR to fix it.

1

Troubleshooting and Adding Log Files to SigNoz POC

Noor has requested help with incorporating log files into their SigNoz POC. In collaboration with vishal-signoz and nitya-signoz, they managed to successfully setup and resolve their issues.

Log Monitoring Connection Error in Signoz

Utkarsh encountered a connection error while adding log monitoring using Signoz. nitya-signoz identified the incorrect otlp exporter port and suggested using 4317.

1

Configuring OTel Collector to Export Data to SigNoz

James asked how to configure the otel collector to send data to SigNoz. Ankit suggested using the otlp exporter in the collector configuration. James found that SigNoz uses its own otel collector and adjusted the port bindings accordingly.

1

Powered By