Logs View Filter Issue and Troubleshooting Logging Configuration

TLDR Alex experienced filtering issues in logs view. After troubleshooting with Ankit, they discovered a bug in the otelcollector helm chart and intends to file a PR to fix it.

Photo of Alex
Alex
Fri, 24 Mar 2023 09:20:29 UTC

I'm a bit confused about the logs view. When I choose "Last 1 hour", "Last 5 minutes" in the logs view, what does that actually give me? On my installation, choosing either one gives me 3 day old logs. What does it actually filter on?

Photo of Ankit
Ankit
Fri, 24 Mar 2023 10:16:13 UTC

This is not much useful if you have continuous logs coming in. If you specify a filter with a few logs that came the last day only. Such dropdown will be useful.

Photo of Ankit
Ankit
Fri, 24 Mar 2023 10:16:30 UTC

Moreover, it is more useful when you want to view a custom time range

Photo of Alex
Alex
Fri, 24 Mar 2023 10:17:18 UTC

but what field does the filter actually apply to? Intuitively it would filter based on the timestamp.

Photo of Alex
Alex
Fri, 24 Mar 2023 10:17:41 UTC

but clearly it does not since I get logs with timestamps 3 days in the past even when I select 1 hour

Photo of Ankit
Ankit
Fri, 24 Mar 2023 10:17:45 UTC

yes..yes

Photo of Alex
Alex
Fri, 24 Mar 2023 10:18:55 UTC

I have a setup where I run vanilla otel-collector-contrib and export logs via OTLP to signoz

Photo of Ankit
Ankit
Fri, 24 Mar 2023 10:18:57 UTC

I didn't get you. You mean you can see logs that happened 3 days back when the selected filter is 1 hour?

Photo of Alex
Alex
Fri, 24 Mar 2023 10:19:06 UTC

yes

Photo of Alex
Alex
Fri, 24 Mar 2023 10:19:45 UTC

Photo of Ankit
Ankit
Fri, 24 Mar 2023 10:19:46 UTC

It's weird. Can you share the complete API request curl from the network tab?

Photo of Alex
Alex
Fri, 24 Mar 2023 10:19:57 UTC

sec

Photo of Alex
Alex
Fri, 24 Mar 2023 10:20:20 UTC

I pressed refresh, and now I get nothing again

Photo of Ankit
Ankit
Fri, 24 Mar 2023 10:21:09 UTC

> I pressed refresh, and now I get nothing again I think this is a bug. Can you show request payload when you see last 3 days data?

Photo of Alex
Alex
Fri, 24 Mar 2023 10:21:33 UTC

I will try but it's hard to reproduce

Photo of Alex
Alex
Fri, 24 Mar 2023 10:23:12 UTC

I am struggling to get my setup working. I cannot seem to get the logs delivered to signoz reliably, and I can't even debug the problem since there is nothing output to the collector logs (not a signoz problem, but still)

Photo of Ankit
Ankit
Fri, 24 Mar 2023 10:26:18 UTC

you can use logging exporter at signoz to verify logs are being received by signoz

Photo of Alex
Alex
Fri, 24 Mar 2023 10:31:44 UTC

tried that. I even added an operator `-type: stdout` to the `filelog` receiver, but I still get nothing. I tried this locally and it worked fine.

Photo of Alex
Alex
Fri, 24 Mar 2023 10:32:04 UTC

I am totally at my wit's end trying to get logging to work.

Photo of Alex
Alex
Fri, 24 Mar 2023 10:32:41 UTC

I know you don't have any obligation to help me, as this is not a signoz problem per se

Photo of Alex
Alex
Fri, 24 Mar 2023 10:32:51 UTC

(and even if it was)

Photo of Alex
Alex
Fri, 24 Mar 2023 11:09:34 UTC

ok, it looks like `filelog` receiver isn't seeing any logs

Photo of Alex
Alex
Fri, 24 Mar 2023 12:12:24 UTC

this was a bug in the otelcollector helm chart. I'm going to file PRs to fix that.

Photo of Alex
Alex
Fri, 24 Mar 2023 12:13:03 UTC

for posterity, the problem was that the include patterns started with `/var/log/` when they should've been prefixed with `/hostfs/var/log` as the host file system is mounted at `/hostfs`