Log Queries Not Working After Updating to 0.18.1
TLDR Al reports log queries not working after updating to `0.18.1`, specifically filtering by `k8s_namespace_name`. Srikanth suggests Nityananda might have an answer, but no response yet.
Srikanth
Thu, 20 Apr 2023 01:05:09 UTCAre you sure there are cert-manager logs in the selected time range?
Al
Thu, 20 Apr 2023 16:07:37 UTCYes 100% certain, I extended the range to 1 week. I used cert-manager as an example, but filtering by k8s_namespace_name simply is not working for any namespace at all. As an example using a specific log entry with `id '2OZrlPONFe1F2RDNZoX2meXseev'` that has the following fields: `k8s_namespace_name ('cert-manager')` `k8s_node_name ('00000a')` `k8s_pod_name ('cert-manager-9997bf6c9-5t98x')` Filter `k8s_namespace_name ('cert-manager')` does *NOT* work. Filter `id IN ('2OZrlPONFe1F2RDNZoX2meXseev')` works! Filter `id IN ('2OZrlPONFe1F2RDNZoX2meXseev') AND k8s_namespace_name IN ('cert-manager')` does *NOT* work. Filter `k8s_node_name ('00000a')` works! Filter `k8s_node_name ('00000a') AND k8s_namespace_name IN ('cert-manager')` does *NOT* work. Filter `k8s_pod_name ('cert-manager-9997bf6c9-5t98x')` works! Filter `k8s_pod_name ('cert-manager-9997bf6c9-5t98x') AND k8s_namespace_name IN ('cert-manager')` does *NOT* work. Srikanth Thanks for the response.
Srikanth
Fri, 21 Apr 2023 08:14:23 UTCNityananda would be the best person to answer this.
Al
Mon, 24 Apr 2023 21:20:14 UTCHi Nityananda In the UI if I attempt to filter logs by `k8s_namespace_name IN ('lecreuset')` it results in *No logs lines found* I see the following in the query service logs: ```2023-04-24T19:48:02.058Z [35mDEBUG [0m clickhouseReader/reader.go:3612 SELECT toInt64(toUnixTimestamp(toStartOfInterval(toDateTime(timestamp/1000000000), INTERVAL 1 minute))*1000000000) as ts_start_interval, toFloat64(count()) as value FROM signoz_logs.distributed_logs WHERE (timestamp >= '1682362080929000000' AND timestamp <= '1682365680929000000' ) AND ( k8s_namespace_name IN ('lecreuset') ) GROUP BY ts_start_interval ORDER BY ts_start_interval 2023-04-24T19:48:02.060Z [34mINFO [0m app/server.go:277 /api/v1/logs/fields timeTaken: 18.868263ms 2023-04-24T19:48:02.060Z [35mDEBUG [0m clickhouseReader/reader.go:3468 SELECT timestamp, id, trace_id, span_id, trace_flags, severity_text, severity_number, body,CAST((attributes_string_key, attributes_string_value), 'Map(String, String)') as attributes_string,CAST((attributes_int64_key, attributes_int64_value), 'Map(String, Int64)') as attributes_int64,CAST((attributes_float64_key, attributes_float64_value), 'Map(String, Float64)') as attributes_float64,CAST((resources_string_key, resources_string_value), 'Map(String, String)') as resources_string from signoz_logs.distributed_logs where ( timestamp >= '1682362080929000000' and timestamp <= '1682365680929000000' ) and ( k8s_namespace_name IN ('lecreuset') ) order by timestamp desc limit 50 2023-04-24T19:48:02.072Z [34mINFO [0m app/server.go:277 /api/v1/logs/aggregate timeTaken: 31.461604ms 2023-04-24T19:48:02.076Z [34mINFO [0m app/server.go:277 /api/v1/logs timeTaken: 35.186776ms 2023-04-24T19:48:09.901Z [34mINFO [0m app/server.go:277 /api/v1/version timeTaken: 19.7µs 2023-04-24T19:48:09.901Z [34mINFO [0m app/server.go:277 /api/v1/version timeTaken: 47.901µs``` If I query clickhouse directly using `k8s_container_name IN ('bizApp')`it works and notice that the log entry returned contains *'k8s_namespace_name':'lecreuset'* ``` SELECT timestamp, id, trace_id, span_id, trace_flags, severity_text, severity_number, body, CAST((attributes_string_key, attributes_string_value), 'Map(String, String)') AS attributes_string, CAST((attributes_int64_key, attributes_int64_value), 'Map(String, Int64)') AS attributes_int64, CAST((attributes_float64_key, attributes_float64_value), 'Map(String, Float64)') AS attributes_float64, CAST((resources_string_key, resources_string_value), 'Map(String, String)') AS resources_string FROM signoz_logs.distributed_logs WHERE ((timestamp >= '1682363725243000000') AND (timestamp <= '1682367325243000000')) AND (k8s_container_name IN ('bizApp')) ORDER BY timestamp DESC LIMIT 1 Query id: 0a4b97c0-78a0-46aa-991b-b6f703fc2cb5 ┌───────────timestamp─┬─id──────────────────────────┬─trace_id─┬─span_id─┬─trace_flags─┬─severity_text─┬─severity_number─┬─body───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┬─attributes_string──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┬─attributes_int64─┬─attributes_float64─┬─resources_string───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ 1682367322861417200 │ 2OZrlPONFe1F2RDNZoX2meeVMcU │ │ │ 0 │ │ 0 │ <Source>EtwEvent</Source><Time>2023-04-24T20:15:19.000Z</Time><Provider idGuid="{E13C0D23-CCBC-4E12-931B-D9CC2EEE27E4}"/><DecodingSource>DecodingSourceXMLFile</DecodingSource><Execution ProcessID="21848" ThreadID="15420" /><Level>None</Level><Keyword>0x40000000</Keyword><EventID Qualifiers="82">82</EventID><EventData><ClrInstanceID>44</ClrInstanceID><Reserved1>0</Reserved1><Reserved2>0</Reserved2><FrameCount>41</FrameCount><Stack>0x7FFE7978127D</Stack><Stack>0x7FFE797842D8</Stack></EventData> │ {'time':'2023-04-24T20:15:22.8614172Z','logtag':'F','log_file_path':'\\var\\log\\pods\\lecreuset_bizApp-6b59cbf5f7-qzsjf_b8866b01-eb0b-400a-af43-27b132db6d45\\bizApp\\0.log','log_iostream':'stdout','env':'prod','region':'east-us'} │ {} │ {} │ {'k8s_namespace_name':'lecreuset','k8s_pod_name':'bizApp-6b59cbf5f7-qzsjf','k8s_container_restart_count':'0','k8s_pod_uid':'69563f93-a351-4a75-ad18-9994c5e652c5','k8s_container_name':'bizApp','host_name':'agentpool-000000','signoz_component':'otel-agent','k8s_cluster_name':'','k8s_pod_ip':'10.20.0.73','os_type':'windows','k8s_node_name':'agentpool-000000','k8s_pod_start_time':'2023-04-21 01:12:19 +0000 GMT'} │ └─────────────────────┴─────────────────────────────┴──────────┴─────────┴─────────────┴───────────────┴─────────────────┴────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┴────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┴──────────────────┴────────────────────┴────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ 1 row in set. Elapsed: 0.035 sec. Processed 8.01 thousand rows, 9.90 MB (227.97 thousand rows/s., 281.89 MB/s.)``` If I use the same query, but simply replace ~(k8s_container_name IN ('bizApp')~ with (*k8s_namespace_name IN ('lecreuset')* it does not work: ``` SELECT timestamp, id, trace_id, span_id, trace_flags, severity_text, severity_number, body, CAST((attributes_string_key, attributes_string_value), 'Map(String, String)') AS attributes_string, CAST((attributes_int64_key, attributes_int64_value), 'Map(String, Int64)') AS attributes_int64, CAST((attributes_float64_key, attributes_float64_value), 'Map(String, Float64)') AS attributes_float64, CAST((resources_string_key, resources_string_value), 'Map(String, String)') AS resources_string FROM signoz_logs.distributed_logs WHERE ((timestamp >= '1682363725243000000') AND (timestamp <= '1682367325243000000')) AND (k8s_namespace_name IN ('lecreuset')) ORDER BY timestamp DESC LIMIT 1 Query id: f9b2e396-c25c-43fa-9463-faeb855eb002 Ok. 0 rows in set. Elapsed: 0.018 sec.``` Srikanth Nityananda Should the query actually be `WHERE resources_string['k8s_namespace_name'] IN ('lecreuset')` instead of k8s_namespace_name IN ('lecreuset') ? I would really appreciate some support on this topic as we scope our customer resources by namespace and therefore filtering logs by namespace is essential for monitoring and troubleshooting deployments.
Al
Tue, 25 Apr 2023 16:28:02 UTCHi Nityananda and Srikanth sorry to push on this, but It seems that log filter is currently broken for resources_string, because the signoz_logs.distributed_logs query should be: ```WHERE resources_string['key'] IN ('value') ```
Al
Wed, 26 Apr 2023 21:16:33 UTCAnother observation, notice the attached screen cap. The log query builder labels the k8s fields as attributes, but they are part of the resources string. So should the query actually be: ```WHERE resources_string['k8s_namespace_name'] IN ('lecreuset') AND resources_string['k8s_container_name'] IN ('acme')``` Nityananda Srikanth I would really appreciate a response here, signoz logs are completely unusable for us at the moment, because we can't search logs for attributes or resource fields to troubleshoot.
Al
Fri, 28 Apr 2023 16:47:59 UTCCreated
SigNoz Community
Indexed 1061 threads
Built with ClickHouse as datastore, SigNoz is an open-source APM to help you find issues in your deployed applications & solve them quickly | Knowledge Base powered by Struct.AI
View in App
Al
Wed, 19 Apr 2023 21:41:57 UTCSince updating to `0.18.1` I have noticed that log queries are no longer working, with fields that are definitely present. For example `k8s_namespace_name IN ('name-that-exists')` does not work anymore. See attached captures.