SigNoz Log Format and JSON Parsing

Is there a well-defined spec for what format logs should be in to be collected by SigNoz? It seems SigNoz expects the JSON to contain specific keys.

This is the spec for a log body by OTEL https://opentelemetry.io/docs/reference/specification/logs/data-model/#log-and-event-record-definition.

But different operators are supported, which will allow you to convert your legacy logs to the above format.

https://signoz.io/docs/userguide/logs/#operators-for-parsing-and-manipulating-logs
https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/stanza/docs/types/pipeline.md

We are using structlog in Python. Do you find people typically use operators to convert, or just modify their loggers directly to match the format?

People mostly use operators to convert, but we are seeing new users who are directly using the SDK(mostly java) to send logs directly.

Since you are using python you can try the otel sdk for python, though support for logs is experimental as of now. https://github.com/open-telemetry/opentelemetry-python/tree/main/docs/examples/logs

Yes, we use JSON

https://opentelemetry.io/docs/reference/specification/protocol/file-exporter/#examples

These examples seem to use like severityText rather than SeverityText as well. Thats 3 potential variants…

It doesn’t matter, you will have to use the traceParser regardless, here is how you do it. https://github.com/SigNoz/logs-benchmark/blob/0b2451e6108d8fa5fdd5808c4e174bd52b9d55d3/signoz/signoz-client/otel-collector-config.yaml#L22

Hey nitya-signoz, I'm a coworker of Luke's. I wanted to additionally mention that we have deployed Signoz via kubernetes and we're automatically seeing all the pod logs. Which receiver are these logs ingested by? OTLP? I noticed the OTLP receiver doesn't support operators. https://signoz.io/docs/userguide/logs/#operators-for-parsing-and-manipulating-logs

> The receivers FluentForward and OTLP doesn’t have operators. But for parsing them we can use logprocessor. i would have expected this to work:

    processors:
      logstransform:
        operators:
          - type: json_parser
            id: my_new_body
            parse_from: attributes.body

however, after restarting the collector, I'm still not seeing "my_new_body" as a field. any ideas?

I confirmed by checking the losgs that the processor is enabled:

 signoz-otel-collector 2023-03-16T21:26:55.811Z    info    pipelines/pipelines.go:90    Processor is starting...    {"kind": "processor", "name": "logstransform", "pipeline": "logs"}                             │
│ signoz-otel-collector 2023-03-16T21:26:55.811Z    info    pipelines/pipelines.go:94    Processor started.    {"kind": "processor", "name": "logstransform", "pipeline": "logs"}

but i do see a failure, since not all logs contain a body or are valid json (lots of the pod logs are not).

│ signoz-otel-collector 2023-03-16T21:29:03.909Z    error    helper/transformer.go:110    Failed to process entry    {"kind": "processor", "name": "logstransform", "pipeline": "logs", "operator_id": "my_new_body ││ ", "operator_type": "json_parser", "error": {"description": "Entry is missing the expected parse_from field.", "suggestion": "Ensure that all incoming entries contain the parse_from field." ...

a couple of quesetion:
1. is this the right way to go about this? should i be using operators on a receiver instead of using a processor?
2. if this error is preventing me from running logstransform on any logs, is there a way to filter which logs this runs on?

The k8s logs are collected by the filelog/k8s receiver.

ooh i see. i didn't have the filelog/k8s receiver configured in any way -- it just works by default i suppose?

so here's an example log that i'm currently seeing in signoz. i don't see an attributes key.

{
  "timestamp": 1679085202378150700,
  "id": "2N9hfxnx4K6pMEslQ4UBGZL0EWB",
  "trace_id": "",
  "span_id": "",
  "trace_flags": 0,
  "severity_text": "",
  "severity_number": 0,
  "body": "{\"body\": {\"http\": {\"method\": \"GET\", \"request_id\": \"5514ff9e43d94cbca171a6751ccae7ca\", \"version\": \"1.1\", \"user_agent\": \"kube-probe/1.24+\"}, \"network\": {\"client\": {\"ip\": \"10.0.3.226\", \"port\": 33064}}, \"duration\": 427268, \"request_id\": \"5514ff9e43d94cbca171a6751ccae7ca\", \"logger\": \"api.access\", \"filename\": \"main.py\", \"func_name\": \"logging_middleware\", \"lineno\": 74, \"message\": \"10.0.3.226:33064 - \\\"GET /api/v1/healthz HTTP/1.1\\\" 200\"}, \"severityText\": \"info\", \"timestamp\": \"2023-03-17T20:33:22.377798Z\", \"traceId\": \"5514ff9e43d94cbca171a6751ccae7ca\"}",
  "resources_string": {
    "host_name": "<hostname>",
    "k8s_cluster_name": "",
    "k8s_container_name": "mlcore-web",
    "k8s_container_restart_count": "0",
    "k8s_namespace_name": "mlcore",
    "k8s_node_name": "<nodename>",
    "k8s_pod_ip": "<k8s_pod_ip>",
    "k8s_pod_name": "mlcore-web-6876b7c7b9-2cxxx",
    "k8s_pod_start_time": "2023-03-17 13:55:03 +0000 UTC",
    "k8s_pod_uid": "caad5d5e-7a16-471d-8a5f-0459b5aa90c4",
    "os_type": "linux",
    "signoz_component": "otel-agent"
  },
  "attributes_string": {
    "log_file_path": "/var/log/pods/mlcore_mlcore-web-6876b7c7b9-2cxxx_7144c554-5d97-4774-ae17-6c39ef19a518/mlcore-web/0.log",
    "log_iostream": "stderr",
    "logtag": "F",
    "time": "2023-03-17T20:33:22.378150623Z"
  },
  "attributes_int": {},
  "attributes_float": {}
}

and here's my relevant otel-collector-config:

    receivers:
      filelog/k8s:
        include:
          - /var/log/pods/*/*/*.log
        exclude:
          - /var/log/pods/kube-system_*/*/*.log
        operators:
          - type: json_parser
            id: body_parser
            parse_from: attributes.body
            parse_to: attributes.parsed_body

i also have the filelog/k8s set in the pipelines.logs.receivers:

      pipelines:
        logs:
          receivers: [otlp, filelog/k8s]

it seems my json_parser is not working at all. i've tried adding any combination of attributes.body or just body or body.body and with/without parse_to, but i can't seem to see any difference.

as an update, i realized this is in the otel-agent-config.yaml, not the otel-collector-config.yaml.

using operators there does seem to be working!