SigNoz Community

SigNoz Community

Command MenuMenu

K

#support

JSON Parsing Issue with Logs in Signoz

TLDR Androw had trouble parsing logs with JSON in Signoz. nitya-signoz suggested changes to config, which resolved the issue after Androw added the missing field 'logsCollection'.

Powered by Struct AI

1

43

4mo

May 15, 2023 (4 months ago)

Photo of md5-cfa4277d32743d86e1ba236b9482720f

Androw

12:49 PM

Hey team, trying to json parse our logs but cant seem to get it working

need to parse the body

{
  "timestamp": 1684154685443880400,
  "id": "2PpWeXe4qjcHXfMZxu0FL86HjVg",
  "trace_id": "",
  "span_id": "",
  "trace_flags": 0,
  "severity_text": "",
  "severity_number": 0,
  "body": "{\"level\":\"info\",\"ts\":1684154685.4437983,\"logger\":\"controllers.webhook-certs-updater\",\"msg\":\"ignoring webhook due to missing labels\",\"Webhookconfig\":\"/istiod-default-validator\",\"":\"webhook\"}",
  "resources_string": {
    "k8s_cluster_name": "",
    "k8s_container_name": "",
    "k8s_container_restart_count": "",
    "k8s_deployment_name": "",
    "k8s_namespace_name": "",
    "k8s_node_name": "",
    "k8s_pod_name": "",
    "k8s_pod_start_time": "",
    "k8s_pod_uid": ""
  },
  "attributes_string": {
    "log_file_path": "",
    "log_iostream": "stderr",
    "logtag": "F",
    "time": ""
  },
  "attributes_int": {},
  "attributes_float": {}
}

tried adding

                - type: json_parser
                  parse_from: attributes.body
                  if: '$matches "^{.*}$"'

to the k8s-infra -> presets -> operators
using helm+argocd to deploy signoz

12:57

Androw

12:57 PM

cc: Tal

Photo of md5-a52b9d6c34f193d9a1ff940024f36f77

nitya-signoz

03:22 PM

please share your entire pipeline, it will be easier to suggest a change.

Photo of md5-cfa4277d32743d86e1ba236b9482720f

Androw

03:26 PM

I push this file to a git repo that is connected with an argocd and argocd deploys it

03:26

Androw

03:26 PM

Is this enough info for you Nitya? if not please tell me what else i can provide to make it easier

Photo of md5-a52b9d6c34f193d9a1ff940024f36f77

nitya-signoz

03:28 PM

cool, jush change the value of attributes.body to body

so it will be

               - type: json_parser
                  parse_from: body
                  if: '$matches "^{.*}$"'

Photo of md5-cfa4277d32743d86e1ba236b9482720f

Androw

03:28 PM

ok on it

03:54

Androw

03:54 PM

still the same

nitya-signoz how can i verify that the operator that i added to the helm values is in fact added?
where should i look at?

Image 1 for still the same<br><br><span class="s-mention s-user"> <strong>nitya-signoz</strong></span> how can i verify that the operator that i added to the helm values is in fact added?<br>where should i look at?

Photo of md5-a52b9d6c34f193d9a1ff940024f36f77

nitya-signoz

03:56 PM

the parsed values will be present in the attributes map, can you share the entire screenshot as I cannot see the attributes key in the above screenshot.

Photo of md5-cfa4277d32743d86e1ba236b9482720f

Androw

03:57 PM

{
  "timestamp": 1684165920042734300,
  "id": "2PpsKtQJSzrNcUu9JxN7j3cgICK",
  "trace_id": "",
  "span_id": "",
  "trace_flags": 0,
  "severity_text": "",
  "severity_number": 0,
  "body": "{\"level\":\"info\",\"msg\":\"Done LoadModelsTask \",\"time\":\"2023-05-15T15:52:00Z\"}",
  "resources_string": {
    "k8s_cluster_name": "",
    "k8s_container_name": "model-analysis-service",
    "k8s_container_restart_count": "3",
    "k8s_deployment_name": "mod1-model-analysis-service",
    "k8s_namespace_name": "imubit-apps",
    "k8s_node_name": "",
    "k8s_pod_name": "",
    "k8s_pod_start_time": "",
    "k8s_pod_uid": ""
  },
  "attributes_string": {
    "log_file_path": "/var/log/pods/imubit-apps_mod1-model-analysis-service-7b8895f9f6-lsdjs_2ec5f13f-85d1-456f-8ee0-2fd4e7eae052/model-analysis-service/3.log",
    "log_iostream": "stdout",
    "logtag": "F",
    "time": "2023-05-15T15:52:00.042734442Z"
  },
  "attributes_int": {},
  "attributes_float": {}
}

Photo of md5-a52b9d6c34f193d9a1ff940024f36f77

nitya-signoz

03:59 PM

The if condition is wrong 'body matches "^{.*}$"'

03:59

nitya-signoz

03:59 PM

This is the correct one ^

Photo of md5-cfa4277d32743d86e1ba236b9482720f

Androw

03:59 PM

ok

03:59

Androw

03:59 PM

                - type: json_parser
                  parse_from: body
                  if: 'body matches "^{.*}$"'

so this?

Photo of md5-a52b9d6c34f193d9a1ff940024f36f77

nitya-signoz

03:59 PM

right

Photo of md5-cfa4277d32743d86e1ba236b9482720f

Androw

03:59 PM

thank u

Photo of md5-a52b9d6c34f193d9a1ff940024f36f77

nitya-signoz

04:00 PM

let me know how it goes.

Photo of md5-cfa4277d32743d86e1ba236b9482720f

Androw

04:21 PM

same thing 😞

04:21

Androw

04:21 PM

 "attributes_string": {
    "log_file_path": "/var/log/pods/imubit-apps_mod1-model-analysis-service-7b8895f9f6-lsdjs_2ec5f13f-85d1-456f-8ee0-2fd4e7eae052/model-analysis-service/3.log",
    "log_iostream": "stdout",
    "logtag": "F",
    "time": "2023-05-15T16:21:00.109518425Z"
  },

04:22

Androw

04:22 PM

Image 1 for

Photo of md5-a52b9d6c34f193d9a1ff940024f36f77

nitya-signoz

04:22 PM

any logs on the otel collector ?

Photo of md5-cfa4277d32743d86e1ba236b9482720f

Androw

04:26 PM

u want me to check the pod logs for something specific? or in signoz ui and see if its being parsed?

04:28

Androw

04:28 PM

otel collector logs

May 16, 2023 (4 months ago)

Photo of md5-a52b9d6c34f193d9a1ff940024f36f77

nitya-signoz

07:50 AM

I have tested this locally and it seems to be working, can you check if the actual configmap of otel-collector is updated with the pipeline changes ?

Photo of md5-cfa4277d32743d86e1ba236b9482720f

Androw

08:52 AM

how would it look if its correctly added/updated?

08:52

Androw

08:52 AM

can u share your configmap of the otel-collector?

Photo of md5-a52b9d6c34f193d9a1ff940024f36f77

nitya-signoz

08:55 AM

I have used docker to replicate it and test. But I would want you to check the configmap in k8s

Photo of md5-cfa4277d32743d86e1ba236b9482720f

Androw

08:58 AM

Photo of md5-a52b9d6c34f193d9a1ff940024f36f77

nitya-signoz

08:59 AM

There should be one configmap for your k8s infra agent.

Photo of md5-cfa4277d32743d86e1ba236b9482720f

Androw

08:59 AM

from signoz-otel-collector

08:59

Androw

08:59 AM

ok checking

09:00

Androw

09:00 AM

ok i see all the other operators but i dont see the one i added there

09:00

Androw

09:00 AM

09:00

Androw

09:00 AM

maybe im setting it wrong in the helm values?

Photo of md5-a52b9d6c34f193d9a1ff940024f36f77

nitya-signoz

09:00 AM

Yeah thats the issue, your argocd is not able to push the changes

09:01

nitya-signoz

09:01 AM

I haven’t worked with argocd much, but in the end this file should be updated with the operators you want to add

Photo of md5-cfa4277d32743d86e1ba236b9482720f

Androw

09:07 AM

well we can put argo to the side this is probably a helm "issue"
k8s-infra -> presets -> operators
is this not the correct path to add the json_parser?

Photo of md5-a52b9d6c34f193d9a1ff940024f36f77

nitya-signoz

09:08 AM

https://github.com/SigNoz/charts/blob/main/charts/k8s-infra/values.yaml the k8s infra values.yaml

09:09

nitya-signoz

09:09 AM

You can also create an override file https://signoz.io/docs/userguide/collect_kubernetes_pod_logs/#steps-to-filterexclude-logs-collection

Photo of md5-cfa4277d32743d86e1ba236b9482720f

Androw

09:38 AM

https://github.com/SigNoz/charts/blob/main/charts/signoz/values.yaml#L1998
it says here it is ignored

09:38

Androw

09:38 AM

prob why its not updating

Photo of md5-a52b9d6c34f193d9a1ff940024f36f77

nitya-signoz

09:39 AM

Prashant might be able to help with this.

Photo of md5-cfa4277d32743d86e1ba236b9482720f

Androw

10:25 AM

🤦
i was missing a field logsCollection
k8s-infra -> presets -> logsCollection -> operators

sorry and thanks for the help

1

SigNoz Community

Built with ClickHouse as datastore, SigNoz is an open-source APM to help you find issues in your deployed applications & solve them quickly | Knowledge Base powered by Struct.AI

Indexed 825 threads (61% resolved)

Join Our Community Manage subscription

Similar Threads

Extracting Custom Fields as Attributes from Log in SigNoz

Harald wants to have custom fields from their body parsed as attributes in the SigNoz logger. They have tried implementing suggestions from nitya-signoz and Prashant, but the issue remains unsolved due to a potential race condition encountered while executing the code. They have provided a full example using Kind for further assistance.

SigNoz Log Format and JSON Parsing

Luke asked about the log format for SigNoz and using operators to convert loggers. nitya-signoz shared resources about specifications and recommended using operators. Travis had issues with JSON parsing but resolved them by updating `otel-agent-config.yaml`.

Parsing log entry and extracting fields with OTEL

Syed needed help parsing a log entry and extracting fields using OTEL. nitya-signoz provided guidance and the correct configurations, resolving the issue.

Adding Labels to Logs in SigNoz Using Pipelines

Tarek was trying to add labels from a JSON body to SigNoz logs. nitya-signoz and Shivanshu helped by providing pipeline examples and configurations. nitya-signoz's solution eventually resolved the issue.

Debugging JSON Parsing and Logging Issue with Otel-Collector

Hannes was struggling to debug log conversion to JSON and attribute parsing in otel-collector. nitya-signoz provided guidance and updates to the processors which eventually resolved the issue.

Powered By