#support

Parsing JSON from Log Entry Using Regex

TLDR Hannes troubleshoots their regex processor for parsing a JSON log entry. They resolved the issue and shared their updated processor, which is of interest to nitya-signoz and Ashu for documentation.

Powered by Struct AI

1

1

Aug 11, 2023 (4 months ago)
Hannes
Photo of md5-5fd18cb6c0f914cf7a6415b80c54af65
Hannes
05:47 AM
hey hey, me again 🙂 Once again with logs and regex 🙂 🧵
05:47
Hannes
05:47 AM
I have a log entry that looks like this:
"1622 <190>1 2023-08-11T05:46:08.075562+00:00 d.7c9c4867-7cdb-4a9a-b4b8-cb877ed3084c app web.1 - - {\"severity_text\":\"INFO\",\"time\":\"2023-08-11T05:46:08.075+00:00\",\"message\":\"[9e330e7e-0e64-432e-a141-51d61aa1e667]   Parameters: {\\\"operationName\\\"=\\u003e\\\"GetListOptions\\\", \\\"variables\\\"=\\u003e{\\\"page\\\"=\\u003e1, \\\"perPage\\\"=\\u003e50, \\\"filter\\\"=\\u003e{\\\"term\\\"=\\u003e\\\"\\\", \\\"usageModes\\\"=\\u003e[]}}, \\\"query\\\"=\\u003e\\\"query GetListOptions($page: Int, $perPage: Int, $filter: ListFiltering) {\\\\n  lists(page: $page, perPage: $perPage, filter: $filter) {\\\\n    entries {\\\\n      ...ListOption\\\\n      __typename\\\\n    }\\\\n    __typename\\\\n  }\\\\n}\\\\n\\\\nfragment ListOption on List {\\\\n  id\\\\n  name\\\\n  usageMode\\\\n  creator {\\\\n    firstName\\\\n    lastName\\\\n    email\\\\n    __typename\\\\n  }\\\\n  project {\\\\n    title\\\\n    __typename\\\\n  }\\\\n  __typename\\\\n}\\\", \\\"graphql\\\"=\\u003e{\\\"operationName\\\"=\\u003e\\\"GetListOptions\\\", \\\"variables\\\"=\\u003e{\\\"page\\\"=\\u003e1, \\\"perPage\\\"=\\u003e50, \\\"filter\\\"=\\u003e{\\\"term\\\"=\\u003e\\\"\\\", \\\"usageModes\\\"=\\u003e[]}}, \\\"query\\\"=\\u003e\\\"query GetListOptions($page: Int, $perPage: Int, $filter: ListFiltering) {\\\\n  lists(page: $page, perPage: $perPage, filter: $filter) {\\\\n    entries {\\\\n      ...ListOption\\\\n      __typename\\\\n    }\\\\n    __typename\\\\n  }\\\\n}\\\\n\\\\nfragment ListOption on List {\\\\n  id\\\\n  name\\\\n  usageMode\\\\n  creator {\\\\n    firstName\\\\n    lastName\\\\n    email\\\\n    __typename\\\\n  }\\\\n  project {\\\\n    title\\\\n    __typename\\\\n  }\\\\n  __typename\\\\n}\\\"}}\",\"operation\":\"GraphqlController#execute\",\"span_id\":\"7b9a038f77454b7a\",\"trace_id\":\"60c75f5a12c3c9aba3c066473ed69800\"}"
05:47
Hannes
05:47 AM
I'd like to regex out the json
05:47
Hannes
05:47 AM
before parsing the json
05:47
Hannes
05:47 AM
I have the following processor set up
05:48
Hannes
05:48 AM
    operators:
      - type: regex_parser
        regex: '{(?P<body>.*)}'
        parse_from: body
        if: "body matches '^{.*}$'"
      - type: json_parser
        parse_to: attributes
        if: "body matches '^{.*}$'"
      - type: severity_parser
        if: "'severity_text' in attributes"
        parse_from: attributes.severity_text
        output: remove_severity_text
      - type: remove
        id: remove_severity_text
        if: "'severity_text' in attributes"
        field: attributes.severity_text
07:07
Hannes
07:07 AM
  logstransform/heroku_parser:
    operators:
      - type: regex_parser
        regex: '(?P<body>{.*})'
        parse_from: body
        parse_to: attributes
        if: "body matches '{.*}$'"
      - type: json_parser
        parse_from: attributes.body
        parse_to: attributes
        if: "body matches '{.*}$'"
      - type: remove
        id: remove_parsed_string
        if: "'body' in attributes"
        field: attributes.body
      - type: severity_parser
        if: "'severity_text' in attributes"
        parse_from: attributes.severity_text
        output: remove_severity_text
      - type: remove
        id: remove_severity_text
        if: "'severity_text' in attributes"
        field: attributes.severity_text

has resolved the issues, but now I'm not getting trace_id &amp; span_id 😕
07:57
Hannes
07:57 AM
Managed to fix it 🙂

1

07:57
Hannes
07:57 AM
  logstransform/heroku_parser:
    operators:
      - type: regex_parser
        regex: '(?P<body>{.*})'
        parse_from: body
        parse_to: attributes
        if: "body matches '{.*}$'"
      - type: json_parser
        parse_from: attributes.body
        parse_to: attributes
        if: "body matches '{.*}$'"
      - type: severity_parser
        if: "'severity_text' in attributes"
        parse_from: attributes.severity_text
        output: remove_severity_text
      - type: remove
        id: remove_severity_text
        if: "'severity_text' in attributes"
        field: attributes.severity_text
      - type: trace_parser
        if: '"trace_id" in attributes or "span_id" in attributes'
        trace_id:
          parse_from: attributes.trace_id
        span_id:
          parse_from: attributes.span_id
        output: remove_trace_id
      - id: remove_trace_id
        type: remove
        if: '"trace_id" in attributes'
        field: attributes.trace_id
        output: remove_span_id
      - id: remove_span_id
        type: remove
        if: '"span_id" in attributes'
        field: attributes.span_id
      - type: remove
        id: remove_parsed_string
        if: "'body' in attributes"
        field: attributes.body

For future reference to anyone possibly trying to do the same thing

1

nitya-signoz
Photo of md5-a52b9d6c34f193d9a1ff940024f36f77
nitya-signoz
11:06 AM
Great, let me know if you need more help . If this is a generic heroku parser that you have created would love to have a blog from you. Ashu do reach out if Hannes is interested.
Aug 16, 2023 (3 months ago)
Ashu
Photo of md5-c60cdaef3af26ce85163c4c204ce2bd3
Ashu
03:36 AM
Thanks nitya-signoz Will reach out to Hannes for documenting this.

SigNoz Community

Built with ClickHouse as datastore, SigNoz is an open-source APM to help you find issues in your deployed applications & solve them quickly | Knowledge Base powered by Struct.AI

Indexed 1023 threads (61% resolved)

Join Our Community