Parsing JSON from Log Entry Using Regex
TLDR Hannes troubleshoots their regex processor for parsing a JSON log entry. They resolved the issue and shared their updated processor, which is of interest to nitya-signoz and Ashu for documentation.
1
1
Aug 11, 2023 (4 months ago)
Hannes
05:47 AMHannes
05:47 AM"1622 <190>1 2023-08-11T05:46:08.075562+00:00 d.7c9c4867-7cdb-4a9a-b4b8-cb877ed3084c app web.1 - - {\"severity_text\":\"INFO\",\"time\":\"2023-08-11T05:46:08.075+00:00\",\"message\":\"[9e330e7e-0e64-432e-a141-51d61aa1e667] Parameters: {\\\"operationName\\\"=\\u003e\\\"GetListOptions\\\", \\\"variables\\\"=\\u003e{\\\"page\\\"=\\u003e1, \\\"perPage\\\"=\\u003e50, \\\"filter\\\"=\\u003e{\\\"term\\\"=\\u003e\\\"\\\", \\\"usageModes\\\"=\\u003e[]}}, \\\"query\\\"=\\u003e\\\"query GetListOptions($page: Int, $perPage: Int, $filter: ListFiltering) {\\\\n lists(page: $page, perPage: $perPage, filter: $filter) {\\\\n entries {\\\\n ...ListOption\\\\n __typename\\\\n }\\\\n __typename\\\\n }\\\\n}\\\\n\\\\nfragment ListOption on List {\\\\n id\\\\n name\\\\n usageMode\\\\n creator {\\\\n firstName\\\\n lastName\\\\n email\\\\n __typename\\\\n }\\\\n project {\\\\n title\\\\n __typename\\\\n }\\\\n __typename\\\\n}\\\", \\\"graphql\\\"=\\u003e{\\\"operationName\\\"=\\u003e\\\"GetListOptions\\\", \\\"variables\\\"=\\u003e{\\\"page\\\"=\\u003e1, \\\"perPage\\\"=\\u003e50, \\\"filter\\\"=\\u003e{\\\"term\\\"=\\u003e\\\"\\\", \\\"usageModes\\\"=\\u003e[]}}, \\\"query\\\"=\\u003e\\\"query GetListOptions($page: Int, $perPage: Int, $filter: ListFiltering) {\\\\n lists(page: $page, perPage: $perPage, filter: $filter) {\\\\n entries {\\\\n ...ListOption\\\\n __typename\\\\n }\\\\n __typename\\\\n }\\\\n}\\\\n\\\\nfragment ListOption on List {\\\\n id\\\\n name\\\\n usageMode\\\\n creator {\\\\n firstName\\\\n lastName\\\\n email\\\\n __typename\\\\n }\\\\n project {\\\\n title\\\\n __typename\\\\n }\\\\n __typename\\\\n}\\\"}}\",\"operation\":\"GraphqlController#execute\",\"span_id\":\"7b9a038f77454b7a\",\"trace_id\":\"60c75f5a12c3c9aba3c066473ed69800\"}"Hannes
05:47 AMHannes
05:47 AMHannes
05:47 AMHannes
05:48 AM operators:
- type: regex_parser
regex: '{(?P<body>.*)}'
parse_from: body
if: "body matches '^{.*}$'"
- type: json_parser
parse_to: attributes
if: "body matches '^{.*}$'"
- type: severity_parser
if: "'severity_text' in attributes"
parse_from: attributes.severity_text
output: remove_severity_text
- type: remove
id: remove_severity_text
if: "'severity_text' in attributes"
field: attributes.severity_textHannes
07:07 AM logstransform/heroku_parser:
operators:
- type: regex_parser
regex: '(?P<body>{.*})'
parse_from: body
parse_to: attributes
if: "body matches '{.*}$'"
- type: json_parser
parse_from: attributes.body
parse_to: attributes
if: "body matches '{.*}$'"
- type: remove
id: remove_parsed_string
if: "'body' in attributes"
field: attributes.body
- type: severity_parser
if: "'severity_text' in attributes"
parse_from: attributes.severity_text
output: remove_severity_text
- type: remove
id: remove_severity_text
if: "'severity_text' in attributes"
field: attributes.severity_texthas resolved the issues, but now I'm not getting
trace_id & span_id ๐Hannes
07:57 AM1
Hannes
07:57 AM logstransform/heroku_parser:
operators:
- type: regex_parser
regex: '(?P<body>{.*})'
parse_from: body
parse_to: attributes
if: "body matches '{.*}$'"
- type: json_parser
parse_from: attributes.body
parse_to: attributes
if: "body matches '{.*}$'"
- type: severity_parser
if: "'severity_text' in attributes"
parse_from: attributes.severity_text
output: remove_severity_text
- type: remove
id: remove_severity_text
if: "'severity_text' in attributes"
field: attributes.severity_text
- type: trace_parser
if: '"trace_id" in attributes or "span_id" in attributes'
trace_id:
parse_from: attributes.trace_id
span_id:
parse_from: attributes.span_id
output: remove_trace_id
- id: remove_trace_id
type: remove
if: '"trace_id" in attributes'
field: attributes.trace_id
output: remove_span_id
- id: remove_span_id
type: remove
if: '"span_id" in attributes'
field: attributes.span_id
- type: remove
id: remove_parsed_string
if: "'body' in attributes"
field: attributes.bodyFor future reference to anyone possibly trying to do the same thing
1
nitya-signoz
11:06 AMAug 16, 2023 (3 months ago)
Ashu
03:36 AM
SigNoz Community
Indexed 1023 threads (61% resolved)
Similar Threads
Debugging JSON Parsing and Logging Issue with Otel-Collector
Hannes was struggling to debug log conversion to JSON and attribute parsing in otel-collector. nitya-signoz provided guidance and updates to the processors which eventually resolved the issue.
5
Parsing log entry and extracting fields with OTEL
Syed needed help parsing a log entry and extracting fields using OTEL. nitya-signoz provided guidance and the correct configurations, resolving the issue.
Resolving Regex Pattern Issues with Signoz
Noor is encountering issues with their regex pattern in Signoz and receiving errors. Srikanth and nitya-signoz provide suggestions and help troubleshoot, but the issue remains unresolved.
JSON Parsing Issue with Logs in Signoz
Androw had trouble parsing logs with JSON in Signoz. nitya-signoz suggested changes to config, which resolved the issue after Androw added the missing field 'logsCollection'.
Extracting Custom Fields as Attributes from Log in SigNoz
Harald wants to have custom fields from their body parsed as attributes in the SigNoz logger. They have tried implementing suggestions from nitya-signoz and Prashant, but the issue remains unsolved due to a potential race condition encountered while executing the code. They have provided a full example using Kind for further assistance.