Resolving Regex Pattern Issues with Signoz
TLDR Noor is encountering issues with their regex pattern in Signoz and receiving errors. Srikanth and nitya-signoz provide suggestions and help troubleshoot, but the issue remains unresolved.
Srikanth
Fri, 06 Oct 2023 22:06:11 UTCYou can put a condition to avoid this error, `if 'key' in attributes` or something like that
Noor
Sat, 07 Oct 2023 01:31:15 UTCSrikanth one more question this is my operator filelog: include: [ /tmp/startServer.log ] start_at: beginning operators: - type: regex_parser regex: (\D+)(\S+)([0-9]).([0-9])+(\S+)(\D)+(\W+)(\S{8})(\D+\D+)([A-Z]).([0-9]\w+:\D+) timestamp: layout: '%Y-%m-%d,%H:%M:%S %z' parse_from: attributes.timestamp_field - type: remove field: attributes.timestamp when restart the collector I get the following error 2023/10/07 01:28:57 application run finished with error: failed to build pipelines: failed to create "filelog" receiver for data type "logs": no named capture groups in regex pattern. This regex parser tested ok in regex101 is this parser allowed in signoz
Noor
Mon, 09 Oct 2023 16:24:16 UTCwhat is best regex pattern is allowed by signoz to run without giving the following error Error: failed to build pipelines: failed to create "filelog" receiver for data type "logs": no named capture groups in regex pattern 2023/10/09 16:23:51 application run finished with error: failed to build pipelines: failed to create "filelog" receiver for data type "logs": no named capture groups in regex pattern
Srikanth
Tue, 10 Oct 2023 00:11:36 UTCnitya-signoz might be able to help
Noor
Tue, 10 Oct 2023 03:03:29 UTCHello Nitya I am currently using this - type: regex_parser regex: (\S+\d+\D+)(\S+\d+\d+)([0-9]\D+\W+\S+)+([a-z]+\W+\S+\D+) timestamp: parse_from: attributes.timestamp_field layout: '%Y-%m-%d,%H:%M:%S %z' - type: move from: attributes["body"] to: body - type: remove field: attributes.timestamp
Noor
Tue, 10 Oct 2023 03:04:07 UTCI am getting this error filelog" receiver for data type "logs": no named capture groups in regex pattern
Noor
Tue, 10 Oct 2023 03:04:34 UTCPer Signoz all parse are test in regex101
nitya-signoz
Tue, 10 Oct 2023 05:31:37 UTCSince you are extracting `timestamp_field` and `body` form the regex those named group should be present in your regex `(\S+\d+\D+)(\S+\d+\d+)([0-9]\D+\W+\S+)+([a-z]+\W+\S+\D+)`
Noor
Tue, 10 Oct 2023 16:26:00 UTCJust wondering getting closer I made few changes I got this error 2023/10/10 16:20:41 application run finished with error: failed to build pipelines: failed to create "filelog" receiver for data type "logs": compiling regex: error parsing regexp: invalid or unsupported Perl syntax: `(?<`
nitya-signoz
Thu, 12 Oct 2023 05:07:50 UTCCan you choose golang as the language in regex101 and test your regex out ?
Noor
Thu, 12 Oct 2023 12:00:55 UTCI currently have it setup this way filelog: include: [ /tmp/startServer.log ] start_at: beginning operators: - type: regex_parser regex: (\S+\d+\D+)(\S+\d+\d+)([0-9]\D+\W+\S+)+([a-z]+\W+\S+\D+)(?P<body>.*)|(?<timestamp>\S+)| parseTimestamp(field=timestamp) timestamp: parse_from: attributes.timestamp_field layout: '%Y-%m-%d,%H:%M:%S %z' - type: move from: attributes["body"] to: body - type: remove field: attributes.timestamp tcplog/docker: listen_address: "0.0.0.0:2255" operators: - type: regex_parser regex: '^\[(?P<time>\d+\/\d+\/\d+ \d{2}:\d+:\d{2}:\d{3} UTC)\](?P<body>.*)' timestamp: parse_from: attributes.timestamp layout: '%Y-%m-%dT%H:%M:%S.%LZ' - type: move from: attributes["body"] to: body - type: remove field: attributes.timestamp
Noor
Thu, 12 Oct 2023 12:05:02 UTCHello Nitya please review my word doc let me what else I need to do make it work for us thanks
Noor
Thu, 12 Oct 2023 16:35:36 UTCI need to know why it is not working for me if I am able to make this work I will be moving forward to deploy it next. Thanks
nitya-signoz
Thu, 12 Oct 2023 16:37:30 UTCcan you share your config ? the regex in doc and the config doesn’t match. Also provide some sample log lines for testing it out.
Noor
Thu, 12 Oct 2023 16:38:17 UTCeceivers: filelog: include: [ /tmp/startServer.log ] start_at: beginning operators: - type: regex_parser regex: layout: '%Y-%m-%d,%H:%M:%S.%z' - type: move from: attributes["body"] to: body - type: remove field: attributes.timestamp tcplog/docker: listen_address: "0.0.0.0:2255" operators: - type: regex_parser regex: '^\[(?P<time>\d+\/\d+\/\d+ \d{2}:\d+:\d{2}:\d{3} UTC)\](?P<body>.*)' timestamp: parse_from: attributes.timestamp layout: '%Y-%m-%dT%H:%M:%S.%LZ' - type: move from: attributes["body"] to: body - type: remove field: attributes.timestamp
Noor
Thu, 12 Oct 2023 16:39:27 UTCThis is what we or I am trying to parse 4/28/21 12:23:54:157 UTC] 00000001 ManagerAdmin I TRAS0017I: The startup trace state is *=info.
[4/28/21 12:23:54:302 UTC] 00000001 AdminTool A ADMU0128I: Starting tool with the Custom01 profile
[4/28/21 12:23:54:305 UTC] 00000001 AdminTool A ADMU3100I: Reading configuration for server: igawas02
[4/28/21 12:23:54:321 UTC] 00000001 ImplFactory W WSVR0072W: Ignoring undeclared override of interface, com.ibm.websphere.cluster.topography.DescriptionManager, with implementation,
Noor
Thu, 12 Oct 2023 16:41:46 UTCI have been trying to use(?P<timestamp>, <?(message>), (?P,<body>) and even <?P<Date>) but has been showing up with error
Noor
Thu, 12 Oct 2023 16:42:10 UTCmy last error now is this 2023/10/12 13:53:49 application run finished with error: failed to get config: cannot resolve the configuration: cannot retrieve the configuration: yaml: line 8: mapping values are not allowed in this context
Noor
Thu, 12 Oct 2023 16:42:53 UTCI need to know what I am doing in correct so I can educate my team member in the future thank you so much for helping me on this
nitya-signoz
Thu, 12 Oct 2023 16:47:15 UTC• The regex pattern is not present and it should be enclosed in quotes • The regex is not correct it doesn’t work (golang as language)
Noor
Thu, 12 Oct 2023 16:57:51 UTCdo you mean like this
Noor
Thu, 12 Oct 2023 17:08:11 UTCI have this now it is matching now in regex
Noor
Thu, 12 Oct 2023 17:11:36 UTCcape character 2023/10/12 17:10:57 application run finished with error: failed to get config: cannot resolve the configuration: cannot retrieve the configuration: yaml: line 7: found unknown escape character
Noor
Thu, 12 Oct 2023 17:13:01 UTCPlease do tell me where is the issue in my config now receivers: filelog: include: [ /tmp/startServer.log ] start_at: beginning operators: - type: regex_parser regex: "(\S+\d+\D+)(S\+\d+\d+)[0-9]\D+\W+\S+([a-z]+\W+\S+\D+)(?P<bod>.*)>.*|(?P<timestamp>\S+)|parseTimestamp(field=timestamp)" layout: '%Y-%m-%d,%H:%M:%S.%z' - type: move from: attributes["body"] to: body - type: remove field: attributes.timestamp tcplog/docker: listen_address: "0.0.0.0:2255" operators: - type: regex_parser regex: '^\[(?P<time>\d+\/\d+\/\d+ \d{2}:\d+:\d{2}:\d{3} UTC)\](?P<body>.*)' timestamp: parse_from: attributes.timestamp layout: '%Y-%m-%dT%H:%M:%S.%LZ' - type: move from: attributes["body"] to: body - type: remove field: attributes.timestamp
Noor
Thu, 12 Oct 2023 17:15:54 UTC2023/10/12 17:15:02 application run finished with error: failed to get config: cannot resolve the configuration: cannot retrieve the configuration: yaml: line 5: did not find expected key with this config operators: - type: regex_parser regex: '(\S+\d+\D+)(S\+\d+\d+)[0-9]\D+\W+\S+([a-z]+\W+\S+\D+)(?P<bod>.*)>.*|(?P<timestamp>\S+)|parseTimestamp(field=timestamp)' layout: '%Y-%m-%d,%H:%M:%S.%z' - type: move from: attributes["body"] to: body - type: remove field: attributes.timestamp
Noor
Thu, 12 Oct 2023 17:19:59 UTCDoes it have to do with my pass 30 day testing with my log in id just wondering that is all
SigNoz Community
Indexed 1061 threads
Built with ClickHouse as datastore, SigNoz is an open-source APM to help you find issues in your deployed applications & solve them quickly | Knowledge Base powered by Struct.AI
View in App
Noor
Fri, 06 Oct 2023 16:13:20 UTCIs this common error in signoz when testing local host log file: 2023-10-06T15:50:05.809Z error helper/transformer.go:99 Failed to process entry {"kind": "receiver", "name": "filelog", "data_type": "logs", "operator_id": "remove", "operator_type": "remove", "error": "remove: field does not exist", "action": "send", "entry": {"observed_timestamp":"2023-10-06T15:50:05.809132405Z","timestamp":"0001-01-01T00:00:00Z","body":"at.management.tools.AdminTool.executeUtility(AdminTool.java:282)","attributes":{"log.file.name":"startServer.log"},"severity":0,"scope_name":""}}