Masking data in logs for Signoz Clickhouse Backend
TLDR Krishnadas needed to mask data in their logs before injecting into their backend. nitya-signoz provided solutions and helped Krishnadas correct their regex commands.
2
Aug 02, 2023 (2 months ago)
Krishnadas
01:38 PMAug 03, 2023 (2 months ago)
vishal-signoz
05:06 AM1
nitya-signoz
05:10 AMreplace_all_matches in transform processor https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/transformprocessor#exampleKrishnadas
06:59 AM- replace_all_matches(attributes, "[email protected]", "[a-zA-Z0-9.]+@[a-zA-Z]+[.com]+")
Do I need to replace attributes with something else ? As per the json, I am seeing attributes_string instead
![Image 1 for Thanks for the reply <@4K1788>. I have tried this but seems like I am doing something wrong. I am trying to mask email from the body and keys and replace with dummy value. My json response is attached.
- replace_all_matches(attributes, "<mailto:dummy@email.com|dummy@email.com>", "[a-zA-Z0-9.]+@[a-zA-Z]+[.com]+")
Do I need to replace attributes with something else ? As per the json, I am seeing attributes_string instead](https://app.struct.ai/images/md5-9450bca17d9577658461440f4dc20e49/512)
nitya-signoz
10:38 AMShouldn’t it be
replace_all_matches(attributes, "[a-zA-Z0-9.]+@[a-zA-Z]+[.com]+", "<mailto:[email protected]|[email protected]>") ?nitya-signoz
10:39 AMAug 04, 2023 (1 month ago)
Krishnadas
04:12 AMKrishnadas
04:17 AMapplication run finished with error: invalid configuration: processors::transform: couldn't create function: the pattern supplied to replace_match is not a valid pattern: expected close range character
Not sure why it says expected close range character, when it is already close range.
nitya-signoz
12:54 PM transform:
log_statements:
- context: log
statements:
- replace_pattern(attributes["csusername"], "[a-zA-Z0-9.]+@[a-zA-Z]+[.com]+", "") Krishnadas
01:40 PMAlso used replace_pattern(body, "[a-zA-Z0-9.]+@[a-zA-Z]+[.com]+", "[email protected]") to mask content from body
1
SigNoz Community
Indexed 826 threads (62% resolved)
Similar Threads
Extracting Custom Fields as Attributes from Log in SigNoz
Harald wants to have custom fields from their body parsed as attributes in the SigNoz logger. They have tried implementing suggestions from nitya-signoz and Prashant, but the issue remains unsolved due to a potential race condition encountered while executing the code. They have provided a full example using Kind for further assistance.
Redacting Sensitive Info from Log Files Using Open Telemetry
Geoffrey needed assistance to redact certain information from log files, but the suggested redaction didn't help. They later resolved it themselves using transform processor, acting on both attributes message and body.
Parsing log entry and extracting fields with OTEL
Syed needed help parsing a log entry and extracting fields using OTEL. nitya-signoz provided guidance and the correct configurations, resolving the issue.
Debugging JSON Parsing and Logging Issue with Otel-Collector
Hannes was struggling to debug log conversion to JSON and attribute parsing in otel-collector. nitya-signoz provided guidance and updates to the processors which eventually resolved the issue.
5
JSON Parsing Issue with Logs in Signoz
Androw had trouble parsing logs with JSON in Signoz. nitya-signoz suggested changes to config, which resolved the issue after Androw added the missing field 'logsCollection'.