#support

Masking data in logs for Signoz Clickhouse Backend

TLDR Krishnadas needed to mask data in their logs before injecting into their backend. nitya-signoz provided solutions and helped Krishnadas correct their regex commands.

4mo

Join the chat

Aug 02, 2023 (4 months ago)

Photo of md5-12063f62d3a94a65cd24766e394864e6

Krishnadas

01:38 PM

#support Is it possible to mask certain data from logs before injecting to Signoz Clickhouse backend ? I am using filelog receiver

Aug 03, 2023 (4 months ago)

Photo of md5-f936d3e5743d23344d6c60813189716f

vishal-signoz

05:06 AM

nitya-signoz Can you help here?

Photo of md5-a52b9d6c34f193d9a1ff940024f36f77

nitya-signoz

05:10 AM

You can try replace_all_matches in transform processor https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/transformprocessor#example

Krishnadas

06:59 AM

Thanks for the reply nitya-signoz. I have tried this but seems like I am doing something wrong. I am trying to mask email from the body and keys and replace with dummy value. My json response is attached.

- replace_all_matches(attributes, "[email protected]", "[a-zA-Z0-9.]+@[a-zA-Z]+[.com]+")

Do I need to replace attributes with something else ? As per the json, I am seeing attributes_string instead

Image 1 for Thanks for the reply <@4K1788>. I have tried this but seems like I am doing something wrong. I am trying to mask email from the body and keys and replace with dummy value. My json response is attached.

- replace_all_matches(attributes, "<mailto:dummy@email.com|dummy@email.com>", "[a-zA-Z0-9.]+@[a-zA-Z]+[.com]+")

Do I need to replace attributes with something else ? As per the json, I am seeing attributes_string instead

nitya-signoz

10:38 AM

attributes will remain attributes

Shouldn’t it be replace_all_matches(attributes, "[a-zA-Z0-9.]+@[a-zA-Z]+[.com]+", "<mailto:[email protected]|[email protected]>") ?

10:39

nitya-signoz

10:39 AM

Is your email present in body or in attributes ?

Aug 04, 2023 (4 months ago)

Krishnadas

04:12 AM

Email is present inside the body and also showing as attributes_string.csusername since I am using a regex to capture keys from the original log line

04:17

Krishnadas

04:17 AM

nitya-signoz When I try the order you suggested, it gives me this error

application run finished with error: invalid configuration: processors::transform: couldn't create function: the pattern supplied to replace_match is not a valid pattern: expected close range character

Not sure why it says expected close range character, when it is already close range.

nitya-signoz

12:54 PM

Here you go

  transform:
    log_statements:
      - context: log
        statements:
          - replace_pattern(attributes["csusername"], "[a-zA-Z0-9.]+@[a-zA-Z]+[.com]+", "")

Krishnadas

01:40 PM

nitya-signoz Great. I was trying attributes.csusername and wasnt working. Now the value is replaced with dummy value. Thank you so much.

Also used replace_pattern(body, "[a-zA-Z0-9.]+@[a-zA-Z]+[.com]+", "[email protected]") to mask content from body

SigNoz Community

Built with ClickHouse as datastore, SigNoz is an open-source APM to help you find issues in your deployed applications & solve them quickly | Knowledge Base powered by Struct.AI

Indexed 1023 threads (61% resolved)

Join Our Community

Similar Threads

Extracting Custom Fields as Attributes from Log in SigNoz

Harald wants to have custom fields from their body parsed as attributes in the SigNoz logger. They have tried implementing suggestions from nitya-signoz and Prashant, but the issue remains unsolved due to a potential race condition encountered while executing the code. They have provided a full example using Kind for further assistance.

209

5mo

Redacting Sensitive Info from Log Files Using Open Telemetry

Geoffrey needed assistance to redact certain information from log files, but the suggested redaction didn't help. They later resolved it themselves using transform processor, acting on both attributes message and body.

5mo

Parsing log entry and extracting fields with OTEL

Syed needed help parsing a log entry and extracting fields using OTEL. nitya-signoz provided guidance and the correct configurations, resolving the issue.

7mo

Debugging JSON Parsing and Logging Issue with Otel-Collector

Hannes was struggling to debug log conversion to JSON and attribute parsing in otel-collector. nitya-signoz provided guidance and updates to the processors which eventually resolved the issue.

4mo

JSON Parsing Issue with Logs in Signoz

Androw had trouble parsing logs with JSON in Signoz. nitya-signoz suggested changes to config, which resolved the issue after Androw added the missing field 'logsCollection'.

7mo