Redacting Sensitive Info from Log Files Using Open Telemetry
TLDR Geoffrey needed assistance to redact certain information from log files, but the suggested redaction didn't help. They later resolved it themselves using transform processor, acting on both attributes message and body.
Srikanth
Thu, 29 Jun 2023 01:40:55 UTCredaction process doesn’t support logs.
Geoffrey
Fri, 30 Jun 2023 12:39:17 UTCHi Srikanth /anyone else interested, I solved this using transform processor. See below config ``` processors: transform/redact_sensitive_info: error_mode: ignore log_statements: - context: log statements: - replace_pattern(attributes["message"], "XnHyH4QALxXnQwmvw7XtB2brs63K4pby", "******") - replace_pattern(body, "XnHyH4QALxXnQwmvw7XtB2brs63K4pby", "******") service: pipelines: logs: receivers: [xyz] processors: [transform/redact_sensitive_info, xyz] exporters: [xyz]```
Geoffrey
Fri, 30 Jun 2023 13:02:00 UTCPS - In my case, I have the log in the body section as well as in the attributes.message that's why there are two replace pattern statements.
Transform processor documentation -
SigNoz Community
Indexed 1061 threads
Built with ClickHouse as datastore, SigNoz is an open-source APM to help you find issues in your deployed applications & solve them quickly | Knowledge Base powered by Struct.AI
View in App
Geoffrey
Wed, 28 Jun 2023 10:33:48 UTCHi Guys, I'm trying to redact certain sensitive information bits that are in the log files before exporting using the Open Telemetry collector. Problem - I can't seem to figure out how to redact the value "XnHyH4QALxXnQwmvw7XtB2brs63K4pby" I have tried to use both the hash attribute processor and redaction processor without success. See my yaml config below. ```processors: attributes: actions: - key: body pattern: 'XnHyH4QALxXnQwmvw7XtB2brs63K4pby' action: hash redaction: blocked_values: - "XnHyH4QALxXnQwmvw7XtB2brs63K4pby"``` See log below - I've applied json formatting to the log so i can apply json_parser on the collector config - that part is working. ```{ "body":"Request full data: {\n \"username\": \"user\",\n \"key\": \"XnHyH4QALxXnQwmvw7XtB2brs63K4pby\",\n \"operator\": \"123\",\n \"no\": \"123\",\n \"units\": 20\n}", "attributes":{ "host_and_client":{ }, "channel":"production", "context":{ }, "datetime":"2023-06-28T12:51:56.962453+03:00", "extra":{ }, "level":200, "level_name":"INFO" } }``` Is there anything I'm missing? How can I achieve the redaction? Thanks a lot in advance:pray: