SLS data masking encryption plugins - Log Service - Simple Log Service

Example

The following table compares a raw log stored in SLS with and without the data masking plugin.

Raw log	Without plugin	With plugin
`[{'account':'1812213231432969','password':'04a23f38'}, {'account':'1812213685634','password':'123a'}]`	Content: "[{'account':'1812213231432969','password':'04a23f38'}, {'account':'1812213685634','password':'123a'}]"	Content: "[{'account':'1812213231432969','password':'****'}, {'account':'1812213685634','password':'****'}]"

Data security plugins

SLS provides the following data security plugins.

Parameter	Type	Description
data masking	native	Masks log fields.
data masking	extension	Replaces sensitive data with a specified string or an MD5 hash.
field encryption	extension	Encrypts the content of specified fields.
data encoding and decoding	extension	Performs Base64 decoding, Base64 encoding, and MD5 encoding on data.

Entry point

To use a Logtail plugin for log processing, add it when you create or modify a Logtail configuration. For more information, see Overview.

Differences between native and extension plugins

Native plugins: Written in C++ for higher performance.

Extension plugins: Written in Go for a richer ecosystem and greater flexibility. Use them for complex business logs that native plugins cannot handle.

Performance limitations of extension plugins
- Extension plugins cause LoongCollector to consume more resources, primarily CPU. Adjust LoongCollector parameters in Configuration Management if needed.
- If raw data exceeds 5 MB/s, avoid complex plugin combinations. Use an extension plugin for basic processing, then offload to Data Transformation.
Log collection limitations
- Extension plugins process text logs in line mode, so file-level metadata such as __tag__:__path__ and __topic__ is stored in each log entry.
- Adding extension plugins affects features related to tags:
  - Contextual query and LiveTail are unavailable unless you add an aggregators configuration.
  - The __topic__ field is renamed to __log_topic__. If you add an aggregators configuration, both the __topic__ and __log_topic__ fields are present in the logs. If you do not need the __log_topic__ field, you can use the drop field plugin to remove it.
  - Fields such as __tag__:__path__ no longer have a native field index. You must create a field index for them.

Data masking plugin (native)

The data masking plugin masks sensitive data in logs.

Configuration

Parameter	Description
Original Field	The field that stores the raw log content before parsing.
Data Masking Method	The masking method. Valid values: const: Replaces sensitive content with a specified string. md5: Replaces sensitive content with its MD5 hash.
Replacement String	The string that replaces sensitive content. This parameter is required only when Data Masking Method is set to const.
Content Expression that Precedes Replaced Content	A regular expression used to locate the sensitive data by matching the content that precedes it. The expression must be in RE2 syntax.
Content Expression to Match Replaced Content	A regular expression that matches the sensitive content to be masked. The expression must be in RE2 syntax.
Replace All Matched Content	If you select Replace All Matched Content, all sensitive content in the field is replaced. If you do not select Replace All Matched Content, only the first part of the field that matches the expression is replaced.

Configuration example

Source field

content:[{'account':'1812213231432969','password':'04a23f38'}, {'account':'1812213685634','password':'123a'}]

Logtail configuration: Set processor type to data masking, masking method to const, replacement string to ********, preceding content expression to password':', and target expression to [^']*. Select replace all matches, and then click confirm.

Processing result

[{'account':'1812213231432969','password':'********'}, {'account':'1812213685634','password':'********'}]

Data desensitization plugin

The processor_desensitize plugin replaces sensitive data in logs with a specified string or an MD5 hash.

Limitations

Form-based configuration: Supported when collecting text logs and container standard output.
JSON-based configuration: Not supported for text log collection.

Configuration

Important

Logtail versions 1.3.0 and later support the processor_desensitize plugin.

Form

Set processor type to Data Masking. The following table describes the related parameters.

Parameters

Parameter	Description
Original Field	The name of the log field.
Data Masking Method	The method used for data desensitization. Valid values: String Replacement: Replaces sensitive content with a string. You can specify the target string by using the Replacement String parameter. md5: Replaces sensitive content with its MD5 hash.
Sensitive Data Content	The method for extracting sensitive content. Valid values: Field Text: Replaces the entire value of the target field. Use Regular Expression: Uses a regular expression to extract sensitive content.
Replacement String	The string used to replace sensitive content. Required if Data Masking Method is String Replacement.
Regex to Match Sensitive Content Prefix	The regular expression to match the prefix of the sensitive content. Required if Sensitive Data Content is regex-based specification.
Regex to Match Sensitive Content	The regular expression to match the sensitive content. Required if Sensitive Data Content is regex-based specification.

Configuration example

Replace the entire value of the target field with a string.
- Raw log
```
"password" : "123abcdefg"
```
- Logtail plugin configuration: Set processor type to data desensitization, and then set the following parameters: source field to password, desensitization method to string replacement, sensitive data content to full field, and replacement string to ********.
- Processing result
```
"password":"********"
```

JSON

Set type to processor_desensitize. The following table describes the parameters within the detail object.

Parameters

Parameter	Type	Required	Description
SourceKey	String	Yes	The name of the log field.
Method	String	Yes	The method used for data desensitization. Valid values: const: Replaces sensitive content with a string. You can use the ReplaceString parameter to specify the target string. md5: Replaces sensitive content with its MD5 hash.
Match	String	No	The method for extracting sensitive content. Valid values: full (default): Replaces the entire value of the target field. regex: Uses a regular expression to extract sensitive content.
ReplaceString	String	No	The string used to replace sensitive content. Required if Method is set to const.
RegexBegin	String	No	The regular expression to match the prefix of the sensitive content. Required if Match is set to regex.
RegexContent	String	No	The regular expression to match the sensitive content. Required if Match is set to regex.

Configuration examples

Example 1

To replace the entire value of the target field with a string, set Method to const and Match to full.

Raw log
```
"password" : "123abcdefg"
```

Logtail plugin configuration

{
  "type" : "processor_desensitize",
  "detail" : {
    "SourceKey" : "password",
    "Method" : "const",
    "Match" : "full",
    "ReplaceString": "********"
  }
}

Processing result
```
"password":"********"
```

Example 2

To use a regular expression to specify sensitive content and replace it with its MD5 hash, set Method to md5 and Match to regex.

Raw log

"content" : "[{'account':'1234567890','password':'abc123'}]"

Logtail plugin configuration

{
   "type" : "processor_desensitize",
   "detail" : {
      "SourceKey" : "content",
      "Method" : "md5",
      "Match" : "regex",
      "RegexBegin": "'password':'",
      "RegexContent": "[^']*"
   }
}

Processing result

"content":"[{'account':'1234567890','password':'e99a18c428cb38d5f260853678922e03'}]"

Field encryption plugin

The processor_encrypt plugin encrypts the content of specified fields.

Configuration

Form

Set Processor Type to Field Encryption. The following table describes the related parameters.

Parameter	Description
Original Field	The source fields to encrypt. You can add multiple fields.
AccessKey Pair	The encryption key. It must be a 64-character hexadecimal string.
Initialization Vector	The initialization vector for encryption. It must be a 32-character hexadecimal string. The default value is `00000000000000000000000000000000`.
Storage Path	The path to the file containing the encryption parameters. If not specified, the plugin uses the File Path from the Input settings of the Logtail configuration.
Retain Raw Data If Processing Fails	If you select this option, the system retains the value of the source field if encryption fails. If this option is not selected, the field value is replaced with `ENCRYPT_ERROR` when encryption fails.

JSON

Set type to processor_encrypt. The following table describes the detail parameter.

Parameter	Type	Required	Description
`SourceKey`	Array of strings	Yes	The source fields to encrypt.
`EncryptionParameters`	Object	Yes	The encryption settings.
`Key`	String	Yes	The encryption key. It must be a 64-character hexadecimal string.
`IV`	String	No	The initialization vector for encryption. It must be a 32-character hexadecimal string. The default value is `00000000000000000000000000000000`.
`KeyFilePath`	String	No	The path to the file containing the encryption parameters. If not specified, the plugin uses the File Path from the Input settings of the Logtail configuration.
`KeepSourceValueIfError`	Boolean	No	Specifies whether to retain the source value if encryption fails. `true`: The source value is retained. `false` (Default): The source value is not retained. Instead, the field value is replaced with `ENCRYPT_ERROR`.

Data encoding and decoding plugin

The processor_base64_encoding, processor_base64_decoding, and processor_md5 plugins encode and decode field values.

Limitations

Form-based configuration: Available when you collect text logs and container standard output.
JSON-based configuration: Not supported for collecting text logs.

Base64 encoding

The processor_base64_encoding plugin Base64-encodes a field value.

Form

Set Processor type to BASE64 (Encoding). The following table describes the parameters.

Parameter	Description
Original Field	The name of the source field.
New Field	The name of the result field after encoding.
Report Original Field Missing Error	Select this option to report an error if the source field is not found in the source log.

JSON

Set type to processor_base64_encoding. The following table describes the parameters in the detail object.

Parameter	Type	Required	Description
`SourceKey`	String	Yes	The name of the source field.
`NewKey`	String	Yes	The name of the result field after encoding.
`NoKeyError`	Boolean	No	Specifies whether to report an error if the source field is not found in the source log. `true`: Reports an error. `false` (default value): Does not report an error.

Base64 decoding

The processor_base64_decoding plugin Base64-decodes a field value.

Form

Set Processor type to BASE64 (Decoding). The following table describes the parameters.

Parameter	Description
Original Field	The name of the source field.
New Field	The name of the result field after decoding.
Report Original Field Missing Error	Select this option to report an error if the source field is not found in the source log.
Report Decoding Failure Error	Select this option to report an error if decoding fails.

JSON

Set type to processor_base64_decoding. The following table describes the parameters in the detail object.

Parameter	Type	Required	Description
`SourceKey`	String	Yes	The name of the source field.
`NewKey`	String	Yes	The name of the result field after decoding.
`NoKeyError`	Boolean	No	Specifies whether to report an error if the source field is not found in the source log. `true`: Reports an error. `false` (default value): Does not report an error.
`DecodeError`	Boolean	No	Specifies whether to report an error if decoding fails. `true`: Reports an error. `false` (default value): Does not report an error.

MD5 encoding

The processor_md5 plugin computes the MD5 hash of a field value.

Form

Set Processor type to MD5. The following table describes the parameters.

Parameter	Description
Original Field	The name of the source field.
New Field	The name of the result field after encoding.
Report Original Field Missing Error	Select this option to report an error if the source field is not found in the source log.

JSON

Set type to processor_md5. The following table describes the parameters in the detail object.

Parameter	Type	Required	Description
`SourceKey`	String	Yes	The name of the source field.
`MD5Key`	String	Yes	The name of the result field after encoding.
`NoKeyError`	Boolean	No	Specifies whether to report an error if the source field is not found in the source log. `true`: Reports an error. `false` (default value): Does not report an error.

References

Manage Logtail pipeline configurations using the API:
Configure processing plugins in the console:
- Continuously collect text logs from hosts
- Collect container logs from clusters (standard output/files) in the console
Collect container logs from clusters (standard output/files) using a Kubernetes CRD