Log Service:Logtail configurations

sensitive_keys

• Parameters

  Parameter	Type	Required	Example	Description
  key	string	Yes	content	The name of the log field.
  type	string	Yes	const	The method that is used to mask the content of the log field. Valid values: const: The sensitive content is replaced by the value of the const field. md5: The sensitive content is replaced by the MD5 hash value that is generated for the content.
  regex_begin	string	Yes	'password':'	The regular expression that is used to match the prefix of sensitive content. The regular expression is used to find sensitive content. You must use the RE2 syntax to specify the regular expression. For more information, visit RE2 syntax.
  regex_content	string	Yes	[^']*	The regular expression that is used to match sensitive content. You must use the RE2 syntax to specify the regular expression. For more information, visit RE2 syntax.
  all	boolean	Yes	true	Specifies whether to replace all sensitive content in the log field. Valid values: true: replaces all sensitive content in the log field. This is the recommended value. false: replaces only the sensitive content that the specified regular expressions match for the first time in the log field.
  const	string	No	"********"	If you set the type parameter to const, you must configure this parameter.

• Configuration example
  If you want to mask password in the log field content whose value is [{'account':'1812213231432969','password':'04a23f38'}, {'account':'1812213685634','password':'123a'}] and replace the values of password with ********, use the following settings for the sensitive_keys parameter:

  sensitive_keys = [{"all": true,
  "const": "********",
  "regex_content": "[^']*",
  "regex_begin": "'password':'",
  "type": "const",
  "key": "content"}]                    

• Sample log

  [{'account':'1812213231432969','password':'********'}, {'account':'1812213685634','password':'********'}]

advanced

Basic parameters

Configurations that are specific to log collection in full regex mode and simple mode

• Parameters

  Parameter	Type	Required	Example	Description
  key	array	Yes	["content"]	The fields that are specified for raw logs.
  logBeginRegex	string	No	.*	The regular expression that is used to match the beginning of the first line of a log.
  regex	string	No	(.*)	The regular expression that is used to extract the value of a field.

• Configuration example

  {
      "configName": "logConfigName", 
      "outputType": "LogService", 
      "inputType": "file", 
      "inputDetail": {
          "logPath": "/logPath", 
          "filePattern": "*", 
          "logType": "common_reg_log", 
          "topicFormat": "default", 
          "discardUnmatch": false, 
          "enableRawLog": true, 
          "fileEncoding": "utf8", 
          "maxDepth": 10, 
          "key": [
              "content"
          ], 
          "logBeginRegex": ".*", 
          "regex": "(.*)"
      }, 
      "outputDetail": {
          "projectName": "test-project", 
          "logstoreName": "test-logstore"
      }
  }

Configurations that are specific to log collection in JSON mode

Configurations that are specific to log collection in delimiter mode

• Parameters

  Parameter	Type	Required	Example	Description
  separator	string	No	,	The delimiter. You must select a delimiter based on the format of logs that you want to collect. For more information, see Collect logs in delimiter mode.
  quote	string	Yes	\	The quote. If a log field contains delimiters, you must specify a quote to enclose the field. Log Service parses the content that is enclosed in a pair of quotes into a complete field. You must select a quote based on the format of logs that you want to collect. For more information, see Collect logs in delimiter mode.
  key	array	Yes	[ "ip", "time"]	The fields that are specified for raw logs.
  timeKey	string	Yes	time	The time field. You must specify a field in the value of key as the time field.
  autoExtend	boolean	No	true	Specifies whether to upload a log if the number of fields parsed from the log is less than the number of specified keys. For example, if you specify a vertical bar (|) as the delimiter, the log 11|22|33|44|55 is parsed into the following fields: 11, 22, 33, 44, and 55. You can set the keys to A, B, C, D, and E. true: The log 11|22|33|55 is uploaded to Log Service, and 55 is uploaded as the value of the D key. false: The log 11|22|33|55 is discarded because the number of fields parsed from the log does not match the number of specified keys.

• Configuration example

  {
      "configName": "logConfigName", 
      "logSample": "testlog", 
      "inputType": "file", 
      "outputType": "LogService", 
      "inputDetail": {
          "logPath": "/logPath", 
          "filePattern": "*", 
          "logType": "delimiter_log", 
          "topicFormat": "default", 
          "discardUnmatch": true, 
          "enableRawLog": true, 
          "fileEncoding": "utf8", 
          "maxDepth": 999, 
          "separator": ",", 
          "quote": "\"", 
          "key": [
              "ip", 
              "time"
          ], 
          "autoExtend": true
      }, 
      "outputDetail": {
          "projectName": "test-project", 
          "logstoreName": "test-logstore"
      }
  }