Simple Log Service:Alibaba Cloud Log Service Add-on for Splunk

How it works

The add-on:

• Creates a Log Service consumer group through a Splunk data input to ingest logs from Log Service in real time.

• Delivers the collected logs to a Splunk indexer by using the Splunk private protocol or HTTP Event Collector (HEC).

Note

This is a data collection add-on. Install it only on a Splunk heavy forwarder. Do not install it on indexers or search heads.

Key concepts

• A data input acts as a consumer that ingests logs.

• A consumer group contains multiple consumers that jointly ingest data from a logstore. Each log is processed by only one consumer to prevent duplicates.

• A logstore contains multiple shards.

  • Each shard is assigned to only one consumer.

  • A consumer can be assigned multiple shards.

• Each consumer name combines the consumer group name, hostname, process name, and event protocol type, ensuring uniqueness within the group.

Use consumer groups to ingest logs.

Prerequisites

• Obtain an AccessKey pair for Log Service.

  Use Alibaba Cloud Resource Access Management (RAM) to obtain an AccessKey pair for your Log Service project. AccessKey pair.

  Use the permission assistant to configure RAM permissions. Configure the permission assistant. The following code shows a common RAM policy:

  Note

  Replace <Project name> and <logstore name> with your actual names. Wildcards (*) are supported.

  {
    "Version": "1",
    "Statement": [
      {
        "Action": [
          "log:ListShards",
          "log:GetCursorOrData",
          "log:GetConsumerGroupCheckPoint",
          "log:UpdateConsumerGroup",
          "log:ConsumerGroupHeartBeat",
          "log:ConsumerGroupUpdateCheckPoint",
          "log:ListConsumerGroup",
          "log:CreateConsumerGroup"
        ],
        "Resource": [
          "acs:log:*:*:project/<Project name>/logstore/<LogStore name>",
          "acs:log:*:*:project/<Project name>/logstore/<LogStore name>/*"
        ],
        "Effect": "Allow"
      }
    ]
  }

• Check your Splunk version and runtime environment.

  • Ensure that you use the latest version of the add-on.

  • Operating system: Linux, macOS, or Windows.

  • Splunk version: Splunk heavy forwarder 8.0 or later and Splunk indexer 7.0 or later.

• Configure the Splunk HTTP Event Collector (HEC). Configure HTTP Event Collector on Splunk Enterprise.

  Required only if you use HEC. Skip this step if you use the Splunk private protocol.

  Note

  When you create an Event Collector token, the indexer acknowledgment feature is not supported.

Install the add-on

Configure the add-on

If Splunk is not running on an Alibaba Cloud ECS instance, use an AccessKey pair to access Log Service:

1. On the Splunk web UI, click Alibaba Cloud Log Service Add-on for Splunk.

2. Configure a global account.

   Go to the Configure > Account tab and configure the AccessKey pair for Log Service.

   Note

   The Username and Password fields map to the AccessKey ID and AccessKey Secret of Log Service.

3. Configure the logging level for the add-on.

   Go to the Configure > Logging tab and configure the logging level for the add-on.

4. Create a data input.

   1. Click Input to go to the Input page.

   2. Click Create New Input to create a new data input.

      Table 1. Data input parameters

      Parameter	Required and format	Description	Example
      Name	Yes, String	A globally unique name for the data input.	None
      Interval	Yes, Integer	Interval in seconds before the data input process restarts after exiting.	Default value: 10
      Index	Yes, String	The Splunk index.	None
      SLS AccessKey	Yes, String	The Alibaba Cloud AccessKey pair (AccessKey ID and AccessKey Secret). Note The Username and Password fields map to the AccessKey ID and AccessKey Secret of Log Service.	The username and password configured for the global account.
      SLS endpoint	Yes, String	The Log Service endpoint. Endpoints.	cn-huhehaote.log.aliyuncs.com https://cn-huhehaote.log.aliyuncs.com
      SLS project	Yes, String	The Log Service project. Manage a project.	None
      SLS logstore	Yes, String	The Log Service logstore. For more information, see Manage a logstore.	None
      SLS consumer group	Yes, String	The Log Service consumer group. To scale out, use the same consumer group name across multiple data inputs. Use consumer groups to ingest logs.	None
      SLS cursor start time	Yes, String	Log ingestion start time. Applies only when the consumer group is first created. Existing consumer groups resume from the last saved checkpoint. Note This time is based on when the log arrives at the server.	Valid values: begin, end, or a time in ISO 8601 format (for example, 2018-12-26 0:0:0+8:00).
      SLS heartbeat interval	Yes, Integer	Heartbeat interval in seconds between the consumer and the server.	Default value: 60
      SLS data fetch interval	Yes, Integer	Interval in seconds for pulling logs. Avoid small values when log frequency is low.	Default value: 1
      Consumer processor	No, String	A consumer processor containing SPL (for example, consume-processor-1). Must be created in your project in advance. Linux x86_64 only. Manage consumer processors.	None
      Topic filter	No, String	Semicolon-separated list of topics to filter. Logs matching any listed topic are not sent to Splunk.	"TopicA;TopicB" means that logs with the topic "TopicA" or "TopicB" are ignored.
      Unfolded fields	No, JSON	A JSON-formatted mapping of topics to field lists: {"topicA": ["field_nameA1", "field_nameA2", ...], "topicB": ["field_nameB1", "field_nameB2", ...], ...}	{"actiontrail_audit_event": ["event"] } means that for logs with the topic actiontrail_audit_event, the string in the log's event field is expanded into a JSON object.
      Event source	No, String	The data source for the Splunk event.	None
      Event source type	No, String	The source type for the Splunk event.	None
      Event retry times	No, Integer	Set to 0 for unlimited retries.	Default value: 0
      Event protocol	Yes	Protocol for sending Splunk events. If you select Private protocol, the HEC parameters below do not apply.	HTTP for HEC HTTPS for HEC Private protocol
      HEC host	Yes, String. Valid only when Event protocol is set to an HEC option.	The HEC host. Set up and use HTTP Event Collector in Splunk Web.	None
      HEC port	Yes, Integer. Valid only when Event protocol is set to an HEC option.	The HEC port.	None
      HEC token	Yes, String. Valid only when Event protocol is set to an HEC option.	The HEC token. About Event Collector tokens.	None
      HEC timeout	Yes, Integer. Valid only when Event protocol is set to an HEC option.	The HEC timeout period in seconds.	Default value: 120

1. Attach a RAM role to the instance.

   Create a RAM role, grant it permissions, and attach it to the ECS instance. Configure an instance RAM role.

   For the permission policy of the RAM role, see the policy in the Prerequisites section.

2. On the Splunk Web UI, click Alibaba Cloud Log Service Add-on for Splunk.

3. Configure a global account.

   Go to the Configure > Account tab and configure the RAM role information for the ECS instance. Add an account with the Username set to ECS_RAM_ROLE and the Password set to the name of the RAM role that you created in Step 1.

4. Create a data input.

   1. Click Input to go to the Input page.

   2. Click Create New Input to create a new data input.

      Set the SLS AccessKey parameter to the global account that you created in Step 3. For information about the other parameters, see Data input parameters.

Search and view logs

• Search data

  Make sure the data input is enabled. On the Splunk home page, click Search & Reporting to query the collected audit logs.

• View internal logs

  • Run index="_internal" | search "SLS info" to view Log Service-related information.

  • Run index="_internal" | search "error" to view runtime error messages.

Performance and security

• Performance specifications

  The following factors affect add-on throughput:

  • Endpoint: Use a public, classic network, VPC, or global acceleration endpoint. A classic network or VPC endpoint is recommended. Endpoints.

  • Bandwidth: Between Log Service and the heavy forwarder, and between the heavy forwarder and the indexer.

  • Splunk indexer processing capability.

  • Number of shards: More shards increase data transfer capability. Set the number based on your source log generation rate. Manage shards.

  • Number of Splunk data inputs: More data inputs with the same consumer group increase throughput.

    Note

    The concurrency of consumers is affected by the number of shards in the Log Service logstore.

  • CPU and memory: Each data input typically uses 1–2 GB of memory and one CPU core.

  A single data input can ingest logs at 1–2 MB/s. Set the number of shards based on your source log generation rate.

  For example, a logstore producing 10 MB/s requires at least 10 shards and 10 data inputs. On a single machine, this requires 10 CPU cores and 12 GB of memory.

• High availability

  Consumer groups save checkpoints on the server, so if a consumer stops, another consumer takes over from the last checkpoint. Distribute data inputs across machines for redundancy. For additional fault tolerance, start more data inputs than shards.

• Use HTTPS

  • Log Service

    If the endpoint starts with https://, such as https://cn-beijing.log.aliyuncs.com, the connection to Log Service is automatically encrypted using HTTPS.

    The server certificate *.aliyuncs.com is issued by GlobalSign. By default, most Linux and Windows machines automatically trust this certificate. In the rare case that a machine does not trust this certificate, see Install a trusted root CA or self-signed certificate.

  • Splunk

    To use HEC over HTTPS, enable SSL in the Splunk HEC global settings. Configure HTTP Event Collector on Splunk Enterprise.

• AccessKey pair storage protection

  The Log Service AccessKey pair and Splunk HEC token are stored in Splunk encrypted storage to prevent credential leaks.

Troubleshooting

• Configuration errors

  • A basic configuration check runs when you create or modify a data input. Data input parameters.

  • A Log Service configuration error occurs, such as a failure to create a consumer group.

    • Command:index="_internal" | search "error"

    • Error log:

      aliyun.log.consumer.exceptions.ClientWorkerException: 
      error occour when create consumer group, 
      errorCode: LogStoreNotExist, 
      errorMessage: logstore xxxx does not exist

    • ConsumerGroupQuotaExceed error

      A logstore supports up to 20 consumer groups. Delete unneeded consumer groups on the Log Service console if you approach the limit. Exceeding 20 triggers a ConsumerGroupQuotaExceed error.

• Permission errors

  • No permission to access Alibaba Cloud Log Service

    • Command:index="_internal" | search "error"

    • Error log:

      aliyun.log.consumer.exceptions.ClientWorkerException: 
      error occour when create consumer group, 
      errorCode: SignatureNotMatch, 
      errorMessage: signature J70VwxYH0+W/AciA4BdkuWxK6W8= not match

  • ECS RAM authentication fails

    • Command:index="_internal" | search "error"

    • Error log:

      ECS RAM Role detected in user config, but failed to get ECS RAM credentials. Please check if ECS instance and RAM role 'ECS-Role' are configured appropriately.

      The name ECS-Role is a placeholder for the actual name of your RAM role.

    • Possible causes:

      • Confirm that the SLS AccessKey parameter in the data input is configured to use the global account that has the RAM role.

      • Confirm that the global account is correctly configured for the RAM role. The username must be ECS_RAM_ROLE, and the password must be the RAM role name.

      • Confirm that the RAM role is attached to the ECS instance.

      • Confirm that the trusted entity for the RAM role is an Alibaba Cloud service, and the trusted service is Elastic Compute Service (ECS).

      • Confirm that the ECS instance with the attached RAM role is the same machine that is running Splunk.

  • No permission to access HEC

    • Command:index="_internal" | search "error"

    • Error log:

      ERROR HttpInputDataHandler - Failed processing http input, token name=n/a, channel=n/a, source_IP=127.0.0.1, reply=4, events_processed=0, http_input_body_size=369
      
      WARNING pid=48412 tid=ThreadPoolExecutor-0_1 file=base_modinput.py:log_warning:302 | 
      SLS info: Failed to write [{"event": "{\"__topic__\": \"topic_test0\", \"__source__\": \"127.0.0.1\", \"__tag__:__client_ip__\": \"10.10.10.10\", \"__tag__:__receive_time__\": \"1584945639\", \"content\": \"goroutine id [0, 1584945637]\", \"content2\": \"num[9], time[2020-03-23 14:40:37|1584945637]\"}", "index": "main", "source": "sls log", "sourcetype": "http of hec", "time": "1584945637"}] remote Splunk server (http://127.0.0.1:8088/services/collector) using hec. 
      Exception: 403 Client Error: Forbidden for url: http://127.0.0.1:8088/services/collector, times: 3

    • Possible causes:

      • HEC is not configured or enabled.

      • The HEC configuration in the data input is incorrect. For example, if you use HTTPS, you must enable the SSL configuration.

      • Check whether the indexer acknowledgment feature is disabled for the HEC token.

• Consumption latency

  View the consumer group status on the Log Service console. Step 2: View the status of the consumer group.

  To resolve consumption latency, increase the number of shards or create more data inputs with the same consumer group. Performance specifications.

• Network jitter

  • Command:index="_internal" | search "SLS info: Failed to write"

  • Error log:

    WARNING pid=58837 tid=ThreadPoolExecutor-0_0 file=base_modinput.py:log_warning:302 |
    SLS info: Failed to write [{"event": "{\"__topic__\": \"topic_test0\", \"__source__\": \"127.0.0.1\", \"__tag__:__client_ip__\": \"10.10.10.10\", \"__tag__:__receive_time__\": \"1584951417\", \"content2\": \"num[999], time[2020-03-23 16:16:57|1584951417]\", \"content\": \"goroutine id [0, 1584951315]\"}", "index": "main", "source": "sls log", "sourcetype": "http of hec", "time": "1584951417"}] remote Splunk server (http://127.0.0.1:8088/services/collector) using hec. 
    Exception: ('Connection aborted.', ConnectionResetError(54, 'Connection reset by peer')), times: 3

  The add-on automatically retries. If the problem persists, contact your network administrator.

• Change the consumption start time

  Note

  The SLS cursor start time parameter applies only when the consumer group is first created. Existing groups resume from the last checkpoint.

  1. On the Input page in the Splunk web UI, disable the target data input.

  2. Log on to the Log Service console. In the Data Consumption section of the target logstore, delete the corresponding consumer group.

  3. On the Input page in the Splunk web UI, click Actions > Edit next to the target data input. Modify the SLS cursor start time parameter and then restart the data input.