Simple Log Service:Use Function Compute to ship logs to an SIEM platform over the syslog protocol

Step 1: Create a function in Function Compute

1. Log on to the Function Compute console. Click Back to Function Compute 2.0 in the upper-right corner.

   

2. In the left-side navigation pane, click Services & Functions.

3. In the top navigation bar, select a region. On the Services page, click Create Service.

   Configure the Name and Service Role parameters. If you create a role to manage Simple Log Service resources, grant the role the permissions to consume data in Logstores. For other parameters, retain the default settings. You can click Show Advanced Options to find the Service Role parameter. For more information, see Grant Function Compute permissions to access other Alibaba Cloud services and Examples of using custom policies to grant permissions to a RAM user.

4. After the service is created, click Create Function on the Functions page.

   

   Perform the following steps to configure the function:

   1. Select Use Built-in Runtime to create a function.

   2. Basic Settings

      • Enter a name in the Function Name field.

      • Select Event Handler for Handler Type.

   3. Code

      • Select Python 3.9 for Runtime.

      • Select Use Sample Code for Code Upload Method. In the Sample Code Description section, select Functions triggered by Log Service.

   4. Environment Variables

      1. Click Form Editor.

      2. Click + Add Variable.

      3. Configure key-value pairs as environment variables.

      • SYSLOG_HOST: the host of the syslog server, which is the receiver of data. Format: hostName.Simple Log Service endpoint. For more information about endpoints, see Endpoints.

      • SYSLOG_PORT: the port of the syslog server, which is the receiver of data. Default value: 10009.

      • SYSLOG_PROTOCOL: the transport-layer protocol for syslog. Default value: tcp.

        

   5. Trigger Configurations

      For more information, see Simple Log Service triggers.

      

      1. Select Log Service for Trigger Type.

      2. Configure the Simple Log Service Project, Logstore, Trigger Interval, Retries, and Trigger Log parameters based on your business requirements.

      3. Retain the default setting for Invocation Parameters.

      4. Select AliyunLogETLRole for Role Name.

         Note

         The first time you create a trigger of this type, click Authorize Now in the dialog box that appears after you click OK. Then, complete authorization to create the AliyunLogETLRole system role for your Simple Log Service trigger.

5. Click Create.

   

Step 2: Configure function code

1. On the Code tab, replace index.py with the following code:

   """
   The sample code is used to implement the following features:
   * Parse the event parameter to obtain the trigger information of Simple Log Service events.
   * Initialize the Simple Log Service client based on the obtained information.
   * Obtain real-time logs from the source Logstore.
   * Ship the logs over syslog.
   
   
   This sample code is mainly doing the following things:
   * Get SLS processing related information from event
   * Initiate SLS client
   * Pull logs from source log store
   * Send logs with syslog
   
   """
   # !/usr/bin/env python
   # -*- coding: utf-8 -*-
   
   import json
   from aliyun.log import LogClient
   import os
   import logging
   import six
   from datetime import datetime
   
   from aliyun.log import PullLogResponse
   from aliyun.log.ext import syslogclient
   from pysyslogclient import SyslogClientRFC5424 as SyslogClient
   
   logger = logging.getLogger()
   
   
   def get_syslog_config():
       config = {
           'host': os.environ.get('SYSLOG_HOST', ''),
           'port': int(os.environ.get('SYSLOG_PORT', '514')),
           'protocol': os.environ.get('SYSLOG_PROTOCOL', 'tcp'),
           "facility": syslogclient.FAC_USER,  # Optional. You can refer to the values of other syslogclient.FAC_* parameters. 
           "severity": syslogclient.SEV_INFO,  # Optional. You can refer to the values of other syslogclient.SEV_* parameters. 
           "hostname": "aliyun.example.com",  # Optional. Specify a hostname. The default value is the hostname of the local host. 
           "tag": "tag"  # Optional. Specify a tag. The default value is a hyphen (-). 
       }
       return config
   
   
   def init_syslog_client(config):
       client = SyslogClient(config.get('host'), config.get('port'), config.get('protocol'))
       return client
   
   
   def process(shard_id, log_groups, config):
       logs = PullLogResponse.loggroups_to_flattern_list(log_groups, time_as_str=True, decode_bytes=True)
       logger.info("Get data from shard {0}, log count: {1}".format(shard_id, len(logs)))
       try:
           client = init_syslog_client(config)
           for log in logs:
               # suppose we only care about audit log
               timestamp = datetime.fromtimestamp(int(log[u'__time__']))
               del log['__time__']
   
               io = six.StringIO()
               # Modify the content format based on your business requirements. In this example, data is transmitted by using key-value pairs that are separated by two consecutive vertical bars (||). 
               for k, v in six.iteritems(log):
                   io.write("{0}{1}={2}".format('||', k, v))
   
               data = io.getvalue()
   
               # Modify the facility and severity parameters based on your business requirements. 
               client.log(data,
                          facility=config.get("facility", None),
                          severity=config.get("severity", None),
                          timestamp=timestamp,
                          program=config.get("tag", None),
                          hostname=config.get("hostname", None))
   
       except Exception as err:
           logger.debug("Failed to connect to remote syslog server ({0}). Exception: {1}".format(config, err))
           # Add code to handle errors. For example, you can add code to retry requests or report errors. 
           raise err
   
       logger.info("Complete send data to remote")
   
   
   def handler(event, context):
       # Use context.credentials.to obtain information about keys.
       # Access keys can be fetched through context.credentials
       print("The content in context entity is: \n")
       print(context)
       creds = context.credentials
       access_key_id = creds.access_key_id
       access_key_secret = creds.access_key_secret
       security_token = creds.security_token
   
       # Parse the event parameter to the object data type.
       # parse event in object
       event_obj = json.loads(event.decode())
       print("The content in event entity is: \n")
       print(event_obj)
   
       # Use event.source to obtain the following information: project name, Logstore name, Simple Log Service endpoint, start cursor, end cursor, and shard ID.
       # Get the name of log project, the name of log store, the endpoint of sls, begin cursor, end cursor and shardId from event.source
       source = event_obj['source']
       log_project = source['projectName']
       log_store = source['logstoreName']
       endpoint = source['endpoint']
       begin_cursor = source['beginCursor']
       end_cursor = source['endCursor']
       shard_id = source['shardId']
   
       # Initialize the Simple Log Service client.
       # Initialize client of sls
       client = LogClient(endpoint=endpoint,
                          accessKeyId=access_key_id,
                          accessKey=access_key_secret,
                          securityToken=security_token)
   
       syslog_config = get_syslog_config()
   
       # Read logs based on the start and end cursors in the source Logstore. In this example, the specified cursors cover all logs that trigger the invocation of the function.
       # Read data from source logstore within cursor: [begin_cursor, end_cursor) in the example, which contains all the logs trigger the invocation
       while True:
           response = client.pull_logs(project_name=log_project,
                                       logstore_name=log_store,
                                       shard_id=shard_id,
                                       cursor=begin_cursor,
                                       count=100,
                                       end_cursor=end_cursor,
                                       compress=False)
           log_group_cnt = response.get_loggroup_count()
           if log_group_cnt == 0:
               break
           logger.info("get %d log group from %s" % (log_group_cnt, log_store))
           process(shard_id, response.get_loggroup_list(), syslog_config)
   
           begin_cursor = response.get_next_cursor()
   
       return 'success'
   

   

Step 3: Create a custom layer

In this topic, logs are shipped over the syslog protocol. Therefore, the dependency library of syslog is required. The following procedure describes how to create a custom layer to import the dependency library:

1. Click Edit Layer.

2. In the panel that appears, choose Add Layer > Add Custom Layer.

3. Click Create Custom Layer. Then, configure the following parameters.

   1. Select Python 3.9 for Compatible Runtime.

   2. Select Build Dependency Layer Online for Layer Upload Method.

   3. Select Python 3.9 for Build Environment.

   4. Enter pysyslogclient in the requirements.txt File field.

   5. Click Create.

4. Return to the layer modification panel, select the custom layer that you created in 3 for Layer 1, and then click OK.

Step 4: Test the function and deploy code

1. Click Test Function. If success is returned, the log shipping is successful.

   

2. Click Deploy to deploy code.

FAQ

What do I do if the UnhandleInvocationError error is reported when I test my function?

1. If the error is reported because your service role does not have the required permissions, see Configure a role.

2. If the error is reported because the configured environment variables are invalid, see Configure environment variables.

3. If you have more questions, join the DingTalk group 11721331 to contact Function Compute engineers.

What to do next

• Check whether logs are received on the syslog server.

• View the trigger and invocation information of the function on the Logs tab.