Function Compute:Simple Log Service triggers

Use cases

• Data cleaning and processing

  Use SLS to quickly collect, process, query, and analyze logs.

  

• Data shipping

  Deliver log data to destinations such as big data products on the cloud or third-party services, and build data pipelines between them.

  

How it works

An SLS trigger corresponds to one SLS ETL job in Function Compute. After you create an SLS ETL job, SLS starts a timer based on the ETL job configuration. The timer polls the shard information in the Logstore. When new data is written, SLS generates a <shard_id, begin_cursor, end_cursor> triplet as the function event and invokes the function.

When the storage system is upgraded, a cursor change may occur even if no new data is written. In this case, each shard is triggered one extra time with no data. You can use the cursor in the function to try to obtain the shard data. If no data is returned, the invocation is an empty trigger and you can ignore it in the function. For more information, see Custom function development guide.

The trigger mechanism is time-based. For example, if you set the trigger interval of the ETL job to 60 seconds and data is continuously written to Shard0 of the Logstore, the shard triggers a function invocation every 60 seconds. If no new data is written to the shard, no invocation is triggered. The input of each invocation is the cursor range of the last 60 seconds. In the function, you can read the Shard0 data based on the cursor for further processing.

Trigger frequency has the following characteristics:

• Each shard is triggered separately. The total number of invocations that you see for a Logstore may be high, but the actual trigger time of each shard still matches the configured interval. For example, if a Logstore has 10 shards, real-time data processing with no trigger delay results in 10 function invocations every 60 seconds.

• The trigger interval of a single shard is the same as the time range of the data processed each time. Assume that the trigger interval is 60 seconds. When the function runs, the trigger interval falls into the following two cases:

  • No trigger delay: the function is triggered every 60 seconds as scheduled, and the processed data range is [now -60s, now).

  • Trigger delay, which occurs when the current processing position of the SLS shard falls more than 10 seconds behind the latest written data: the trigger catches up and may fire once every 2 seconds. Each invocation still processes a 60-second window.

  

Data processing functions

The function that an SLS trigger invokes can be one of the following types:

• Template functions

  For more information, see aliyun-log-fc-functions.

• Custom functions

  The configuration format depends on how you implement the function. For more information, see ETL function development guide.

Prerequisites

• Function Compute

  • Create an event-triggered function

• Simple Log Service (SLS)

  • Create a Project and a Logstore

  • Create one Project and two Logstores. One Logstore stores the collected logs. Function Compute is triggered by incremental logs, so make sure that logs can be continuously collected into this Logstore. The other Logstore stores the logs generated by the SLS trigger.

  The Project must be in the same region as the Function Compute service.

Step 1: Create an SLS trigger

You can configure an SLS trigger that periodically obtains updated data and invokes a function to incrementally consume the data in an SLS Logstore. In the function, you perform custom processing tasks, such as data cleaning and processing, and deliver the data to third-party services. This example demonstrates only how to obtain log data and print it. The function used for data processing can be a template provided by SLS or a custom function. The following steps use a custom function.

1. Log on to the Function Compute console. In the left-side navigation pane, choose Functions > Functions.

2. In the top navigation bar, select a region. On the Functions page, click the function that you want to manage.

3. On the Function Details page, click the Trigger tab and click Create Trigger. In the Create Trigger panel, set Trigger Type to Log Service, configure the other parameters, and then click OK.

After the trigger is created, it appears on the Triggers tab. To modify or delete the trigger, see "Manage Triggers" in the Function Compute User Guide.

Step 2: Configure permissions

The function role provides the SLS permissions that your function needs when it is invoked.

1. On the Function Details page, click the Configuration tab. In the Advanced Settings section, click Modify. In the Advanced Settings panel, select a Function Role.

   • If your function only reads log data, you can use the default role AliyunFCServerlessDevsRole, which has read-only permissions on SLS by default.

   • If your function requires permissions beyond read-only access to SLS, create a custom RAM role that meets the following two requirements:

     a. When you create the RAM role, set Trusted entity to Cloud Service and set Trusted service to Function Compute. For more information, see Create a RAM role for a trusted Alibaba Cloud service.

     b. Grant the RAM role the SLS permissions that your function requires. For more information, see Examples of custom RAM policies.

2. Click Deploy.

Step 3: Deploy the function and view printed logs

1. On the Code tab of the Function Details page, write your code in the code editor and then click Deploy.

   This example deploys a Python function that does the following:

   • Obtains SLS event information, such as endpoint, projectName, logstoreName, and beginCursor, from event.

   • Obtains the credential information accessKeyId, accessKeySecret, and securityToken from context.

   • Initializes the SLS client based on the obtained information.

   • Obtains the log data at the specified cursor position from the source Logstore.

     You can use the following sample code as a starting template for most log processing scenarios.

   """
   This sample code is mainly doing the following things:
   * Get SLS processing related information from event
   * Initiate SLS client
   * Pull logs from source log store
   """
   import logging
   import json
   from aliyun.log import LogClient
   
   logger = logging.getLogger()
   
   
   def handler(event, context):
       # Access keys can be fetched through context.credentials
       print("The content in context entity is: ", context)
       creds = context.credentials
       access_key_id = creds.access_key_id
       access_key_secret = creds.access_key_secret
       security_token = creds.security_token
   
       # parse event in object
       event_obj = json.loads(event.decode())
       print("The content in event entity is: ", event_obj)
   
       # Get the name of log project, the name of log store, the endpoint of sls, begin cursor, end cursor and shardId from event.source
       source = event_obj['source']
       log_project = source['projectName']
       log_store = source['logstoreName']
       endpoint = source['endpoint']
       begin_cursor = source['beginCursor']
       end_cursor = source['endCursor']
       shard_id = source['shardId']
   
       # Initialize client of sls
       client = LogClient(endpoint=endpoint, accessKeyId=access_key_id, accessKey=access_key_secret, securityToken=security_token)
   
       # Read data from source logstore within cursor: [begin_cursor, end_cursor) in the example, which contains all the logs trigger the invocation
       while True:
           response = client.pull_logs(project_name=log_project, logstore_name=log_store, shard_id=shard_id, cursor=begin_cursor, count=100, end_cursor=end_cursor, compress=False)
           log_group_cnt = response.get_loggroup_count()
           if log_group_cnt == 0:
               break
           logger.info("get %d log group from %s" % (log_group_cnt, log_store))
           logger.info(response.get_loggroup_list())
           begin_cursor = response.get_next_cursor()
   
       return 'success'

2. On the Function Details page, choose Logs > Function Logs to view the latest data obtained when the function runs. If The logging feature is not enabled for the current function. is displayed, click Enable.

   You have completed the SLS trigger configuration. After the trigger interval elapses and new data is written to the source Logstore, the trigger invokes the function, and the function logs appear on the Function Logs page. You can also check the Logstore that you specified for the Trigger Log parameter in Step 1 to confirm that the trigger fired. To debug the code in the console, complete the following steps.

(Optional) Step 4: Test the function with a simulated event

1. On the Code tab of the Function Details page, click the icon to the right of Test Function and select Configure Test Parameters from the drop-down list.

2. In the Configure Test Parameters panel, select Create New Test Event or Modify Existing Test Event, enter an event name and the event content, and then click OK. If you create a new test event, we recommend that you select the Log Service event template. For more information about the test data, see event parameter.

3. After you configure the simulated event, click Test Function.

   After the invocation is complete, you can view the result above the Code tab.

Limits

The number of SLS triggers associated with a single Project cannot exceed five times the number of existing Logstores in that Project.

Configure no more than five SLS triggers for each Logstore. Otherwise, the efficiency of data delivery to Function Compute may be affected.

Input parameters

{
    "parameter": {},
    "source": {
        "endpoint": "http://cn-hangzhou-intranet.log.aliyuncs.com",
        "projectName": "fc-test-project",
        "logstoreName": "fc-test-logstore",
        "shardId": 0,
        "beginCursor": "MTUyOTQ4MDIwOTY1NTk3ODQ2Mw==",
        "endCursor": "MTUyOTQ4MDIwOTY1NTk3ODQ2NA=="
    },
    "jobName": "1f7043ced683de1a4e3d8d70b5a412843d81****",
    "taskId": "c2691505-38da-4d1b-998a-f1d4bb8c****",
    "cursorTime": 1529486425
}

FAQ