Simple Log Service:Use Function Compute to consume log data

Scenarios

• Data cleaning and transformation

  Collect, transform, query, and analyze logs with SLS.

• Data Transfer Scenarios

  Ship data to various destinations and build data pipelines between big data products on the cloud.

Data transformation functions

• Function types

  • Template functions

    Available templates: aliyun-log-fc-functions.

  • User-defined functions

    The configuration format varies by implementation. ETL function development guide.

• Function Compute trigger mechanism

  Each trigger task maps to a Function Compute trigger. SLS starts a timer that polls shard information in the Logstore. When new data arrives, SLS generates a <shard_id,begin_cursor,end_cursor> triple as the function event and triggers execution.

  Note

  Storage upgrades may shift cursors without new data, causing one extra empty invocation per shard. Handle this by fetching data with the cursor in your function — if no data returns, skip the invocation. User-defined function development guide.

  Triggers are time-based. For example, with a 60-second interval and continuous writes to Shard 0, a function triggers every 60 seconds for that shard. Shards with no new data are skipped. The function receives the cursor range from the last 60 seconds and uses it to read shard data for processing.

Step 1: Create an SLS trigger

1. Log on to the Function Compute console. In the left-side navigation pane, click Functions.

2. In the top navigation bar, select a region. On the Functions page, click the function you want to manage.

3. On the function details page, click the Trigger tab, then click Create Trigger. In the Create Trigger panel, set Trigger Type to Log Service, configure the following parameters, and click OK.

After the trigger is created, it appears on the Triggers tab. To modify or delete a trigger, see Trigger Management.

Step 2: Configure permissions

1. On the function details page, select the Configuration tab. In the Advanced Settings section, click Modify.

2. In the Advanced Settings panel, select a Function Role:

   • Default role — AliyunFCServerlessDevsRole grants read-only access to SLS and covers most use cases.

   • Custom RAM role — if your function writes to SLS or accesses other services, create a RAM role with:

     1. Trusted entity: Cloud Service → Function Compute. See Create a RAM role for a trusted Alibaba Cloud service.

     2. Permissions: the SLS permissions your function needs. See Examples of custom RAM policies.

3. Click Deploy.

Step 3: Deploy and view logs

1. On the Code tab, paste your function code into the editor and click Deploy. The following Python example reads log data from the source Logstore using the cursor range from the trigger event. It uses temporary credentials from context.credentials to initialize the SLS client — no hardcoded keys required.

   """
   Reads log data from an SLS Logstore within the cursor range
   provided by the trigger event.
   """
   import logging
   import json
   from aliyun.log import LogClient
   
   logger = logging.getLogger()
   
   
   def handler(event, context):
       # Get temporary credentials from the function's execution role.
       creds = context.credentials
       access_key_id = creds.access_key_id
       access_key_secret = creds.access_key_secret
       security_token = creds.security_token
   
       # Parse the trigger event.
       event_obj = json.loads(event.decode())
       source = event_obj['source']
       endpoint = source['endpoint']
       log_project = source['projectName']
       log_store = source['logstoreName']
       shard_id = source['shardId']
       begin_cursor = source['beginCursor']
       end_cursor = source['endCursor']
   
       # Initialize the SLS client with temporary credentials.
       client = LogClient(
           endpoint=endpoint,
           accessKeyId=access_key_id,
           accessKey=access_key_secret,
           securityToken=security_token
       )
   
       # Pull all log groups in the range [begin_cursor, end_cursor).
       while True:
           response = client.pull_logs(
               project_name=log_project,
               logstore_name=log_store,
               shard_id=shard_id,
               cursor=begin_cursor,
               count=100,
               end_cursor=end_cursor,
               compress=False
           )
           log_group_cnt = response.get_loggroup_count()
           if log_group_cnt == 0:
               break
           logger.info("Got %d log group(s) from %s", log_group_cnt, log_store)
           logger.info(response.get_loggroup_list())
           begin_cursor = response.get_next_cursor()
   
       return 'success'

2. On the function details page, go to Logs > Function Logs to verify that the function is receiving and processing data. If the message "The logging feature is not enabled for the current function." appears, click Enable.

(Optional) Step 4: Test with a simulated event

1. On the Code tab, click the icon next to Test Function and select Configure Test Parameters.

2. In the Configure Test Parameters panel, select Create New Test Event or Modify Existing Test Event. Select the Log Service template, enter an event name, and click OK.

3. Click Test Function. After execution completes, view the result above the Code tab.

event parameter

When the trigger fires, SLS passes the following JSON object to the function's event parameter:

{
    "parameter": {},
    "source": {
        "endpoint": "http://cn-hangzhou-intranet.log.aliyuncs.com",
        "projectName": "fc-test-project",
        "logstoreName": "fc-test-logstore",
        "shardId": 0,
        "beginCursor": "MTUyOTQ4MDIwOTY1NTk3ODQ2Mw==",
        "endCursor": "MTUyOTQ4MDIwOTY1NTk3ODQ2NA=="
    },
    "jobName": "1f7043ced683de1a4e3d8d70b5a412843d81****",
    "taskId": "c2691505-38da-4d1b-998a-f1d4bb8c****",
    "cursorTime": 1529486425
}

User-configurable fields

Source fields (the log block your function reads)

System-generated fields (auto-populated by Function Compute, no configuration needed)

Related operations

• Query trigger logs

  To view trigger execution statistics, create an index on the trigger log Logstore. Create an index.

• View function execution logs

  Use the CLI to inspect function execution details. View invocation logs.

FAQ

SLS trigger doesn't invoke the function

Check that new data is being written to the configured Logstore — the trigger only fires when shard data changes. If data is flowing, check the trigger logs and function run logs for error messages.

Why is the invocation count high?

Each shard triggers independently. A Logstore with 10 shards produces 10 invocations per interval during normal operation. If the trigger is catching up after a delay (when processing falls more than 10 seconds behind), it accelerates to roughly one invocation every 2 seconds per shard until it catches up.

Error: "denied by sts or ram"

The function role is missing required SLS permissions. Review and update the role as described in Step 2: Configure permissions.