Function Compute:Configure a Log Service trigger

Scenario

You can configure a Log Service trigger to periodically obtain updated data and trigger functions. Triggers like this are ideal for scenarios where you want to incrementally consume data from a Logstore. You can trigger functions to perform post-processing tasks, such as a data cleansing task and a data processing task, and deliver data to a third-party service. This example only shows how to obtain and print log data.

Prerequisites

• Function Compute
  • Create a Service
  • Create a function
• Log Service
  • Create a Log Service project and Logstores
  Note You must create a Log Service project and two Logstores. One Logstore is used to process logs and data sources, and the other Logstore is used to store logs that are generated by Function Compute.

Step 1: Create a Log Service trigger

1. Log on to the Function Compute console.
2. In the left-side navigation pane, click Services and Functions.
3. In the top navigation bar, select the region where the service resides.
4. On the Services page, find the desired service and click Functions in the Actions column.
5. On the Functions page, click the function that you want to manage.
6. On the function details page, click the Triggers tab, select a version or alias from the Version or Alias drop-down list, and then click Create Trigger.
7. In the Create Trigger panel, specify related parameters. After you specify the parameters, click OK.

   Field	Description	Example
   Trigger Type	Select Log Service.	Log Service
   Name	Enter a custom trigger name.	log_trigger
   Version or Alias	The default value is LATEST. If you want to create a trigger for another version or alias, select a version or alias in the upper-right corner of the function details page. For more information about versions and aliases of a service, see Manage versions and Manage aliases.	LATEST
   Log Service Project	Select a Log Service project.	aliyun-fc-cn-chengdu-2238f0df-a742-524f-9f90-976ba457****
   Logstore	Select a Logstore. The trigger created in this example subscribes to the data in the Logstore at a scheduled time and sends the data to Function Compute for custom processing.	function-log
   Trigger Interval	Enter the time interval at which Log Service triggers the function. Valid values: [3,600]. Unit: seconds. Default value: 60.	60
   Retries	Enter the maximum number of retries that are allowed for a single trigger operation. Valid values: [0,100]. Default value: 3	3
   Trigger Log	Select a Logstore to which you want to store the logs that are generated when Log Service triggers a function.	function-log2
   Invocation Parameters	You can configure a custom parameter in this field. This parameter is passed to the function as the parameter field of the event parameter. The value of this parameter must be a string in JSON format. By default, this parameter is empty.	None
   Role Name	Select AliyunLogETLRole. Note After you configure the preceding parameters, click OK. If this is the first time that you create a trigger of this type, click Authorize Now in the message that appears.	AliyunLogETLRole

   After the trigger is created, it is displayed on the Triggers tab. To modify or delete an existing trigger, see Manage triggers.

Step 2: Configure the input parameter of the function

1. On the function details page, click the Code tab and click the icon. From the drop-down list that appears, select Configure Test Parameters.
2. In the Configure Test Parameters panel, click the Create New Test Event or Modify Existing Test Event tab, and specify Event Name and the event content. After you specify the parameters, click OK.
   The event parameter is the input parameter of Function Compute. The following code shows the format of the event parameter:

   {
       "parameter": {},
       "source": {
           "endpoint": "http://cn-shanghai-intranet.log.aliyuncs.com",
           "projectName": "log-com",
           "logstoreName": "log-en",
           "shardId": 0,
           "beginCursor": "MTUyOTQ4MDIwOTY1NTk3ODQ2Mw==",
           "endCursor": "MTUyOTQ4MDIwOTY1NTk3ODQ2NA=="
       },
       "jobName": "1f7043ced683de1a4e3d8d70b5a412843d81****",
       "taskId": "c2691505-38da-4d1b-998a-f1d4bb8c****",
       "cursorTime": 1529486425
   }                       

   Field	Description	Example
   parameter	The value of Invocation Parameters.	None
   source	The log block information that you want the function to read from Log Service. endpoint: the endpoint of the Alibaba Cloud region in which the Log Service project is located. projectName: the name of the Log Service project. logstoreName: the name of the Logstore. shardId: the ID of a specified shard in the Logstore. beginCursor: the offset where data consumption starts. endCursor: the offset where data consumption ends.	{ "endpoint":"http://cn-shanghai-intranet.log.aliyuncs.com", "projectName":"log-com", "logstoreName":"log-en", "shardId":0, "beginCursor":"MTUyOTQ4MDIwOTY1NTk3ODQ2Mw==", "endCursor":"MTUyOTQ4MDIwOTY1NTk3ODQ2NA==" }
   jobName	The name of an extract, transform, and load (ETL) job in Log Service. Log Service triggers must correspond to ETL jobs in Log Service.	1f7043ced683de1a4e3d8d70b5a412843d81****
   taskId	For an ETL job, taskId is the identifier for a deterministic function invocation.	c2691505-38da-4d1b-998a-f1d4bb8c****
   cursorTime	The Unix timestamp of the time when the last log arrives at Log Service. Unit: seconds.	1529486425

Step 3: Write and test the function

After you create the Log Service trigger, you can write function code and test the function to verify whether the code is correct. Functions are triggered when Log Service collects incremental logs. Function Compute obtains the logs and prints the collected logs.

1. On the function details page, click the Code tab, enter function code in the code editor, and then click Save.
   In this example, the function code is written in Python. The following code provides a template on how to extract most logical logs. You can specify accessKeyId and accessKey in the code or obtain them from context and creds.

   # -*- coding: utf-8 -*-
   import logging
   import json
   from aliyun.log import LogClient
   from time import time
   
   def logClient(endpoint, creds):
     logger = logging.getLogger()
     logger.info('creds info')
     logger.info(creds.access_key_id)
     logger.info(creds.access_key_secret)
     logger.info(creds.security_token)
     accessKeyId = 'your accessKeyId'
     accessKey = 'your accessKeyId scr'
     client = LogClient(endpoint, accessKeyId, accessKey)
     return client
   
   def handler(event, context):
     logger = logging.getLogger()
     logger.info('start deal SLS data')
     logger.info(event.decode().encode())
     info_arr = json.loads(event.decode())
     fetchdata(info_arr['source'],context)
     return 'hello world'
   
    def fetchdata(event,context):
     logger = logging.getLogger()
     endpoint = event['endpoint']
     creds = context.credentials
     client = logClient(endpoint, creds)
     if client == None :
         logger.info("client create failed")
         return False
     project = event['projectName']
     logstore = event['logstoreName']
     start_cursor = event['beginCursor']
     end_cursor = event['endCursor']
     loggroup_count = 10
     shard_id = event['shardId']
     while True:
         res = client.pull_logs(project, logstore, shard_id, start_cursor, loggroup_count, end_cursor)
         res.log_print()
         next_cursor = res.get_next_cursor()
         if next_cursor == start_cursor :
             break
         start_cursor = next_cursor
   
       #log_data =  res.get_loggroup_json_list()
     return True                        

2. Click the Code tab and click Test Function.
   After the function is executed, you can view the result on the Code tab.

References

To modify or delete an existing trigger, see Manage triggers.