All Products
Search
Document Center

Log Service trigger

Last Updated: Oct 16, 2018

The Log Service trigger regularly subscribes to incremental data from the Logstore. The incremental data triggers Extract, Transform, and Load (ETL) of the log updates in Function Compute.

Scenarios

The Log Service trigger is suitable for Data scrubbing and ETL. Log Service allows you to quickly perform log collection, ETL, query, and analysis, as shown in the following figure:

Data ETL

The trigger supports shipping data to the destination, and building data pipelines between big data services in the cloud, as shown in the following figure:

Data shipping

Description:

  1. Build and deliver Column-based storage.
  2. Pre-process and deliver fields.
  3. Customize and store response.

Configure Log Service trigger

Trigger example

The following slsTrigger.yml is a configuration template of the Log Service trigger.

  1. triggerConfig:
  2. sourceConfig:
  3. project: "etl"
  4. logstore: "etl-log"
  5. jobConfig:
  6. maxRetryTime: 3
  7. triggerInterval: 60
  8. functionParameter:
  9. a: "b"
  10. c: "d"
  11. logConfig:
  12. project: "ali-fc-test"
  13. logstore: "test-store"
  14. enable: true

Trigger parameters

  • sourceConfig is the data source configuration parameter, and includes the following properties:
    • Logstore: The name of the data source (Logstore). The trigger regularly subscribes to data from this Logstore and sends the data to Function Compute for ETL. You cannot modify this parameter after you have created it.
  • jobConfig is the task configuration parameter, and includes the following properties:
    • triggerInterval: The interval at which the function that is triggered by Log Service runs. Value range: [3, 600], unit: second. For example, triggerInterval: 60 indicates that this function reads the locations of data that occurs in the last 60 seconds in each shard at intervals of 60 seconds. The incremental data triggers executing this function. If your Logstore shards have a high traffic volume that is more than 1 MiB/s, we recommend that you use a shorter trigger interval. Therefore, the function can process data of a reasonable size.
    • maxRetryTime: The maximum number of retries allowed for a single trigger. Value range: [0, 100]. An error may occur when Log Service triggers function execution at the specified trigger interval, such as insufficient permissions, network failures, or function execution exceptions. The function execution may still fail after the maximum number of retries. In these conditions, Log Service triggers function execution again after the trigger interval. The impact of retries on the business varies according to the specific function code logic.
  • functionParameter: The event parameter directly uses this parameter from the .yml file. Function usage depends on the custom logic of this function. Each function may require different function configurations. You must enter parameters for most default function templates according to the instructions. The default value is empty ({}).
  • logConfig is the log configuration of the trigger, and includes the following properties:
    • project: The project name in Log Service.
    • logstore: The name of the Logstore that stores log files.
  • enable is used to enable the trigger. Value range: true|false.

Event format

  1. {
  2. "parameter":{
  3. "a":"b",
  4. "c":"d"
  5. },
  6. "source":{
  7. "endpoint":"http://cn-shanghai-intranet.log.aliyuncs.com",
  8. "projectName":"vangie-fc-test",
  9. "logstoreName":"fc-test",
  10. "shardId":0,
  11. "beginCursor":"MTUyMzI2NzI5NDY1NjI4MzgzNg==",
  12. "endCursor":"MTUyMzI2NzI5NDY1NjI4MzgzNw=="
  13. },
  14. "jobName":"05c79f637c6b46eaa85911cae032cf47551af7bb",
  15. "taskId":"d22697c0-2a41-4d35-b27c-dccec8856768",
  16. "cursorTime":1523323454
  17. }
  • parameter: The function parameter that you configure for the trigger.
  • source: The information of the log block that Function Compute reads from Log Service.
    • endpoint: The region of the Log Service project.
    • projectName: The project name.
    • logstoreName: The Logstore name.
    • shardId: The specified shard in the Logstore.
    • beginCursor: The location on the shard where the triggered function starts consuming data.
    • endCursor: The location on the shard where the triggered function stops consuming data.
  • jobName: The name of the ETL job in Log Service. A Log Service trigger in Function Compute corresponds to an ETL job in Log Service.
  • taskId: The identifier of the execution of the specified function. This execution corresponds to an ETL job.
  • cursorTime: The unix_timestamp of the last log entry that reaches the backend of Log Service. The function reads this data when retrieving log entries.

Create Log Service trigger

Sample 1. By using Function Compute console

Make sure that Function Compute and Log Service are deployed in the same region. Otherwise, Function Service cannot locate Log Service when you configure the Log Service trigger. For more information, see Regions and zones.

  1. Log on to the Alibaba Cloud console.

  2. Navigate to the AliyunLogETLRole role management page, click the Confirm Authorization Policy button to grant Function Compute the AliyunLogETLRole access policy.

  3. Log on to Log Service console, create one Logstore to process log files and data sources, and create another Logstore to store the log files that are generated by Function Compute. For more information, see Log Service topic Preparation.

  4. Log on to the Function Compute console and create a Service. In the Create Service dialog box that appears:

    1. Select a region, in this sample, we use the China (Shanghai) region.

    2. Click Create Service.

    3. In the Create Service dialog box that appears, enter the service name, in this sample, the service name is log-com.

    4. Enable the Advanced Settings option.

    5. In the Log Configs, set your available Log Project and Logstore.

    6. In the Role Config, select Create new role from the Role Operation drop-down list, select AliyunLogFullAccess and AliyunLogReadOnlyAccess from the System Policies drop-down list.

    7. Click Authorize and OK to confirm your action.

  5. In the left-side navigation pane, select the new service you created.

  6. Click Create Function to go to the Create Function page.

    1. Click Select All, and select python2.7 from the drop-down list. This sample takes Python code execution for example.
    2. Click Select in Empty Function.

      Note: You can create the trigger during or after function creation. Then, you can configure the trigger. For more information, see Basic operations.

    3. Select Log Service (Log) from the Trigger Type drop-down list, set the Trigger Name, Log Project Name, Trigger Log, Invocation Interval, Retry Count, and Function Configuration configuration. In this sample, we set the trigger as follows:
      Create a trigger

      You can set the Function Configuration field according to the event parameter in your function code. In this sample, we set the Function Configuration as follows:

      1. {
      2. "source":{
      3. "endpoint": "http://cn-shanghai-intranet.log.aliyuncs.com"
      4. },
      5. "target": {
      6. "endpoint": "http://cn-shanghai-intranet.log.aliyuncs.com",
      7. "projectName": "etl-test",
      8. "logstoreName": "nginx_access_log_rep"
      9. }
      10. }
    4. Configure the Service Name, Function Name, Function Description, Runtime, and Runtime Environment parameters.

    5. Click Next.
    6. Make sure that all the settings are correct, and then click Create.

Sample 2. By using Pyhton SDK

This section takes fc-python-sdk for example:

  1. import fc2
  2. client = fc2.Client(
  3. endpoint = '<Your Endpoint>',
  4. accessKeyID = '<Your AccessKeyID>',
  5. accessKeySecret = '<Your AccessKeySecret>')
  6. service_name = '<service_name>'
  7. function_name = '<function_name>'
  8. trigger_name = '<trigger_name>'
  9. # Create log trigger
  10. log_trigger_config = {
  11. 'sourceConfig': {
  12. 'logstore': 'log_store_source'
  13. },
  14. 'jobConfig': {
  15. 'triggerInterval': 60,
  16. 'maxRetryTime': 10
  17. },
  18. 'functionParameter': {},
  19. 'logConfig': {
  20. 'project': 'log_project',
  21. 'logstore': 'log_store'
  22. },
  23. 'enable': False
  24. }
  25. source_arn = 'acs:log:cn-shanghai:12345678:project/log_project'
  26. invocation_role = 'acs:ram::12345678:role/aliyunlogetlrole'
  27. client.create_trigger('service_name', 'function_name', 'trigger_name', 'oss',
  28. log_trigger_config, source_arn, invocation_role)

References

Log Service trigger use case Demo overview.