All Products
Search
Document Center

Simple Log Service:Use Function Compute to consume log data

Last Updated:Jan 02, 2024

Simple Log Service allows you to use Function Compute to transform streaming data. You can configure a Function Compute trigger to detect data updates and call a function. This allows you to consume the incremental data in a Logstore and transform data. You can use template functions or user-defined functions to transform data.

Prerequisites

Scenarios

  • Data cleansing and transformation

    You can use Simple Log Service to collect, transform, query, and analyze logs in an efficient manner.

  • Data shipping

    You can use Simple Log Service to ship data to a specified destination. In this scenario, Simple Log Service serves as a data channel for big data services in the cloud.

Data transformation functions

  • Function types

  • Function calling mechanism

    A Function Compute trigger is used to call a function. After you create a Function Compute trigger for a Logstore in Simple Log Service, a timer is started to monitor the shards of the Logstore based on the trigger configuration. If new data is written to the Logstore, a three-tuple data record in the <shard_id,begin_cursor,end_cursor > format is generated as a function event. Then, the function is called.

    Note

    If no new data is written to a Logstore and the storage system is updated, the cursor information may change. As a result, a function is called for each shard but no data is transformed. In this case, you can use the cursor information to obtain data from the shards. If no data is obtained, a function is called but no data is transformed. You can ignore the function calls. For more information, see Create a custom function.

    A Function Compute trigger calls a function based on time. For example, the call interval in a Function Compute trigger for a Logstore is set to 60 seconds. If data is continuously written to Shard 0, the function is called at 60-second intervals to transform data based on the cursor range of the previous 60 seconds.

Procedure

  1. Create a Function Compute service.

    For more information, see Create a service.

    Note

    The service must reside in the same region as your Simple Log Service project.

  2. Create a function.

    For more information, see Create a function.

  3. Create a trigger.

    For more information, see Configure a Simple Log Service trigger.

What to do next

  • Query trigger logs

    You can create indexes for your Logstore in which trigger logs are stored and view the execution results of the trigger. For more information, see Create indexes.

  • View the operational logs of the function

    You can use a CLI to view the detailed information about function calls. For more information, see View function invocation logs.

FAQ

  • What do I do if I create a trigger but no function is called?

    You can troubleshoot the issue by using the following methods:

    • Check whether new data is written to the Logstore for which your Function Compute trigger is configured. If new data is written to the Logstore, the function is called.

    • Check whether exceptions can be found in trigger logs and operational logs of functions.

  • Why is the call interval of a function larger than expected?

    A function is separately called for each shard. Even if the number of times that a function is called for shards in a Logstore is large, the interval at which the function is called for each shard can be consistent with the call interval that is specified.

    The call interval at which the function is called for a shard is the same as the time interval that is specified for data transformation. When a function is called, latency may exist. This may cause the call interval to be larger than expected. The following list describes two scenarios with a specified call interval of 60 seconds.

    • Scenario 1: The function is called, and latency does not exist. The function is called at 60-second intervals to transform data that is generated in the following time range: [now -60s, now).

      Note

      A function is separately called for each shard. If a Logstore contains 10 shards and latency does not exist when the function is called, the function is called 10 times at 60-second intervals to transform data in real time.

    • Scenario 2: The function is called, and latency exists. The time difference between the point in time at which data in a Simple Log Service shard is transformed and the point in time at which the latest data is written to Simple Log Service is greater than 10 seconds. In this case, the trigger shortens the call interval. For example, the function can be called at 2-second intervals to transform data that is generated within 60 seconds.