All Products
Search

This Product

    Function Compute:Overview

    Document Center

    Function Compute:Overview

    Last Updated:Dec 07, 2023

    Simple Log Service allows you to use Function Compute to transform streaming data. You can configure an extract-transform-load (ETL) job to detect data updates and invoke functions. Then the incremental data in a Logstore is consumed and transformed.

    Scenarios

    You can use Simple Log Service triggers to integrate Function Compute and Simple Log Service in the following scenarios:

    • Data cleansing and processing

      Simple Log Service allows you to quickly collect, process, query, and analyze logs.

      Scenarios_Data scrubbing

    • Data shipping

      Simple Log Service allows you to ship data to its destination and build data pipelines between big data services on the cloud.

      Scenarios_shipping data

      1. Field preprocessing and shipping

      2. Column store creation and shipping

      3. Custom processing and result storage

    ETL functions

    • Function types

    • Trigger mechanisms

      An ETL job corresponds to a Simple Log Service trigger and is used to invoke a function. After you create an ETL job for a Logstore in Simple Log Service, a timer is started to poll data from the shards of the Logstore based on the job configurations. If data is written to the Logstore, a triple data record in the <shard_id,begin_cursor,end_cursor > format is generated as a function event. Then, the associated ETL function is invoked.

      Note

      If no new data is written to the Logstore and the storage system is updated, the cursor information will change. The ETL function is invoked for each shard but no data is transformed. In this case, you can use the cursor information to obtain data from the shards. If no data is obtained, the ETL function is invoked but no data is transformed. You can ignore the function invocations. For more information, see Create a custom function.

      An ETL job invokes functions based on the time mechanism. For example, you set the invocation interval in an ETL job to 60 seconds for a Logstore. If data is continuously written to Shard 0, the ETL function is invoked every 60 seconds to transform data that is located in the cursor range of the last 60 seconds.

    FAQ

    If you create a trigger but the ETL function is not called, you can troubleshoot the issue by using one of the following methods:
    • Check whether new data is written to the Logstore for which an ETL job is configured. If new data is written to the Logstore, the function will be called.
    • Check trigger logs and logs of function callings.