Simple Log Service:Create a custom function

Function event

A function event is a JSON-serialized string that contains the parameters for function execution. For more information, see Use Function Compute to consume log data.

• Parameters

  Parameter	Description
  jobName	The extract, transform, and load (ETL) job name in Log Service. Each ETL job corresponds to a Function Compute trigger.
  taskId	Unique identifier for a function invocation in the ETL job.
  cursorTime	UNIX timestamp of the last log entry received during the function invocation.
  source	Data range for the current function invocation. Log Service generates this value on a regular basis for each invocation. endpoint: the endpoint of the region where the project resides. Endpoints. projectName: the name of the project. logstoreName: the name of the Logstore. shardId: the ID of a shard in the Logstore. beginCursor: the start position for data consumption in the shard. endCursor: the end position for data consumption in the shard. Note The consumption interval is left-closed and right-open, in the format of [beginCursor,endCursor).
  parameter	The Function Configuration value specified when creating a trigger. This JSON object is parsed by the custom ETL function at runtime. See Simple Log Service triggers.

• Example

  {
      "source": {
          "endpoint": "http://cn-shanghai-intranet.log.aliyuncs.com", 
          "projectName": "fc-****************", 
          "logstoreName": "demo", 
          "shardId": 0, 
          "beginCursor": "MTUwNTM5MDI3NTY1ODcwNzU2Ng==", 
          "endCursor": "MTUwNTM5MDI3NTY1ODcwNzU2OA=="
      }, 
      "parameter": {
          ...
      }, 
      "jobName": "fedad35f51a2a97b466da57fd71f315f539d2234", 
      "taskId": "9bc06c96-e364-4f41-85eb-b6e579214ae4",
      "cursorTime": 1511429883
  }

  To debug a custom function, call the GetCursor operation to obtain cursors, then construct an event in the preceding format to test the function.

Function development

Log Service provides SDKs for Java, Python, Node.js, and other languages to develop custom functions. SDK reference.

The following example uses JRE 8 to demonstrate ETL function development. Java programming guide for Function Compute.

• Java function template

  Log Service provides a JRE 8-based custom function template as a starting point for your custom logic.

  Built-in features:

  • Parses the source, taskId, and jobName fields of events.

  • Uses the Log Service SDK for Java to retrieve data based on the source parameter and calls processData to process it.

  Required customizations:

  • Parse the parameter field of events by using the UserDefinedFunctionParameter.java file.

  • Define custom business logic in the processData method of the UserDefinedFunction.java file.

  • Rename UserDefinedFunction to a descriptive name that identifies your ETL function.

• processData method

  The processData method handles data consumption, transformation, and delivery. For example, LogstoreReplication.java reads data from one Logstore and writes it to another.

  Note

  • processData returns true on success or false on failure. When false is returned, the ETL function continues to run, Log Service treats the task as successful, and exception data is ignored.

  • If a fatal error or logic exception triggers throw Exception, the ETL function stops. Log Service treats the function as failed and re-invokes it based on the ETL job retry rule.

  • For high-throughput shards, allocate sufficient memory to prevent out-of-memory (OOM) errors.

  • For time-consuming functions or high-throughput shards, set a short trigger interval and a long timeout period.

  • Grant required permissions to Function Compute. For example, grant OSS write permission if the function writes data to OSS.

ETL logs

• ETL scheduling logs

  ETL scheduling logs record the start time, end time, success status, and return information of each ETL task. Errors generate log entries and trigger email or SMS alerts to the system administrator. When creating a trigger, specify a Logstore for trigger logs and enable indexes. See Create indexes.

  Use the outputStream function in JRE 8 to retrieve execution results. The function template outputs a JSON-serialized string recorded in the scheduling log for querying and analysis.

• ETL processing logs

  ETL processing logs record key information and errors for each step, including start time, end time, initialization status, and error details. Use these logs to monitor and troubleshoot ETL functions.

  Record processing logs with the context.getLogger() method. The logs are stored in a Logstore under a specified project. We recommend that you enable indexing for this Logstore.