DataHub:Create a Function Compute synchronization

Create a synchronous function

1. Create a Function Compute function

1.1 Create a service

Go to the Function Compute console and create a service. If you have already created a function, skip this step.

1.2 Create a function

To create a function in the service that you just created, click Create Function and enter your code. For more information, see Function overview. On the Triggers page, click Skip, and then click Finish.

2. Service-linked role authorization

When you select the Security Token Service (STS) temporary account method, a Service-linked role for DataHub is automatically created. The DataHub service uses this role to access FC to perform data synchronization.

3. Create a DataHub topic

For more information, see Get started with DataHub synchronization.

4. Create a DataHub Function Compute connector

4.1 Go to the topic details page

4.2 Create an FC connector

4.3 Enter the configuration information

• Endpoint: The service endpoint of Function Compute. You must specify the internal service endpoint in the format https://<account_id>.fc.<region>.aliyuncs.com. For example, the endpoint of Function Compute in the China (Shanghai) region is https://12423423992.fc.cn-shanghai-internal.aliyuncs.com. For more information, see Service endpoints.

• Service: The name of the destination Function Compute service.

• Function: The name of the destination Function Compute function.

• Start Time: The position in the DataHub topic from which to start calling the function.

  5. Event structure definition

  The data sent to Function Compute has the following structure:

{
  "eventSource": "acs:datahub",
  "eventName": "acs:datahub:putRecord",
  "eventSourceARN": "/projects/test_project_name/topics/test_topic_name",
  "region": "cn-hangzhou",
  "records": [
    {
      "eventId": "0:12345",
      "systemTime": 1463000123000,
      "data": "[\"col1's value\",\"col2's value\"]"
    },
    {
      "eventId": "0:12346",
      "systemTime": 1463000156000,
      "data": "[\"col1's value\",\"col2's value\"]"
    }
  ]
}

Where:

• eventSource: The event source. This is the constant string acs:datahub.

• eventName: The event name. For data from DataHub, the event name is acs:datahub:putRecord.

• eventSourceARN: The event source identifier. It includes the DataHub project and topic names, such as /projects/test_project_name/topics/test_topic_name.

• region: The region where the source DataHub service is located, such as cn-hangzhou.

• records: The list of records that the event contains.

  • eventId: The ID of the record. The format is shardId:SequenceNumber.

  • systemTime: The UNIX timestamp in milliseconds when the event was stored in DataHub.

  • data: The data content of the event. For a Tuple topic, this field is a list. Each element in the list is a string that corresponds to the value of a field in the topic. For a BLOB topic, this field is a string.

    6. Notes

• The service domain name must be an internal service domain name. The Function Compute service name and the function name must exist.

• DataHub supports only invoking functions in synchronous mode to ensure data processing order.

• If a function has a runtime error, DataHub retries the task after 1 s, and suspends the task if it fails 512 times.

• You can view the task running status, points, detailed error messages, and more on the web console.