All Products
Search
Document Center

DataHub:API reference

Last Updated:Jul 24, 2023

API reference

I. API specifications

1. Common request headers

Request header

Description

x-datahub-client-version

The version number of the API.

x-datahub-security-token

The security token, if any.

Date

The time when the request was sent, in GMT. Format: EEE, dd MMM yyyy HH:mm:ss z.

Authorization

The signature string of the request.

Content-Type

The MIME type of the request.

2. Common response headers

Response header

Description

Content-Type

The MIME type of the returned message.

Content-Length

The size of the message body.

x-datahub-request-id

The globally unique identifier (GUID) of the request.

3. Error codes

Error code

Description

Remarks

InvalidParameter

The error code returned because one or more parameters are invalid.

InvalidCursor

The error code returned because the cursor is invalid.

NoSuchXXX

The error code returned because the server cannot find the requested resource.

XXXAlreadyExist

The error code returned because the specified resource already exists.

Unauthorized

The error code returned because the authentication failed.

The AccessKey pair is invalid, or the timestamp of the request does not match the timestamp recorded on the server.

NoPermission

The error code returned because you are not authorized to perform the operation.

The RAM user that you use to initiate the request does not have the associated permissions.

OperationDenied

The error code returned because the specified operation is forbidden.

The specified operation is forbidden. For example, you are not allowed to delete a project that contains one or more topics.

LimitExceeded

The error code returned because the request exceeds one or more defined limits.

The limits for the server are exceeded, such as the QPS or traffic limit.

InvalidShardOperation

The error code returned because the shard is closed for writing due to the split or merge operation.

MalformedRecord

The error code returned because the format of the requested data is invalid.

OffsetReseted

The error code returned because the consumer offset was reset.

OffsetSessionChanged

The error code returned because the subscription ID is occupied by another user.

SubscriptionOffline

The error code returned because the subscription was canceled.

InternalServerError

The error code returned because an internal error occurred.

Other error codes

Other non-retryable error codes, which may be removed in the future.

4. Unified error response format

Parameter

Description

ErrorCode

The error code returned.

ErrorMessage

The error message returned.

Sample error responses:

    HTTP/1.1 403
    x-datahub-request-id: 2018050817492199d6650a00000039
    Content-Type: application/json
    Content-Length: xxx

    {
        "ErrorCode": "Unauthorized",
        "ErrorMessage": "Authroize failed"
    }

5. Limits

Parameter

Limit

ProjectName

The project name must be 3 to 32 characters in length and can contain only letters, digits, and underscores (_). It must start with a letter. The value is not case-sensitive.

TopicName

The topic name must be 3 to 128 characters in length and can contain only letters, digits, and underscores (_). It must start with a letter. The value is not case-sensitive.

II. Signature calculations for the Authorization header

Authorization = "DATAHUB " + AccessId + ":" + Signature
Signature = base64(hmac-sha1(AccessKey,
            HTTPMethod + "\n"
            + Content-Type + "\n"
            + Date + "\n"
            + CanonicalizedDataHubHeaders + "\n"
            + CanonicalizedResource))

  • AccessKey: the AccessKey secret used to calculate the signature.

  • HTTPMethod: the method of the HTTP request, such as PUT, GET, POST, HEAD, and DELETE.

  • \n: the line feed.

  • Content-Type: the MIME type of the request body. In most cases, this parameter is set to application/json.

  • Date: the time when the request was sent, in GMT. For example, you can set this parameter to Sun, 22 Nov 2015 08:16:38 GMT.

  • CanonicalizedDataHubHeaders: the HTTP request headers prefixed with x-datahub-. The request headers are arranged in alphabetical order.

  • CanonicalizedResource: the URL of the requested DataHub resource. If the URL contains one or more parameters, the parameters must be arranged in alphabetical order.

Construction of the CanonicalizedDataHubHeaders string

All HTTP request headers prefixed with x-datahub- are called canonicalized DataHub headers. The following procedure describes how to construct the CanonicalizedDataHubHeaders string:

  1. Convert the names of all HTTP request headers that are prefixed with x-datahub- to lowercase letters. For example, convert X-DATAHUB-Client-Version:1.1 to x-datahub-client-version:1.1.

  2. If the request is sent by using the AccessKey pair provided by Security Token Service (STS), add the returned token to the signature string in the format of x-datahub-security-token:token.

  3. Sort all obtained HTTP request headers in alphabetical order.

  4. Delete all spaces on either side of a delimiter between the header and its value. For example, convert x-datahub-client-version : 1.1 to x-datahub-client-version:1.1.

  5. Separate each two pairs of headers and their values with the delimiter (\n) to obtain the CanonicalizedDataHubHeaders string.

Construction of the CanonicalizedResource string

The DataHub resources specified in the request are called canonicalized resources. The following procedure describes how to construct the CanonicalizedResource string:

  1. Set the CanonicalizedResource string to an empty string.

  2. Enter the URL of the DataHub resource that you want to access, such as /projects/<ProjectName>/topics/<TopicName>.

  3. If the URL contains one or more parameters, arrange them in alphabetical order and separate them with ampersands (&). Add a question mark (?) and the parameters at the end of the string. Example: /projects/<ProjectName>/topics/<TopicName>/connectors/sink_odps?donetime

Calculation of the signature

  • A signature string must be encoded in UTF-8. A signature string that contains Chinese characters must be encoded in UTF-8. The encoded signature string is used together with the AccessKey secret to calculate the signature.

  • Use the SHA1 algorithm defined in RFC 2104 to calculate the HMAC value of the signature. The AccessKey secret is used as the key for the HMAC calculation.

  • Headers prefixed with x-datahub- must comply with the following requirements:

  • The header names must be in lowercase.

  • The headers are sorted in alphabetical order.

  • No spaces exist on either side of a colon (:) between the header name and the value.

  • Each header is followed by a line feed (\n). If no header is used, set the CanonicalizedDataHubHeaders string to an empty string.

Examples

Request

Calculation formula of the signature string

Signature string

POST /projects/test_project/topics/test_topic HTTP/1.1 Host: http://dh-cn-hangzhou.aliyuncs.com

User-Agent: customer x-datahub-client-version: 1.1 Content-Type: application/json Date: Thu, 10 Jan 2019 07:28:29 GMT

Signature = base64(hmac-sha1(AccessKey,HTTPMethod + "\n" + Content-Type + "\n" + Date + "\n" + CanonicalizedDataHubHeaders+ CanonicalizedResource))

POST\napplication/json\nThu, 10 Jan 2019 07:28:29 GMT\nx-datahub-client-version:1.1\n/projects/test_project/topics/test_topic

The following code shows you how to calculate the signature in Python:

import base64
import hmac
import sha
h = hmac.new("****your accessKey*****",
"POST\napplication/json\nThu, 10 Jan 2019 07:28:29 GMT\nx-datahub-client-version:1.1\n/projects/test_project/topics/test_topic", sha)
Signature = base64.b64encode(h.digest())
print("Signature: %s" % Signature)

Sample request headers:

The value of the Authorization header is in the DATAHUB AccessId:Signature format.

POST /projects/test_project/topics/test_topic HTTP/1.1
Authorization: DATAHUB AccessId:Signature
Content-Type: application/json
Date: Thu, 10 Jan 2019 07:28:29 GMT
Host: http://dh-cn-hangzhou.aliyuncs.com
User-Agent: customer
x-datahub-client-version: 1.1

The following code shows you how to calculate the signature in Java 8:

import javax.crypto.Mac;
import javax.crypto.spec.SecretKeySpec;
import java.util.Base64;
import java.util.HashMap;
import java.util.Iterator;
import java.util.Map;
import java.util.SortedMap;
import java.util.TreeMap;

public abstract class Authorization {
    private static final String DEFAULT_ENCODING = "UTF-8";
    private static final String DEFAULT_HASH = "HmacSHA1";
    private static final String X_DATAHUB_PREFIX = "x-datahub";
    private static final String HEADER_CONTENT_TYPE = "Content-Type";
    private static final String HEADER_DATE = "Date";

    static public String getAkAuthorization(Request request) {
        String canonicalURL = request.getUrlPath();
        String canonicalQueryString = request.getQueryStrings();
        String canonicalHeaderString = getCanonicalHeaders(
                getSortedHeadersToSign(request.getHeaders()));

        String canonicalRequest = request.getMethod().toUpperCase() + "\n" +
                canonicalHeaderString + "\n" + canonicalURL;
        if (canonicalQueryString != null && !canonicalQueryString.isEmpty()) {
            canonicalRequest += "?" + canonicalQueryString;
        }

        String signature = HMAC1Sign(request.getAccessKey(), canonicalRequest);

        return "DATAHUB " + request.getAccessId() + ":" + signature;
    }

    static private String HMAC1Sign(String accessKey, String canonicalRequest) {
        try {
            SecretKeySpec signingKey = new SecretKeySpec(accessKey.getBytes(), DEFAULT_HASH);
            Mac mac = Mac.getInstance(DEFAULT_HASH);
            mac.init(signingKey);
            return Base64.getEncoder().encodeToString(
                    mac.doFinal(canonicalRequest.getBytes(DEFAULT_ENCODING))
            ).trim();
        } catch (Exception e) {
            throw new RuntimeException(e.getMessage(), e);
        }
    }

    static private String getCanonicalHeaders(Map<String, String> headers) {
        StringBuilder sb = new StringBuilder();
        Iterator<Map.Entry<String, String>> pairs = headers.entrySet().iterator();
        while (pairs.hasNext()) {
            Map.Entry<String, String> pair = pairs.next();
            if (pair.getKey().startsWith(X_DATAHUB_PREFIX)) {
                sb.append(pair.getKey());
                sb.append(":");
                sb.append(pair.getValue());
            } else {
                sb.append(pair.getValue());
            }

            if (pairs.hasNext()) {
                sb.append("\n");
            }
        }
        return sb.toString();
    }

    static private SortedMap<String, String> getSortedHeadersToSign(Map<String, String> headers) {
        SortedMap<String, String> sortedHeaders = new TreeMap<>();

        for (Map.Entry<String, String> entry : headers.entrySet()) {
            String lowerKey = entry.getKey().toLowerCase();
            if (lowerKey.equalsIgnoreCase(HEADER_CONTENT_TYPE) ||
                    lowerKey.equalsIgnoreCase(HEADER_DATE) ||
                    lowerKey.startsWith(X_DATAHUB_PREFIX)) {
                if (!entry.getValue().isEmpty()) {
                    sortedHeaders.put(lowerKey, entry.getValue());
                }
            }
        }

        if (!sortedHeaders.containsKey(HEADER_CONTENT_TYPE.toLowerCase())) {
            sortedHeaders.put(HEADER_CONTENT_TYPE.toLowerCase(), "");
        }

        return sortedHeaders;
    }

    public static class Request {
        private String accessId;
        private String accessKey;
        private String urlPath;
        private String method;
        private Map<String, String> headers;
        private String queryStrings;

        public String getAccessId() {
            return accessId;
        }

        public Request setAccessId(String accessId) {
            this.accessId = accessId;
            return this;
        }

        public String getAccessKey() {
            return accessKey;
        }

        public Request setAccessKey(String accessKey) {
            this.accessKey = accessKey;
            return this;
        }

        public String getUrlPath() {
            return urlPath;
        }

        public Request setUrlPath(String urlPath) {
            this.urlPath = urlPath;
            return this;
        }

        public String getMethod() {
            return method;
        }

        public Request setMethod(String method) {
            this.method = method;
            return this;
        }

        public Map<String, String> getHeaders() {
            return headers;
        }

        public Request setHeaders(Map<String, String> headers) {
            this.headers = headers;
            return this;
        }

        public String getQueryStrings() {
            return queryStrings;
        }

        public Request setQueryStrings(String queryStrings) {
            this.queryStrings = queryStrings;
            return this;
        }
    }

    public static void main(String[] args) {
        Map<String, String> headers = new HashMap<>();
        headers.put("Date", "Thu, 10 Jan 2019 07:28:29 GMT");
        headers.put("x-datahub-client-version", "1.1");
        headers.put("Content-type", "application/json");

        String accessId = "testKeyID";
        String accessKey = "testKeySecret";
        String method = "POST";
        String path = "/projects/test_project/topics/test_topic";
        String canonicalQueryString = ""; // Parameters are arranged in alphabetical order and separated by ampersands (&). Example: a=x&b=y.

        Authorization.Request authRequest = new Authorization.Request()
                .setAccessId(accessId)
                .setAccessKey(accessKey)
                .setMethod(method.toUpperCase())
                .setUrlPath(path)
                .setHeaders(headers)
                .setQueryStrings(canonicalQueryString);
        System.out.println(Authorization.getAkAuthorization(authRequest));
    }
}

III. API operations

Create a project

Request

Request syntax

  POST /projects/<ProjectName> HTTP/1.1

Request parameters

Parameter

Type

Description

Comment

String

The description of the project. The description cannot exceed 1,024 bytes.

Response

Response syntax

  HTTP/1.1 201 Created

Examples

Sample requests

  POST /projects/<ProjectName> HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString
  Content-Type: application/json
  Content-Length: xxx

  {
      "Comment": "test project"
  }

Sample success responses

  HTTP/1.1 201 Created
  x-datahub-request-id: 2018050817492199d6650a00000039
  Content-Length: 0

Query the information about a project

Request

Request syntax

  GET /projects/<ProjectName> HTTP/1.1

Response

Response syntax

  HTTP/1.1 200 OK

Response parameters

Parameter

Type

Description

CreateTime

long

The time when the project was created. Unit: seconds.

LastModifyTime

long

The last time when the project was updated. Unit: seconds.

Comment

String

The description of the project.

Examples

Sample requests

  GET /projects/<ProjectName> HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString

Sample success responses

  HTTP/1.1 200 OK
  x-datahub-request-id: 2018050817492199d6650a00000039
  Content-Type: applicaton/json
  Content-Length: xxx

  {
      "Comment": "test project",
      "CreateTime": 1525763481,
      "LastModifyTime": 1525763481
  }

Query the information about projects

Request

Request syntax

  GET /projects HTTP/1.1

Response

Response syntax

  HTTP/1.1 200 OK

Response parameters

Parameter

Type

Description

ProjectNames

List

The names of the projects.

Examples

Sample requests

  GET /projects HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString

Sample success responses

  HTTP/1.1 200 OK
  x-datahub-request-id: 2018050817492199d6650a00000039
  Content-Type: applicaton/json
  Conent-Length: xxx

  {
      "ProjectNames": [
          "project1",
          "projcet2"
      ]
  }

Update a project

Request

Request syntax

  PUT /projects/<ProjectName> HTTP/1.1

Request parameters

Parameter

Type

Description

Comment

String

The description of the project.

Response

Response syntax

  HTTP/1.1 200 OK

Examples

Sample requests

  PUT /projects/<ProjectName> HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString
  Content-Type: application/json
  Content-Length: xxx

  {
      "Comment": "update comment"
  }

Sample success responses

  HTTP/1.1 200 OK
  x-datahub-request-id: 2018050817492199d6650a00000039
  Content-Length: 0

Delete a project

Request

Request syntax

  DELETE /projects/<ProjectName> HTTP/1.1

Response

Response syntax

  HTTP/1.1 200 OK

Examples

Sample requests

  DELETE /projects/<ProjectName> HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString

Sample success responses

  HTTP/1.1 200 OK
  x-datahub-request-id: 2018050817492199d6650a00000039
  Content-Length: 0

Create a topic

Request

Request syntax

  POST /projects/<ProjectName>/topics/<TopicName> HTTP/1.1

Request parameters

Parameter

Type

Description

Action

String

The operation that you want to perform.

ShardCount

int

The initial number of shards.

Lifecycle

int

The time to live (TTL) of data.

RecordType

String

The record type. Valid values: BLOB and TUPLE. The BLOB data type is designed for unstructured data storage, and the TUPLE data type is designed for structured data storage.

RecordSchema

String

The record schema. If you set the RecordType parameter to TUPLE, you must specify the record schema. This parameter is not required if you set the RecordType parameter to BLOB.

Comment

String

The description of the topic.

ExpandMode

String

The extension mode. If you enable the extension mode, set this parameter to extend. In other cases, this parameter is not required.

Response

Response syntax

  HTTP/1.1 201 Created

Examples

Sample requests

  POST /projects/<ProjectName>/topics/<TopicName> HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString
  Content-Type: application/json
  Content-Length: xxx

  {
      "Action": "create",
      "ShardCount": 1,
      "Lifecycle": 1,
      "RecordType": "TUPLE",
      "RecordSchema": "{\"fields\":[{\"name\":\"field1\",\"type\":\"STRING\"},{\"name\":\"field2\",\"type\":\"BIGINT\"}]}}",
      "Comment": "create topic", 
      "ExpandMode": "extend"
  }

Sample success responses

  HTTP/1.1 201 Created
  x-datahub-request-id: 2018050817492199d6650a00000039
  Content-Type: applicaton/json
  Content-Length: 0

Query the information about a topic

Request

Request syntax

  GET /projects/<ProjectName>/topics/<TopicName> HTTP/1.1

Response

Response syntax

  HTTP/1.1 200 OK

Response parameters

Parameter

Type

Description

ShardCount

int

The initial number of shards.

Lifecycle

int

The TTL of data.

RecordType

String

The record type. Valid values: BLOB and TUPLE. The BLOB data type is designed for unstructured data storage, and the TUPLE data type is designed for structured data storage.

RecordSchema

String

The record schema. If you set the RecordType parameter to TUPLE, you must specify the record schema. This parameter is not required if you set the RecordType parameter to BLOB.

Comment

String

The description of the topic.

CreateTime

long

The time when the topic was created.

LastModifyTime

long

The last time when the topic was updated.

Examples

Sample requests

  GET /projects/<ProjectName>/topics/<TopicName> HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString

Sample success responses

  HTTP/1.1 201 Created
  x-datahub-request-id: 2018050817492199d6650a00000039
  Content-Type: applicaton/json
  Content-Length: xxx

  {
      "ShardCount": 1,
      "Lifecycle": 1,
      "RecordType": "TUPLE",
      "RecordSchema": "{\"fields\":[{\"name\":\"field1\",\"type\":\"STRING\"},{\"name\":\"field2\",\"type\":\"BIGINT\"}]}",
      "Comment": "create topic",
      "CreateTime": 1525763481,
      "LastModifyTime": 1525763481
  }

Query the information about topics

Request

Request syntax

  GET /projects/<ProjectName>/topics HTTP/1.1

Response

Response syntax

  HTTP/1.1 200 OK

Response parameters

Parameter

Type

Description

TopicNames

List

The names of the topics.

Examples

Sample requests

  GET /projects/<ProjectNames>/topics HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString

Sample success responses

  HTTP/1.1 200 OK
  x-datahub-request-id: 2018050817492199d6650a00000039
  Content-Type: applicaton/json
  Conent-Length: xxx

  {
      "TopicNames": [
          "topic1",
          "topic2"
      ]
  }

Update a topic

Request

Request syntax

  PUT /projects/<ProjectName>/topics/<TopicName> HTTP/1.1

Request parameters

Parameter

Type

Description

Comment

String

The description of the topic.

Response

Response syntax

  HTTP/1.1 200 OK

Examples

Sample requests

  PUT /projects/<ProjectName>/topics/<TopicName> HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString
  Content-Type: application/json
  Content-Length: xxx
  
  {
          "Comment": "update comment"
  }

Delete a topic

Request

Request syntax

  DELETE /projects/<ProjectName>/topics/<TopicName> HTTP/1.1

Response

Response syntax

  HTTP/1.1 200 OK

Examples

Sample requests

  DELETE /projects/<ProjectName>/topics/<TopicName> HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString

Sample success responses

  HTTP/1.1 200 OK
  x-datahub-request-id: 2018050817492199d6650a00000039
  Content-Length: 0

Query the information about shards

Request

Request syntax

  GET /projects/<ProjectName>/topics/<TopicName>/shards HTTP/1.1

Response

Response syntax

  HTTP/1.1 200 OK

Response parameters

Parameter

Type

Description

ShardId

String

Shard Id

State

String

The status of the shard. Examples: OPENING, ACTIVE, and CLOSED.

BeginHashKey

String

The start hash key of the hash key range.

EndHashKey

String

The end hash key of the hash key range.

ParentShardIds

List

The information about the parent shard.

Examples

Sample requests

  GET /projects/<ProjectName>/topics/<TopicName>/shards HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString

Sample success responses

  HTTP/1.1 200 OK
  x-datahub-request-id: 2018050817492199d6650a00000039
  Content-Type: applicaton/json
  Conent-Length: xxx

  {
      "Shards": [
              {
                      "ShardId": "0",
                      "State": "ACTIVE",
                      "BeginHashKey":"00000000000000000000000000000000",
                      "EndHashKey":"FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF",
                      "ParentShardIds:[]
              }
      ]
  }

Split a shard

Request

Request syntax

  POST /projects/<ProjectName>/topics/<TopicName>/shards HTTP/1.1

  • Request parameters

Parameter

Type

Description

Action

String

The operation that you want to perform. Set the value to split.

ShardId

String

The ID of the shard to be split.

SplitKey

String

The key to use for splitting the shard. Formula: SplitKey = BeginHashKey + (EndHashKey - BeginHashKey)/2.

Response

Response syntax

  HTTP/1.1 200 OK

Response parameters

Parameter

Type

Description

NewShards

List

The information about the new shards.

Examples

Sample requests

  POST /projects/<ProjectName>/topics/<TopicName>/shards HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString
  Content-Type: application/json
  Content-Length: xxx
  
  {
          "Action": "split",
          "ShardId": "0",
          "SplitKye": "7FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF"
  }

Sample success responses

  HTTP/1.1 200 OK
  x-datahub-request-id: 2018050817492199d6650a00000039
  Content-Type: applicaton/json
  Conent-Length: xxx

  {
          "NewShards": [
                  {
                          "ShardId": "1",
                          "BeginHashKey":"00000000000000000000000000000000",
                          "EndHashKey":"7FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF"
                  },
                  {
                          "ShardId":"0",
                          "BeginHashKey":"7FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF",
                          "EndHashKey":"FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF"
                  }
          ]
  }

Merge shards

Request

Request syntax

  POST /projects/<ProjectName>/topics/<TopicName>/shards HTTP/1.1

Request parameters

Parameter

Type

Description

Action

String

The operation that you want to perform. Set the value to merge.

ShardId

String

The ID of the shard to be merged.

AdjacentShardId

String

The ID of the adjacent shard for the merge operation.

Response

Response syntax

  HTTP/1.1 200 OK

Response parameters

Parameter

Type

Description

ShardId

String

The ID of the new shard.

BeginHashKey

String

The start hash key of the hash key range.

EndHashKey

String

The end hash key of the hash key range.

Examples

Sample requests

  POST /projects/<ProjectName>/topics/<TopicName>/shards HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString
  Content-Type: application/json
  Content-Length: xxx
  
  {
          "Action": "merge",
          "ShardId": "0",
          "AdjacentShardId": "1"
  }

Sample success responses

  HTTP/1.1 200 OK
        x-datahub-request-id: 2018050817492199d6650a00000039
        Content-Type: applicaton/json
        Conent-Length: xxx
        
        {
                "ShardId":"2",
                "BeginHashKey":"00000000000000000000000000000000",
                "EndHashKey":"FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF"
        }

Query a cursor

Request

Request syntax

  POST /projects/<ProjectName>/topics/<TopicName>/shards/<ShardId> HTTP/1.1

Request parameters

Parameter

Type

Description

Action

String

The operation that you want to perform. Set the value to cursor.

Type

String

The method used to obtain the cursor. Valid values: OLDEST, LATEST, SYSTEM_TIME, and SEQUENCE.

SystemTime

Int64

The system time of the cursor. Specify this parameter if you set the Type parameter to SYSTEM_TIME. Unit: milliseconds.

Sequence

Int64

The sequence number of the cursor. Specify this parameter if you set the Type parameter to SEQUENCE.

Response

Response syntax

  HTTP/1.1 200 OK

Response parameters

Parameter

Type

Description

Cursor

String

The information about the cursor.

RecordTime

Int64

The time when the data was written to DataHub. Unit: milliseconds.

Sequence

Int64

The sequence number in which the data was written. The sequence number is unique in a single shard.

Examples

Sample requests

  POST /projects/<ProjectName>/topics/<TopicName>/shards/<ShardId> HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString
  Content-Type: application/json
  Content-Length: xxx
  
  {
          "Action": "cursor",
          "Type": "SEQUENCE",
          "Sequence": 1
  }

Sample success responses

  HTTP/1.1 200 OK
  x-datahub-request-id: 2018050817492199d6650a00000039
  Content-Type: applicaton/json
  Conent-Length: xxx

  {
          "Cursor": "30005af19b3800000000000000000000",
          "RecordTime": 1525783352873,
          "Sequence": 1
  }

Write data without specifying shards

Request

Request syntax

  POST /projects/<ProjectName>/topics/<TopicName>/shards HTTP/1.1

Request parameters

Parameter

Type

Description

Action

String

The operation that you want to perform. Set the value to pub.

ShardId

String

ShardId

Attributes

Map<String, String>

The user attributes.

Data

The data to be written. For the BLOB data type, the value is a Base64-encoded string. For the TUPLE data type, the value is an array of strings.

Response

Response syntax

  HTTP/1.1 200 OK

Response parameters

Parameter

Type

Description

FailedRecordCount

Int

The number of the failed records.

FailedRecords

Array

The detailed information about the failed records.

Examples

Sample requests

  POST /projects/<ProjectName>/topics/<TopicName>/shards HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString
  Content-Type: applicaton/json
  Conent-Length: xxx

  {
          "Action": "pub",
          "Records": [
                  {
                          "ShardId": "0",
                          "Attributes": {
                                  "attr1": "value1",
                                  "attr2": "value2"
                          },
                          "Data": ["A","B","3","4"]
                  }
          ]
  }

  // BLOB
  {
          "Action": "pub",
          "Record": [
                  {
                          "ShardId": "0",
                          "Attributes": {
                                  "attr1": "value1",
                                  "attr2": "value2"
                          },
                          "Data": "Base64String"
                  }
          ]
  }

  • Sample success responses

  HTTP/1.1 200 OK
  x-datahub-request-id: 2018050817492199d6650a00000039
  Content-Type: applicaton/json
  Conent-Length: xxx

  {
      "FailedRecordCount": 1,
      "FailedRecords": [
                      {
                              "Index": 0,
                              "ErrorCode": "errorCode",
                              "ErrorMessage": "errormsg"
                      }
      ]
  }

Read data

Request

  • Request syntax

  POST /projects/<ProjectName>/topics/<TopicName>/shards/<ShardId> HTTP/1.1

  • Request parameters

Parameter

Type

Description

Action

String

The operation that you want to perform. Set the value to sub.

Cursor

String

The start cursor used to read data.

Limit

Int

The number of records to read at a time.

Response

  • Response syntax

  HTTP/1.1 200 OK

  • Response parameters

Parameter

Type

Description

NextCursor

String

The information about the cursor used to read the next record.

SystemTime

Int64

The time when the record was written to DataHub. Unit: milliseconds.

Cursor

String

The information about the cursor corresponding to the record.

Sequence

Int64

The sequence number in which the record was written to DataHub.

Attributes

Map

The user attributes.

Data

The information about the requested data.

Examples

  • Sample requests

  POST /projects/<ProjectName>/topics/<TopicName>/shards HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString
  Content-Type: applicaton/json
  Conent-Length: xxx
  
  {
          "Action": "sub",
          "Cursor": "30005af19b3800000000000000000000",
          "Limit": 1
  }

  • Sample success responses

  HTTP/1.1 200 OK
  x-datahub-request-id: 2018050817492199d6650a00000039
  Content-Type: applicaton/json
  Conent-Length: xxx

  {
      "NextCursor": "30005af19b3800000000000000090001",
      "Records": [
                      {
                              "Cursor": "30005af19b3800000000000000000000",
                              "SystemTime": 1525783352873,
                              "Sequence": 1,
                              "Attributes": {
                                      "key1": "value1",
                                      "key2": "value2"
                              },
                              "Data": ["AAA", "100"]
                      }
      ]
  }

Add a field

Request

  • Request syntax

  POST /projects/<ProjectName>/topics/<TopicName> HTTP/1.1

  • Request parameters

Parameter

Type

Description

Action

String

The operation that you want to perform. Set the value to appendfield.

FieldName

String

The name of the field.

FieldType

String

The data type of the field. Examples: STRING and BIGINT.

Response

  • Response syntax

  HTTP/1.1 200 OK

Examples

  • Sample requests

  POST /projects/<ProjectName>/topics/<TopicName> HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString
  Content-Type: applicaton/json
  Conent-Length: xxx
  
  {
          "Action": "appendfield",
          "FieldName": "field1",
          "FieldType": "BIGINT"
  }

Create a connector

Request

  • Request syntax

  POST /projects/<ProjectName>/topics/<TopicName>/connectors/<ConnectorType> HTTP/1.1

  • Request parameters

Parameter

Type

Description

Type

String

The type of the connector. Example: SINK_ODPS.

ColumnFields

Array

The fields to be synchronized.

Config

Map

The configurations of the connector.

Response

  • Response syntax

  HTTP/1.1 201 Created

Examples

  • Sample request for creating a connector of the SINK_ODPS type

  POST /projects/<ProjectName>/topics/<TopicName>/connectors/sink_odps HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString
  Content-Type: applicaton/json
  Conent-Length: xxx
  
  {
          "Type": "SINK_ODPS",
          "ColumnFields": ["field1", "field2"],
          "Config": {
                  "Project": "odpsProject",
                  "Topic": "odpsTopic",
                  "OdpsEndpoint": "xxx",
                  "TunnelEndpoint": "xxx",
                  "AccessId": "xxx",
                  "AccessKey": "xxx",
                  "PartitionMode": "SYSTEM_TIME",
                  "TimeRange": 60,
                  "PartitionConfig": {
                          "pt": "%Y%m%d",
                          "ct": "%H%M"
                  }
          }
  }

Query the information about a connector

Request

  • Request syntax

  GET /projects/<ProjectName>/topics/<TopicName>/connectors/<ConnectorType> HTTP/1.1

Response

  • Response syntax

  HTTP/1.1 200 OK

Query the information about connectors

Request

  • Request syntax

  GET /projects/<ProjectName>/topics/<TopicName>/connectors HTTP/1.1

Response

  • Response syntax

  HTTP/1.1 200 OK

Examples

  • Sample requests

  GET /projects/<ProjectName>/topics/<TopicName>/connectors HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString

  • Sample success responses

  HTTP/1.1 200 OK
  x-datahub-request-id: 2018050817492199d6650a00000039
  Content-Type: applicaton/json
  Conent-Length: xxx

  {
          "Connectors": [
                  "sink_odps", "sink_ads"
          ]
  }

Delete a connector

Request

  • Request syntax

  DELETE /projects/<ProjectName>/topics/<TopicName>/connectors/<ConnectorType> HTTP/1.1

Response

  • Response syntax

  HTTP/1.1 200 OK

Reload Connector

Request

  • Request syntax

  POST /projects/<ProjectName>/topics/<TopicName>/connectors/<ConnectorType> HTTP/1.1

  • Request parameters

Parameter

Type

Description

Action

String

The operation that you want to perform. Set the value to reload.

ShardId

String

The ID of the shard to be reloaded. If this parameter is left empty, all shards specified by the connector are reloaded.

Response

  • Response syntax

  HTTP/1.1 200 OK

Examples

  • Sample requests

  POST /projects/<ProjectName>/topics/<TopicName>/connectors/<ConnectorType> HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString
  Content-Type: applicaton/json
  Conent-Length: xxx
  
  {
          "Action": "reload"
  }

Query the status of a shard specified by a connector

Request

  • Request syntax

  POST /projects/<ProjectName>/topics/<TopicName>/connectors/<ConnectorType> HTTP/1.1

  • Request parameters

Parameter

Type

Description

Action

String

The operation that you want to perform. Set the value to status.

ShardId

String

The ID of the shard.

Response

  • Response syntax

  HTTP/1.1 200 OK

  • Response parameters

Parameter

Type

Description

CurrentSequence

Int64

The sequence number of the record.

State

Enum

The status of the shard.

LastErrorMessage

String

The error message returned if the status of the shard is not CONTEXT_EXECUTING.

DiscardCount

Int64

The total number of records dropped since the connector started.

Examples

  • Sample requests

  POST /projects/<ProjectName>/topics/<TopicName>/connectors/<ConnectorType> HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString
  Content-Type: applicaton/json
  Conent-Length: xxx
  
  {
          "Action": "status",
          "ShardId": "0"
  }

  • Sample success responses

  HTTP/1.1 200 OK
  x-datahub-request-id: 2018050817492199d6650a00000039
  Content-Type: applicaton/json
  Conent-Length: xxx

  {
          "State": "CONTEXT_EXECUTING",
          "CurrentSequence": 10,
          "DiscardCount": 0,
          "LastErrorMessage": ""
  }

Append Connector Field

Request

  • Request syntax

POST /projects//topics//connectors/ HTTP/1.1
  • Request parameters

Parameter

Type

Description

Action

String

The operation that you want to perform. Set the value to appendfield.

FieldName

String

The name of the field.

Response

  • Response syntax

  HTTP/1.1 200 OK

  • Sample requests

  POST /projects/<ProjectName>/topics/<TopicName>/connectors/<ConnectorType> HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString
  Content-Type: applicaton/json
  Conent-Length: xxx
  
  {
          "Action": "appendfiled",
          "FieldName": "field1"
  }

Create a subscription

Request

  • Request syntax

  POST /projects/<ProjectName>/topics/<TopicName>/subscriptions HTTP/1.1

  • Request parameters

Parameter

Type

Description

Action

String

The operation that you want to perform. Set the value to create.

Comment

String

The description of the subscription.

Response

  • Response syntax

  HTTP/1.1 201 Created

  • Response parameters

Parameter

Type

Description

SubId

String

The ID of the subscription.

Examples

  • Sample requests

  POST /projects/<ProjectName>/topics/<TopicName>/subscriptions HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString
  Content-Type: applicaton/json
  Conent-Length: xxx
  
  {
          "Action": "create",
          "Comment": "xxxx"
  }

  • Sample success responses

  HTTP/1.1 200 OK
        x-datahub-request-id: 2018050817492199d6650a00000039
         Content-Type: applicaton/json
  Conent-Length: xxx

        {
          "SubId": "1542078393028fzsZx"
  }

Query the information about a subscription

Request

  • Request syntax

  GET /projects/<ProjectName>/topics/<TopicName>/subscriptions/<SubId> HTTP/1.1

Response

  • Response syntax

  HTTP/1.1 200 OK

  • Response parameters

Parameter

Type

Description

SubId

String

The ID of the subscription.

State

Int

0: online, 1: offline

Comment

String

The description of the subscription.

Examples

  • Sample requests

  GET /projects/<ProjectName>/topics/<TopicName>/subscriptions/<SubId> HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT

  • Sample success responses

  HTTP/1.1 200 OK
  x-datahub-request-id: 2018050817492199d6650a00000039
  Authorization: AuthorizationString
  Content-Type: applicaton/json
  Conent-Length: xxx
  
  {
          "SubId": "xxxx",
          "Comment": "xxxx"
          "State": 1
  }

Query the information about subscriptions

Request

  • Request syntax

  POST /projects/<ProjectName>/topics/<TopicName>/subscriptions HTTP/1.1

  • Request parameters

Parameter

Type

Description

Action

String

The operation that you want to perform. Set the value to list.

PageIndex

Int

The number of the page to return.

PageSize

Int

The number of entries to return on each page.

Response

  • Response syntax

  HTTP/1.1 200 OK

  • Response parameters

Parameter

Type

Description

TotalCount

Int

The total number of pages returned.

  • Sample requests

  POST /projects/<ProjectName>/topics/<TopicName>/subscriptions HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString
  Content-Type: applicaton/json
  Conent-Length: xxx
  
  {
          "Action": "list",
          "PageIndex": 1,
          "PageSize": 10
  }

  • Sample success responses

  HTTP/1.1 200 OK
  x-datahub-request-id: 2018050817492199d6650a00000039
  Content-Type: applicaton/json
  Conent-Length: xxx
  
  {
          "Subscriptions": [
                  {
                           "Comment": "xxxx",
                            "State": 1,
                            "SubId": "1542079169844gH8HM"
                  },
                  {
                          "Comment": "xxxx",
                          "State": 1,
                          "SubId": "1542078393028fzsZx"
                  }
          ],
          "TotalCount": 2
  }

Delete a subscription

Request

  • Request syntax

  DELETE /projects/<ProjectName>/topics/<TopicName>/subscriptions/<SubId> HTTP/1.1

Examples

  • Sample requests

  DELETE /projects/<ProjectName>/topics/<TopicName>/subscriptions/<SubId> HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString

Update the status of a subscription

Request

  • Request syntax

  PUT /projects/<ProjectName>/topics/<TopicName>/subscriptions/<SubId> HTTP/1.1

Response

  • Response syntax

  HTTP/1.1 200 OK

Examples

  • Sample requests

  PUT /projects/<ProjectName>/topics/<TopicName>/subscriptions/<SubId> HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString
  Content-Type: applicaton/json
  Conent-Length: xxx
  
  {
          "State": 0
  }

Open a session for a consumer offset

Request

  • Request syntax

  POST /projects/<ProjectName>/topics/<TopicName>/subscriptions/<SubId>/offsets HTTP/1.1

  • Request parameters

Parameter

Type

Description

Action

String

The operation that you want to perform. Set the value to open.

ShardIds

Array

The IDs of the shards.

Response

  • Response syntax

  HTTP/1.1 200 OK

  • Response parameters

Parameter

Type

Description

Timestamp

Int64

The timestamp of the consumer offset. Unit: milliseconds.

Sequence

Int64

The sequence number of the consumer offset.

Version

Int64

Session VersionId

SessionId

String

SessionId

Examples

  • Sample requests

  POST /projects/<ProjectName>/topics/<TopicName>/subscriptions/<SubId>/offsets HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString
  Content-Type: applicaton/json
  Conent-Length: xxx
  
  {
          "Action": "open",
          "ShardIds": ["0"]
  }

  • Sample success responses

  HTTP/1.1 200 OK
        x-datahub-request-id: 2018050817492199d6650a00000039
  Content-Type: applicaton/json
  Conent-Length: xxx
  
  {
          "Offsets": {
                  "0": {
                          "Timestamp": 1000,
                          "Sequence": 1,
                          "Version": 1,
                          "SessionId": "xxx"
                  }
          }
  }

Query the information about consumer offsets

Request

  • Request syntax

  POST /projects/<ProjectName>/topics/<TopicName>/subscriptions/<SubId>/offsets HTTP/1.1

  • Request parameters

Parameter

Type

Description

Action

String

The operation that you want to perform. Set the value to get.

ShardIds

Array

The IDs of the shards.

Response

  • Response syntax

  HTTP/1.1 200 OK

  • Response parameters

Parameter

Type

Description

Timestamp

Int64

The timestamp of the consumer offset. Unit: milliseconds.

Sequence

Int64

The sequence number of the consumer offset.

Version

Int64

Session VersionId

SessionId

String

SessionId

Examples

  • Sample requests

  POST /projects/<ProjectName>/topics/<TopicName>/subscriptions/<SubId>/offsets HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString
  Content-Type: applicaton/json
  Conent-Length: xxx
  
  {
          "Action": "get",
          "ShardIds": ["0"]
  }

  • Sample success responses

  HTTP/1.1 200 OK
        x-datahub-request-id: 2018050817492199d6650a00000039
  Content-Type: applicaton/json
  Conent-Length: xxx
  
  {
          "Offsets": {
                  "0": {
                          "Timestamp": 1000,
                          "Sequence": 1,
                          "Version": 1,
                          "SessionId": "xxx"
                  }
          }
  }

Commit a consumer offset

Request

  • Request syntax

  PUT /projects/<ProjectName>/topics/<TopicName>/subscriptions/<SubId>/offsets HTTP/1.1

  • Request parameters

Parameter

Type

Description

Action

String

The operation that you want to perform. Set the value to commit.

Timestamp

Int64

The timestamp of the consumer offset. Unit: milliseconds.

Sequence

Int64

The sequence number of the consumer offset.

Version

Int64

Session VersionId

SessionId

Int64

SessionId

Response

  • Response syntax

  HTTP/1.1 200 OK

Examples

  • Sample requests

  PUT /projects/<ProjectName>/topics/<TopicName>/subscriptions/<SubId>/offsets HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString
  Content-Type: applicaton/json
  Conent-Length: xxx
  
  {
          "Action": "commit",
          "Offsets": {
                  "0": {
                          "Timestamp": 1000,
                          "Sequence": 1,
                          "Version": 1,
                          "SessionId": 1
                  }
          }
  }