API reference

I. API specifications

1. Common request headers

Request header	Description
x-datahub-client-version	The version number of the API.
x-datahub-security-token	The security token, if any.
Date	The time when the request was sent, which must be in the standard GMT format: EEE, dd MMM yyyy HH:mm:ss z.
Authorization	The signature string of the request.
Content-Type	The data serialization type of the request.

2. Common response headers

Response header	Description
Content-Type	The data serialization type of the returned message.
Content-Length	The size of the message body.
x-datahub-request-id	The globally unique identifier (GUID) of the request.

3. Error codes

Error code	Description	Remarks
InvalidParameter	The error code is returned because one or more parameters are invalid.
InvalidCursor	The error code is returned because the cursor is invalid.
NoSuchXXX	The error code is returned because the requested resource does not exist.
XXXAlreadyExist	The error code is returned because the requested resource already exists.
Unauthorized	The error code is returned because the authentication failed.	The AccessKey pair is invalid, or the timestamp of the request does not match the timestamp recorded on the server.
NoPermission	The error code is returned because you are not authorized to perform the operation.	The Resource Access Management (RAM) user that you use to initiate the request does not have the associated permissions.
OperationDenied	The error code is returned because the specified operation is forbidden.	The specified operation is forbidden. For example, you are not allowed to delete a project that contains one or more topics.
LimitExceeded	The error code is returned because traffic exceeds one or more defined limits.	The limits for the server are exceeded, such as the queries per second (QPS) and traffic limit.
InvalidShardOperation	The error code is returned because an error occurs when a shard is split or merged.
MalformedRecord	The error code is returned because the format of the requested data is invalid.
OffsetReseted	The error code is returned because the consumer offset was reset.
OffsetSessionChanged	The error code is returned because the subscription ID is occupied by another user.
SubscriptionOffline	The error code is returned because the subscription was canceled.
InternalServerError	The error code is returned because an internal error occurred.
Other error codes	Other non-retryable error codes, which may be removed in the future.

4. Unified error response format

Parameter	Description
ErrorCode	The error code that is returned.
ErrorMessage	The error message that is returned.

Sample error response:

    HTTP/1.1 403
    x-datahub-request-id: 2018050817492199d6650a00000039
    Content-Type: application/json
    Content-Length: xxx

    {
        "ErrorCode": "Unauthorized",
        "ErrorMessage": "Authroize failed"
    }

5. Limits

Parameter	Description
ProjectName	The project name must be 3 to 32 characters in length and can contain only letters, digits, and underscores (_). It must start with a letter. The value is not case-sensitive.
TopicName	The topic name must be 3 to 128 characters in length and can contain only letters, digits, and underscores (_). It must start with a letter. The value is not case-sensitive.

II. Signature calculations for the Authorization header

Authorization = "DATAHUB " + AccessId + ":" + Signature
Signature = base64(hmac-sha1(AccessKey,
            HTTPMethod + "\n"
            + Content-Type + "\n"
            + Date + "\n"
            + CanonicalizedDataHubHeaders + "\n"
            + CanonicalizedResource))

AccessKey: the AccessKey secret used to calculate the signature.
HTTPMethod: the method of the HTTP request, such as PUT, GET, POST, HEAD, and DELETE.
\n: the line feed.
Content-Type: the data serialization type of the request. In most cases, this parameter is set to application/json.
Date: the time when the request was sent, in GMT. For example, you can set this parameter to Sun, 22 Nov 2015 08:16:38 GMT.
CanonicalizedDataHubHeaders: the HTTP request headers prefixed with x-datahub-. The request headers are arranged in alphabetical order.
CanonicalizedResource: the URL of the requested DataHub resource. If the URL contains one or more parameters, the parameters must be arranged in alphabetical order.

Construction of the CanonicalizedDataHubHeaders string

All HTTP request headers prefixed with x-datahub- are called canonicalized DataHub headers. The following procedure describes how to construct the CanonicalizedDataHubHeaders string:

Convert the names of all HTTP request headers that are prefixed with x-datahub- to lowercase letters. For example, convert X-DATAHUB-Client-Version:1.1 to x-datahub-client-version:1.1.
If the requests are sent by using the AccessKey pair provided by Security Token Service (STS), add the returned token to the signature string in the format of x-datahub-security-token:token.
Sort all obtained HTTP request headers in alphabetical order.
Delete all spaces on both sides of the colon (:) between the request headers and their values. For example, convert x-datahub-client-versionn : 1.1 to x-datahub-client-version:1.1.
Separate a pair of request header and its value from another pair of request header and its value with the delimiter (\n) to obtain the CanonicalizedDataHubHeaders string.

Construction of the CanonicalizedResource string

The DataHub resources specified in the request are called canonicalized resources. The following procedure describes how to construct the CanonicalizedResource string:

Set the CanonicalizedResource string to an empty string.
Enter the URL of the DataHub resource that you want to access, such as /projects/<ProjectName>/topics/<TopicName>.
If the URL contains one or more parameters, arrange them in alphabetical order and separate them with ampersands (&). Add a question mark (?) and the parameters at the end of the string. Example: /projects/<ProjectName>/topics/<TopicName>/connectors/sink_odps?donetime

Calculation of the signature string

A signature string must be encoded in UTF-8. A signature string that contains Chinese characters must be encoded in UTF-8. Then, the encoded signature string is used together with the AccessKey secret to calculate the signature.
Use the SHA1 algorithm defined in RFC 2104 to calculate the hash-based message authentication code (HMAC) value of the signature. The AccessKey secret is used as the key for the HMAC calculation.
Headers prefixed with x-datahub- must comply with the following requirements:
The header names must be in lowercase.
The headers are sorted in alphabetical order.
No spaces exist on either side of a colon (:) between the header name and the value.
Each header is followed by a line feed (\n). If no header is used, set the CanonicalizedDataHubHeaders string to an empty string.

Examples

Request

Calculation formula of the signature string

Signature string

POST /projects/test_project/topics/test_topic HTTP/1.1 Host: https://dh-cn-hangzhou.aliyuncs.com

User-Agent: customer x-datahub-client-version: 1.1 Content-Type: application/json Date: Thu, 10 Jan 2019 07:28:29 GMT

Signature = base64(hmac-sha1(AccessKey,HTTPMethod + "\n" + Content-Type + "\n" + Date + "\n" + CanonicalizedDataHubHeaders+ CanonicalizedResource))

POST\napplication/json\nThu, 10 Jan 2019 07:28:29 GMT\nx-datahub-client-version:1.1\n/projects/test_project/topics/test_topic

The following code shows you how to calculate the signature in Python:

import base64
import hmac
import sha
h = hmac.new("****your accessKey*****",
"POST\napplication/json\nThu, 10 Jan 2019 07:28:29 GMT\nx-datahub-client-version:1.1\n/projects/test_project/topics/test_topic", sha)
Signature = base64.b64encode(h.digest())
print("Signature: %s" % Signature)

Sample request headers:

The value of the Authorization header is in the DATAHUB AccessId:Signature format.

POST /projects/test_project/topics/test_topic HTTP/1.1
Authorization: DATAHUB AccessId:Signature
Content-Type: application/json
Date: Thu, 10 Jan 2019 07:28:29 GMT
Host: http://dh-cn-hangzhou.aliyuncs.com
User-Agent: customer
x-datahub-client-version: 1.1

The following code shows you how to calculate the signature in Java 8:

import javax.crypto.Mac;
import javax.crypto.spec.SecretKeySpec;
import java.util.Base64;
import java.util.HashMap;
import java.util.Iterator;
import java.util.Map;
import java.util.SortedMap;
import java.util.TreeMap;

public abstract class Authorization {
    private static final String DEFAULT_ENCODING = "UTF-8";
    private static final String DEFAULT_HASH = "HmacSHA1";
    private static final String X_DATAHUB_PREFIX = "x-datahub";
    private static final String HEADER_CONTENT_TYPE = "Content-Type";
    private static final String HEADER_DATE = "Date";

    static public String getAkAuthorization(Request request) {
        String canonicalURL = request.getUrlPath();
        String canonicalQueryString = request.getQueryStrings();
        String canonicalHeaderString = getCanonicalHeaders(
                getSortedHeadersToSign(request.getHeaders()));

        String canonicalRequest = request.getMethod().toUpperCase() + "\n" +
                canonicalHeaderString + "\n" + canonicalURL;
        if (canonicalQueryString != null && !canonicalQueryString.isEmpty()) {
            canonicalRequest += "?" + canonicalQueryString;
        }

        String signature = HMAC1Sign(request.getAccessKey(), canonicalRequest);

        return "DATAHUB " + request.getAccessId() + ":" + signature;
    }

    static private String HMAC1Sign(String accessKey, String canonicalRequest) {
        try {
            SecretKeySpec signingKey = new SecretKeySpec(accessKey.getBytes(), DEFAULT_HASH);
            Mac mac = Mac.getInstance(DEFAULT_HASH);
            mac.init(signingKey);
            return Base64.getEncoder().encodeToString(
                    mac.doFinal(canonicalRequest.getBytes(DEFAULT_ENCODING))
            ).trim();
        } catch (Exception e) {
            throw new RuntimeException(e.getMessage(), e);
        }
    }

    static private String getCanonicalHeaders(Map<String, String> headers) {
        StringBuilder sb = new StringBuilder();
        Iterator<Map.Entry<String, String>> pairs = headers.entrySet().iterator();
        while (pairs.hasNext()) {
            Map.Entry<String, String> pair = pairs.next();
            if (pair.getKey().startsWith(X_DATAHUB_PREFIX)) {
                sb.append(pair.getKey());
                sb.append(":");
                sb.append(pair.getValue());
            } else {
                sb.append(pair.getValue());
            }

            if (pairs.hasNext()) {
                sb.append("\n");
            }
        }
        return sb.toString();
    }

    static private SortedMap<String, String> getSortedHeadersToSign(Map<String, String> headers) {
        SortedMap<String, String> sortedHeaders = new TreeMap<>();

        for (Map.Entry<String, String> entry : headers.entrySet()) {
            String lowerKey = entry.getKey().toLowerCase();
            if (lowerKey.equalsIgnoreCase(HEADER_CONTENT_TYPE) ||
                    lowerKey.equalsIgnoreCase(HEADER_DATE) ||
                    lowerKey.startsWith(X_DATAHUB_PREFIX)) {
                if (!entry.getValue().isEmpty()) {
                    sortedHeaders.put(lowerKey, entry.getValue());
                }
            }
        }

        if (!sortedHeaders.containsKey(HEADER_CONTENT_TYPE.toLowerCase())) {
            sortedHeaders.put(HEADER_CONTENT_TYPE.toLowerCase(), "");
        }

        return sortedHeaders;
    }

    public static class Request {
        private String accessId;
        private String accessKey;
        private String urlPath;
        private String method;
        private Map<String, String> headers;
        private String queryStrings;

        public String getAccessId() {
            return accessId;
        }

        public Request setAccessId(String accessId) {
            this.accessId = accessId;
            return this;
        }

        public String getAccessKey() {
            return accessKey;
        }

        public Request setAccessKey(String accessKey) {
            this.accessKey = accessKey;
            return this;
        }

        public String getUrlPath() {
            return urlPath;
        }

        public Request setUrlPath(String urlPath) {
            this.urlPath = urlPath;
            return this;
        }

        public String getMethod() {
            return method;
        }

        public Request setMethod(String method) {
            this.method = method;
            return this;
        }

        public Map<String, String> getHeaders() {
            return headers;
        }

        public Request setHeaders(Map<String, String> headers) {
            this.headers = headers;
            return this;
        }

        public String getQueryStrings() {
            return queryStrings;
        }

        public Request setQueryStrings(String queryStrings) {
            this.queryStrings = queryStrings;
            return this;
        }
    }

    public static void main(String[] args) {
        Map<String, String> headers = new HashMap<>();
        headers.put("Date", "Thu, 10 Jan 2019 07:28:29 GMT");
        headers.put("x-datahub-client-version", "1.1");
        headers.put("Content-type", "application/json");

        String accessId = "testKeyID";
        String accessKey = "testKeySecret";
        String method = "POST";
        String path = "/projects/test_project/topics/test_topic";
        String canonicalQueryString = ""; // Parameters are arranged in alphabetical order and separated by ampersands (&). Example: a=x&b=y.

        Authorization.Request authRequest = new Authorization.Request()
                .setAccessId(accessId)
                .setAccessKey(accessKey)
                .setMethod(method.toUpperCase())
                .setUrlPath(path)
                .setHeaders(headers)
                .setQueryStrings(canonicalQueryString);
        System.out.println(Authorization.getAkAuthorization(authRequest));
    }
}

III. API operations

Create a project

Request

Request syntax

  POST /projects/<ProjectName> HTTP/1.1

Request parameters

Parameter	Type	Description
Comment	String	The description of the project. The description cannot exceed 1,024 bytes.

Response

Response syntax

  HTTP/1.1 201 Created

Examples

Sample requests

  POST /projects/<ProjectName> HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString
  Content-Type: application/json
  Content-Length: xxx

  {
      "Comment": "test project"
  }

Sample success responses

  HTTP/1.1 201 Created
  x-datahub-request-id: 2018050817492199d6650a00000039
  Content-Length: 0

Query the information about a project

Request

Request syntax

  GET /projects/<ProjectName> HTTP/1.1

Response

Response syntax

  HTTP/1.1 200 OK

Response parameters

Parameter	Type	Description
CreateTime	long	The time when the project was created. Unit: seconds.
LastModifyTime	long	The time when the project was last updated. Unit: seconds.
Comment	String	The description of the project.

Examples

Sample requests

  GET /projects/<ProjectName> HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString

Sample success responses

  HTTP/1.1 200 OK
  x-datahub-request-id: 2018050817492199d6650a00000039
  Content-Type: applicaton/json
  Content-Length: xxx

  {
      "Comment": "test project",
      "CreateTime": 1525763481,
      "LastModifyTime": 1525763481
  }

Query the information about projects

Request

Request syntax

  GET /projects HTTP/1.1

Response

Response syntax

  HTTP/1.1 200 OK

Response parameters

Parameter	Type	Description
ProjectNames	List	The names of the projects.

Examples

Sample requests

  GET /projects HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString

Sample success responses

  HTTP/1.1 200 OK
  x-datahub-request-id: 2018050817492199d6650a00000039
  Content-Type: applicaton/json
  Conent-Length: xxx

  {
      "ProjectNames": [
          "project1",
          "projcet2"
      ]
  }

Update a project

Request

Request syntax

  PUT /projects/<ProjectName> HTTP/1.1

Request parameters

Parameter	Type	Description
Comment	String	The description of the project.

Response

Response syntax

  HTTP/1.1 200 OK

Examples

Sample requests

  PUT /projects/<ProjectName> HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString
  Content-Type: application/json
  Content-Length: xxx

  {
      "Comment": "update comment"
  }

Sample success responses

  HTTP/1.1 200 OK
  x-datahub-request-id: 2018050817492199d6650a00000039
  Content-Length: 0

Delete a project

Request

Request syntax

  DELETE /projects/<ProjectName> HTTP/1.1

Response

Response syntax

  HTTP/1.1 200 OK

Examples

Sample requests

  DELETE /projects/<ProjectName> HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString

Sample success responses

  HTTP/1.1 200 OK
  x-datahub-request-id: 2018050817492199d6650a00000039
  Content-Length: 0

Create a topic

Request

Request syntax

  POST /projects/<ProjectName>/topics/<TopicName> HTTP/1.1

Request parameters

Parameter	Type	Description
Action	String	The operation that you want to perform.
ShardCount	int	The initial number of shards.
Lifecycle	int	The time to live (TTL) of data.
RecordType	String	The record type. Valid values: BLOB and TUPLE. The BLOB data type is designed for unstructured data storage, and the TUPLE data type is designed for structured data storage.
RecordSchema	String	The record schema. If you set the RecordType parameter to TUPLE, you must specify the record schema. If you set the RecordType parameter to BLOB, this parameter is not required.
Comment	String	The description of the topic.
ExpandMode	String	The extension mode. If you enable the extension mode, set this parameter to extend. In other cases, this parameter is not required.

Response

Response syntax

  HTTP/1.1 201 Created

Examples

Sample requests

  POST /projects/<ProjectName>/topics/<TopicName> HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString
  Content-Type: application/json
  Content-Length: xxx

  {
      "Action": "create",
      "ShardCount": 1,
      "Lifecycle": 1,
      "RecordType": "TUPLE",
      "RecordSchema": "{\"fields\":[{\"name\":\"field1\",\"type\":\"STRING\"},{\"name\":\"field2\",\"type\":\"BIGINT\"}]}}",
      "Comment": "create topic", 
      "ExpandMode": "extend"
  }

Sample success responses

  HTTP/1.1 201 Created
  x-datahub-request-id: 2018050817492199d6650a00000039
  Content-Type: applicaton/json
  Content-Length: 0

Query the information about a topic

Request

Request syntax

  GET /projects/<ProjectName>/topics/<TopicName> HTTP/1.1

Response

Response syntax

  HTTP/1.1 200 OK

Response parameters

Parameter	Type	Description
ShardCount	int	The initial number of shards.
Lifecycle	int	The TTL of data.
RecordType	String	The record type. Valid values: BLOB and TUPLE. The BLOB data type is designed for unstructured data storage, and the TUPLE data type is designed for structured data storage.
RecordSchema	String	The record schema. If you set the RecordType parameter to TUPLE, you must specify the record schema. If you set the RecordType parameter to BLOB, this parameter is not required.
Comment	String	The description of the topic.
CreateTime	long	The time when the topic was created.
LastModifyTime	long	The time when the topic was last updated.

Examples

Sample requests

  GET /projects/<ProjectName>/topics/<TopicName> HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString

Sample success responses

  HTTP/1.1 201 Created
  x-datahub-request-id: 2018050817492199d6650a00000039
  Content-Type: applicaton/json
  Content-Length: xxx

  {
      "ShardCount": 1,
      "Lifecycle": 1,
      "RecordType": "TUPLE",
      "RecordSchema": "{\"fields\":[{\"name\":\"field1\",\"type\":\"STRING\"},{\"name\":\"field2\",\"type\":\"BIGINT\"}]}",
      "Comment": "create topic",
      "CreateTime": 1525763481,
      "LastModifyTime": 1525763481
  }

Query the information about topics

Request

Request syntax

  GET /projects/<ProjectName>/topics HTTP/1.1

Response

Response syntax

  HTTP/1.1 200 OK

Response parameters

Parameter	Type	Description
TopicNames	List	The names of the topics.

Examples

Sample requests

  GET /projects/<ProjectNames>/topics HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString

Sample success responses

  HTTP/1.1 200 OK
  x-datahub-request-id: 2018050817492199d6650a00000039
  Content-Type: applicaton/json
  Conent-Length: xxx

  {
      "TopicNames": [
          "topic1",
          "topic2"
      ]
  }

Update a topic

Request

Request syntax

  PUT /projects/<ProjectName>/topics/<TopicName> HTTP/1.1

Request parameters

Parameter	Type	Description
Comment	String	The description of the topic.

Response

Response syntax

  HTTP/1.1 200 OK

Examples

Sample requests

  PUT /projects/<ProjectName>/topics/<TopicName> HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString
  Content-Type: application/json
  Content-Length: xxx
  
  {
          "Comment": "update comment"
  }

Delete a topic

Request

Request syntax

  DELETE /projects/<ProjectName>/topics/<TopicName> HTTP/1.1

Response

Response syntax

  HTTP/1.1 200 OK

Examples

Sample requests

  DELETE /projects/<ProjectName>/topics/<TopicName> HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString

Sample success responses

  HTTP/1.1 200 OK
  x-datahub-request-id: 2018050817492199d6650a00000039
  Content-Length: 0

Query the information about shards

Request

Request syntax

  GET /projects/<ProjectName>/topics/<TopicName>/shards HTTP/1.1

Response

Response syntax

  HTTP/1.1 200 OK

Response parameters

Parameter	Type	Description
ShardId	String	The shard ID.
State	String	The shard state. Example: OPENING, ACTIVE, and CLOSED.
BeginHashKey	String	The start hash key of the hash key range.
EndHashKey	String	The end hash key of the hash key range.
ParentShardIds	List	The information about the parent shard.

Examples

Sample requests

  GET /projects/<ProjectName>/topics/<TopicName>/shards HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString

Sample success responses

  HTTP/1.1 200 OK
  x-datahub-request-id: 2018050817492199d6650a00000039
  Content-Type: applicaton/json
  Conent-Length: xxx

  {
      "Shards": [
              {
                      "ShardId": "0",
                      "State": "ACTIVE",
                      "BeginHashKey":"00000000000000000000000000000000",
                      "EndHashKey":"FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF",
                      "ParentShardIds:[]
              }
      ]
  }

Split a shard

Request

Request syntax

  POST /projects/<ProjectName>/topics/<TopicName>/shards HTTP/1.1

Request parameters

Parameter	Type	Description
Action	String	The operation that you want to perform. Set the value to split.
ShardId	String	The ID of the shard that you want to split.
SplitKey	String	The key to use for splitting the shard. Formula: SplitKey = BeginHashKey + (EndHashKey - BeginHashKey)/2.

Response

Response syntax

  HTTP/1.1 200 OK

Response parameters

Parameter	Type	Description
NewShards	List	The information about the new shards.

Examples

Sample requests

  POST /projects/<ProjectName>/topics/<TopicName>/shards HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString
  Content-Type: application/json
  Content-Length: xxx
  
  {
          "Action": "split",
          "ShardId": "0",
          "SplitKye": "7FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF"
  }

Sample success responses

  HTTP/1.1 200 OK
  x-datahub-request-id: 2018050817492199d6650a00000039
  Content-Type: applicaton/json
  Conent-Length: xxx

  {
          "NewShards": [
                  {
                          "ShardId": "1",
                          "BeginHashKey":"00000000000000000000000000000000",
                          "EndHashKey":"7FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF"
                  },
                  {
                          "ShardId":"0",
                          "BeginHashKey":"7FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF",
                          "EndHashKey":"FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF"
                  }
          ]
  }

Merge shards

Request

Request syntax

  POST /projects/<ProjectName>/topics/<TopicName>/shards HTTP/1.1

Request parameters

Parameter	Type	Description
Action	String	The operation that you want to perform. Set the value to merge.
ShardId	String	The ID of the shard that you want to merge.
AdjacentShardId	String	The ID of the adjacent shard for the merge operation.

Response

Response syntax

  HTTP/1.1 200 OK

Response parameters

Parameter	Type	Description
ShardId	String	The ID of the new shard.
BeginHashKey	String	The start hash key of the hash key range.
EndHashKey	String	The end hash key of the hash key range.

Examples

Sample requests

  POST /projects/<ProjectName>/topics/<TopicName>/shards HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString
  Content-Type: application/json
  Content-Length: xxx
  
  {
          "Action": "merge",
          "ShardId": "0",
          "AdjacentShardId": "1"
  }

Sample success responses

  HTTP/1.1 200 OK
        x-datahub-request-id: 2018050817492199d6650a00000039
        Content-Type: applicaton/json
        Conent-Length: xxx
        
        {
                "ShardId":"2",
                "BeginHashKey":"00000000000000000000000000000000",
                "EndHashKey":"FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF"
        }

Query a cursor

Request

Request syntax

  POST /projects/<ProjectName>/topics/<TopicName>/shards/<ShardId> HTTP/1.1

Request parameters

Parameter	Type	Description
Action	String	The operation that you want to perform. Set the value to cursor.
Type	String	The method that you use to query the cursor. Valid values: OLDEST, LATEST, SYSTEM_TIME, and SEQUENCE.
SystemTime	Int64	The system time of the cursor. Specify this parameter if you set the Type parameter to SYSTEM_TIME. Unit: millisecond.
Sequence	Int64	The sequence number of the cursor. Specify this parameter if you set the Type parameter to SEQUENCE.

Response

Response syntax

  HTTP/1.1 200 OK

Response parameters

Parameter	Type	Description
Cursor	String	The information about the cursor.
RecordTime	Int64	The time when the data was written to DataHub. Unit: millisecond.
Sequence	Int64	The sequence number in which the data was written. The sequence number is unique in a single shard.

Examples

Sample requests

  POST /projects/<ProjectName>/topics/<TopicName>/shards/<ShardId> HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString
  Content-Type: application/json
  Content-Length: xxx
  
  {
          "Action": "cursor",
          "Type": "SEQUENCE",
          "Sequence": 1
  }

Sample success responses

  HTTP/1.1 200 OK
  x-datahub-request-id: 2018050817492199d6650a00000039
  Content-Type: applicaton/json
  Conent-Length: xxx

  {
          "Cursor": "30005af19b3800000000000000000000",
          "RecordTime": 1525783352873,
          "Sequence": 1
  }

Write data without specifying shards

Request

Request syntax

  POST /projects/<ProjectName>/topics/<TopicName>/shards HTTP/1.1

Request parameters

Parameter	Type	Description
Action	String	The operation that you want to perform. Set the value to pub.
ShardId	String	The shard ID.
Attributes	Map<String, String>	The user attributes.
Data		The data that you want to write. For the BLOB data type, the value is a Base64-encoded string. For the TUPLE data type, the value is an array of strings.

Response

Response syntax

  HTTP/1.1 200 OK

Response parameters

Parameter	Type	Description
FailedRecordCount	Int	The number of the failure records.
FailedRecords	Array	The detailed information about the failure records.

Examples

Sample requests

  POST /projects/<ProjectName>/topics/<TopicName>/shards HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString
  Content-Type: applicaton/json
  Conent-Length: xxx

  {
          "Action": "pub",
          "Records": [
                  {
                          "ShardId": "0",
                          "Attributes": {
                                  "attr1": "value1",
                                  "attr2": "value2"
                          },
                          "Data": ["A","B","3","4"]
                  }
          ]
  }

  // BLOB
  {
          "Action": "pub",
          "Record": [
                  {
                          "ShardId": "0",
                          "Attributes": {
                                  "attr1": "value1",
                                  "attr2": "value2"
                          },
                          "Data": "Base64String"
                  }
          ]
  }

Sample success responses

  HTTP/1.1 200 OK
  x-datahub-request-id: 2018050817492199d6650a00000039
  Content-Type: applicaton/json
  Conent-Length: xxx

  {
      "FailedRecordCount": 1,
      "FailedRecords": [
                      {
                              "Index": 0,
                              "ErrorCode": "errorCode",
                              "ErrorMessage": "errormsg"
                      }
      ]
  }

Read data

Request

Request syntax

  POST /projects/<ProjectName>/topics/<TopicName>/shards/<ShardId> HTTP/1.1

Request parameters

Parameter	Type	Description
Action	String	The operation that you want to perform. Set the value to sub.
Cursor	String	The start cursor that is used to read records.
Limit	Int	The number of records to read at a time.

Response

Response syntax

  HTTP/1.1 200 OK

Response parameters

Parameter	Type	Description
NextCursor	String	The information about the cursor used to read the next record.
SystemTime	Int64	The time when the record was written to DataHub. Unit: millisecond.
Cursor	String	The information about the cursor that corresponds to the record.
Sequence	Int64	The sequence number in which the record was written to DataHub.
Attributes	Map	The user attributes.
Data		The information about the requested data.

Examples

Sample requests

  POST /projects/<ProjectName>/topics/<TopicName>/shards HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString
  Content-Type: applicaton/json
  Conent-Length: xxx
  
  {
          "Action": "sub",
          "Cursor": "30005af19b3800000000000000000000",
          "Limit": 1
  }

Sample success responses

  HTTP/1.1 200 OK
  x-datahub-request-id: 2018050817492199d6650a00000039
  Content-Type: applicaton/json
  Conent-Length: xxx

  {
      "NextCursor": "30005af19b3800000000000000090001",
      "Records": [
                      {
                              "Cursor": "30005af19b3800000000000000000000",
                              "SystemTime": 1525783352873,
                              "Sequence": 1,
                              "Attributes": {
                                      "key1": "value1",
                                      "key2": "value2"
                              },
                              "Data": ["AAA", "100"]
                      }
      ]
  }

Add a field

Request

Request syntax

  POST /projects/<ProjectName>/topics/<TopicName> HTTP/1.1

Request parameters

Parameter	Type	Description
Action	String	The operation that you want to perform. Set the value to appendfield.
FieldName	String	The field name.
FieldType	String	The data type of the field. Example: STRING and BIGINT.

Response

Response syntax

  HTTP/1.1 200 OK

Examples

Sample requests

  POST /projects/<ProjectName>/topics/<TopicName> HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString
  Content-Type: applicaton/json
  Conent-Length: xxx
  
  {
          "Action": "appendfield",
          "FieldName": "field1",
          "FieldType": "BIGINT"
  }

Create a connector

Request

Request syntax

  POST /projects/<ProjectName>/topics/<TopicName>/connectors/<ConnectorType> HTTP/1.1

Request parameters

Parameter	Type	Description
Type	String	The connector type. Example: SINK_ODPS.
ColumnFields	Array	The fields that you want to synchronize.
Config	Map	The configurations of the connector.

Response

Response syntax

  HTTP/1.1 201 Created

Examples

Sample request for creating a connector of the SINK_ODPS type

  POST /projects/<ProjectName>/topics/<TopicName>/connectors/sink_odps HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString
  Content-Type: applicaton/json
  Conent-Length: xxx
  
  {
          "Type": "SINK_ODPS",
          "ColumnFields": ["field1", "field2"],
          "Config": {
                  "Project": "odpsProject",
                  "Topic": "odpsTopic",
                  "OdpsEndpoint": "xxx",
                  "TunnelEndpoint": "xxx",
                  "AccessId": "xxx",
                  "AccessKey": "xxx",
                  "PartitionMode": "SYSTEM_TIME",
                  "TimeRange": 60,
                  "PartitionConfig": {
                          "pt": "%Y%m%d",
                          "ct": "%H%M"
                  }
          }
  }

Query the information about a connector

Request

Request syntax

  GET /projects/<ProjectName>/topics/<TopicName>/connectors/<ConnectorType> HTTP/1.1

Response

Response syntax

  HTTP/1.1 200 OK

Query the information about connectors

Request

Request syntax

  GET /projects/<ProjectName>/topics/<TopicName>/connectors HTTP/1.1

Response

Response syntax

  HTTP/1.1 200 OK

Examples

Sample requests

  GET /projects/<ProjectName>/topics/<TopicName>/connectors HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString

Sample success responses

  HTTP/1.1 200 OK
  x-datahub-request-id: 2018050817492199d6650a00000039
  Content-Type: applicaton/json
  Conent-Length: xxx

  {
          "Connectors": [
                  "sink_odps", "sink_ads"
          ]
  }

Delete a connector

Request

Request syntax

  DELETE /projects/<ProjectName>/topics/<TopicName>/connectors/<ConnectorType> HTTP/1.1

Response

Response syntax

  HTTP/1.1 200 OK

Reload a connector

Request

Request syntax

  POST /projects/<ProjectName>/topics/<TopicName>/connectors/<ConnectorType> HTTP/1.1

Request parameters

Parameter	Type	Description
Action	String	The operation that you want to perform. Set the value to reload.
ShardId	String	The ID of the shard that you want to reload. If this parameter is left empty, reload all shards specified by the connector.

Response

Response syntax

  HTTP/1.1 200 OK

Examples

Sample requests

  POST /projects/<ProjectName>/topics/<TopicName>/connectors/<ConnectorType> HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString
  Content-Type: applicaton/json
  Conent-Length: xxx
  
  {
          "Action": "reload"
  }

Query the shard state for a connector

Request

Request syntax

  POST /projects/<ProjectName>/topics/<TopicName>/connectors/<ConnectorType> HTTP/1.1

Request parameters

Parameter	Type	Description
Action	String	The operation that you want to perform. Set the value to status.
ShardId	String	The shard ID.

Response

Response syntax

  HTTP/1.1 200 OK

Response parameters

Parameter	Type	Description
CurrentSequence	Int64	The sequence number of the consumer offset.
State	Enum	The shard state.
LastErrorMessage	String	The error message that is returned if the shard state is not CONTEXT_EXECUTING.
DiscardCount	Int64	The total number of records that were dropped since the connector started.

Examples

Sample requests

  POST /projects/<ProjectName>/topics/<TopicName>/connectors/<ConnectorType> HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString
  Content-Type: applicaton/json
  Conent-Length: xxx
  
  {
          "Action": "status",
          "ShardId": "0"
  }

Sample success responses

  HTTP/1.1 200 OK
  x-datahub-request-id: 2018050817492199d6650a00000039
  Content-Type: applicaton/json
  Conent-Length: xxx

  {
          "State": "CONTEXT_EXECUTING",
          "CurrentSequence": 10,
          "DiscardCount": 0,
          "LastErrorMessage": ""
  }

Add a field for a connector

Request

Request syntax

POST /projects//topics//connectors/ HTTP/1.1

Request parameters

Parameter	Type	Description
Action	String	The operation that you want to perform. Set the value to appendfield.
FieldName	String	The field name.

Response

Response syntax

  HTTP/1.1 200 OK

Sample requests

  POST /projects/<ProjectName>/topics/<TopicName>/connectors/<ConnectorType> HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString
  Content-Type: applicaton/json
  Conent-Length: xxx
  
  {
          "Action": "appendfiled",
          "FieldName": "field1"
  }

Create a subscription

Request

Request syntax

  POST /projects/<ProjectName>/topics/<TopicName>/subscriptions HTTP/1.1

Request parameters

Parameter	Type	Description
Action	String	The operation that you want to perform. Set the value to create.
Comment	String	The description of the subscription.

Response

Response syntax

  HTTP/1.1 201 Created

Response parameters

Parameter	Type	Description
SubId	String	The subscription ID.

Examples

Sample requests

  POST /projects/<ProjectName>/topics/<TopicName>/subscriptions HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString
  Content-Type: applicaton/json
  Conent-Length: xxx
  
  {
          "Action": "create",
          "Comment": "xxxx"
  }

Sample success responses

  HTTP/1.1 200 OK
        x-datahub-request-id: 2018050817492199d6650a00000039
         Content-Type: applicaton/json
  Conent-Length: xxx

        {
          "SubId": "1542078393028fzsZx"
  }

Query the information about a subscription

Request

Request syntax

  GET /projects/<ProjectName>/topics/<TopicName>/subscriptions/<SubId> HTTP/1.1

Response

Response syntax

  HTTP/1.1 200 OK

Response parameters

Parameter	Type	Description
SubId	String	The subscription ID.
State	Int	The subscription state. Valid values 0 and 1. 0 indicates that the subscription is online, and 1 indicates that the subscription is offline.
Comment	String	The description of the subscription.

Examples

Sample requests

  GET /projects/<ProjectName>/topics/<TopicName>/subscriptions/<SubId> HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT

Sample success responses

  HTTP/1.1 200 OK
  x-datahub-request-id: 2018050817492199d6650a00000039
  Authorization: AuthorizationString
  Content-Type: applicaton/json
  Conent-Length: xxx
  
  {
          "SubId": "xxxx",
          "Comment": "xxxx"
          "State": 1
  }

Query the information about subscriptions

Request

Request syntax

  POST /projects/<ProjectName>/topics/<TopicName>/subscriptions HTTP/1.1

Request parameters

Parameter	Type	Description
Action	String	The operation that you want to perform. Set the value to list.
PageIndex	Int	The page number.
PageSize	Int	The number of entries per page.

Response

Response syntax

  HTTP/1.1 200 OK

Response parameters

Parameter	Type	Description
TotalCount	Int	The total number of entries returned.

Sample requests

  POST /projects/<ProjectName>/topics/<TopicName>/subscriptions HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString
  Content-Type: applicaton/json
  Conent-Length: xxx
  
  {
          "Action": "list",
          "PageIndex": 1,
          "PageSize": 10
  }

Sample success responses

  HTTP/1.1 200 OK
  x-datahub-request-id: 2018050817492199d6650a00000039
  Content-Type: applicaton/json
  Conent-Length: xxx
  
  {
          "Subscriptions": [
                  {
                           "Comment": "xxxx",
                            "State": 1,
                            "SubId": "1542079169844gH8HM"
                  },
                  {
                          "Comment": "xxxx",
                          "State": 1,
                          "SubId": "1542078393028fzsZx"
                  }
          ],
          "TotalCount": 2
  }

Delete a subscription

Request

Request syntax

  DELETE /projects/<ProjectName>/topics/<TopicName>/subscriptions/<SubId> HTTP/1.1

Examples

Sample requests

  DELETE /projects/<ProjectName>/topics/<TopicName>/subscriptions/<SubId> HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString

Update the subscription status

Request

Request syntax

  PUT /projects/<ProjectName>/topics/<TopicName>/subscriptions/<SubId> HTTP/1.1

Response

Response syntax

  HTTP/1.1 200 OK

Examples

Sample requests

  PUT /projects/<ProjectName>/topics/<TopicName>/subscriptions/<SubId> HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString
  Content-Type: applicaton/json
  Conent-Length: xxx
  
  {
          "State": 0
  }

Create a session for a consumer offset

Request

Request syntax

  POST /projects/<ProjectName>/topics/<TopicName>/subscriptions/<SubId>/offsets HTTP/1.1

Request parameters

Parameter	Type	Description
Action	String	The operation that you want to perform. Set the value to open.
ShardIds	Array	The IDs of the shards.

Response

Response syntax

  HTTP/1.1 200 OK

Response parameters

Parameter	Type	Description
Timestamp	Int64	The timestamp of the consumer offset. Unit: millisecond.
Sequence	Int64	The sequence number of the consumer offset.
Version	Int64	The session version.
SessionId	String	The session ID.

Examples

Sample requests

  POST /projects/<ProjectName>/topics/<TopicName>/subscriptions/<SubId>/offsets HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString
  Content-Type: applicaton/json
  Conent-Length: xxx
  
  {
          "Action": "open",
          "ShardIds": ["0"]
  }

Sample success responses

  HTTP/1.1 200 OK
        x-datahub-request-id: 2018050817492199d6650a00000039
  Content-Type: applicaton/json
  Conent-Length: xxx
  
  {
          "Offsets": {
                  "0": {
                          "Timestamp": 1000,
                          "Sequence": 1,
                          "Version": 1,
                          "SessionId": "xxx"
                  }
          }
  }

Query the information about a consumer offset

Request

Request syntax

  POST /projects/<ProjectName>/topics/<TopicName>/subscriptions/<SubId>/offsets HTTP/1.1

Request parameters

Parameter	Type	Description
Action	String	The operation that you want to perform. Set the value to get.
ShardIds	Array	The IDs of the shards.

Response

Response syntax

  HTTP/1.1 200 OK

Response parameters

Parameter	Type	Description
Timestamp	Int64	The timestamp of the consumer offset. Unit: millisecond.
Sequence	Int64	The sequence number of the consumer offset.
Version	Int64	The session version.
SessionId	String	The session ID.

Examples

Sample requests

  POST /projects/<ProjectName>/topics/<TopicName>/subscriptions/<SubId>/offsets HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString
  Content-Type: applicaton/json
  Conent-Length: xxx
  
  {
          "Action": "get",
          "ShardIds": ["0"]
  }

Sample success responses

  HTTP/1.1 200 OK
        x-datahub-request-id: 2018050817492199d6650a00000039
  Content-Type: applicaton/json
  Conent-Length: xxx
  
  {
          "Offsets": {
                  "0": {
                          "Timestamp": 1000,
                          "Sequence": 1,
                          "Version": 1,
                          "SessionId": "xxx"
                  }
          }
  }

Commit a consumer offset

Request

Request syntax

  PUT /projects/<ProjectName>/topics/<TopicName>/subscriptions/<SubId>/offsets HTTP/1.1

Request parameters

Parameter	Type	Description
Action	String	The operation that you want to perform. Set the value to commit.
Timestamp	Int64	The timestamp of the consumer offset. Unit: millisecond.
Sequence	Int64	The sequence number of the consumer offset.
Version	Int64	The session version.
SessionId	Int64	The session ID.

Response

Response syntax

  HTTP/1.1 200 OK

Examples

Sample requests

  PUT /projects/<ProjectName>/topics/<TopicName>/subscriptions/<SubId>/offsets HTTP/1.1
  x-datahub-client-version: 1.1
  Date: Tue, 08 May 2018 09:47:48 GMT
  Authorization: AuthorizationString
  Content-Type: applicaton/json
  Conent-Length: xxx
  
  {
          "Action": "commit",
          "Offsets": {
                  "0": {
                          "Timestamp": 1000,
                          "Sequence": 1,
                          "Version": 1,
                          "SessionId": 1
                  }
          }
  }