All Products
Search
Document Center

API reference

Last Updated: Aug 02, 2021

API reference

1. API specifications

Common request headers

Request header

Description

x-datahub-client-version

The version number of the API.

x-datahub-security-token

The security token, if any.

Date

The time when the request was sent, in GMT. Format: EEE, dd MMM yyyy HH:mm:ss z.

Authorization

The signature string of the request.

Content-Type

The MIME type of the request.

Common response headers

Response header

Description

Content-Type

The MIME type of the returned message.

Content-Length

The size of the message body.

x-datahub-request-id

The globally unique identifier (GUID) of the request.

Error codes

Error code

Description

Remarks

InvalidParameter

The error code returned because one or more parameters are invalid.

InvalidCursor

The error code returned because the cursor is invalid.

NoSuchXXX

The error code returned because the server cannot find the requested resource.

XXXAlreadyExist

The error code returned because the specified resource already exists.

Unauthorized

The error code returned because the authentication failed.

The AccessKey pair is invalid, or the timestamp of the request does not match the timestamp recorded on the server.

NoPermission

The error code returned because you are not authorized to perform the operation.

The RAM user that you use to initiate the request does not have the associated permissions.

OperationDenied

The error code returned because the specified operation is forbidden.

The specified operation is forbidden. For example, you are not allowed to delete a project that contains one or more topics.

LimitExceeded

The error code returned because the request exceeds one or more defined limits.

The limits for the server are exceeded, such as the QPS or traffic limit.

InvalidShardOperation

The error code returned because the shard is closed for writing due to the split or merge operation.

MalformedRecord

The error code returned because the format of the requested data is invalid.

OffsetReseted

The error code returned because the consumer offset was reset.

OffsetSessionChanged

The error code returned because the subscription specified by SubId is occupied.

SubscriptionOffline

The error code returned because the subscription was canceled.

InternalServerError

The error code returned because an internal error occurred.

Other error codes

The error codes that do not support retries will be removed.

Unified error response format

Parameter

Description

ErrorCode

The error code returned.

ErrorMessage

The error message returned.

Sample error responses:

    HTTP/1.1 403
    x-datahub-request-id: 2018050817492199d6650a00000039
    Content-Type: application/json
    Content-Length: xxx

    {
        "ErrorCode": "Unauthorized",
        "ErrorMessage": "Authroize failed"
    }

Limits

Parameter

Limit

ProjectName

The project name must be 3 to 32 characters in length and can contain only letters, digits, and underscores (_). It must start with a letter. The value is not case-sensitive.

TopicName

The topic name must be 3 to 128 characters in length and can contain only letters, digits, and underscores (_). It must start with a letter. The value is not case-sensitive.

2. Signature calculations for the Authorization header

Authorization = "DATAHUB " + AccessId + ":" + Signature
Signature = base64(hmac-sha1(AccessKey,
            HTTPMethod + "\n"
            + Content-Type + "\n"
            + Date + "\n"
            + CanonicalizedDataHubHeaders + "\n"
            + CanonicalizedResource))
  • AccessKey: the AccessKey secret used to sign the request.

  • HTTPMethod: the method of the HTTP request, such as PUT, GET, POST, HEAD, and DELETE.

  • \n: the line feed.

  • Content-Type: the MIME type of the request body. In most cases, this parameter is set to application/json.

  • Date: the time when the request was sent, in GMT. For example, you can set this parameter to Sun, 22 Nov 2015 08:16:38 GMT.

  • CanonicalizedDataHubHeaders: the HTTP request headers prefixed with x-datahub-. The request headers are arranged in alphabetical order.

  • CanonicalizedResource: the URL of the requested DataHub resource. If the URL contains one or more parameters, the parameters must be arranged in alphabetical order.

    Construction of the CanonicalizedDataHubHeaders string

    All HTTP request headers prefixed with x-datahub- are called canonicalized DataHub headers. The following procedure describes how to construct the CanonicalizedDataHubHeaders string:

  1. Convert the names of all HTTP request headers that are prefixed with x-datahub- to lowercase letters. For example, convert X-DATAHUB-Client-Version:1.1 to x-datahub-client-version:1.1.

  2. If the request is sent by using the AccessKey pair provided by Security Token Service (STS), add the returned token to the signature string in the format of x-datahub-security-token:token.

  3. Sort all obtained HTTP request headers in alphabetical order.

  4. Delete all spaces on either side of the delimiter that separates the name and value of a header. For example, convert x-datahub-client-version : 1.1 to x-datahub-client-version:1.1.

  5. Separate each two pairs of headers and their values with the delimiter (\n) to obtain the CanonicalizedDataHubHeaders string.

    Construction of the CanonicalizedResource string

    The DataHub resources specified in the request are called canonicalized resources. The following procedure describes how to construct the CanonicalizedResource string:

  6. Set the CanonicalizedResource string to an empty string.

  7. Enter the URL of the DataHub resource that you want to access, such as /projects/test_project/topics/test_topic.

  8. If the URL contains one or more parameters, arrange them in alphabetical order and separate them with ampersands (&). Add a question mark (?) and the parameters at the end of the string. Example: /projects/test_project/topics/test_topic/connectors/sink_odps?donetime.

    Calculation of the signature

  • A signature string must be encoded in UTF-8. A signature string that contains Chinese characters must be encoded in UTF-8. The encoded signature string is used together with the AccessKey secret to calculate the signature.

  • Use the SHA1 algorithm defined in RFC 2104 to calculate the HMAC value of the string to sign. The AccessKey secret is used as the key for the HMAC calculation.

  • Headers prefixed with x-datahub- must comply with the following requirements:

    • The header names must be in lowercase.

    • The headers are sorted in alphabetical order.

    • No spaces exist on either side of the colon (:) that separates the name and value of a header.

    • Each header is followed by a line feed (\n). If no header is used, set the CanonicalizedDataHubHeaders string to an empty string.

      Examples

      In the following examples, the AccessKey ID is 44CF9590006BF252F707, and the AccessKey secret is OtxrzxIsfpFjA7SwPzILwy8Bw21TLhquhboDYROV.

Request

Calculation formula of the signature string

Signature string

POST /projects/test_project/topics/test_topic HTTP/1.1 Host: http://dh-cn-hangzhou.aliyuncs.com User-Agent: customer x-datahub-client-version: 1.1 Content-Type: application/json Date: Thu, 10 Jan 2019 07:28:29 GMT

Signature = base64(hmac-sha1(AccessKey,HTTPMethod + "\n" + Content-Type + "\n" + Date + "\n" + CanonicalizedDataHubHeaders+ CanonicalizedResource))

POST\napplication/json\nThu, 10 Jan 2019 07:28:29 GMT\nx-datahub-client-version:1.1\n/projects/test_project/topics/test_topic

The following code shows you how to calculate the signature in Python:

import base64
import hmac
import sha
h = hmac.new("OtxrzxIsfpFjA7SwPzILwy8Bw21TLhquhboDYROV",
"POST\napplication/json\nThu, 10 Jan 2019 07:28:29 GMT\nx-datahub-client-version:1.1\n/projects/test_project/topics/test_topic", sha)
Signature = base64.b64encode(h.digest())
print("Signature: %s" % Signature)

The calculated signature is 2ZOa0YVc6PwOrqaOYzpNGb/3peU=. Therefore, the value of the Authorization header is DATAHUB 44CF9590006BF252F707:2ZOa0YVc6PwOrqaOYzpNGb/3peU=. You can obtain the following request headers:

POST /projects/connector_test/topics/event_time_test HTTP/1.1
Authorization: DATAHUB 44CF9590006BF252F707:2ZOa0YVc6PwOrqaOYzpNGb/3peU=
Content-Type: application/json
Date: Thu, 10 Jan 2019 07:28:29 GMT
Host: http://dh-cn-hangzhou.aliyuncs.com
User-Agent: customer
x-datahub-client-version: 1.1

The following code shows you how to calculate the signature in Java 8:

import javax.crypto.Mac;
import javax.crypto.spec.SecretKeySpec;
import java.util.Base64;
import java.util.HashMap;
import java.util.Iterator;
import java.util.Map;
import java.util.SortedMap;
import java.util.TreeMap;

public abstract class Authorization {
    private static final String DEFAULT_ENCODING = "UTF-8";
    private static final String DEFAULT_HASH = "HmacSHA1";
    private static final String X_DATAHUB_PREFIX = "x-datahub";
    private static final String HEADER_CONTENT_TYPE = "Content-Type";
    private static final String HEADER_DATE = "Date";

    static public String getAkAuthorization(Request request) {
        String canonicalURL = request.getUrlPath();
        String canonicalQueryString = request.getQueryStrings();
        String canonicalHeaderString = getCanonicalHeaders(
                getSortedHeadersToSign(request.getHeaders()));

        String canonicalRequest = request.getMethod().toUpperCase() + "\n" +
                canonicalHeaderString + "\n" + canonicalURL;
        if (canonicalQueryString != null && !canonicalQueryString.isEmpty()) {
            canonicalRequest += "?" + canonicalQueryString;
        }

        String signature = HMAC1Sign(request.getAccessKey(), canonicalRequest);

        return "DATAHUB " + request.getAccessId() + ":" + signature;
    }

    static private String HMAC1Sign(String accessKey, String canonicalRequest) {
        try {
            SecretKeySpec signingKey = new SecretKeySpec(accessKey.getBytes(), DEFAULT_HASH);
            Mac mac = Mac.getInstance(DEFAULT_HASH);
            mac.init(signingKey);
            return Base64.getEncoder().encodeToString(
                    mac.doFinal(canonicalRequest.getBytes(DEFAULT_ENCODING))
            ).trim();
        } catch (Exception e) {
            throw new RuntimeException(e.getMessage(), e);
        }
    }

    static private String getCanonicalHeaders(Map<String, String> headers) {
        StringBuilder sb = new StringBuilder();
        Iterator<Map.Entry<String, String>> pairs = headers.entrySet().iterator();
        while (pairs.hasNext()) {
            Map.Entry<String, String> pair = pairs.next();
            if (pair.getKey().startsWith(X_DATAHUB_PREFIX)) {
                sb.append(pair.getKey());
                sb.append(":");
                sb.append(pair.getValue());
            } else {
                sb.append(pair.getValue());
            }

            if (pairs.hasNext()) {
                sb.append("\n");
            }
        }
        return sb.toString();
    }

    static private SortedMap<String, String> getSortedHeadersToSign(Map<String, String> headers) {
        SortedMap<String, String> sortedHeaders = new TreeMap<>();

        for (Map.Entry<String, String> entry : headers.entrySet()) {
            String lowerKey = entry.getKey().toLowerCase();
            if (lowerKey.equalsIgnoreCase(HEADER_CONTENT_TYPE) ||
                    lowerKey.equalsIgnoreCase(HEADER_DATE) ||
                    lowerKey.startsWith(X_DATAHUB_PREFIX)) {
                if (!entry.getValue().isEmpty()) {
                    sortedHeaders.put(lowerKey, entry.getValue());
                }
            }
        }

        if (!sortedHeaders.containsKey(HEADER_CONTENT_TYPE.toLowerCase())) {
            sortedHeaders.put(HEADER_CONTENT_TYPE.toLowerCase(), "");
        }

        return sortedHeaders;
    }

    public static class Request {
        private String accessId;
        private String accessKey;
        private String urlPath;
        private String method;
        private Map<String, String> headers;
        private String queryStrings;

        public String getAccessId() {
            return accessId;
        }

        public Request setAccessId(String accessId) {
            this.accessId = accessId;
            return this;
        }

        public String getAccessKey() {
            return accessKey;
        }

        public Request setAccessKey(String accessKey) {
            this.accessKey = accessKey;
            return this;
        }

        public String getUrlPath() {
            return urlPath;
        }

        public Request setUrlPath(String urlPath) {
            this.urlPath = urlPath;
            return this;
        }

        public String getMethod() {
            return method;
        }

        public Request setMethod(String method) {
            this.method = method;
            return this;
        }

        public Map<String, String> getHeaders() {
            return headers;
        }

        public Request setHeaders(Map<String, String> headers) {
            this.headers = headers;
            return this;
        }

        public String getQueryStrings() {
            return queryStrings;
        }

        public Request setQueryStrings(String queryStrings) {
            this.queryStrings = queryStrings;
            return this;
        }
    }

    public static void main(String[] args) {
        Map<String, String> headers = new HashMap<>();
        headers.put("Date", "Wed, 20 Jan 2021 14:01:56 GMT");
        headers.put("x-datahub-client-version", "1.1");
        headers.put("Content-type", "application/json");

        String accessId = "xx";
        String accessKey = "yy";
        String method = "POST";
        String path = "/projects/test1/topics/test2/shards";
        String canonicalQueryString = ""; // Parameters are arranged in alphabetical order and separated by ampersands (&). Example: a=x&b=y.

        Authorization.Request authRequest = new Authorization.Request()
                .setAccessId(accessId)
                .setAccessKey(accessKey)
                .setMethod(method.toUpperCase())
                .setUrlPath(path)
                .setHeaders(headers)
                .setQueryStrings(canonicalQueryString);
        System.out.println(Authorization.getAkAuthorization(authRequest));
    }
}

3. API operations

Create a project

Request

  • Request syntax

    POST /projects/<ProjectName> HTTP/1.1
  • Request parameters

Parameter

Type

Description

Comment

String

The description of the project. The description cannot exceed 1,024 bytes.

Response

  • Response syntax

    HTTP/1.1 201 Created

Examples

  • Sample requests

    POST /projects/<ProjectName> HTTP/1.1
    x-datahub-client-version: 1.1
    Date: Tue, 08 May 2018 09:47:48 GMT
    Authorization: AuthorizationString
    Content-Type: application/json
    Content-Length: xxx
    
    {
        "Comment": "test project"
    }
  • Sample success responses

    HTTP/1.1 201 Created
    x-datahub-request-id: 2018050817492199d6650a00000039
    Content-Length: 0

Query the information about a project

Request

  • Request syntax

    GET /projects/<ProjectName> HTTP/1.1

Response

  • Response syntax

    HTTP/1.1 200 OK
  • Response parameters

Parameter

Type

Description

CreateTime

long

The time when the project was created. Unit: seconds.

LastModifyTime

long

The last time when the project was updated. Unit: seconds.

Comment

String

The description of the project.

Examples

  • Sample requests

    GET /projects/<ProjectName> HTTP/1.1
    x-datahub-client-version: 1.1
    Date: Tue, 08 May 2018 09:47:48 GMT
    Authorization: AuthorizationString
  • Sample success responses

    HTTP/1.1 200 OK
    x-datahub-request-id: 2018050817492199d6650a00000039
    Content-Type: applicaton/json
    Content-Length: xxx
    
    {
        "Comment": "test project",
        "CreateTime": 1525763481,
        "LastModifyTime": 1525763481
    }

Query the information about projects

Request

  • Request syntax

    GET /projects HTTP/1.1

Response

  • Response syntax

    HTTP/1.1 200 OK
  • Response parameters

Parameter

Type

Description

ProjectNames

List

The names of the projects.

Examples

  • Sample requests

    GET /projects HTTP/1.1
    x-datahub-client-version: 1.1
    Date: Tue, 08 May 2018 09:47:48 GMT
    Authorization: AuthorizationString
  • Sample success responses

    HTTP/1.1 200 OK
    x-datahub-request-id: 2018050817492199d6650a00000039
    Content-Type: applicaton/json
    Conent-Length: xxx
    
    {
        "ProjectNames": [
            "project1",
            "projcet2"
        ]
    }

Update a project

Request

  • Request syntax

    PUT /projects/<ProjectName> HTTP/1.1
  • Request parameters

Parameter

Type

Description

Comment

String

The description of the project.

Response

  • Response syntax

    HTTP/1.1 200 OK

Examples

  • Sample requests

    PUT /projects/<ProjectName> HTTP/1.1
    x-datahub-client-version: 1.1
    Date: Tue, 08 May 2018 09:47:48 GMT
    Authorization: AuthorizationString
    Content-Type: application/json
    Content-Length: xxx
    
    {
        "Comment": "update comment"
    }
  • Sample success responses

    HTTP/1.1 200 OK
    x-datahub-request-id: 2018050817492199d6650a00000039
    Content-Length: 0

Delete a project

Request

  • Request syntax

    DELETE /projects/<ProjectName> HTTP/1.1

Response

  • Response syntax

    HTTP/1.1 200 OK

Examples

  • Sample requests

    DELETE /projects/<ProjectName> HTTP/1.1
    x-datahub-client-version: 1.1
    Date: Tue, 08 May 2018 09:47:48 GMT
    Authorization: AuthorizationString
  • Sample success responses

    HTTP/1.1 200 OK
    x-datahub-request-id: 2018050817492199d6650a00000039
    Content-Length: 0

Create a topic

Request

  • Request syntax

    POST /projects/<ProjectName>/topics/<TopicName> HTTP/1.1
  • Request parameters

Parameter

Type

Description

Action

String

The operation that you want to perform.

ShardCount

int

The initial number of shards.

Lifecycle

int

The time to live (TTL) of data.

RecordType

String

The record type. Valid values: BLOB and TUPLE. The BLOB data type is designed for unstructured data storage, and the TUPLE data type is designed for structured data storage.

RecordSchema

String

The record schema. If you set the RecordType parameter to TUPLE, you must specify the record schema. This parameter is not required if you set the RecordType parameter to BLOB.

Comment

String

The description of the topic.

Response

  • Response syntax

    HTTP/1.1 201 Created

Examples

  • Sample requests

    POST /projects/<ProjectName>/topics/<TopicName> HTTP/1.1
    x-datahub-client-version: 1.1
    Date: Tue, 08 May 2018 09:47:48 GMT
    Authorization: AuthorizationString
    Content-Type: application/json
    Content-Length: xxx
    
    {
        "Action": "create",
        "ShardCount": 1,
        "Lifecycle": 1,
        "RecordType": "TUPLE",
        "RecordSchema": "{\"fields\":[{\"name\":\"field1\",\"type\":\"STRING\"},{\"name\":\"field2\",\"type\":\"BIGINT\"}]}}",
        "Comment": "create topic"
    }
  • Sample success responses

    HTTP/1.1 201 Created
    x-datahub-request-id: 2018050817492199d6650a00000039
    Content-Type: applicaton/json
    Content-Length: 0

Query the information about a topic

Request

  • Request syntax

    GET /projects/<ProjectName>/topics/<TopicName> HTTP/1.1

Response

  • Response syntax

    HTTP/1.1 200 OK
  • Response parameters

Parameter

Type

Description

ShardCount

int

The initial number of shards.

Lifecycle

int

The TTL of data.

RecordType

String

The record type. Valid values: BLOB and TUPLE. The BLOB data type is designed for unstructured data storage, and the TUPLE data type is designed for structured data storage.

RecordSchema

String

The record schema. This parameter is returned if the RecordType parameter is set to TUPLE, and is not returned if the RecordType parameter is set to BLOB.

Comment

String

The description of the topic.

CreateTime

long

The time when the topic was created.

LastModifyTime

long

The last time when the topic was updated.

Examples

  • Sample requests

    GET /projects/<ProjectName>/topics/<TopicName> HTTP/1.1
    x-datahub-client-version: 1.1
    Date: Tue, 08 May 2018 09:47:48 GMT
    Authorization: AuthorizationString
  • Sample success responses

    HTTP/1.1 201 Created
    x-datahub-request-id: 2018050817492199d6650a00000039
    Content-Type: applicaton/json
    Content-Length: xxx
    
    {
        "ShardCount": 1,
        "Lifecycle": 1,
        "RecordType": "TUPLE",
        "RecordSchema": "{\"fields\":[{\"name\":\"field1\",\"type\":\"STRING\"},{\"name\":\"field2\",\"type\":\"BIGINT\"}]}}",
        "Comment": "create topic",
        "CreateTime": 1525763481,
        "LastModifyTime": 1525763481
    }

Query the information about topics

Request

  • Request syntax

    GET /projects/<ProjectName>/topics HTTP/1.1

Response

  • Response syntax

    HTTP/1.1 200 OK
  • Response parameters

Parameter

Type

Description

TopicNames

List

The names of topics.

Examples

  • Sample requests

    GET /projects/<ProjectNames>/topics HTTP/1.1
    x-datahub-client-version: 1.1
    Date: Tue, 08 May 2018 09:47:48 GMT
    Authorization: AuthorizationString
  • Sample success responses

    HTTP/1.1 200 OK
    x-datahub-request-id: 2018050817492199d6650a00000039
    Content-Type: applicaton/json
    Conent-Length: xxx
    
    {
        "TopicNames": [
            "topic1",
            "topic2"
        ]
    }

Update a topic

Request

  • Request syntax

    PUT /projects/<ProjectName>/topics/<TopicName> HTTP/1.1
  • Request parameters

Parameter

Type

Description

Comment

String

The description of the topic.

Response

  • Response syntax

    HTTP/1.1 200 OK

Examples

  • Sample requests

    PUT /projects/<ProjectName>/topics/<TopicName> HTTP/1.1
    x-datahub-client-version: 1.1
    Date: Tue, 08 May 2018 09:47:48 GMT
    Authorization: AuthorizationString
    Content-Type: application/json
    Content-Length: xxx
    
    {
        "Comment": "update comment"
    }

Delete a topic

Request

  • Request syntax

    DELETE /projects/<ProjectName>/topics/<TopicName> HTTP/1.1

Response

  • Response syntax

    HTTP/1.1 200 OK

Examples

  • Sample requests

    DELETE /projects/<ProjectName>/topics/<TopicName> HTTP/1.1
    x-datahub-client-version: 1.1
    Date: Tue, 08 May 2018 09:47:48 GMT
    Authorization: AuthorizationString
  • Sample success responses

    HTTP/1.1 200 OK
    x-datahub-request-id: 2018050817492199d6650a00000039
    Content-Length: 0

Query the information about shards

Request

  • Request syntax

    GET /projects/<ProjectName>/topics/<TopicName>/shards HTTP/1.1

Response

  • Response syntax

    HTTP/1.1 200 OK
  • Response parameters

Parameter

Type

Description

ShardId

String

The ID of the shard.

State

String

The status of the shard. Examples: OPENING, ACTIVE, and CLOSED.

BeginHashKey

String

The start hash key of the hash key range.

EndHashKey

String

The end hash key of the hash key range.

ParentShardIds

List

The information about the parent shard.

Examples

  • Sample requests

    GET /projects/<ProjectName>/topics/<TopicName>/shards HTTP/1.1
    x-datahub-client-version: 1.1
    Date: Tue, 08 May 2018 09:47:48 GMT
    Authorization: AuthorizationString
  • Sample success responses

    HTTP/1.1 200 OK
    x-datahub-request-id: 2018050817492199d6650a00000039
    Content-Type: applicaton/json
    Conent-Length: xxx
    
    {
        "Shards": [
            {
                "ShardId": "0",
                "State": "ACTIVE",
                "BeginHashKey":"00000000000000000000000000000000",
                "EndHashKey":"FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF",
                "ParentShardIds:[]
            }
        ]
    }

Split a shard

Request

  • Request syntax

    POST /projects/<ProjectName>/topics/<TopicName>/shards HTTP/1.1
  • Request parameters

Parameter

Type

Description

Action

String

The operation that you want to perform. Set the value to split.

ShardId

String

The ID of the shard to be split.

SplitKey

String

The key to use for splitting the shard. Formula: SplitKey = BeginHashKey + (EndHashKey - BeginHashKey)/2.

Response

  • Response syntax

    HTTP/1.1 200 OK
  • Response parameters

Parameter

Type

Description

NewShards

List

The information about the new shards.

Examples

  • Sample requests

    POST /projects/<ProjectName>/topics/<TopicName>/shards HTTP/1.1
    x-datahub-client-version: 1.1
    Date: Tue, 08 May 2018 09:47:48 GMT
    Authorization: AuthorizationString
    Content-Type: application/json
    Content-Length: xxx
    
    {
        "Action": "split",
        "ShardId": "0",
        "SplitKye": "7FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF"
    }
  • Sample success responses

    HTTP/1.1 200 OK
    x-datahub-request-id: 2018050817492199d6650a00000039
    Content-Type: applicaton/json
    Conent-Length: xxx
    
    {
        "NewShards": [
            {
                "ShardId": "1",
                "BeginHashKey":"00000000000000000000000000000000",
                "EndHashKey":"7FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF"
            },
            {
                "ShardId":"0",
                "BeginHashKey":"7FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF",
                "EndHashKey":"FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF"
            }
        ]
    }

Merge shards

Request

  • Request syntax

    POST /projects/<ProjectName>/topics/<TopicName>/shards HTTP/1.1
  • Request parameters

Parameter

Type

Description

Action

String

The operation that you want to perform. Set the value to merge.

ShardId

String

The ID of the shard to be merged.

AdjacentShardId

String

The ID of the adjacent shard for the merge operation.

Response

  • Response syntax

    HTTP/1.1 200 OK
  • Response parameters

Parameter

Type

Description

ShardId

String

The ID of the new shard.

BeginHashKey

String

The start hash key of the hash key range.

EndHashKey

String

The end hash key of the hash key range.

Examples

  • Sample requests

    POST /projects/<ProjectName>/topics/<TopicName>/shards HTTP/1.1
    x-datahub-client-version: 1.1
    Date: Tue, 08 May 2018 09:47:48 GMT
    Authorization: AuthorizationString
    Content-Type: application/json
    Content-Length: xxx
    
    {
        "Action": "merge",
        "ShardId": "0",
        "AdjacentShardId": "1"
    }
  • Sample success responses

    HTTP/1.1 200 OK
      x-datahub-request-id: 2018050817492199d6650a00000039
      Content-Type: applicaton/json
      Conent-Length: xxx
    
      {
          "ShardId":"2",
          "BeginHashKey":"00000000000000000000000000000000",
          "EndHashKey":"FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF"
      }

Query a cursor

Request

  • Request syntax

    POST /projects/<ProjectName>/topics/<TopicName>/shards/<ShardId> HTTP/1.1
  • Request parameters

Parameter

Type

Description

Action

String

The operation that you want to perform. Set the value to cursor.

Type

String

The method to get the cursor. Valid values: OLDEST, LATEST, SYSTEM_TIME, and SEQUENCE.

SystemTime

Int64

The system time of the cursor. Specify this parameter if you set the Type parameter to SYSTEM_TIME. Unit: milliseconds.

Sequence

Int64

The sequence number of the cursor. Specify this parameter if you set the Type parameter to SEQUENCE.

Response

  • Response syntax

    HTTP/1.1 200 OK
  • Response parameters

Parameter

Type

Description

Cursor

String

The information about the cursor.

RecordTime

Int64

The time when the data was written to DataHub. Unit: milliseconds.

Sequence

Int64

The sequence number in which the data was written. The sequence number is unique in a single shard.

Examples

  • Sample requests

    POST /projects/<ProjectName>/topics/<TopicName>/shards/<ShardId> HTTP/1.1
    x-datahub-client-version: 1.1
    Date: Tue, 08 May 2018 09:47:48 GMT
    Authorization: AuthorizationString
    Content-Type: application/json
    Content-Length: xxx
    
    {
        "Action": "cursor",
        "Type": "SEQUENCE",
        "Sequence": 1
    }
  • Sample success responses

    HTTP/1.1 200 OK
    x-datahub-request-id: 2018050817492199d6650a00000039
    Content-Type: applicaton/json
    Conent-Length: xxx
    
    {
        "Cursor": "30005af19b3800000000000000000000",
        "RecordTime": 1525783352873,
        "Sequence": 1
    }

Write data without specifying shards

Request

  • Request syntax

    POST /projects/<ProjectName>/topics/<TopicName>/shards HTTP/1.1
  • Request parameters

Parameter

Type

Description

Action

String

The operation that you want to perform. Set the value to pub.

ShardId

String

The ID of the shard.

Attributes

Map<string, string=""></string,>

The user attributes.

Data

The data to be written. For the BLOB data type, the value is a Base64-encoded string. For the TUPLE data type, the value is an array of strings.

Response

  • Response syntax

    HTTP/1.1 200 OK
  • Response parameters

Parameter

Type

Description

FailedRecordCount

Int

The number of the failed records.

FailedRecords

Array

The detailed information about the failed records.

Examples

  • Sample requests

    POST /projects/<ProjectName>/topics/<TopicName>/shards HTTP/1.1
    x-datahub-client-version: 1.1
    Date: Tue, 08 May 2018 09:47:48 GMT
    Authorization: AuthorizationString
    Content-Type: applicaton/json
    Conent-Length: xxx
    
    {
        "Action": "pub",
        "Records": [
            {
                "ShardId": "0",
                "Attributes": {
                    "attr1": "value1",
                    "attr2": "value2"
                },
                "Data": ["A","B","3","4"]
            }
        ]
    }
    // BLOB
    {
        "Action": "pub",
        "Record": [
            {
                "ShardId": "0",
                "Attributes": {
                    "attr1": "value1",
                    "attr2": "value2"
                },
                "Data": "Base64String"
            }
        ]
    }
  • Sample success responses

    HTTP/1.1 200 OK
    x-datahub-request-id: 2018050817492199d6650a00000039
    Content-Type: applicaton/json
    Conent-Length: xxx
    
    {
        "FailedRecordCount": 1,
        "FailedRecords": [
                {
                    "Index": 0,
                    "ErrorCode": "errorCode",
                    "ErrorMessage": "errorMsg"
                }
        ]
    }

Read data

Request

  • Request syntax

    POST /projects/<ProjectName>/topics/<TopicName>/shards/<ShardId> HTTP/1.1
  • Request parameters

Parameter

Type

Description

Action

String

The operation that you want to perform. Set the value to sub.

Cursor

String

The start cursor used to read data.

Limit

Int

The number of records to read at a time.

Response

  • Response syntax

    HTTP/1.1 200 OK
  • Response parameters

Parameter

Type

Description

NextCursor

String

The information about the cursor used to read the next record.

SystemTime

Int64

The time when the record was written to DataHub. Unit: milliseconds.

Cursor

String

The information about the cursor corresponding to the record.

Sequence

Int64

The sequence number in which the record was written to DataHub.

Attributes

Map

The user attributes.

Data

The information about the requested data.

Examples

  • Sample requests

    POST /projects/<ProjectName>/topics/<TopicName>/shards HTTP/1.1
    x-datahub-client-version: 1.1
    Date: Tue, 08 May 2018 09:47:48 GMT
    Authorization: AuthorizationString
    Content-Type: applicaton/json
    Conent-Length: xxx
    
    {
        "Action": "sub",
        "Cursor": "30005af19b3800000000000000000000",
        "Limit": 1
    }
  • Sample success responses

    HTTP/1.1 200 OK
    x-datahub-request-id: 2018050817492199d6650a00000039
    Content-Type: applicaton/json
    Conent-Length: xxx
    
    {
        "NextCursor": "30005af19b3800000000000000090001",
        "Records": [
                {
                    "Cursor": "30005af19b3800000000000000000000",
                    "SystemTime": 1525783352873,
                    "Sequence": 1,
                    "Attributes": {
                        "key1": "value1",
                        "key2": "value2"
                    },
                    "Data": ["AAA", "100"]
                }
        ]
    }

Add a field

Request

  • Request syntax

    POST /projects/<ProjectName>/topics/<TopicName> HTTP/1.1
  • Request parameters

Parameter

Type

Description

Action

String

The operation that you want to perform. Set the value to appendfield.

FieldName

String

The name of the field.

FieldType

String

The data type of the field. Examples: STRING and BIGINT.

Response

  • Response syntax

    HTTP/1.1 200 OK

Examples

  • Sample requests

    POST /projects/<ProjectName>/topics/<TopicName> HTTP/1.1
    x-datahub-client-version: 1.1
    Date: Tue, 08 May 2018 09:47:48 GMT
    Authorization: AuthorizationString
    Content-Type: applicaton/json
    Conent-Length: xxx
    
    {
        "Action": "appendfield",
        "FieldName": "field1",
        "FieldType": "BIGINT"
    }

Create a connector

Request

  • Request syntax

    POST /projects/<ProjectName>/topics/<TopicName>/connectors/<ConnectorType> HTTP/1.1
  • Request parameters

Parameter

Type

Description

Type

String

The type of the connector. Example: SINK_ODPS.

ColumnFields

Array

The fields to be synchronized.

Config

Map

The configurations of the connector.

Response

  • Response syntax

    HTTP/1.1 201 Created

Examples

  • Sample request for creating a connector of the SINK_ODPS type

    POST /projects/<ProjectName>/topics/<TopicName>/connectors/sink_odps HTTP/1.1
    x-datahub-client-version: 1.1
    Date: Tue, 08 May 2018 09:47:48 GMT
    Authorization: AuthorizationString
    Content-Type: applicaton/json
    Conent-Length: xxx
    
    {
        "Type": "SINK_ODPS",
        "ColumnFields": ["field1", "field2"],
        "Config": {
            "Project": "odpsProject",
            "Topic": "odpsTopic",
            "OdpsEndpoint": "xxx",
            "TunnelEndpoint": "xxx",
            "AccessId": "xxx",
            "AccessKey": "xxx",
            "PartitionMode": "SYSTEM_TIME",
            "TimeRange": 60,
            "PartitionConfig": {
                "pt": "%Y%m%d",
                "ct": "%H%M"
            }
        }
    }

Query the information about a connector

Request

  • Request syntax

    GET /projects/<ProjectName>/topics/<TopicName>/connectors/<ConnectorType> HTTP/1.1

Response

  • Response syntax

    HTTP/1.1 200 OK

Query the information about connectors

Request

  • Request syntax

    GET /projects/<ProjectName>/topics/<TopicName>/connectors HTTP/1.1

Response

  • Response syntax

    HTTP/1.1 200 OK

Examples

  • Sample requests

    GET /projects/<ProjectName>/topics/<TopicName>/connectors HTTP/1.1
    x-datahub-client-version: 1.1
    Date: Tue, 08 May 2018 09:47:48 GMT
    Authorization: AuthorizationString
  • Sample success responses

    HTTP/1.1 200 OK
    x-datahub-request-id: 2018050817492199d6650a00000039
    Content-Type: applicaton/json
    Conent-Length: xxx
    
    {
        "Connectors": [
            "sink_odps", "sink_ads"
        ]
    }

Delete a connector

Request

  • Request syntax

    DELETE /projects/<ProjectName>/topics/<TopicName>/connectors/<ConnectorType> HTTP/1.1

Response

  • Response syntax

    HTTP/1.1 200 OK

Reload one or more shards specified by a connector

Request

  • Request syntax

    POST /projects/<ProjectName>/topics/<TopicName>/connectors/<ConnectorType> HTTP/1.1
  • Request parameters

Parameter

Type

Description

Action

String

The operation that you want to perform. Set the value to reload.

ShardId

String

The ID of the shard to be reloaded. If this parameter is left empty, all shards specified by the connector are reloaded.

Response

  • Response syntax

    HTTP/1.1 200 OK

Examples

  • Sample requests

    POST /projects/<ProjectName>/topics/<TopicName>/connectors/<ConnectorType> HTTP/1.1
    x-datahub-client-version: 1.1
    Date: Tue, 08 May 2018 09:47:48 GMT
    Authorization: AuthorizationString
    Content-Type: applicaton/json
    Conent-Length: xxx
    
    {
        "Action": "reload"
    }

Query the status of a shard specified by a connector

Request

  • Request syntax

    POST /projects/<ProjectName>/topics/<TopicName>/connectors/<ConnectorType> HTTP/1.1
  • Request parameters

Parameter

Type

Description

Action

String

The operation that you want to perform. Set the value to status.

ShardId

String

The ID of the shard.

Response

  • Response syntax

    HTTP/1.1 200 OK
  • Response parameters

Parameter

Type

Description

CurrentSequence

Int64

The sequence number of the record.

State

Enum

The status of the shard.

LastErrorMessage

String

The error message returned if the status of the shard is not CONTEXT_EXECUTING.

DiscardCount

Int64

The total number of records dropped since the connector started.

Examples

  • Sample requests

    POST /projects/<ProjectName>/topics/<TopicName>/connectors/<ConnectorType> HTTP/1.1
    x-datahub-client-version: 1.1
    Date: Tue, 08 May 2018 09:47:48 GMT
    Authorization: AuthorizationString
    Content-Type: applicaton/json
    Conent-Length: xxx
    
    {
        "Action": "status",
        "ShardId": "0"
    }
  • Sample success responses

    HTTP/1.1 200 OK
    x-datahub-request-id: 2018050817492199d6650a00000039
    Content-Type: applicaton/json
    Conent-Length: xxx
    
    {
        "State": "CONTEXT_EXECUTING",
        "CurrentSequence": 10,
        "DiscardCount": 0,
        "LastErrorMessage": ""
    }

Add a connector field

Request

  • Request syntaxPOST /projects//topics//connectors/ HTTP/1.1

  • Request parameters

Parameter

Type

Description

Action

String

The operation that you want to perform. Set the value to appendfield.

FieldName

String

The name of the field.

Response

  • Response syntax

    HTTP/1.1 200 OK
  • Sample requests

    POST /projects/<ProjectName>/topics/<TopicName>/connectors/<ConnectorType> HTTP/1.1
    x-datahub-client-version: 1.1
    Date: Tue, 08 May 2018 09:47:48 GMT
    Authorization: AuthorizationString
    Content-Type: applicaton/json
    Conent-Length: xxx
    
    {
        "Action": "appendfiled",
        "FieldName": "field1"
    }

Create a subscription

Request

  • Request syntax

    POST /projects/<ProjectName>/topics/<TopicName>/subscriptions HTTP/1.1
  • Request parameters

Parameter

Type

Description

Action

String

The operation that you want to perform. Set the value to create.

Comment

String

The description of the subscription.

Response

  • Response syntax

    HTTP/1.1 201 Created
  • Response parameters

Parameter

Type

Description

SubId

String

The ID of the subscription.

Examples

  • Sample requests

    POST /projects/<ProjectName>/topics/<TopicName>/subscriptions HTTP/1.1
    x-datahub-client-version: 1.1
    Date: Tue, 08 May 2018 09:47:48 GMT
    Authorization: AuthorizationString
    Content-Type: applicaton/json
    Conent-Length: xxx
    
    {
        "Action": "create",
        "Comment": "xxxx"
    }
  • Sample success responses

    HTTP/1.1 200 OK
      x-datahub-request-id: 2018050817492199d6650a00000039
       Content-Type: applicaton/json
    Conent-Length: xxx
    
      {
        "SubId": "1542078393028fzsZx"
    }

Query the information about a subscription

Request

  • Request syntax

    GET /projects/<ProjectName>/topics/<TopicName>/subscriptions/<SubId> HTTP/1.1

Response

  • Response syntax

    HTTP/1.1 200 OK
  • Response parameters

Parameter

Type

Description

SubId

String

The ID of the subscription.

State

Int

The status of the subscription. The value of 0 indicates that the subscription is online. The value of 1 indicates that the subscription is offline.

Comment

String

The description of the subscription.

Examples

  • Sample requests

    GET /projects/<ProjectName>/topics/<TopicName>/subscriptions/<SubId> HTTP/1.1
    x-datahub-client-version: 1.1
    Date: Tue, 08 May 2018 09:47:48 GMT
  • Sample success responses

    HTTP/1.1 200 OK
    x-datahub-request-id: 2018050817492199d6650a00000039
    Authorization: AuthorizationString
    Content-Type: applicaton/json
    Conent-Length: xxx
    
    {
        "SubId": "xxxx",
        "Comment": "xxxx"
        "State": 1
    }

Query the information about subscriptions

Request

  • Request syntax

    POST /projects/<ProjectName>/topics/<TopicName>/subscriptions HTTP/1.1
  • Request parameters

Parameter

Type

Description

Action

String

The operation that you want to perform. Set the value to list.

PageIndex

Int

The number of the page to return.

PageSize

Int

The number of entries to return on each page.

Response

  • Response syntax

    HTTP/1.1 200 OK
  • Response parameters

Parameter

Type

Description

TotalCount

Int

The total number of pages returned.

  • Sample requests

    POST /projects/<ProjectName>/topics/<TopicName>/subscriptions HTTP/1.1
    x-datahub-client-version: 1.1
    Date: Tue, 08 May 2018 09:47:48 GMT
    Authorization: AuthorizationString
    Content-Type: applicaton/json
    Conent-Length: xxx
    
    {
        "Action": "list",
        "PageIndex": 1,
        "PageSize": 10
    }
  • Sample success responses

    HTTP/1.1 200 OK
    x-datahub-request-id: 2018050817492199d6650a00000039
    Content-Type: applicaton/json
    Conent-Length: xxx
    
    {
        "Subscriptions": [
            {
                 "Comment": "xxxx",
                  "State": 1,
                  "SubId": "1542079169844gH8HM"
            },
            {
                "Comment": "xxxx",
                "State": 1,
                "SubId": "1542078393028fzsZx"
            }
        ],
        "TotalCount": 2
    }

Delete a subscription

Request

  • Request syntax

    DELETE /projects/<ProjectName>/topics/<TopicName>/subscriptions/<SubId> HTTP/1.1

Examples

  • Sample requests

    DELETE /projects/<ProjectName>/topics/<TopicName>/subscriptions/<SubId> HTTP/1.1
    x-datahub-client-version: 1.1
    Date: Tue, 08 May 2018 09:47:48 GMT
    Authorization: AuthorizationString

Update the status of a subscription

Request

  • Request syntax

    PUT /projects/<ProjectName>/topics/<TopicName>/subscriptions/<SubId> HTTP/1.1

Response

  • Response syntax

    HTTP/1.1 200 OK

Examples

  • Sample requests

    PUT /projects/<ProjectName>/topics/<TopicName>/subscriptions/<SubId> HTTP/1.1
    x-datahub-client-version: 1.1
    Date: Tue, 08 May 2018 09:47:48 GMT
    Authorization: AuthorizationString
    Content-Type: applicaton/json
    Conent-Length: xxx
    
    {
        "State": 0
    }

Open a session for a consumer offset

Request

  • Request syntax

    POST /projects/<ProjectName>/topics/<TopicName>/subscriptions/<SubId>/offsets HTTP/1.1
  • Request parameters

Parameter

Type

Description

Action

String

The operation that you want to perform. Set the value to open.

ShardIds

Array

The IDs of the shards.

Response

  • Response syntax

    HTTP/1.1 200 OK
  • Response parameters

Parameter

Type

Description

Timestamp

Int64

The timestamp of the consumer offset. Unit: milliseconds.

Sequence

Int64

The sequence number of the consumer offset.

Version

Int64

The version number of the session.

SessionId

String

The ID of the session.

Examples

  • Sample requests

    POST /projects/<ProjectName>/topics/<TopicName>/subscriptions/<SubId>/offsets HTTP/1.1
    x-datahub-client-version: 1.1
    Date: Tue, 08 May 2018 09:47:48 GMT
    Authorization: AuthorizationString
    Content-Type: applicaton/json
    Conent-Length: xxx
    
    {
        "Action": "open",
        "ShardIds": ["0"]
    }
  • Sample success responses

    HTTP/1.1 200 OK
      x-datahub-request-id: 2018050817492199d6650a00000039
    Content-Type: applicaton/json
    Conent-Length: xxx
    
    {
        "Offsets": {
            "0": {
                "Timestamp": 1000,
                "Sequence": 1,
                "Version": 1,
                "SessionId": "xxx"
            }
        }
    }

Query the information about consumer offsets

Request

  • Request syntax

    POST /projects/<ProjectName>/topics/<TopicName>/subscriptions/<SubId>/offsets HTTP/1.1
  • Request parameters

Parameter

Type

Description

Action

String

The operation that you want to perform. Set the value to get.

ShardIds

Array

The IDs of the shards.

Response

  • Response syntax

    HTTP/1.1 200 OK
  • Response parameters

Parameter

Type

Description

Timestamp

Int64

The timestamp of the consumer offset. Unit: milliseconds.

Sequence

Int64

The sequence number of the consumer offset.

Version

Int64

The version number of the session.

SessionId

String

The ID of the session.

Examples

  • Sample requests

    POST /projects/<ProjectName>/topics/<TopicName>/subscriptions/<SubId>/offsets HTTP/1.1
    x-datahub-client-version: 1.1
    Date: Tue, 08 May 2018 09:47:48 GMT
    Authorization: AuthorizationString
    Content-Type: applicaton/json
    Conent-Length: xxx
    
    {
        "Action": "get",
        "ShardIds": ["0"]
    }
  • Sample success responses

    HTTP/1.1 200 OK
      x-datahub-request-id: 2018050817492199d6650a00000039
    Content-Type: applicaton/json
    Conent-Length: xxx
    
    {
        "Offsets": {
            "0": {
                "Timestamp": 1000,
                "Sequence": 1,
                "Version": 1,
                "SessionId": "xxx"
            }
        }
    }

Commit a consumer offset

Request

  • Request syntax

    PUT /projects/<ProjectName>/topics/<TopicName>/subscriptions/<SubId>/offsets HTTP/1.1
  • Request parameters

Parameter

Type

Description

Action

String

The operation that you want to perform. Set the value to commit.

Timestamp

Int64

The timestamp of the consumer offset. Unit: milliseconds.

Sequence

Int64

The sequence number of the consumer offset.

Version

Int64

The version number of the session.

SessionId

Int64

The ID of the session.

Response

  • Response syntax

    HTTP/1.1 200 OK

Examples

  • Sample requests

    PUT /projects/<ProjectName>/topics/<TopicName>/subscriptions/<SubId>/offsets HTTP/1.1
    x-datahub-client-version: 1.1
    Date: Tue, 08 May 2018 09:47:48 GMT
    Authorization: AuthorizationString
    Content-Type: applicaton/json
    Conent-Length: xxx
    
    {
        "Action": "commit",
        "Offsets": {
            "0": {
                "Timestamp": 1000,
                "Sequence": 1,
                "Version": 1,
                "SessionId": 1
            }
        }
    }