DataHub:Create and configure topics

Create a topic with a custom schema or by importing a MaxCompute table schema, then manage fields and data retention directly from the DataHub console or via the DataHub SDK.

Create a topic

On the details page of a project, click Create Topic. DataHub supports two creation methods.

Create a topic with a custom schema

The Shard Extension Mode setting controls whether the topic's Shards can scale horizontally. When enabled:

• Shards support horizontal scaling but cannot be merged or split.

• The Shard count can only be increased, never decreased.

• The topic becomes consumable via Kafka.

Enable shard extension mode when you need to increase throughput by adding Shards over time, or when you want Kafka-compatible consumption on the same topic.

Create a topic by importing the schema of a MaxCompute table

1. Set the parameters to access the MaxCompute table.

1. Click Next Step to import the MaxCompute table schema. Optionally update the topic name, field descriptions, and null-allow settings for each field.

If shard extension mode is enabled, the same constraints apply: Shards support horizontal scaling only, the Shard count can only be increased, and Kafka consumption is available.

View the details of a topic

On the details page of a project, find the topic and click View in the Actions column.

Delete a topic

On the Topic List tab, find the topic and click Delete in the Actions column.

Deleting a topic permanently removes all data, Shards, and DataConnectors associated with it. This action cannot be undone.

Create a field

Add fields to a topic schema from the console or via the DataHub SDK.

Create a field on the details page of a topic

1. Go to the topic details page and click the Schema Details tab.

2. Click Create Schema in the lower-right corner.

1. In the Create Schema panel, set the Name, Type, Allow Null, and Description parameters.

1. Click OK. The new field appears on the Schema Details tab.

The console creates one field per operation. To add multiple fields at once, use the DataHub SDK.

Create a field using the DataHub SDK

The appendField method adds a field to an existing topic schema.

Exceptions:

Example:

public static void appendNewField() {
    try {
        Field newField = new Field("newField", FieldType.STRING, true);
        datahubClient.appendField(Constant.projectName, Constant.topicName, newField);
        System.out.println("append field successful");
    } catch (InvalidParameterException e) {
        System.out.println("invalid parameter, please check your parameter");
        System.exit(1);
    } catch (AuthorizationFailureException e) {
        System.out.println("AK error, please check your accessId and accessKey");
        System.exit(1);
    } catch (ResourceNotFoundException e) {
        System.out.println("project or topic not found");
        System.exit(1);
    } catch (DatahubClientException e) {
        System.out.println("other error");
        System.exit(1);
    }
}

Modify the TTL of a topic

Use the updateTopic method to change the time-to-live (TTL) and description of a topic.

Exceptions:

Example:

public static void updateTopic() {
    try {
        int lifeCycle = 7;
        String comment = "test";
        datahubClient.updateTopic(Constant.projectName, Constant.topicName, lifeCycle, comment);
        System.out.println("update topicLifeCycle success!");
    } catch (InvalidParameterException e) {
        System.out.println("invalid parameter, please check your parameter");
        System.exit(1);
    } catch (AuthorizationFailureException e) {
        System.out.println("AK error, please check your accessId and accessKey");
        System.exit(1);
    } catch (ResourceNotFoundException e) {
        System.out.println("project or topic not found");
        System.exit(1);
    } catch (DatahubClientException e) {
        System.out.println("other error");
        System.exit(1);
    }
}