DataHub:Manage topics

Create a topic

Click Create Topic on the details page of a project to create a topic. You can create topics in two ways.

Create a topic with a custom schema

Note: If the shard extension mode is enabled, the shards of the current topic support horizontal expansion, but do not support merging and splitting. The number of shards can only be increased but cannot be decreased. After this mode is enabled, you can use Kafka to consume the current topic.

You can create topics based on your business requirements. To understand the parameters used to create topics, you can view the terms in DataHub documentation.

Create a topic by importing the schema of a MaxCompute table

Step 1

Set the parameters used to access the MaxCompute table.

Step 2

Click Next Step to import the schema of the MaxCompute table. You can modify the topic name and field description, and specify whether the fields in the schema can be null as needed. Note: If the shard extension mode is enabled, the shards of the current topic support horizontal expansion, but do not support merging and splitting. The number of shards can only be increased but cannot be decreased. After this mode is enabled, you can use Kafka to consume the current topic.

View the details of a topic

1.On the details page of a project, find the topic whose details you want to view and click View in the Actions column.

Delete a topic

On the Topic List tab, find the topic that you want to delete and click Delete in the Actions column. Note that if a topic is deleted, the data and resources including shards and DataConnectors in the topic are deleted and cannot be restored. Proceed with caution.

Create a field

You can create a field for a topic on the details page of the topic or by using a DataHub SDK.

Create a field on the details page of a topic

1. Go to the details page of the topic. Click the Schema Details tab. Click Create Schema in the lower-right corner of the details page.

2.In the Create Schema panel, set the Name, Type, Allow Null, and Description parameters.

3.Click OK. After the field is created, view the field on the Schema Details tab.

Note: You can create only one field at a time. To create multiple fields at a time, use the console command-line tool or a DataHub SDK.

Create a field by using a DataHub SDK

Sample code:

• Parameters

  • projectName: the name of the project in which you want to add a field.

  • topicName: the name of the topic.

• Exceptions

  • DatahubClientException

  • InvalidParameterException

  • AuthorizationFailureException

  • ResourceNotFoundException

• Example

  public static void appendNewField() {
    try {
        Field newField = new Field("newField", FieldType.STRING, true);
        datahubClient.appendField(Constant.projectName, Constant.topicName, newField);
        System.out.println("append field successful");
    } catch (InvalidParameterException e) {
        System.out.println("invalid parameter, please check your parameter");
        System.exit(1);
    } catch (AuthorizationFailureException e) {
        System.out.println("AK error, please check your accessId and accessKey");
        System.exit(1);
    } catch (ResourceNotFoundException e) {
        System.out.println("project or topic not found");
        System.exit(1);
    } catch (DatahubClientException e) {
        System.out.println("other error");
        System.exit(1);
    }
  }

Modify the TTL period of a topic

You can modify the time-to-live (TTL) period of a topic in DataHub. Sample code:

• Parameters

  • projectName: the name of the project in which you want to modify theTTL of a topic.

  • topicName: the name of the topic.

  • lifeCycle: the TTL of the topic.

  • comment: the description of the topic.

• Exceptions

  • DatahubClientException

  • InvalidParameterException

  • AuthorizationFailureException

  • ResourceNotFoundException

• Example

  public static void updatetopic() {
    try {
        int lifeCycle = 7;
        String comment = "test";
        datahubClient.updateTopic(Constant.projectName, Constant.topicName,liefCycle,comment );
        System.out.println("update topicLifeCycle success!");
    } catch (InvalidParameterException e) {
        System.out.println("invalid parameter, please check your parameter");
        System.exit(1);
    } catch (AuthorizationFailureException e) {
        System.out.println("AK error, please check your accessId and accessKey");
        System.exit(1);
    } catch (ResourceNotFoundException e) {
        System.out.println("project or topic not found");
        System.exit(1);
    } catch (DatahubClientException e) {
        System.out.println("other error");
        System.exit(1);
    }
  }