1. Overview
You can use the data distribution module to push instrumentation data collected in Quick Tracking to a destination in real time.
This feature is only available in the private deployment version. To use data subscription in the Software as a Service (SaaS) version, see SaaS Data Log Redemption.
This feature only supports pushing real-time instrumentation data. It does not support pushing historical data. If you restart a stopped task, data from the period when the task was stopped cannot be backfilled.
2. Instructions
2.1 Accessing the feature
Go to the Quick Tracking console. In the navigation menu or from the nut icon in the upper-right corner, click Management Console. Then, click Data Distribution to open the feature page.
2.2 Feature components
The Data Distribution feature includes the following components:
Task List: The task list includes Task Name, Distribution Method, Remarks, Last Status Updater, Last Status Update Time, Status, and Operations.
Task Name: A custom name for the task that you set when you create it.
Distribution Method: The channel selected when the task was created, either Simple Log Service (SLS) or Kafka.
Remarks: An optional field for adding a description when you create the task.
Last Status Updater: The account name of the user who last started or stopped the task.
Last Status Update Time: The time when the task was last started or stopped.
Status: The current running status of the task, which is either `Running` or `Stopped`. You can change the task status in this column. This column also indicates if a task is abnormal.
Operations: When a task's status is `Running`, all operation buttons are disabled. To perform operations on the task, you must first change its status to `Stopped`.
i. Edit: Allows you to edit the task information.
ii. Delete: You cannot delete a task that has a `Running` status. You can only delete a task if its status is `Stopped`. Deleted tasks cannot be recovered.
2.3 Create a distribution task
Go to the Management Console > Data Distribution page. In the upper-right corner, click Create Distribution Task to open the task creation interface. (Note: You must have the Management Console - Data Distribution permission to view and use this module.)
2.4 Information for a new distribution task
Basic Information
Task Name: A custom name for the task. The name can be up to 500 characters long.
Content to Distribute: Select the content to distribute from the drop-down list. Currently, only Raw Logs is available.
Remarks: This field is optional. You can add a custom description.
Channel Configuration
Channel Configuration: You can select either SLS or Kafka as the distribution channel.
i. If you select SLS, configure the following parameters:
1) endpoint: This parameter is required. Enter your public SLS endpoint.
2) AccessKey ID: This parameter is required. Enter your AccessKey ID.
3) AccessKey Secret: This parameter is required. Enter your AccessKey secret.
4) Project: This parameter is required. Enter your SLS project name.
5) LogStore: This parameter is required. Enter the name of the Logstore.
6) Advanced Configuration: This parameter is optional. Enter advanced configurations in JSON format. These configurations apply to SLS. The following table describes the advanced configuration parameters.
Name | Value (numbers only) |
memory.queue.size (number of memory queues) | Default value: 20000. This value is user-configurable. |
batchbytesize (size of each batch) | Default value: 2097152. This value is user-configurable. |
batch.count (number of logs per batch) | Default value: 40000. This value is user-configurable. |
queue.timeout (queue wait time) | Default value: 5000. This value is user-configurable. |
ii. If you select Kafka, configure the following parameters:
1) bootstrap.servers: This parameter is required. Enter the address of your Kafka broker servers.
2) TopicName: This parameter is required. Enter the name of the topic.
3) Advanced Configuration: This parameter is optional. Enter advanced configurations in JSON format. These configurations apply to Kafka. The following table describes the advanced configuration parameters.
Name | Description |
kafka.acks | The number of replicas that must acknowledge the message before the producer considers the send successful. |
kafka.retries | The number of times the producer retries sending a message if it fails. |
kafka.batch.size | The size of a batch of messages sent by the producer. The unit is bytes. The default value is 16384 (16 KB). |
kafka.linger.ms | The maximum time in milliseconds that the producer waits for more messages to join the current batch before sending. |
kafka.buffer.memory | The amount of available memory in bytes that the producer uses to cache messages waiting to be sent. The default value is 33554432 (32 MB). |
kafka.max.in.flight.requests.per.connection | The maximum number of unacknowledged messages that a producer can send before receiving an acknowledgment from the server. The default value is 5. |
Number of threads (processes) | The default number of processes is 2. |
batchbytesize | The size of each batch. |
batch.count | The number of logs in each batch. |
queue.timeout | The queue wait time. |
Distribution Rules
Distribution Rule: For the scope, you can select either All organizations and applications or Select organizations. If you select All organizations and applications, the rule applies to all applications across all organizations. If you select Select organizations and applications, the rule applies only to the specified organizations and applications.
Select Events: Choose the events to forward. The All Events toggle is on by default, which means all events are selected. If you turn off the All Events toggle, you can select specific events from the drop-down list.
Global Filter: Set global property rules for forwarding data. The filter condition only supports "equals". You can select multiple "equals" values. You can set up to five groups, with each group containing up to two values. You can also import values in batches.
Blocked Fields: Click the input box to open the property selection window. Select the system, global, or event properties that you do not want to forward. You can select multiple properties, but you cannot select more than 20.
After you complete these steps, click OK. The Quick Tracking system automatically tests the connection. If the connection test is successful, the task is created and started. (Note: The distribution task takes approximately 15 minutes to take effect after it is created.)
2.5 Start or stop a distribution task
By default, a task's status is `Running` after it is created, and data is distributed normally. To stop data distribution, click the Start/Stop toggle to change the task status. (Note: The status change takes approximately 15 minutes to take effect.)
2.6 Edit or delete a task
When a task's status is `Running`, you cannot edit or delete it from the Operations area. To edit or delete a task, you must first change its status to `Stopped` using the Start/Stop toggle. When the status is `Stopped`, you can click Edit to go to the task details page and modify the task. Before you delete a task, confirm that it is obsolete or invalid. Deleted tasks cannot be recovered. (Note: Task changes take approximately 15 minutes to take effect.)