DataWorks:Single-table real-time synchronization: O&M and tuning

Start and stop tasks

• After starting a task, first confirm that schema preparation is complete. Then, observe whether full initialization has started and is making progress. Finally, verify that incremental synchronization is continuously reading and writing data. For message source tasks, also monitor the consumption offset and message backlog. For database log-based tasks, pay attention to the retention period for Binlog, WAL, or other logs.

• Before stopping a task, confirm that the source log or message retention period can cover the downtime window. If the downtime exceeds the retention period, the task may not be able to resume from its original offset. This often requires resetting the offset, skipping the historical backlog, or re-initializing the task. Before performing these actions, assess the risks of duplicate writes and data loss.

Configuration changes

Common modifications for single-table real-time tasks include adjusting field mappings, primary keys, write modes, partitioning rules, parallelism, resources, offsets, and dirty data policies.

Alert configuration

For single-table real-time tasks, monitor at least the task status, business latency, and failover events. Depending on the channel capabilities and task configuration, you can also add alerts for resource utilization, write errors, DDL notifications, and message backlog size.

For message source tasks, you also need to monitor the backlog in the source topic, shard, Logstore, or consumer group. For database log-based tasks, you need to monitor the retention of Binlog, WAL, archive logs, or replication slots. Configuring an alert only for task failure is not enough, as it cannot detect issues such as continuously increasing latency, slow writes, or an offset that is not advancing, even while the task is still running.

No data output

If a task shows as running but no new data appears at the destination, first confirm whether the source is generating new data. Then, determine if the reader is not consuming data, the pipeline is failing to process it, or the writer is not committing it.

Follow these steps to troubleshoot:

1. Check if new data exists at the source table, topic, Logstore, or log offset.

2. Check if the task is in the full initialization, incremental synchronization, or failover state.

3. Check if read metrics, write metrics, business latency, and checkpoints are advancing normally.

4. Check if the destination table, destination partitions, primary key, and write mode match the configuration.

5. Check for dirty data, field mapping failures, or destination permission issues.

If the DML/DDL statistics in the runtime details show no data for a short period but logs and destination writes appear normal, also investigate potential metric collection delays or errors. This helps avoid misdiagnosing a monitoring display issue as a synchronization interruption.

Slow or failed full initialization

When full initialization is slow, first identify whether the bottleneck is in source reading, the network, the resource group, destination writing, or uneven data splitting.

During the full initialization phase, do not rely solely on average throughput. A few large splits, wide tables, large fields, or hot partitions can determine the overall completion time.

High real-time latency

When real-time latency increases, first check if the latency is continuously decreasing. If the task has just completed full initialization, the real-time pipeline might be catching up on data that accumulated during full initialization. If latency continues to grow, identify whether the bottleneck is the reader, writer, resources, network, or an external system.

Follow this troubleshooting sequence:

1. In the task's runtime details, check the window wait time to determine if the bottleneck is primarily at the reader or writer.

2. Examine the task logs from the period of high latency. Search for keywords such as Error, Exception, and OutOfMemory.

3. Review failover records to check for frequent failover events, out-of-memory (OOM) errors, or external service exceptions.

4. Examine running metrics and events, focusing on throughput, backpressure, checkpoints, JVM memory, GC, DDL events, and dirty data.

5. Further troubleshoot the issue by checking the source, destination, resources, parallelism, and network. Do not add parallelism or resources until you identify the bottleneck.

The window wait time can serve as a first-level diagnostic:

After identifying the bottleneck, you can investigate further based on the symptoms:

• For message sources like Kafka, DataHub, and LogHub, focus on the number of partitions or shards. A single partition or shard can typically be consumed by only one parallel instance. If data is concentrated in a few partitions, simply increasing the total parallelism may not be effective. Check for hot partitions or shards by combining the cumulative byte counts of different reader threads with source monitoring data.

• For database log-based sources like MySQL, focus on large transactions, high DML volume, frequent DDL, and Binlog growth rate. If synchronization speed is low but latency continues to increase, check for source database pressure or large volumes of Binlog from unsynchronized tables by examining source audit logs, CPU, I/O, connection counts, and Binlog growth.

• When writing to MaxCompute, if logs show the error uploader map size has reached uploaderMapMaximumSize, it usually means that too many dynamic partition values were generated within a single flush interval. Prioritize adjusting the partition granularity or dynamic partition field. Avoid using high-cardinality fields such as second-level timestamps, order IDs, or user IDs as partition values. Then, assess whether to increase parallelism or resources.

DDL processing issues

If a task fails, latency increases, or the destination schema becomes inconsistent after a DDL operation, first confirm whether the current channel supports the DDL type and what the task's processing strategy is for that DDL.

Key areas to check:

• Which DDL operations occurred at the source, such as adding a column, deleting a column, changing a type, renaming, creating a table, deleting a table, or changing an index.

• Whether the destination supports the same DDL or an equivalent schema change.

• Whether the task's strategy is to apply, ignore, alert on, or stop on the DDL.

• Whether the destination account has permission to alter the table.

• Whether field mappings, the primary key, and partitions remain consistent after the DDL.

Do not simply ignore unsupported DDL. Ignoring a DDL might allow the task to continue running, but subsequent DML writes could fail due to field mismatches, primary key changes, or an inconsistent destination schema.

UPDATE or DELETE issues

If data at the destination does not match expectations after UPDATE or DELETE operations, first check if the destination has a primary or unique key to locate records, and verify that the primary key mapping between the source and destination is consistent.

Common causes include:

• The destination table lacks a primary or unique key, preventing record-level updates or deletions.

• The source primary key and destination primary key are inconsistent.

• The primary key field is transformed, renamed, or written as null.

• The destination write mode does not support updates or deletes.

• The task is configured to ignore DELETE operations or has a special write policy.

• Write failures are recorded as dirty data, but the task continues to run.

To resolve this, first use a few sample records to verify the source events, task mappings, and destination results. Then, decide whether to adjust the primary key, change the write mode, or re-initialize the destination table.

Increased dirty data

When the amount of dirty data increases, do not simply raise the dirty data threshold. First, confirm whether the dirty data will cause missing data, null fields, field truncation, primary key conflicts, or partition errors at the destination.

Key areas to check:

• Whether field types, lengths, precision, and non-null constraints match.

• Whether the destination's primary key, unique key, or partition field meets requirements.

• Whether the source has introduced oversized strings, invalid encodings, special JSON, unparsable timestamps, or unsupported data types.

• Whether the destination is experiencing throttling, write failures, or server-side filtering.

Only consider temporarily increasing the dirty data threshold if you can confirm that your business can tolerate discarding these error records. Plan to fix the source data or destination schema later.