edit-icon download-icon

Sync log check

Last Updated: Apr 11, 2018

Overview

Data Integration is a stable, efficient, and auto scalable data synchronization platform provided by the Alibaba Group. It is designed to implement fast and stable data movement and synchronization between various heterogeneous data sources in complex network environments. A wide range of data sources: text storage (FTP/SFTP/OSS/multimedia files), databases (RDS/DRDS/MySQL/PostgreSQL), NoSQL (Memcache/Redis/MongoDB/HBase), big data (MaxCompute/AnalyticDB/HDFS), and MPP database (HybridDB for MySQL).

As Data Integration enables interconnection between different types of databases in complex network environments, errors may occur while you use it. Next, let us analyze the log composition of Data Integration.

Where does a task start

1

As shown in the preceding screenshot: Start Job means starting the job. The log under Start Job > running in Pipeline[XXXXX]“ is mainly used to identify the server on which the task is running. If XXXXX contains basecommon_group_XXXX, it means that the task is running on a server of a public resource group. If not, it runs on a server of your custom resource group. To view the name of the server running a task, see the description in section Task running status.

Actual task running code

After a task starts, the log displays the actual task running code (for security reasons, sensitive information is indicated with an asterix (*)) as shown in the following figure.

2

The actual code sample for the task in the figure is as follows.

  1. Reader: odps
  2. shared=[false ]
  3. bindingCalcEngineId=[9617 ]
  4. column=[["t_name","t_password","pt"] ]
  5. description=[connection from odps calc engine 9617]
  6. project=[XXXXXXXXX ]
  7. *accessKey=[******** ]
  8. gmtCreate=[2016-10-13 16:42:19 ]
  9. type=[odps ]
  10. accessId=[XXXXXXXXX ]
  11. datasourceType=[odps ]
  12. odpsServer=[http://service.xxx.aliyun.com/api]
  13. endpoint=[http://service.xxx.aliyun.com/api]
  14. partition=[pt=20170425 ]
  15. datasourceBackUp=[odps_first ]
  16. name=[odps_first ]
  17. tenantId=[168418089343600 ]
  18. subType=[ ]
  19. id=[30525 ]
  20. authType=[1 ]
  21. projectId=[27474 ]
  22. table=[t_name ]
  23. status=[1 ]
  24. Writer: odps
  25. shared=[false ]
  26. bindingCalcEngineId=[9617 ]
  27. column=[["id","name","pt"] ]
  28. description=[connection from odps calc engine 9617]
  29. project=[XXXXXXXXX ]
  30. *accessKey=[******** ]
  31. gmtCreate=[2016-10-13 16:42:19 ]
  32. type=[odps ]
  33. accessId=[XXXXXXXXX ]
  34. datasourceType=[odps ]
  35. odpsServer=[http://service.xxx.aliyun.com/api]
  36. endpoint=[http://service.xxx.aliyun.com/api]
  37. partition=[ ]
  38. truncate=[true ]
  39. datasourceBackUp=[odps_first ]
  40. name=[odps_first ]
  41. tenantId=[XXXXXXXXX ]
  42. subType=[ ]
  43. id=[30525 ]
  44. authType=[1 ]
  45. projectId=[27474 ]
  46. table=[test_pm ]
  47. status=[1 ]

This is a typical task code used to synchronize a MaxCompute (formerly known as ODPS) data source to a MaxCompute data source. For analysis on the task code, see MaxCompute Reader and MaxCompute Writer.

Task running status

We have introduced the actual task running code, which is followed by the running status of the task, as shown in the following figure.

3

The content included in the larger red box records the start and end time of the task.

  • When Status: 2(WAIT) is displayed, it means that the task is waiting.

  • When Status: 3(RUN) is displayed, it means that the task is running.

  • When Status: 0(SUCCESS) is displayed, it means that the task is completed successfully.

Note: Under the record indicating task completion is INFO Start run shell on node XXXXXXX, which means that the task actually runs on XXXXXXX server.

Troubleshooting assistant: Dirty data cannot be written in, and the log reports the following error in case of dirty data

Detailed running logs

In fact, the task log of data synchronization has ended already. The following log is the detailed running log of DataX (Data Integration applies encapsulation to the open-source project DataX of Alibaba), as shown in the following figure.

1

If the problem still persists even after referring to the common errors article, contact us.

Thank you! We've received your feedback.