Migrate data from Prometheus to a Lindorm time series database

• Linux
• Java Development Kit (JDK): You can use JDK 1.8 or later. We recommend that you use JDK 1.8.
• Python: We recommend that you use Python 2.6.x.
• Prometheus: Only Prometheus versions 2.9.x are supported. Earlier versions are not fully compatible with DataX.
• Lindorm time series engine: Only V2.4.x and later are supported. Earlier versions are not fully compatible with DataX.

Click DataX to download DataX and the Prometheus Reader and TSDB Writer plug-ins.

The plug-ins used in the test are Stream Reader and Stream Writer. These plug-ins do not require external dependencies. Stream Reader and Stream Writer are used to simulate a data migration process. Stream Reader generates random strings. Stream Writer receives the strings and then prints them on your CLI.

• Install DataX

  Decompress the installation package to a specified directory and then run the datax.py script to start the migration task.

  $ cd ${DATAX_HOME}
  $ python bin/datax.py job/job.json        

• Check the migration result

  If the following information is returned, the data is migrated:

  Task start time                    : 2019-04-26 11:18:07
  Task end time                    : 2019-04-26 11:18:17
  Time consumed                    :                 10s
  Average traffic                    :          253.91KB/s
  Write rate                    :          10000rec/s
  Number of records obtained                    :              100000
  Number of write and read failures                    :                   0
                          

  For more information, watch Data migration quick start.

In Step 3, the Stream Reader and Stream Writer plug-ins are used to test the migration process of DataX. The test results indicate that DataX can migrate data. The following parts describe how to use the Prometheus Reader and TSDB Writer plug-ins to migrate data from Prometheus to a Lindorm time series database.

• Configure a migration task

  Configure a task to migrate data from Prometheus to a Lindorm time series database. In this example, the task name is opentsdb2tsdb.json. The following code provides an example of the task configuration. For more information about the parameters, see the "Parameter description" section.

  {
    "job": {
      "content": [
        {
          "reader": {
            "name": "prometheusreader",
            "parameter": {
              "endpoint": "http://localhost:9090",
              "column": [
                "up"
              ],
              "beginDateTime": "2019-05-20T16:00:00Z",
              "endDateTime": "2019-05-20T16:00:10Z"
            }
          },
          "writer": {
            "name": "tsdbhttpwriter",
            "parameter": {
              "endpoint": "http://localhost:8242"
            }
          }
        }
      ],
      "setting": {
        "speed": {
          "channel": 1
        }
      }
    }
  }
                          

• Start the task

  $ cd ${DATAX_HOME}/..
  $ ls
    datax/  datax.tar.gz  prometheus2tsdb.json
  $ python datax/bin/datax.py prometheus2tsdb.json
                          

• Check the migration result

  If the following information is returned, the data is migrated:

  Task start time                    : 2019-05-20 20:22:39
  Task end time                    : 2019-05-20 20:22:50
  Time consumed                    :                 10s
  Average traffic                    :          122.07KB/s
  Write rate                    :          1000rec/s
  Number of records obtained                    :              10000
  Number of write and read failures                    :                   0
                          

  For more information, watch Migrate data from Prometheus to a Lindorm time series database.

TSDB Writer writes data by calling the /api/put HTTP API operation. Therefore, to ensure successful data migration, make sure that the processes of migration tasks can access the HTTP API provided by Lindorm times series engine. Otherwise, a connection exception is reported.

Prometheus Reader reads data by calling the /api/v1/query_range operation. Therefore, to ensure successful data migration, make sure that the processes of migration tasks can access the HTTP API provided by Prometheus. Otherwise, a connection exception is reported.