All Products
Search
Document Center

Use TSDB for InfluxDB®️ to collect and analyze data - Unlock the full potential of valuable data

Last Updated: Aug 05, 2020

Background information

Substantial value lies in the large amounts of data generated in each industry. Alibaba Cloud launches TSDB for InfluxDB® to help you obtain business insights from large amounts of time series data. TSDB for InfluxDB® is a database service that is widely used in production and development environments to store and analyze time series data. Currently, Alibaba Cloud TSDB for InfluxDB® is in public preview. This topic describes how to migrate your data to the cloud for maximizing the data value.

TSDB for InfluxDB®

TSDB for InfluxDB® is a time series database service that handles high write and query loads. It allows you to store large amounts of time series data, including DevOps monitoring data, application metrics, and Internet of things (IoT) sensor data. It also allows you to analyze these time series data in real time. TSDB for InfluxDB® offers the following benefits:

  • Provides high-performance storage that is specific to time series data. Uses the Time-Structured Merge Tree (TSM) engine to provide features, such as high-speed data reads or writes and high-performance data compression.
  • Provides an easy and efficient way to write and query data by using the HTTP API.
  • Provides an easy way to query and aggregate time series data by using a SQL-like query language.
  • Allows you to index tags to improve query performance.
  • Provides retention policies. Retention policies automatically clear obsolete data.

After you purchase TSDB for InfluxDB® by following the instructions in Purchase process, you must create a database account and a password, and configure retention policies. For more information, see Manage user accounts and databases. On the Instance Details page, you can view two endpoints that are used to access TSDB for InfluxDB® over HTTP: VPC endpoint and public endpoint. Cloud-based instances that are connected to the same virtual private cloud (VPC) can access each other. You can apply for a public endpoint when you purchase a TSDB for InfluxDB® instance. This allows you to access the instance from both the internal network and the Internet. HTTPS-based encryption and authentication are enabled for VPCs and the Internet. Therefore, you must access TSDB for InfluxDB® over HTTPS channels.

connect

After you create an instance, you can create a user named test and a database named test on the Accounts page. You must ensure that the test user has read/write access to the test database.

test

Telegraf: Data transmission tool

Telegraf is a server agent that is written in Go. It collects data from systems and services, and writes the data to TSDB for InfluxDB®. Telegraf requires a small memory footprint. The plug-in architecture of Telegraf provides an easy way for developers to add extensions for other services. After you install Telegraf, you can use it to write data to TSDB for InfluxDB® over HTTP.

To write data to a TSDB for InfluxDB® instance, set the urls parameter to the public endpoint or the VPC endpoint that is displayed on the Instance Details page of the instance. Then, change the values of the database and retention_policy parameters that are marked in the preceding figure. If you do not specify a retention policy, the system writes data to the default retention policy of the database. The system authenticates your account when you write data to TSDB for InfluxDB®. Therefore, make sure that your account has write access to the database before you set the username and password parameters. To configure the access permissions, log on to the TSDB for InfluxDB® console, click Instances, find the target instance, and then click Manage in the Actions column. In the left-side navigation pane of the page that appears, click Accounts. After you install Telegraf and configure the parameters, you can write the data collected by Telegraf to TSDB for InfluxDB®.

Start Telegraf

Run the following command in Ubuntu, Debian, Red Hat, and CentOS operating systems whose versions are earlier than Ubuntu 15.04, Debian 8, RHEL 7, and CentOS 7, respectively:

  1. sudo service telegraf start

Run the following command in Ubuntu, Debian, Red Hat, and CentOS operating systems whose versions are Ubuntu 15.04, Debian 8, RHEL 7, and CentOS 7 or later, respectively:

  1. sudo systemctl start telegraf

High-speed data migration

If your InfluxDB instances run in your on-premises environment or on other platforms, you can migrate your InfluxDB services to TSDB for InfluxDB®. TSDB for InfluxDB® offers a data migration tool that provides a quick method for you to migrate your services. For more information, see Data migration. The data migration tool automatically creates and selects a destination database in TSDB for InfluxDB®. Therefore, do not create a database before you migrate data. Otherwise, write conflicts occur.
After the data is migrated, TSDB for InfluxDB® handles database operations and maintenance (O&M). This allows you to spend more time on writing new data, and querying and analyzing data that is stored in TSDB for InfluxDB® instances.

Various methods of writing data from clients

The InfluxDB open source community provides a variety of SDKs that are written in major programming languages. TSDB for InfluxDB® is fully compatible with a wide array of clients. This topic introduces the major methods of writing data:

Go SDK

  1. package main
  2. import (
  3. "fmt"
  4. "log"
  5. "math/rand"
  6. "net/url"
  7. "time"
  8. client "github.com/influxdata/influxdb1-client"
  9. )
  10. func main() {
  11. host, err := url.Parse(fmt.Sprintf("https://%s:%d", "xxx.influxdata.rds.aliyuncs.com", 3242))
  12. if err != nil {
  13. log.Fatal(err)
  14. }
  15. config := client.Config{
  16. URL: *host,
  17. Username: "test",
  18. Password: "test",
  19. }
  20. con, err := client.NewClient(config)
  21. if err != nil {
  22. log.Fatal(err)
  23. }
  24. _, _, err = con.Ping()
  25. if err != nil {
  26. log.Fatal(err)
  27. }
  28. var (
  29. shapes = []string{"circle", "rectangle", "square", "triangle"}
  30. colors = []string{"red", "blue", "green"}
  31. sampleSize = 1000
  32. pts = make([]client.Point, sampleSize)
  33. )
  34. rand.Seed(42)
  35. for i := 0; i < sampleSize; i++ {
  36. pts[i] = client.Point{
  37. Measurement: "shapes",
  38. Tags: map[string]string{
  39. "color": colors[rand.Intn(len(colors))],
  40. "shape": shapes[rand.Intn(len(shapes))],
  41. },
  42. Fields: map[string]interface{}{
  43. "value": rand.Intn(sampleSize),
  44. },
  45. Time: time.Now(),
  46. }
  47. }
  48. bps := client.BatchPoints{
  49. Points: pts,
  50. Database: "test",
  51. RetentionPolicy: "autogen",
  52. }
  53. _, err = con.Write(bps)
  54. if err != nil {
  55. log.Fatal(err)
  56. }
  57. }

When you configure an HTTP connection, you must specify the public endpoint or VPC endpoint of the instance, the database user name, and the password. To find these details, you can go to the Instance Details page. Before you write data, specify the destination database and a retention policy. If you do not specify the retention policy, the system writes data to the default retention policy. You can write data by row or in batches. To improve the write throughput, we recommend that you write data in batches. You can write 100 to 1,000 rows in each batch based on the data size.
For more information about how to prepare the Go environment, see Install the Go environment. You can download the Go SDK by using this link: Download the SDK. To compile and run the program, you can run the go run main.go command.

Java SDK

  1. package main;
  2. import java.util.concurrent.TimeUnit;
  3. import org.influxdb.InfluxDB;
  4. import org.influxdb.BatchOptions;
  5. import org.influxdb.InfluxDBFactory;
  6. import org.influxdb.dto.Point;
  7. public class StartMain {
  8. public static void main(String[] args) throws Exception {
  9. StartMain startMain = new StartMain();
  10. try {
  11. startMain.run();
  12. } catch (Exception e) {
  13. System.out.println(e);
  14. }
  15. }
  16. public void run() throws Exception {
  17. InfluxDB influxDB = InfluxDBFactory.connect("https://xxx.influxdata.rds.aliyuncs.com:3242", "test", "test");
  18. String dbName = "test";
  19. influxDB.setDatabase(dbName);
  20. String rpName = "autogen";
  21. influxDB.setRetentionPolicy(rpName);
  22. influxDB.enableBatch(BatchOptions.DEFAULTS);
  23. influxDB.write(Point.measurement("cpu")
  24. .time(System.currentTimeMillis(), TimeUnit.MILLISECONDS)
  25. .addField("idle", 90L)
  26. .addField("user", 9L)
  27. .addField("system", 1L)
  28. .build());
  29. influxDB.write(Point.measurement("disk")
  30. .time(System.currentTimeMillis(), TimeUnit.MILLISECONDS)
  31. .addField("used", 80L)
  32. .addField("free", 1L)
  33. .build());
  34. influxDB.close();
  35. }
  36. }

You can download the Java SDK by using this link: Download the Java SDK. You can configure the exceptionHandler() function in enableBatch to handle write exceptions.

  1. influxDB.enableBatch(BatchOptions.DEFAULTS.exceptionHandler(
  2. (failedPoints, throwable) -> { /* custom error handling here */ })
  3. );

CLI: influx

TSDB for InfluxDB® supports the influx command-line interface (CLI). If you use the influx CLI to write data, you must specify the following parameters: ssl, host, port, username, and password.

  1. influx -ssl -host xxx.influxdata.rds.aliyuncs.com -port 3242 -username admin -password admin

Clients can write data to TSDB for InfluxDB® by using the line protocol. This topic provides an example of the line protocol format.

  1. > INSERT cpu,host=serverA,region=us_west value=0.64

Shell scripts

  1. curl -i -XPOST -u test:test 'https://xxx.influxdata.rds.aliyuncs.com:3242/write?db=test' --data-binary 'cpu_load_short,host=server01,region=us-west value=0.64 1434055562000000000'

For more information about how to use shell scripts, see HTTP API.

Data analysis

TSDB for InfluxDB® supports a wide range of easy-to-use SQL-like queries. For more information about queries, see the TSDB for InfluxDB® documentation. You can use query statements and functions to explore the value that lies in data.

The following example queries the average metric value for each group of the color tag.

  1. > SELECT MEAN("value") FROM "shapes" GROUP BY "color"
  2. name: shapes
  3. tags: color=blue
  4. time mean
  5. ---- ----
  6. 0 216.25
  7. name: shapes
  8. tags: color=green
  9. time mean
  10. ---- ----
  11. 0 434.25
  12. name: shapes
  13. tags: color=red
  14. time mean
  15. ---- ----
  16. 0 540.25


InfluxDB® is a trademark registered by InfluxData, which is not affiliated with, and does not endorse, TSDB for InfluxDB®.