Elasticsearch:Integrate aliyun-timestream with Prometheus and Grafana

Prerequisites

Before you begin, ensure that you have:

• An Alibaba Cloud Elasticsearch cluster that meets one of the following version requirements:

  • Cluster version V7.10, kernel version V1.8.0 or later

  • Cluster version V7.16 or later, kernel version V1.7.0 or later

For instructions on creating a cluster, see Create an Alibaba Cloud Elasticsearch cluster.

Procedure

1. Step 1: Prepare the environment

2. Step 2: Download and start node_exporter

3. Step 3: Download, configure, and start Prometheus

4. Step 4: Download, start, and configure a Grafana dashboard

Step 1: Prepare the environment

1. Create a General-purpose Commercial Edition Alibaba Cloud Elasticsearch cluster of version 7.16.

   For more information, see Create an Alibaba Cloud Elasticsearch cluster.

2. Create an ECS instance in the same VPC as the Elasticsearch instance that you created in Step 1, and select Linux as the operating system.

   For details on creating an ECS instance, see Create an instance with Custom Launch. This ECS instance is used to access the Alibaba Cloud Elasticsearch cluster and deploy Prometheus and Grafana.

3. Create a time series index in Elasticsearch to store data from Prometheus.

   1. Log on to the Kibana console for your Alibaba Cloud Elasticsearch cluster.

      For more information, see Log on to the Kibana console.

   2. In the left-side navigation pane, click the icon and choose Management > Dev Tools.

   3. In the console, run the PUT _time_stream/prom_index command to create a time series index named prom_index.

Step 2: Download and start node_exporter

node_exporter collects various hardware- and kernel-related metrics and exposes them for Prometheus to scrape. For more information, see node_exporter.

1. Connect to the ECS instance.

   For more information, see Log on to a Linux instance by using a password or key.

   Note

   This topic uses a non-root user for all operations.

2. Download the node_exporter installation package.

   This guide uses node_exporter 1.3.1 as an example. Run the following command to download it:

   wget https://github.com/prometheus/node_exporter/releases/download/v1.3.1/node_exporter-1.3.1.linux-amd64.tar.gz

3. Unpack the installation package and start node_exporter.

   tar xvfz node_exporter-1.3.1.linux-amd64.tar.gz
   cd node_exporter-1.3.1.linux-amd64
   ./node_exporter

Step 3: Download, configure, and start Prometheus

1. Connect to the ECS instance.

   For more information, see Log on to a Linux instance by using a password or key.

2. In the home directory, download the Prometheus installation package.

   This guide uses Prometheus 2.36.2 as an example. Run the following command to download it:

   cd ~
   wget https://github.com/prometheus/prometheus/releases/download/v2.36.2/prometheus-2.36.2.linux-amd64.tar.gz

3. Unpack the Prometheus installation package.

   tar xvfz prometheus-2.36.2.linux-amd64.tar.gz

4. In the prometheus.yml file, configure node_exporter and remote_write.

   cd prometheus-2.36.2.linux-amd64
   vim prometheus.yml

   The following code provides a configuration example:

   scrape_configs:
     # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
     - job_name: 'prometheus'
   
       # metrics_path defaults to '/metrics'
       # scheme defaults to 'http'.
   
       static_configs:
       - targets: ['localhost:9090']
   
     # Configure node_exporter.
     - job_name: "node"
       static_configs:
       - targets: ["127.0.0.1:9100"]
   
   # Configure remote_write. Ensure that Prometheus can access the Elasticsearch cluster.
   remote_write:
     - url: "http://xxx:9200/_time_stream/prom_write/prom_index"
       basic_auth:
        username: elastic
        password: xxxx

   Parameter	Description
   node_exporter	Configure the connection information for node_exporter. Set targets to the access address and port of node_exporter. Because this guide uses the same ECS instance to deploy Prometheus and node_exporter, the IP address is 127.0.0.1 and the default port 9100 is used.
   remote_write	The connection settings for the time series index in your Alibaba Cloud Elasticsearch cluster. You must configure the following basic parameters. For more advanced parameters, see remote_write. url: The endpoint URL for the time series index, in the following format: http://<public_or_internal_endpoint_of_your_Elasticsearch_cluster>:9200/_time_stream/prom_write/<your_time_series_index>. Note Public or internal endpoint of your Elasticsearch cluster: Find this on the Basic Information page of your Alibaba Cloud Elasticsearch cluster console. If the ECS instance where Prometheus is deployed is in the same VPC as your Elasticsearch cluster, use the internal endpoint, as in this example. Otherwise, you must use the public endpoint and configure a public IP address whitelist. For more information, see Manage IP address whitelists. <your_time_series_index>: The name of the time series index that receives data from Prometheus. You must create this index in advance. This guide uses prom_index as an example. username: The username to access the time series index. The default is elastic. You can also use a custom user, but you must ensure the user has the required permissions to access and manage the time series index. For more information, see Manage access control with Elasticsearch X-Pack roles. password: The password for the specified user. The password for the elastic account is set when you create the cluster. If you forget it, you can reset it. For more information, see Reset an Elasticsearch cluster password.

5. Start Prometheus.

   ./prometheus

6. Verify that Prometheus data is being sent to the time series index in Elasticsearch.

   In the Kibana console for your Alibaba Cloud Elasticsearch cluster, run the following commands to verify:

   • Check if the prom_index index contains data.

     GET _cat/indices/prom_index?v&s=i

     Expected result:

     PUT _time_stream/prom_index
     
     GET _cat/indices/prom_index?v&s=i
     
     health status index                              uuid                pri rep docs.count docs.deleted store.size pri.store.size
     green  open   .ds-prom_index-2022.06.29-000001 U-y2z4j4RyWlKmxxx   1   1        178            0      5.6kb          226b

   • Confirm that you can query and view the data.

     GET prom_index/_search

     Expected result:

     PUT _time_stream/prom_index
     GET _cat/indices/prom_index?v&s=i
     GET prom_index/_search
     
     {
       "took" : 354,
       "timed_out" : false,
       "_shards" : {
         "total" : 1,
         "successful" : 1,
         "skipped" : 0,
         "failed" : 0
       },
       "hits" : {
         "total" : {
           "value" : 3036,
           "relation" : "eq"
         },
         "max_score" : 1.0,
         "hits" : [
           {
             "_index" : ".ds-prom_index-2022.06.29-000001",
             "_type" : "_doc",
             "_id" : "kBXXroEBMnT0XiRApR78",
             "_score" : 1.0,
             "_source" : {
               "labels" : {
                 "instance" : "localhost:9090",
                 "job" : "prometheus"
               },
               "metrics" : {
                 "prometheus_remote_storage_exemplars_in_total" : 0.0,
                 "prometheus_target_scrape_pools_failed_total" : 0.0,
                 "prometheus_tsdb_data_replay_duration_seconds" : 6.66077E-4,
                 "prometheus_tsdb_wal_completed_pages_total" : 1.0,
                 "prometheus_tsdb_lowest_timestamp" : 1.656495558477E12,
                 "prometheus_tsdb_head_gc_duration_seconds_count" : 0.0,
                 "prometheus_tsdb_out_of_bound_samples_total" : 0.0,
                 "prometheus_tsdb_isolation_high_watermark" : 30.0,
                 "prometheus_tsdb_storage_blocks_bytes" : 0.0,
                 "up" : 1.0,
                 "go_gc_duration_seconds_sum" : 9.66942E-4,
                 "process_cpu_seconds_total" : 2.96,
                 "prometheus_tsdb_wal_writes_failed_total" : 0.0,
                 "prometheus_tsdb_head_max_time_seconds" : 1.656495768E9,
                 "prometheus_tsdb_head_chunks_created_total" : 455.0,
                 "prometheus_sd_kuma_fetch_duration_seconds_sum" : 0.0
               }
             }
           }
         ]
       }
     }

Step 4: Download, start, and configure Grafana

1. Connect to the ECS instance.

   For more information, see Log on to a Linux instance by using a password or key.

2. In the home directory, download the Grafana installation package.

   This guide uses Grafana 9.0.2 as an example. Run the following command to download it:

   cd ~
   wget https://dl.grafana.com/enterprise/release/grafana-enterprise-9.0.2.linux-amd64.tar.gz

3. Unpack the Grafana installation package and start Grafana.

   tar xvfz grafana-enterprise-9.0.2.linux-amd64.tar.gz
   cd grafana-9.0.2
   ./bin/grafana-server

4. In a web browser, go to http://<your_ECS_public_IP>:3000 to open the Grafana logon page. Enter your username and password to access the Grafana console.

   • When you log on for the first time, use admin for both the username and password. You will be prompted to change the password.

   • To find the public IP address of your ECS instance, go to the ECS console and find the address in the IP Address column for your instance.

   • Grafana's default port is 3000. To access this port, add an inbound rule to the security group of your ECS instance. Set the Destination port to 3000 and the Source to your client's IP address. For more information, see Add a security group rule.

5. Add a Prometheus data source in Grafana.

   1. In the left navigation bar of the Grafana console, select > Data sources.

   2. On the Data sources tab, click Add data source.

   3. In the Time series databases list, click Prometheus.

   4. On the Settings tab, configure the Prometheus data source.

      The following table describes the parameters that must be configured for this example.

      Parameter	Description
      URL	The endpoint URL for the time series index, in the following format: http://<public_or_internal_endpoint_of_your_Elasticsearch_cluster>:9200/_time_stream/prom/<your_time_series_index>. Note Public or internal endpoint of your Elasticsearch cluster: Find this on the Basic Information page of your Alibaba Cloud Elasticsearch cluster console. If the ECS instance where Prometheus is deployed is in the same VPC as your Elasticsearch cluster, use the internal endpoint, as in this example. Otherwise, you must use the public endpoint and configure a public IP address whitelist. For more information, see Manage IP address whitelists. <your_time_series_index>: The name of the time series index that receives data from Prometheus. You must create this index in advance. This guide uses prom_index as an example.
      Basic auth	Toggles Basic Authentication for the cluster. When enabled, you must provide a username and password.
      User	The username to access the time series index. The default is elastic. You can also use a custom user, but you must ensure the user has the required permissions to access and manage the time series index. For more information, see Manage access control with Elasticsearch X-Pack roles.
      Password	The password for the specified user. The password for the elastic account is set when you create the cluster. If you forget it, you can reset it. For more information, see Reset an Elasticsearch cluster password.

   5. Click Save & test.

      If the configuration is successful, a Data source is working message appears.

6. In Grafana, create a dashboard for the Prometheus data source.

   1. In the left navigation bar of the Grafana console, select > New dashboard.

   2. Click Add a new panel.

   3. Select the data source, choose a time range, and then click Run queries.

      For example, set Data source to test_datastream, set Metric to go_gc_duration_seconds, and set the time range to Last 2 days.

   4. Click Save in the upper-right corner to save the dashboard.

7. In Grafana, import the default node_exporter dashboard and connect it to your Prometheus data source to monitor metrics.

   1. In the left navigation bar of the Grafana console, select > Import.

   2. In the Import via grafana.com text box, enter the dashboard URL https://grafana.com/grafana/dashboards/1860 or the ID 1860.

   3. Click Load.

   4. On the configuration page, from the Prometheus data source drop-down list, select the data source that you created.

   5. Click Import.

   6. On the dashboard, select a time range in the upper-right corner to view the metrics for that period.

      

      Note

      For more detailed information about Grafana, see the Grafana documentation.

References

• Quick start: Manage Elasticsearch time series data with aliyun-timestream

• aliyun-timestream API reference

• Integrate aliyun-timestream with Prometheus