Application Real-Time Monitoring Service:Manage Prometheus monitoring with Terraform

Prerequisites

Before you begin, make sure that you have:

• A Prometheus instance for Container Service or ECS. See

• Terraform 0.12.28 or later. Run terraform --version to check your version

  • Cloud Shell includes Terraform by default with your account pre-configured

  • To install Terraform locally, see Install and configure Terraform

• Alibaba Cloud credentials configured through one of the methods described in the following section

export ALICLOUD_ACCESS_KEY="<your-access-key>"
export ALICLOUD_SECRET_KEY="<your-secret-key>"
export ALICLOUD_REGION="cn-beijing"

provider "alicloud" {
  access_key = "<your-access-key>"
  secret_key = "<your-secret-key>"
  region     = "cn-beijing"
}

Supported monitoring types

The monitoring types available depend on your Prometheus instance type:

Health check agent constraints:

• The status parameter is not supported for Probe resources.

• Name format: <custom-name>-{tcp|http|ping}-blackbox. For example, name1-tcp-blackbox indicates a TCP health check.

• For ECS instances (fully managed), the namespace must be empty or follow the format <vpc-id>-<user-id>. For example, vpc-0jl4q1q2of2tagvwxxxx-11032353609xxxx.

Argument reference

All monitoring types use the alicloud_arms_prometheus_monitoring resource with the following arguments:

Deploy a monitoring resource

All monitoring types share the same Terraform workflow. Each section below provides the config_yaml for a specific type.

1. Create a working directory and add a main.tf file with the provider block:

      provider "alicloud" {
      }

2. Initialize Terraform: Expected output:

      terraform init

      Initializing the backend...
   
      Initializing provider plugins...
      - Checking for available provider plugins...
      - Downloading plugin for provider "alicloud" (hashicorp/alicloud) 1.90.1...
      ...
   
      You may now begin working with Terraform. Try running "terraform plan" to see
      any changes that are required for your infrastructure. All Terraform commands
      should now work.
   
      If you ever set or change modules or backend configuration for Terraform,
      rerun this command to reinitialize your working directory. If you forget, other
      commands will detect it and remind you to do so if necessary.

3. Add the resource configuration for your monitoring type to main.tf. See the configuration examples in the following sections.

4. Preview the changes: The output shows the resources that Terraform will create. For example:

      terraform plan

      Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
        + create
   
      Terraform will perform the following actions:
      ...
      Plan: 1 to add, 0 to change, 0 to destroy.

5. Apply the configuration: The output shows the execution plan and prompts for confirmation: If yes is returned, the monitoring resource is created for the current Prometheus instance.

      terraform apply

      ...
      Plan: 1 to add, 0 to change, 0 to destroy.
   
      Do you want to perform these actions?
        Terraform will perform the actions described above.
        Only 'yes' will be accepted to approve.
   
        Enter a value: yes

Add a ServiceMonitor

Add the following resource block to main.tf:

resource "alicloud_arms_prometheus_monitoring" "myServiceMonitor1" {
  cluster_id  = "c77e1106f429e4b46b0ee1720cxxxxx"   # The ID of the Prometheus instance.
  status      = "run"
  type        = "serviceMonitor"
  config_yaml = <<-EOT
    apiVersion: monitoring.coreos.com/v1
    kind: ServiceMonitor
    metadata:
      name: tomcat-demo
      namespace: default
    spec:
      endpoints:
        - interval: 30s
          path: /metrics
          port: tomcat-monitor
      namespaceSelector:
        any: true
      selector:
        matchLabels:
          app: tomcat
  EOT
}

Key fields in config_yaml:

Add a PodMonitor

Add the following resource block to main.tf:

resource "alicloud_arms_prometheus_monitoring" "myPodMonitor1" {
  cluster_id  = "c77e1106f429e4b46b0ee1720cxxxxx"   # The ID of the Prometheus instance.
  status      = "run"
  type        = "podMonitor"
  config_yaml = <<-EOT
    apiVersion: "monitoring.coreos.com/v1"
    kind: "PodMonitor"
    metadata:
      name: "podmonitor-demo"
      namespace: "default"
    spec:
      namespaceSelector:
        any: true
      podMetricsEndpoints:
        - interval: "30s"
          path: "/metrics"
          port: "tomcat-monitor"
      selector:
        matchLabels:
          app: "nginx2-exporter"
  EOT
}

Key fields in config_yaml:

Add a custom job

Add the following resource block to main.tf:

resource "alicloud_arms_prometheus_monitoring" "myCustomJob1" {
  cluster_id  = "c77e1106f429e4b46b0ee1720cxxxxx"   # The ID of the Prometheus instance.
  status      = "run"
  type        = "customJob"
  config_yaml = <<-EOT
    scrape_configs:
      - job_name: prometheus1
        honor_timestamps: false
        honor_labels: false
        scheme: http
        metrics_path: /metric
        static_configs:
          - targets:
              - 127.0.0.1:9090
  EOT
}

Key fields in config_yaml:

Add a health check agent

Add the following resource block to main.tf:

resource "alicloud_arms_prometheus_monitoring" "myProbe1" {
  cluster_id  = "c77e1106f429e4b46b0ee1720cxxxxx"   # The ID of the Prometheus instance.
  type        = "probe"
  config_yaml = <<-EOT
    apiVersion: monitoring.coreos.com/v1
    kind: Probe
    metadata:
      name: name1-tcp-blackbox
      namespace: arms-prom
    spec:
      interval: 30s
      jobName: blackbox
      module: tcp_connect
      prober:
        path: /blackbox/probe
        scheme: http
        url: 'localhost:9335'
      targets:
        staticConfig:
          static:
            - 'arms-prom-admin.arms-prom:9335'
  EOT
}

Key fields in config_yaml:

Delete monitoring resources

To remove all monitoring resources managed by the current Terraform configuration, run:

terraform destroy

Enter yes when prompted to confirm. Expected output:

...
Do you really want to destroy all resources?
  Terraform will destroy all your managed infrastructure, as shown above.
  There is no undo. Only 'yes' will be accepted to confirm.

  Enter a value: yes
...
Destroy complete! Resources: 1 destroyed.

What's next

• Use Terraform to manage Prometheus instances

• Create a Terraform template

• Install and configure Terraform