Create and manage Elasticsearch clusters with Terraform - Elasticsearch

Prerequisites

A RAM user with least-privilege permissions. Use a RAM user with only the required permissions to reduce AccessKey leak risk. Create a RAM user and Grant permissions to a RAM user. Attach the following custom policy:

  {
    "Version": "1",
    "Statement": [
      {
        "Effect": "Allow",
        "Action": [
          "vpc:CreateVpc",
          "vpc:DeleteVpc",
          "vpc:CreateVSwitch",
          "vpc:DeleteVSwitch",
          "ecs:CreateSecurityGroup",
          "ecs:ModifySecurityGroupPolicy",
          "ecs:DescribeSecurityGroups",
          "ecs:ListTagResources",
          "ecs:DeleteSecurityGroup",
          "ecs:DescribeSecurityGroupAttribute"
        ],
        "Resource": "*"
      },
      {
        "Effect": "Allow",
        "Action": [
          "vpc:DescribeVpcAttribute",
          "vpc:DescribeRouteTableList",
          "vpc:DescribeVSwitchAttributes"
        ],
        "Resource": "*"
      },
      {
        "Effect": "Allow",
        "Action": [
          "elasticsearch:CreateInstance",
          "elasticsearch:DescribeInstance",
          "elasticsearch:ListAckClusters",
          "elasticsearch:UpdateDescription",
          "elasticsearch:ListInstance",
          "elasticsearch:ListAvailableEsInstanceIds"
        ],
        "Resource": "*"
      }
    ]
  }

A Terraform runtime environment set up using one of the following methods:
- Terraform Explorer (recommended): Browser-based Terraform environment. No installation required. Log on to Terraform Explorer to get started.
- Cloud Shell: Terraform preinstalled with credentials configured. Run commands directly.
- On-premises machine: Install and configure Terraform locally. Best for restricted networks or custom setups.

Important

This tutorial creates billable resources. Release them when no longer needed.

Resources created

Resource	Terraform resource type	Description
Virtual private cloud (VPC)	alicloud_vpc	Network isolation for the Elasticsearch cluster
Security group	alicloud_security_group	Network access control rules
vSwitch	alicloud_vswitch	Subnet within the VPC
Elasticsearch cluster	alicloud_elasticsearch_instance	Managed Elasticsearch cluster (billable). Billing overview.

Step 1: Write the Terraform configuration

Create a working directory and a file named main.tf.

Copy the following code into main.tf. This configuration creates a VPC, a security group, a vSwitch, and an Elasticsearch cluster with these parameters:

Important

Do not hardcode AccessKey credentials in .tf files. Use environment variables (ALIBABA_CLOUD_ACCESS_KEY_ID and ALIBABA_CLOUD_ACCESS_KEY_SECRET) or the Terraform provider credential configuration. Credentials committed to version control risk being leaked.

Parameter	Value	Description
`instance_charge_type`	`PostPaid`	Pay-as-you-go billing method.
`data_node_amount`	`2`	Number of data nodes in the cluster.
`data_node_spec`	`elasticsearch.sn2ne.large`	Instance type for data nodes.
`data_node_disk_size`	`20`	Disk size for each data node, in GB.
`data_node_disk_type`	`cloud_ssd`	Disk type for data nodes.
`data_node_disk_performance_level`	`PL1`	Performance level for cloud_ssd disks.
`version`	`6.7_with_X-Pack`	Elasticsearch version with X-Pack enabled.
`zone_count`	`1`	Number of zones for cluster deployment.
`master_node_spec`	`elasticsearch.sn2ne.large`	Instance type for dedicated master nodes.
`kibana_node_spec`	`elasticsearch.sn2ne.large`	Instance type for the Kibana node.

Important

The node_spec variable is assigned to data_node_spec, master_node_spec, and kibana_node_spec at the same time. When you select node specifications:

For data nodes and master nodes, use a new-generation specification with the .new suffix, such as elasticsearch.sn1ne.large.new (recommended).
For the Kibana node, you must use an old-format specification, such as elasticsearch.sn1ne.large. Otherwise, cluster creation fails.

Use a consistent specification format across all three node types. Mixing new-generation and old-format specifications causes an order validation failure. For the node specifications supported by Elasticsearch clusters, see Node specifications.

   # ------------------------------------
   # Variables
   # ------------------------------------
   variable "region" {
     default = "cn-qingdao"
   }

   variable "vpc_cidr_block" {
     default = "172.16.0.0/16"
   }

   variable "vsw_cidr_block" {
     default = "172.16.0.0/24"
   }

   variable "node_spec" {
     default = "elasticsearch.sn2ne.large"
   }

   # ------------------------------------
   # Provider and data sources
   # ------------------------------------
   provider "alicloud" {
     region = var.region
   }

   # Query available zones that support vSwitch creation and cloud_ssd disks
   data "alicloud_zones" "default" {
     available_resource_creation = "VSwitch"
     available_disk_category     = "cloud_ssd"
   }

   # Generate a random suffix to avoid resource name conflicts
   resource "random_integer" "default" {
     min = 10000
     max = 99999
   }

   # ------------------------------------
   # Network resources
   # ------------------------------------

   # Create a VPC
   resource "alicloud_vpc" "vpc" {
     vpc_name   = "vpc-test_${random_integer.default.result}"
     cidr_block = var.vpc_cidr_block
   }

   # Create a security group in the VPC
   resource "alicloud_security_group" "group" {
     name   = "test_${random_integer.default.result}"
     vpc_id = alicloud_vpc.vpc.id
   }

   # Create a vSwitch in the first available zone
   resource "alicloud_vswitch" "vswitch" {
     vpc_id       = alicloud_vpc.vpc.id
     cidr_block   = var.vsw_cidr_block
     zone_id      = data.alicloud_zones.default.zones[0].id
     vswitch_name = "vswitch-test-${random_integer.default.result}"
   }

   # ------------------------------------
   # Elasticsearch cluster
   # ------------------------------------
   resource "alicloud_elasticsearch_instance" "instance" {
     description          = "test_Instance"
     instance_charge_type = "PostPaid"                   # Pay-as-you-go billing
     data_node_amount     = "2"                          # Number of data nodes
     data_node_spec       = var.node_spec                # Data node instance type
     data_node_disk_size  = "20"                         # Data node disk size in GB
     data_node_disk_type  = "cloud_ssd"                  # Data node disk type
     vswitch_id           = alicloud_vswitch.vswitch.id  # Deploy in the created vSwitch
     password             = "es_password_01"             # Cluster access password
     version              = "6.7_with_X-Pack"            # Elasticsearch version with X-Pack
     master_node_spec     = var.node_spec                # Dedicated master node instance type
     zone_count           = "1"                          # Single-zone deployment
     master_node_disk_type = "cloud_ssd"                 # Master node disk type
     kibana_node_spec     = var.node_spec                # Kibana node instance type
     data_node_disk_performance_level = "PL1"            # SSD performance level

     tags = {
       Created = "TF",
       For     = "example",
     }
   }

Step 2: Initialize Terraform

Run the following command to initialize Terraform:

terraform init

Expected output on success:

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

Step 3: Create the Elasticsearch cluster

Run the following command to create the resources:

terraform apply

When prompted, enter yes and press Enter. Cluster creation takes 10 to 30 minutes.

Expected output on success:

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes


Apply complete!  Resources: 5 added, 0 changed, 0 destroyed.

Step 4: Verify the result

Run the terraform show command

Run the following command to view the created resources:

terraform show

shell@Alicloud:~/es$ terraform show
# alicloud_elasticsearch_instance.instance: (tainted)
resource "alicloud_elasticsearch_instance" "instance" {
    id    = "es-cn-e3bb xxx"
    tags = {
        "Created" = "TF"
        "For"     = "example"
    }
}

# alicloud_security_group.group:
resource "alicloud_security_group" "group" {
    id                  = "sg-m5ee xxx"
    inner_access        = true
    inner_access_policy = "Accept"
    name                = "test_ xxx"
    security_group_type = "normal"
    vpc_id              = "vpc-m5e09u xxx"
}

# alicloud_vpc.vpc:
resource "alicloud_vpc" "vpc" {
    cidr_block            = "172.16.0.0/16"
    classic_link_enabled  = false
    create_time           = "2024-11-04T08:25:51Z"
    enable_ipv6           = false
    id                    = "vpc-m5e09u xxx"
    ipv6_cidr_blocks      = []
    name                  = "vpc-test_87753"
    resource_group_id     = "rg-acfm xxx"
    route_table_id        = "vtb-m5e5hqk8bwsrmw xxx"
    router_id             = "vrt-m5e0kbfyjtyceye xxx"
    router_table_id       = "vtb-m5e5hqk8bwsrmw9k5o76s"
    secondary_cidr_blocks = []
    status                = "Available"
    tags                  = {}
    user_cidrs            = []
    vpc_name              = "vpc-test_ xxx"
}

Log on to the Elasticsearch console

Log on to the Elasticsearch console to view the cluster.

In the instance list, you can view information about the created instance, including Instance ID/Name, Status, Version, Instance Type, Number of Data Nodes, Specifications, and Billing Method. In this example, the instance name is testInstanceName, the version is 6.7.0, the instance type is Enhanced Edition, the number of data nodes is 2, the specifications are 2 vCPUs, 8 GiB memory, and a 20 GiB standard SSD, the billing method is Pay-as-you-go, and the status is Active.

Step 5: Release resources

When no longer needed, release all Terraform-managed resources with terraform destroy. Common commands.

terraform destroy

When prompted, enter yes and press Enter to confirm.

Complete sample code

Run the sample code with a few clicks in Terraform Explorer.

Sample code

variable "region" {
  default = "cn-qingdao"
}

data "alicloud_zones" "default" {
  available_resource_creation = "VSwitch"
  available_disk_category     = "cloud_ssd"
}

variable "vpc_cidr_block" {
  default = "172.16.0.0/16"
}

variable "vsw_cidr_block" {
  default = "172.16.0.0/24"
}

variable "node_spec" {
  default = "elasticsearch.sn2ne.large"
}
provider "alicloud" {
  region = var.region
}

resource "random_integer" "default" {
  min = 10000
  max = 99999
}

resource "alicloud_vpc" "vpc" {
  vpc_name   = "vpc-test_${random_integer.default.result}"
  cidr_block = var.vpc_cidr_block
}

resource "alicloud_security_group" "group" {
  name   = "test_${random_integer.default.result}"
  vpc_id = alicloud_vpc.vpc.id
}

resource "alicloud_vswitch" "vswitch" {
  vpc_id       = alicloud_vpc.vpc.id
  cidr_block   = var.vsw_cidr_block
  zone_id      = data.alicloud_zones.default.zones[0].id
  vswitch_name = "vswitch-test-${random_integer.default.result}"
}

resource "alicloud_elasticsearch_instance" "instance" {
  description           = "test_Instance"
  instance_charge_type  = "PostPaid"
  data_node_amount      = "2"
  data_node_spec        = var.node_spec
  data_node_disk_size   = "20"
  data_node_disk_type   = "cloud_ssd"
  vswitch_id            = alicloud_vswitch.vswitch.id
  password              = "es_password_01"
  version               = "6.7_with_X-Pack"
  master_node_spec      = var.node_spec
  zone_count            = "1"
  master_node_disk_type = "cloud_ssd"
  kibana_node_spec      = var.node_spec
  data_node_disk_performance_level = "PL1"
  tags = {
    Created = "TF",
    For     = "example",
  }
}

For more complete examples, visit the Terraform landing zone quickstarts on GitHub.