Elasticsearch:Configure data transmission over the public network using NAT Gateway

Step 1: Open the NAT gateway configuration

1. Go to the Logstash Clusters page of the Alibaba Cloud Elasticsearch console.

2. In the top navigation bar, select the region where the cluster resides.

3. On the Logstash Clusters page, find the cluster and click its ID.

4. In the left-side navigation pane, click Network and Security.

5. In the Network Settings section, click Configure NAT Gateway.

Step 2: Create an Internet NAT gateway

On the Internet NAT Gateway page, click Create Internet NAT Gateway. Select the region and VPC where the Logstash cluster resides.

For detailed parameter descriptions, see Step 1: Create an Internet NAT gateway. For an overview of NAT gateway configurations, see Create and manage Internet NAT gateways.

Step 3: Associate an EIP with the NAT gateway

1. On the Internet NAT Gateway page, find the NAT gateway and choose More > Associate EIP in the Actions column.

2. In the Associate EIP dialog box, select Select Existing EIP and choose an available EIP. If no EIPs are available, select Purchase and Associate EIP and click OK.

3. Click OK.

Step 4: Create a DNAT entry

DNAT entries allow Internet services to send data to your Logstash cluster.

1. On the Internet NAT Gateway page, find the NAT gateway and click Configure DNAT in the Actions column.

2. Click the DNAT Management tab.

3. In the DNAT Entry List section, click Create DNAT Entry.

4. Configure the following parameters:

   Parameter	Description
   Select EIP	Select an available public IP address. An EIP used in an SNAT entry cannot be reused here.
   Select Private IP Address	Select Manual Input and enter the IP address of a Logstash cluster node. To find node IP addresses, go to the Basic Information page of the cluster. See View the basic information of a cluster.
   Port Settings	Select a DNAT mapping method: Any Port for IP address mapping (all requests to the public IP are forwarded to the cluster), or Custom Port for port mapping (requests are forwarded based on specific protocols and ports). If you select Custom Port, also configure Public Port, Private Port, and Protocol.
   Entry Name	Enter a name for the DNAT entry. The name must be 2-128 characters, start with a letter, and can contain letters, digits, underscores (_), and hyphens (-).

5. Click OK.

Step 5: Create an SNAT entry

SNAT entries allow your Logstash cluster to access the Internet.

1. On the Internet NAT Gateway page, find the NAT gateway and click Configure SNAT in the Actions column.

2. Click the SNAT Management tab.

3. In the SNAT Entry List section, click Create SNAT Entry.

4. For SNAT Entry, select Specify vSwitch and configure the following parameters:

   Parameter	Description
   Select vSwitch	Select the vSwitch in the VPC where the Logstash cluster resides. All Elastic Compute Service (ECS) instances in the vSwitch can then access the Internet through this SNAT entry.
   Select EIP	Select the public IP address for Internet access. To build an SNAT IP address pool, select multiple EIPs. If you select multiple EIPs, they must all be associated with the same EIP bandwidth plan. See Associate EIPs with and disassociate EIPs from Internet Shared Bandwidth instances.

5. Click OK.

For the full list of SNAT parameters, see Create and manage SNAT entries.

Step 6: Configure the Logstash pipeline

Return to the Basic Information page of the Logstash cluster in the Elasticsearch console and configure a pipeline to transmit data over the Internet. See Use configuration files to manage pipelines.

The following example uses Beats as the input and an Elasticsearch cluster as the output. The Beats server is in the same VPC as the Logstash cluster, while the Elasticsearch cluster is in a different VPC — so an SNAT entry is required for Logstash to reach the Elasticsearch cluster over the Internet.

input {
    beats {
        port => 8000
        host => "118.11.xx.xx"
    }
}
filter {

}
output {
    elasticsearch {
        hosts => ["http://es-cn-o40xxxxxxxxxx****.elasticsearch.aliyuncs.com:9200"]
        index => "logstash_test_1"
        password => "es_password"
        user => "elastic"
    }
    file_extend {
        path => "/ssd/1/ls-cn-v0h1kzca****/logstash/logs/debug/test"
    }
}