Elasticsearch:Configure a NAT gateway for data transmission over the Internet

Background information

Alibaba Cloud Logstash clusters are deployed in VPCs. If you want to use a Logstash cluster to migrate data, but the Logstash cluster resides in a VPC different from the source or destination, data cannot be transmitted between the source and destination. This is because a network connection cannot be established between the source and destination. In this case, you can use the Destination Network Address Translation (DNAT) and Source Network Address Translation (SNAT) features provided by the Alibaba Cloud NAT Gateway service to establish the network connection. The DNAT feature can enable services on the Internet to send data to Logstash. The SNAT feature can enable Logstash to access the Internet. For more information about the NAT Gateway service, see What is NAT Gateway?.

Prerequisites

• A VPC and a vSwitch are created.

  For more information, see Create a VPC with an IPv4 CIDR block.

• An Alibaba Cloud Logstash cluster is created.

  For more information, see Create a cluster.

Procedure

1. Go to the Logstash Clusters page of the Alibaba Cloud Elasticsearch console.
2. Navigate to the desired cluster.
   1. In the top navigation bar, select the region where the cluster resides.
   2. On the Logstash Clusters page, find the cluster and click its ID.

3. In the left-side navigation pane of the page that appears, click Networks and Security.

4. In the Network Settings section, click Configure NAT Gateway.

   For more information about the descriptions and configurations of NAT gateways, see Create and manage Internet NAT gateways. DNAT entries allow services on the Internet to send data to Logstash. SNAT entries allow Logstash to access the Internet.

5. On the Internet NAT Gateway page, click Create NAT Gateway.

   When you create a NAT gateway, select the region and VPC where the Logstash cluster resides. For more information, see Step 1: Create an Internet NAT gateway.

6. Associate an elastic IP address (EIP) with the NAT gateway.

   1. On the Internet NAT Gateway page, find the NAT gateway that you created, click in the Actions column, and then select Bind Elastic IP Address.

   2. In the Associate EIP dialog box, select Select Existing EIP.

      If no EIPs are available, select Purchase and Associate EIP and click OK. Then, an EIP is associated with the NAT gateway.

   3. Select an EIP and click OK.

      Important

      You can associate a maximum of 20 EIPs with a NAT gateway. A maximum of 10 pay-as-you-go EIPs can be associated with a NAT gateway, and each of the pay-as-you-go EIPs supports a peak throughput of 200 Mbit/s. If you want to increase the number of EIPs that can be associated, you can submit a ticket.

7. Create a DNAT entry.

   1. On the Internet NAT Gateway page, find the NAT gateway that you created and click Configure DNAT in the Actions column.

   2. Click the DNAT Management tab.

   3. In the DNAT Entry List section, click Create DNAT Entry.

   4. On the Create DNAT Entry page, configure the following parameters.

      Parameter	Description
      Select Public IP Address	Select an available public IP address. Note A public IP address specified in an SNAT entry cannot be specified in a DNAT entry.
      Select Private IP Address	Select Manual Input and enter the IP address of a node in your Logstash cluster. You can obtain the IP addresses of the nodes in your Logstash cluster on the Basic Information page of the Logstash cluster. For more information, see View the basic information of a cluster.
      Port Settings	Select a DNAT mapping method. Any Port: specifies IP address mapping. The selected public IP address is associated with the Logstash cluster. All requests that are destined for the public IP address are forwarded to the Logstash cluster. Specific Port: specifies port mapping. The NAT gateway forwards requests to the specified node based on the specified protocol and ports. If you select Specific Port, you must configure Public Port, Private Port, and Protocol Type. Public Port specifies the external port that is used for request forwarding. Private Port specifies the internal port that is used for request forwarding. Protocol Type specifies the protocol used by the ports.
      Entry Name	Enter a name for the DNAT entry. The name must be 2 to 128 characters in length and can contain letters, digits, underscores (_), and hyphens (-). The name must start with a letter.

   5. Click OK.

8. Create an SNAT entry.

   1. Go to the Internet NAT Gateway page. Find the NAT gateway that you created and click Configure SNAT in the Actions column.

   2. Click the SNAT Management tab.

   3. In the SNAT Entry List section, click Create SNAT Entry.

   4. On the Create SNAT Entry page, select Specify vSwitch for SNAT Entry and configure other parameters.

      Parameter	Description
      Select VSwitch	Select a vSwitch in the VPC where the Logstash cluster resides. All Elastic Compute Service (ECS) instances that belong to the specified vSwitch can access the Internet by using the SNAT feature.
      Select Public IP Address	Select the public IP address that is used to access the Internet. You can select multiple public IP addresses to build an SNAT IP address pool. If you select multiple public IP addresses to build an SNAT IP address pool, make sure that these public IP addresses are associated with the same Internet Shared Bandwidth. For more information, see Associate EIPs with and disassociate EIPs from Internet Shared Bandwidth instances.

      For more information about the parameters, see Create and manage SNAT entries.

   5. Click OK.

9. Return to the Basic Information page of the Logstash cluster in the Elasticsearch console and configure a pipeline to transmit data over the Internet.

   For more information, see Use configuration files to manage pipelines. The following code provides a configuration example:

   input {
       beats {
           port => 8000
           host => "118.11.xx.xx"
       }
   }
   filter {
   
   }
   output {
       elasticsearch {
           hosts => ["http://es-cn-o40xxxxxxxxxx****.elasticsearch.aliyuncs.com:9200"]
           index => "logstash_test_1"
           password => "es_password"
           user => "elastic"
       }
       file_extend {
           path => "/ssd/1/ls-cn-v0h1kzca****/logstash/logs/debug/test"
       }
   }

   In the example, the source is Beats, the destination is an Elasticsearch cluster, the server on which Beats is deployed resides in the same VPC as the Logstash cluster, and the Elasticsearch cluster resides in a VPC different from the Logstash cluster. In this case, you must configure an SNAT entry for the Logstash cluster to enable the Logstash cluster to transmit data to the Elasticsearch cluster over the Internet.

FAQ

• How do I import or export data to Logstash over the Internet?

• FAQ about data transfer by using Logstash