Elasticsearch:Configure a NAT gateway for public data transfer

Background

Alibaba Cloud Logstash instances are deployed in a Virtual Private Cloud (VPC). If the source or destination is not in the same VPC as your Alibaba Cloud Logstash instance, a direct network connection cannot be established, and data cannot be transmitted. In this case, you can use a NAT gateway. A DNAT entry on the NAT gateway allows services on the public network to push data to your Logstash nodes. An SNAT entry allows your Logstash instance to access the public network. For more information, see What is a NAT gateway?.

Prerequisites

Procedure

1. Go to the Logstash Clusters page.

2. Navigate to the target cluster.

   1. In the top navigation bar, select the region where the cluster resides.

   2. On the Logstash Clusters page, find the cluster and click its ID.

3. In the left-side navigation pane, click Network and Security.

4. In the Network Configuration section, click Configure NAT Gateway.

   For more information about NAT gateways and the configuration process, see Internet NAT gateways. Use DNAT entries to allow services on the public network to push data to Logstash nodes. Use SNAT entries to allow Logstash to access the public network.

5. On the NAT gateway configuration page, create a NAT gateway.

   When you create the NAT gateway, make sure that the selected region and VPC ID match those of the Alibaba Cloud Logstash instance. For detailed instructions, see Procedure.

6. Associate an elastic IP address (EIP).

   1. In the NAT gateway list, find the target gateway. In the Actions column, choose > Associate with EIP.

   2. On the Associate with EIP page, select Select Existing EIP.

      If no EIP is available, select Purchase and Associate EIP and follow the on-screen instructions.

   3. Select an available EIP and click OK.

      Important

      You can associate up to 20 EIPs with a NAT gateway. A maximum of 10 of these can be pay-as-you-go EIPs, and the peak bandwidth of each pay-as-you-go EIP cannot exceed 200 Mbps. You can submit a ticket to request a higher quota.

7. Create a DNAT entry.

   1. In the NAT gateway list, find the target gateway and click Configure DNAT in the Actions column.

   2. Click the DNAT tab.

   3. In the DNAT Entry List area, click Create DNAT Entry.

   4. On the Create DNAT Entry page, configure the parameters.

      Parameter	Description
      Select public IP address	Select an available public IP address. Note A public IP address cannot be used for a DNAT entry if it is already used in an SNAT entry.
      Select Private IP Address	Select Manually Enter and enter the IP address of your Logstash instance. You can find the IP address on the basic information page of the instance. For more information, see View the basic information of an instance.
      Ports	Select a DNAT mapping method: Any Port: This option enables IP mapping, which is equivalent to associating an elastic IP address with the Logstash instance. All requests to this public IP address are forwarded to the instance's private IP address. Specific Port: This option enables port mapping. The NAT gateway forwards requests from the specified public IP address and port to the corresponding private IP address and port of the Logstash instance. If you select Specific Port, you must specify the Public network port (the external port for forwarding), Internal Port (the internal port for forwarding), and Protocol (the protocol for port forwarding) based on your business needs.
      Entry Name	Enter a name for the DNAT entry. The name must be 2 to 128 characters in length, start with a letter (uppercase or lowercase) or a Chinese character, and can contain digits, underscores (_), and hyphens (-).

   5. Click Confirm Creation to create the entry.

8. Create an SNAT entry.

   1. Return to the NAT gateway list page. Find the target gateway and click Configure SNAT in the Actions column.

   2. Click the SNAT tab.

   3. In the SNAT Entry List area, click Create SNAT Entry.

   4. On the Create SNAT Entry page, select Select vSwitch in the SNAT Entry section, and then configure the parameters.

      Parameter	Description
      Select Switch	Select the vSwitch in the VPC where the Alibaba Cloud Logstash instance resides. All ECS instances in this vSwitch can access the public network using the SNAT feature.
      Select public IP address	Select the public IP addresses that you want to use for public network access. You can select multiple public IP addresses to create an SNAT IP address pool. If you select multiple public IP addresses to create an SNAT IP address pool, make sure that each public IP address is added to an Internet Shared Bandwidth instance. For more information, see Add EIPs to and remove EIPs from an Internet Shared Bandwidth instance.

      For more information about the parameters, see Create and manage SNAT entries.

   5. Click Confirm Creation to create the entry.

9. Return to the Logstash console and configure a pipeline to enable data transfer over the public network.

   For more information, see Manage pipelines by using configuration files. The following is a configuration example:

   input {
       beats {
           port => 8000
           host => "118.11.xx.xx"
       }
   }
   filter {
   
   }
   output {
       elasticsearch {
           hosts => ["http://es-cn-o40xxxxxxxxxx****.elasticsearch.aliyuncs.com:9200"]
           index => "logstash_test_1"
           password => "es_password"
           user => "elastic"
       }
       file_extend {
           path => "/ssd/1/ls-cn-v0h1kzca****/logstash/logs/debug/test"
       }
   }

   In this example, the input is beats and the output is elasticsearch. The Beats server is in the same VPC as the Alibaba Cloud Logstash instance, but the destination Elasticsearch cluster is not. Sending data from Logstash to Elasticsearch requires Logstash to access the public network. This action requires a configured SNAT entry.

Related documents

• How do I use Logstash to import data from or export data to the public network?

• Troubleshoot Logstash data write issues

• If you need to use HTTPS, see Use the HTTPS protocol.