Elasticsearch:Use a Fleet agent to collect NetFlow log data

• Create an Alibaba Cloud Elasticsearch cluster. For more information, see Create an Alibaba Cloud Elasticsearch cluster. In this example, an Alibaba Cloud Elasticsearch V8.5 cluster is created.

• Create an Elastic Compute Service (ECS) instance in the same virtual private cloud (VPC) as the Elasticsearch cluster. For more information, see Create an instance by using the wizard.

  Note

  The ECS instance is used as the source server. A Fleet agent collects data from the ECS instance.

Step 1: Create an agent policy

1. Log on to the Kibana console of the Elasticsearch cluster. For more information, see Log on to the Kibana console.

2. Click the icon in the upper-left corner. In the left-side navigation pane, choose Management > Fleet.

3. On the Fleet page, click the Agent policies tab.

4. Click Create agent policy. In the Create agent policy panel, configure the agent policy.

   1. Enter the name netflow-log in the Name field.

   2. Clear Collect system logs and metrics.

   3. Click Advanced options. In the Agent monitoring section, clear Collect agent logs and Collect agent metrics.

      
      Note

      In this example, only NetFlow Records logs need to be collected. Therefore, Collect system logs and metrics, Collect agent logs, and Collect agent metrics do not need to be selected.

5. Click Create agent policy.

Step 2: Add a Fleet server integration

1. On the Agent policies tab of the Fleet page, find the netflow-log agent policy and click its name.

2. On the Integrations tab of the page that appears, click Add integration.

3. On the Browse integrations tab of the Integrations page, enter Fleet Server in the search box. Then, click the Fleet Server card that is displayed.

4. Install the Fleet server integration.

   1. On the Fleet Server page, click the Settings tab.

   2. Click Install Fleet Server assets. In the Install Fleet Server message, click Install Fleet Server.

      Note

      After the integration is installed, the version of the integration is displayed on the Settings tab of the Fleet Server page.

5. In the upper-right corner of the Fleet Server page, click Add Fleet Server.

6. On the Add Fleet Server integration page, enter a name for the integration in the Integration name field in the Configure integration section and select netflow-log from the Agent policy drop-down list in the Where to add this integration section.

7. In the lower-right corner of the Add Fleet Server integration page, click Save and continue. In the Fleet Server integration added message, click Add Elastic Agent later.

Step 3: Add a NetFlow integration

1. On the Integrations tab of the agent policy netflow-log, click Add integration.

2. On the Browse integrations tab of the Integrations page, enter NetFlow Records in the search box. Then, click the NetFlow Records card that is displayed.

3. Install the NetFlow Records integration.

   1. On the NetFlow Records page, click the Settings tab.

   2. Click Install NetFlow Records assets. In the Install NetFlow Records message, click Install NetFlow Records.

      Note

      After the integration is installed, the version of the integration is displayed on the Settings tab of the NetFlow Records page.

4. In the upper-right corner of the NetFlow Records page, click Add NetFlow Records.

5. On the Add NetFlow Records integration page, configure the integration.

   1. In the Configure integration section, enter netflow-1 in the Integration name field.

   2. Click Change defaults next to Collect NetFlow logs. Enter 0.0.0.0 in the UDP host to listen on field and retain the default value 2055 in the UDP port to listen on field.

      

   3. On the Existing hosts tab of the Where to add this integration section, select netflow-log from the Agent policy drop-down list.

6. In the lower-right corner of the Add NetFlow Records integration page, click Save and continue. In the NetFlow Records integration added message, click Add Elastic Agent later.

Step 1: Configure a host for the Fleet server

1. Log on to the Kibana console of the Elasticsearch cluster. For more information, see Log on to the Kibana console.

2. Click the icon in the upper-left corner. In the left-side navigation pane, choose Management > Fleet.

3. On the Fleet page, click the Settings tab. On the Settings tab, configure parameters for Fleet.

   1. In the Fleet server hosts section, click Edit hosts.

   2. In the Fleet Server hosts panel, enter the URL of the source from which you want to collect data in the Specify host URL field. The URL must be in the https://<Private IP address of the source>:<Port number> format, such as https://172.16.*.***:8220. Then, click Save and apply settings. In the Save and deploy changes message, click Save and deploy.

      Note

      In this example, a URL that contains the primary private IP address of the ECS instance is entered. For more information about the configurations, see Fleet Server hosts.

   3. In the Outputs section of the Fleet page, click the icon in the Actions column.

   4. In the Edit output panel, enter the URL of the Elasticsearch cluster in the Hosts field. The URL must be in the http://<Internal endpoint of the Elasticsearch cluster>:<Port number> format, such as http://es-cn-uqm3auln80001****.elasticsearch.aliyuncs.com:9200.

   5. Click Save and apply settings. In the Save and deploy changes message, click Save and deploy.

Step 2: Add a Fleet agent

Add a Fleet agent to the Fleet server.

1. Click the icon in the upper-left corner. In the left-side navigation pane, choose Management > Fleet.

2. On the Fleet page, click the Agent policies tab.

3. On the Agent policies tab, find the agent policy netflow-log, click the icon in the Actions column, and then select Add agent.

4. On the Enroll in Fleet tab of the Add agent panel, click Add Fleet Server. In the Add a Fleet Server panel, click Advanced. In the Select a policy for Fleet Server section, retain the default value netflow-log.

5. In the Choose a deployment mode for security section, retain the default value Quick start.

6. In the Add your Fleet Server host section, click Add host.

7. In the Generate a service token section, click Generate service token.

8. In the Install Fleet Server to a centralized host section, copy the code that is automatically generated and run the code in the ECS instance.

   

   If Successfully is displayed after you run the code, the Fleet agent is installed on the ECS instance and is started.

Step 3: Configure the NetFlow service

In this example, softflowd is used to generate NetFlow logs. You must run the following code to start softflowd in the ECS instance.

1. Download the source code package of softflowd.

   wget https://storage.googleapis.com/google-code-archive-downloads/v2/code.google.com/softflowd/softflowd-0.9.9.tar.gz

2. Install the ibpcap-devel environment.

   yum install libpcap-devel

3. Compile code and install softflowd.

   tar -xvf softflowd-0.9.9.tar.gz
   cd softflowd-0.9.9
   ./configure 
   make
   make install

4. Run softflowd.

   nohup softflowd -v 9 -D -i eth0 -t maxlife=1 -n localhost:2055 >/dev/null 2>&1 &