Elasticsearch:Collect NetFlow log data using an Elastic Agent

Step 1: Create an agent policy

1. Log on to the Kibana console of your Elasticsearch cluster. For more information, see Log on to the Kibana console.

2. Click the icon in the upper-left corner. In the left-side navigation pane, choose Management > Fleet.

3. On the Fleet page, click the Agent policies tab.

4. Click Create agent policy. In the Create agent policy panel, configure the policy:

   1. Enter netflow-log in the Name field.

   2. Clear Collect system logs and metrics.

   3. Click Advanced options. In the Agent monitoring section, clear Collect agent logs and Collect agent metrics.

      This tutorial only collects NetFlow Records logs, so system logs, agent logs, and agent metrics are not needed.

5. Click Create agent policy.

Step 2: Add a Fleet Server integration

1. On the Agent policies tab, find the netflow-log policy and click its name.

2. On the Integrations tab, click Add integration.

3. On the Browse integrations tab, search for Fleet Server, then click the Fleet Server card.

4. Install the Fleet Server integration:

   1. On the Fleet Server page, click the Settings tab.

   2. Click Install Fleet Server assets. In the confirmation dialog, click Install Fleet Server.

      After installation, the integration version appears on the Settings tab.

5. In the upper-right corner of the Fleet Server page, click Add Fleet Server.

6. On the Add Fleet Server integration page:

   • In the Configure integration section, enter a name in the Integration name field.

   • In the Where to add this integration section, select netflow-log from the Agent policy drop-down list.

7. Click Save and continue. In the confirmation message, click Add Elastic Agent later.

Step 3: Add a NetFlow Records integration

1. On the Integrations tab of the netflow-log agent policy, click Add integration.

2. On the Browse integrations tab, search for NetFlow Records, then click the NetFlow Records card.

3. Install the NetFlow Records integration:

   1. On the NetFlow Records page, click the Settings tab.

   2. Click Install NetFlow Records assets. In the confirmation dialog, click Install NetFlow Records.

      After installation, the integration version appears on the Settings tab.

4. In the upper-right corner of the NetFlow Records page, click Add NetFlow Records.

5. On the Add NetFlow Records integration page, configure the integration:

   1. In the Configure integration section, enter netflow-1 in the Integration name field.

   2. Click Change defaults next to Collect NetFlow logs. Set the following values:

      Field	Value	Description
      UDP host to listen on	0.0.0.0	Listens on all network interfaces
      UDP port to listen on	2055	Keep the default; softflowd sends to this port

      

   3. On the Existing hosts tab of the Where to add this integration section, select netflow-log from the Agent policy drop-down list.

6. Click Save and continue. In the confirmation message, click Add Elastic Agent later.

Step 1: Configure Fleet Server host and output

1. Log on to the Kibana console. For more information, see Log on to the Kibana console.

2. Click the icon in the upper-left corner. In the left-side navigation pane, choose Management > Fleet.

3. On the Fleet page, click the Settings tab.

4. Configure the Fleet Server host:

   1. In the Fleet server hosts section, click Edit hosts.

   2. In the Fleet Server hosts panel, enter the host URL in the Specify host URL field. Use the format https://<private-IP-of-ECS>:<port>, for example https://172.16.*.***:8220.

      Enter the primary private IP address of your ECS instance. For details on Fleet Server host settings, see Fleet Server hosts.

   3. Click Save and apply settings. In the confirmation dialog, click Save and deploy.

5. Configure the output to point to your Elasticsearch cluster:

   1. In the Outputs section, click the icon in the Actions column.

   2. In the Edit output panel, enter the internal endpoint URL of your Elasticsearch cluster in the Hosts field. Use the format http://<internal-endpoint>:<port>, for example http://es-cn-uqm3auln80001****.elasticsearch.aliyuncs.com:9200.

   3. Click Save and apply settings. In the confirmation dialog, click Save and deploy.

Step 2: Add a Fleet agent

To collect NetFlow traffic from multiple source servers, repeat this step for each server. Each Fleet agent collects data from its respective source server, and Fleet Server manages all agents centrally.

1. Click the icon in the upper-left corner. In the left-side navigation pane, choose Management > Fleet.

2. On the Fleet page, click the Agent policies tab.

3. Find the netflow-log agent policy, click the icon in the Actions column, and select Add agent.

4. On the Enroll in Fleet tab of the Add agent panel, click Add Fleet Server. In the Add a Fleet Server panel, click Advanced. In the Select a policy for Fleet Server section, keep the default value netflow-log.

5. In the Choose a deployment mode for security section, keep the default Quick start.

6. In the Add your Fleet Server host section, click Add host.

7. In the Generate a service token section, click Generate service token.

8. In the Install Fleet Server to a centralized host section, copy the generated command and run it on the ECS instance. If Successfully appears in the output, Fleet agent is installed and running on the ECS instance.

   

Step 3: Configure the NetFlow service

This tutorial uses softflowd to generate NetFlow traffic data. Run the following commands on the ECS instance.

1. Download the softflowd source package:

   wget https://storage.googleapis.com/google-code-archive-downloads/v2/code.google.com/softflowd/softflowd-0.9.9.tar.gz

2. Install the libpcap development library (required for packet capture):

   yum install libpcap-devel

3. Build and install softflowd:

   tar -xvf softflowd-0.9.9.tar.gz
   cd softflowd-0.9.9
   ./configure
   make
   make install

4. Start softflowd:

   Flag	Value	Description
   -v	9	Use NetFlow version 9
   -D	—	Run as a daemon
   -i	eth0	Capture traffic on the eth0 interface
   -t	maxlife=1	Export flows after a maximum lifetime of 1 minute
   -n	localhost:2055	Send NetFlow records to port 2055 (matches the integration config)

   nohup softflowd -v 9 -D -i eth0 -t maxlife=1 -n localhost:2055 >/dev/null 2>&1 &

   The following table explains the key flags:

Method 1: View data in dashboards

1. Click the icon in the upper-left corner. In the left-side navigation pane, choose Management > Fleet.

2. On the Fleet page, click the Data streams tab to see collected NetFlow log data in the dataset list.

3. Find the desired dataset, click the icon in the Actions column, and select View Dashboards. Select a dashboard to explore, for example [Logs Netflow] Overview.

Method 2: Explore data in Discover

Click the icon in the upper-left corner. In the left-side navigation pane, choose Analytics > Discover. On the Discover page, browse the data in the destination index.

Method 3: Query data from Dev Tools

1. Click the icon in the upper-left corner. In the left-side navigation pane, choose Management > Dev Tools.

2. On the Console tab, run the following query:

   GET logs-netflow.log-default/_search