All Products
Search

This Product

    Elasticsearch:Use a Fleet agent to collect NGINX log data

    Document Center

    Elasticsearch:Use a Fleet agent to collect NGINX log data

    Last Updated:Aug 22, 2023

    NGINX is a widely-used high-performance web server. Developers need to monitor the stability of the server. To better monitor and optimize the performance and security of the NGINX server, you can use a Fleet agent to collect NGINX log data and use the Fleet server provided by Kibana to manage the collected data. You can use the Fleet server to transfer the collected data to Kibana for visualized analysis. This topic describes how to use the Fleet server to manage NGINX log data that is collected from two Elastic Compute Service (ECS) instances based on the same agent policy.

    Terms

    Term

    Description

    Fleet

    Fleet is a powerful solution provided by Elasticsearch to manage Fleet agents in a centralized manner.

    Fleet agent

    A Fleet agent is a lightweight data collection agent that is used to collect data from a source.

    A Fleet agent can run on different types of operating systems and collect multiple types of data.

    Fleet server

    The Fleet server is used to transfer data that is collected by a Fleet agent from a source to Elasticsearch.

    Prepare environments

    • Create an Alibaba Cloud Elasticsearch cluster. For more information, see Create an Alibaba Cloud Elasticsearch cluster. In this example, an Alibaba Cloud Elasticsearch V8.5 cluster is created.

      Note

      • The version of the Alibaba Cloud Elasticsearch cluster must be V7.10 or a later version other than V7.16.

    • Create two ECS instances. For more information, see Create an instance by using the wizard.

      • The ECS instances must reside in the same virtual private cloud (VPC) as the Elasticsearch cluster.

      • You must associate elastic IP addresses (EIPs) with the ECS instances. For more information, see Associate or disassociate an EIP.

        Note

        The ECS instances are used as the source servers. A Fleet agent collects NGINX log data from the ECS instances.

    Create an agent policy and add integrations

    Step 1: Create an agent policy

    1. Log on to the Kibana console of the Elasticsearch cluster. For more information, see Log on to the Kibana console.

    2. Click the image.png icon in the upper-left corner. In the left-side navigation pane, choose Management > Fleet.

    3. On the Fleet page, click the Agent policies tab.

    4. Click Create agent policy. In the Create agent policy panel, configure the agent policy.

      1. Enter the name nginx-log in the Name field.

      2. Clear Collect system logs and metrics.

      3. Click Advanced options. In the Agent monitoring section, clear Collect agent logs and Collect agent metrics.

        image.png
        Note

        In this example, only NGINX log data is collected. Therefore, Collect system logs and metrics, Collect agent logs, and Collect agent metrics do not need to be selected.

    5. Click Create agent policy.

    Step 2: Add a Fleet server integration

    1. On the Agent policies tab of the Fleet page, find the nginx-log agent policy and click its name.

    2. On the Integrations tab of the page that appears, click Add integration.

    3. On the Browse integrations tab of the Integrations page, enter Fleet Server in the search box. Then, click the Fleet Server card that is displayed.

    4. Install the Fleet server integration.

      1. On the Fleet Server page, click the Settings tab.

      2. Click Install Fleet Server assets. In the Install Fleet Server message, click Install Fleet Server.

        Note

        After the integration is installed, the version of the integration is displayed on the Settings tab of the Fleet Server page.

    5. In the upper-right corner of the Fleet Server page, click Add Fleet Server.

    6. In the Configure integration section of the Add Fleet Server integration page, enter a name for the integration in the Integration name field. On the Existing hosts tab of the Where to add this integration section of the same page, retain the default value nginx-log for the Agent policy parameter.

    7. In the lower-right corner of the Add Fleet Server integration page, click Save and continue. In the Fleet Server integration added message, click Add Elastic Agent later.

    Step 3: Add an NGINX integration

    1. On the Integrations tab of the nginx-log agent policy, click Add integration.

    2. On the Browse integrations tab of the Integrations page, enter Nginx in the search box. Then, click the Nginx card that is displayed.

    3. Install the NGINX integration.

      1. On the Nginx page, click the Settings tab.

      2. Click Install Nginx assets. In the Install Nginx message, click Install Nginx.

        Note

        After the integration is installed, the version of the integration is displayed on the Settings tab of the Nginx page.

    4. In the upper-right corner of the Nginx page, click Add Nginx.

    5. On the Add Nginx integration page, configure the integration.

      1. In the Configure integration section, retain the default value nginx-1 in the Integration name field.

      2. Click Change defaults next to Collect logs from Nginx instances. Enter a path in the Paths field based on your business requirements.

      3. On the Existing hosts tab of the Where to add this integration section, select nginx-log from the Agent policy drop-down list.

    6. In the lower-right corner of the Add Nginx integration page, click Save and continue. In the Nginx integration added message, click Add Elastic Agent later.

    Add a Fleet agent

    Step 1: Configure a host for the Fleet server

    1. Click the image.png icon in the upper-left corner. In the left-side navigation pane, choose Management > Fleet.

    2. On the Fleet page, click the Settings tab. On the Settings tab, configure parameters for Fleet.

      1. In the Fleet server hosts section, click Edit hosts.

      2. In the Fleet Server hosts panel, enter the URL of the source from which you want to collect data in the Specify host URL field. The URL must be in the https://<Private IP address of the source>:<Port number> format, such as https://172.16.*.***:8220. Then, click Save and apply settings. In the Save and deploy changes message, click Save and deploy.

        Note

        In this example, a URL that contains the primary private IP address of one of the ECS instances is entered. For more information about the configurations, see Fleet Server hosts.

      3. In the Outputs section of the Fleet page, click the image.png icon in the Actions column.

      4. In the Edit output panel, enter the URL of the Elasticsearch cluster in the Hosts field. The URL must be in the http://<Internal endpoint of the Elasticsearch cluster>:<Port number> format, such as http://es-cn-uqm3auln80001****.elasticsearch.aliyuncs.com:9200.

      5. Click Save and apply settings. In the Save and deploy changes message, click Save and deploy.

    Step 2: Add a Fleet agent

    1. On the Fleet page, click the Agent policies tab.

    2. Find the nginx-log agent policy, click the image.png icon in the Actions column, and then select Add agent.

    3. On the Enroll in Fleet tab of the Add agent panel, click Add Fleet Server. In the Add a Fleet Server panel, click Advanced. In the Select a policy for Fleet Server section, retain the default value nginx-log.

    4. In the Choose a deployment mode for security section, retain the default value Quick start.

    5. In the Add your Fleet Server host section, click Add host.

    6. In the Generate a service token section, click Generate service token.

    7. In the Install Fleet Server to a centralized host section, copy the code that is automatically generated and run the code in one of the ECS instances.

      image.png

      If Successfully is displayed after you run the code, the Fleet agent is installed on the ECS instance and is started.

    8. Repeat the preceding substeps to add a Fleet agent for the other ECS instance.

    View the collected data

    You can use one of the following methods to view the collected NGINX log data:

    Method 1: View the collected NGINX log data on the View Dashboards page

    1. Click the image.png icon in the upper-left corner. In the left-side navigation pane, choose Management > Fleet.

    2. On the Fleet page, click the Data streams tab. On the Data streams tab, you can view the collected NGINX log data in the dataset list.

    3. Find the desired dataset, click the image.png icon in the Actions column, and then select View Dashboards. On the page that appears, you can select the item whose information you want to view. For example, you can select [Metrics Nginx] Overview and view information about it.

      image.png

    Method 2: View the collected NGINX log data in the destination index on the Discover page

    1. Click the image.png icon in the upper-left corner. In the left-side navigation pane, choose Analytics > Discover.

    2. In the upper-left corner of the Discover page, select a data view. For example, you can select metrics-*.

      image.png

    Method 3: Query the collected NGINX log data by running a command on the Console tab

    1. Click the image.png icon in the upper-left corner. In the left-side navigation pane, choose Management > Dev Tools.

    2. On the Console tab, run the following commands to query the collected NGINX log data: 

      GET logs-nginx.access-*/_search
GET logs-nginx.error-*/_search