Elasticsearch:Use a self-managed APM server to collect data to Alibaba Cloud Elasticsearch

Prerequisites

• You have created an Alibaba Cloud Elasticsearch V7.10 instance. For more information, see Create an Alibaba Cloud Elasticsearch instance.

  Note

  We recommend enabling Indexing Service and Openstore to support cost-effective, high-volume data ingestion and storage for application performance monitoring.

• You have created an Alibaba Cloud ECS instance that runs a Linux operating system. For more information, see Create an instance by using the custom launch wizard.

• You have installed the Go language environment on the ECS instance.

  Note

  This topic uses the APM agent for Go, which requires the Go language environment.

Procedure

1. Step 1: Set up the APM Server

2. Step 2: Configure the APM agent

3. Step 3: View and analyze data in Kibana

Step 1: Build an APM server

1. Connect to the ECS instance.

   For more information, see Log on to a Linux instance by using Workbench.

   Note

   The operations in this topic are performed using a regular user account.

2. Install an APM server.

   1. Download the installation package of an APM server.

      wget https://artifacts.elastic.co/downloads/apm-server/apm-server-7.10.2-linux-x86_64.tar.gz

   2. Decompress the installation package.

      tar -zxf apm-server-7.10.2-linux-x86_64.tar.gz

3. Modify the configurations of the APM server.

   1. Go to the directory in which the APM server is installed.

      cd apm-server-7.10.2-linux-x86_64/

   2. Modify the YML configuration file.

      vim apm-server.yml

      Modify the YML file as shown in the following example.

      apm-server:
        host: "0.0.0.0:8200"
      output.elasticsearch:
        hosts: ["es-cn-*****.elasticsearch.aliyuncs.com:9200"]
        username: "elastic"
        password: "[pwd]"

      Parameter	Description
      apm-server	The endpoint for the APM Server. The host value must be set to the listening IP address:port of the APM Server. In this example, it is set to 0.0.0.0:8200.
      output.elasticsearch	The Elasticsearch output configuration. hosts: The public or internal endpoint of your Elasticsearch instance. To find the endpoint, see View the basic information of an instance. Important If the Elasticsearch instance and the ECS instance where the APM Server is installed are in the same VPC, use the internal endpoint. Otherwise, you must use the public endpoint and configure an IP address whitelist. For more information, see Configure a public or internal IP address whitelist for an instance. username: The username for accessing the Elasticsearch instance. The default is elastic. If you use a custom user, you must assign the required roles and permissions. For more information, see Use Elasticsearch X-Pack roles to manage user permissions. password: The password for the specified user. The password for the elastic user is set when you create the instance. If you forget it, you can reset it. For more information, see Reset the access password of an instance.

4. Start the APM server.

   sudo nohup ./apm-server -e > apmserver.log 2>&1 &

Step 2: Configure an APM agent

This topic uses Go as an example.

1. Install an APM agent.

   1. Initialize the go.mod file.

      sudo go mod init demo

   2. Install an APM Go agent package.

      sudo go get go.elastic.co/apm
      sudo go get go.elastic.co/apm/module/apmhttp

      Note

      A golang.org xxxx: i/o timeout error indicates that golang.org is inaccessible. To resolve this, change the download source.

2. Configure the APM agent.

   1. Open the configuration file for environment variables.

      vim ~/.bash_profile

   2. In the configuration file, add the following settings to initialize the APM agent. Then, save the modifications.

      # Set the service name. Allowed characters: a-z, A-Z, 0-9, -, _, and space.
      # If ELASTIC_APM_SERVICE_NAME is not set, the executable's file name is used.
      export ELASTIC_APM_SERVICE_NAME=zijian
      
      # Set a custom APM Server URL (default: http://localhost:8200).
      export ELASTIC_APM_SERVER_URL=
      
      # Use a secret token if required by the APM Server.
      export ELASTIC_APM_SECRET_TOKEN=

   3. Run the following command to make the configuration file take effect:

      source ~/.bash_profile

3. Use the APM agent to detect an application.

   1. Create a sample application file named apm.go.

      vim apm.go

   2. Add the following information to the apm.go file and save the file.

      To instrument your Go application, use one of the provided instrumentation modules or the tracer API.

      package main
      
        import (
          "net/http"
      
          "go.elastic.co/apm/module/apmhttp"
        )
      
        func main() {
          mux := http.NewServeMux()
          http.ListenAndServe(":8080", apmhttp.Wrap(mux))
        }

   3. Run the apm.go file.

      sudo go run apm.go

Step 3: View and analyze data that is collected by the APM server in the Kibana console

1. Log on to the Kibana console of the Elasticsearch cluster.

   For more information, see Log on to the Kibana console.

   Note

   This topic uses an Alibaba Cloud Elasticsearch V7.10 instance as an example. The UI and operations may differ for other versions.

2. On the Kibana homepage, click Dev tools in the upper-right corner.

3. On the Console tab, run the following script to enable auto-creation of the APM onboarding index.

   Note

   • Alibaba Cloud Elasticsearch enforces specific naming conventions for automatically created indexes. Because the default name for the APM onboarding index does not comply, you must run this command to allow its creation.

   • In the following script, 7.10.2 is the version of the APM Server you installed.

   PUT _cluster/settings
   {
     "persistent": {
       "action.auto_create_index":"+.*,+apm-7.10.2-onboarding-*,-*"
     }
   }

4. View services that are monitored by APM.

   The APM Services page lists all onboarded services in a table. The table contains the Name, Environment, Avg. response time, Trans. per minute, and Error rate % columns, which show each onboarded service and its performance metrics.

   1. In the upper-left corner, click the icon to expand the left-side navigation pane.

   2. In the navigation pane, under Observability, click APM.

   3. The Services tab displays all APM services.

   4. Click the name of a service and view and analyze data related to the service.