Use a self-managed APM server to collect data to Alibaba Cloud Elasticsearch - Elasticsearch

You can integrate the application performance monitoring (APM) service provided by open source Elasticsearch with Alibaba Cloud Elasticsearch to build an APM system, which helps achieve observability of services and applications. This topic describes how to use a self-managed APM server to collect data to Alibaba Cloud Elasticsearch.

Background information

Observability is the capability to measure the performance of your infrastructures, platforms, and applications, helping you understand how they work. Compared with traditional monitoring services, the mainstream monitoring services focus on issue identification and alerting. Observability can provide explanations for all events that occur on a complex distributed system. The monitoring capability focuses on the service status of software during and after delivery, whereas the observability capability focuses on the entire lifecycle of the R&D and O&M process.

Logs, metrics for infrastructures, and APM are the key elements of observability. APM bridges the gap between metrics and logs. Logs and metrics provide information about infrastructures and components and are partly overlapped. APM focuses on applications and allows IT and development engineers to monitor the application layer of a stack and user experience. You can integrate APM into a system monitoring service to obtain the following benefits:

Helps you understand what takes the longest period of time when services are running, and the reasons why services stop responding.
Helps you understand how services interact with each other and view the bottlenecks of the services.
Helps you understand how services interact with each other and view the bottlenecks of the services.
Improves the productivity of the development team.
Tracks user experience in browsers.

APM is widely used in the following scenarios:

User experience monitoring: APM can improve user experience by monitoring user behaviors. For example, APM can monitor the interactions between users and web interfaces or clients and record the time when the interactions occur.
Runtime application architectures: APM can help you understand the dependencies between services, and the network typologies of interactions between applications.
Business transactions: APM can generate in-depth SLA reports and provide trend information about application performance from the business perspective.
Deep dive component monitoring: If you want to use APM in this scenario, you must install an APM agent. APM is mainly used to monitor the middle layer of a service, including web servers, applications, and message servers. The robust monitoring capability provided by APM can display the clear path of code execution. Deep dive component monitoring is closely correlated with runtime application architectures. In most cases, the scenarios overlap with each other, and APM is concurrently used in the scenarios.
Analytics and reporting: APM can perform analytics on metric data that is collected from applications and display the analysis results in common views in a standard manner.

Prerequisites

An Alibaba Cloud Elasticsearch V7.10 cluster is created. For more information, see Create an Alibaba Cloud Elasticsearch cluster.
An Elastic Compute Service (ECS) instance whose operating system is Linux is created. For more information, see Connect to a Linux instance by using a password or key.
The Go programming language is installed in the ECS instance.
Note In this example, an APM agent that uses the Go programming language is used. You must install the Go programming language in advance.

Procedure

Step 1: Build an APM server
Step 2: Configure an APM agent
Step 3: View and analyze data that is collected by the APM server in the Kibana console

Step 1: Build an APM server

Connect to the ECS instance.
For more information, see Connect to a Linux instance by using a password or key.
Note In this example, a regular user is used to connect to the ECS instance.

Install an APM server.

Download the installation package of an APM server.

wget https://artifacts.elastic.co/downloads/apm-server/apm-server-7.10.2-linux-x86_64.tar.gz

Decompress the installation package.

tar -zxf apm-server-7.10.2-linux-x86_64.tar.gz

Modify the configurations of the APM server.

Go to the directory in which the APM server is installed.
```
cd apm-server-7.10.2-linux-x86_64/
```

Modify the YML configuration file.

vim apm-server.yml

The following code provides an example on how to modify the YML configuration file:

apm-server:
  host: "0.0.0.0:8200"
output.elasticsearch:
  hosts: ["es-cn-*****.elasticsearch.aliyuncs.com:9200"]
  username: "elastic"
  password: "[pwd]"


Parameter	Description
apm-server	The address of the APM server. You must configure the host parameter in the following format: `IP address to be listened by the APM server:Port number`. In this example, the host parameter is set to `0.0.0.0:8200`.
output.elasticsearch	The information about the Elasticsearch cluster to which the output of the APM server is transferred. You must configure the following parameters: hosts: the public or internal endpoint of the Elasticsearch cluster. For information about how to obtain the endpoint of an Elasticsearch cluster, see View the basic information of a cluster. Important If the Elasticsearch cluster resides in the same virtual private cloud (VPC) as the ECS instance in which the APM server is installed, use the internal endpoint of the Elasticsearch cluster. If the Elasticsearch cluster resides in a different VPC from the ECS instance, use the public endpoint of the Elasticsearch cluster and configure a public IP address whitelist for the Elasticsearch cluster. For more information, see Configure a public or private IP address whitelist for an Elasticsearch cluster. username: the username that is used to access the Elasticsearch cluster. The default username is elastic. You can use a custom user account. Before you use such an account, you must create a role for it and grant the required permissions to the role. For more information, see Use the RBAC mechanism provided by Elasticsearch X-Pack to implement access control. password: the password that corresponds to the username. The password of the elastic account is specified when you create the Elasticsearch cluster. If you forget the password, you can reset it. For information about the procedure and precautions for resetting the password, see Reset the access password for an Elasticsearch cluster.

Start the APM server.

nohup ./apm-server -e > apmserver.log 2>&1 &

Step 2: Configure an APM agent

In this example, the Go programming language is used.

Install an APM agent.
1. Initialize the go.mod file.
```
sudo go mod init demo
```
2. Install an APM Go agent package.
```
sudo go get go.elastic.co/apm
sudo go get go.elastic.co/apm/module/apmhttp
```
  Note If the golang.org xxxx: i/o timeout error is reported when you install the package, you cannot access golang.org. In this case, you must change the download source.

Configure the APM agent.

Open the configuration file for environment variables.
```
vim ~/.bash_profile
```

In the configuration file, add the following settings to initialize the APM agent. Then, save the modifications.

# Specify a service name. The name can contain the following characters: letters, digits, hyphens (-), underscores (_), and spaces. 
# If no service name is specified, the name of an executable file is used. 
export ELASTIC_APM_SERVICE_NAME=zijian

# Specify the URL of the APM server. The default URL is http://localhost:8200. 
export ELASTIC_APM_SERVER_URL=

# The token that is required by the APM server. 
export ELASTIC_APM_SECRET_TOKEN=

Run the following command to make the configuration file take effect:
```
source ~/.bash_profile
```

Use the APM agent to detect an application.
1. Create a sample application file named apm.go.
```
vim apm.go
```
2. Add the following information to the apm.go file and save the file.
  You can use one of the following detection modules that are provided or the execution tracer API to detect the application.
```
package main

  import (
    "net/http"

    "go.elastic.co/apm/module/apmhttp"
  )

  func main() {
    mux := http.NewServeMux()
    http.ListenAndServe(":8080", apmhttp.Wrap(mux))
  }
```
3. Run the apm.go file.
```
sudo go run apm.go
```

Step 3: View and analyze data that is collected by the APM server in the Kibana console

Log on to the Kibana console of the Elasticsearch cluster.
For information about how to log on to the Kibana console, see Log on to the Kibana console.
Note In this example, an Elasticsearch V7.10 cluster is used. Different operations may be required to log on to the Kibana console of clusters of other versions.
Go to the homepage of the Kibana console and click Dev tools in the upper-right corner.
On the Console tab of the page that appears, run the following command to enable automatic creation of an APM onboarding index.
Note
- After the APM server is started, the system automatically creates APM-related indexes in the Elasticsearch cluster. You can view the indexes in the Kibana console. Alibaba Cloud Elasticsearch has requirements on the naming convention of indexes that are automatically created. The name of the APM onboarding index that is automatically created does not conform to the naming convention. Therefore, you must manually enable automatic creation of an APM onboarding index.
- 7.10.2 in the following command indicates the version number of the APM server that you installed.
```
PUT _cluster/settings
{
  "persistent": {
    "action.auto_create_index":"+.*,+apm-7.10.2-onboarding-*,-*"
  }
}
```
View services that are monitored by APM.
1. In the upper-left corner, click the icon to expand the left-side navigation pane.
2. In the left-side navigation pane, click APM below Observability.
3. On the Services tab of the page that appears, view all services that are monitored by APM.
4. Click the name of a service and view and analyze data related to the service.