By Alex Muchiri.
In this tutorial, we will explore ways to monitor the activity of a service running on an Alibaba Cloud ECS server. The tools for the job include Prometheus, Alertmanager and Blackbox Exporter. Follow this link to access the latest Prometheus client download. Prometheus is open source and is used by developers and system administrators to track system metrics, monitor services, and record that information in a time-series database. It makes use of an Alertmanager for alerting receivers, which may also duplicate or group different or similar alert types. The advantage of using the manager is its compatibility with messaging services such as email.
While Prometheus is a useful tool all by itself, it becomes even handier when integrated with an exporter. Your infrastructure can be separated into various components including web and database servers, messaging systems or APIs. An exporter such as Blackbox Exporter can have alerts configured for all these individual parts. Blackbox Exporter uses several techniques to run request metrics including HTTP, HTTPS, DNS, TCP, and ICMP. For each request, there are detailed data points such as the success of requests and response times. We shall use both tools to see how well a NGINX server running on Alibaba Cloud responds to our requests. We will also create email alerts if the server fails a response test.
To complete the setup guide in this tutorial, you should have the following:
We will all of these set up in this tutorial. Note that it is also possible to configure Alertmanager to send notifications to Slack.
Prometheus is available by default in Ubuntu 18.04 (Bionic Beaver), but in some versions, which are outdated, you may not have it installed. The first step is to update the package list with the following commands:
sudo apt-get update
sudo apt-get upgrade
Next, you'll want to run the command below to check what version is available in your system.
sudo apt-cache policy prometheus
The output should look like this:
prometheus:
Installed: (none)
Candidate: 2.1.0+ds-1
Version table:
2.1.0+ds-1 500
500 <http://ke.archive.ubuntu.com/ubuntu bionic/universe> amd64 Packages
Now check if Prometheus is installed by executing the command below:
sudo systemctl status prometheus
Below is the expected output.
Unit prometheus.service could not be found.
Next, we will need to install Prometheus on our Alibaba Cloud instance by running the command below.
sudo apt-get install prometheus
If Prometheus is unavailable in your package repository, you will need to fetch the key first. You can do so with this command.
wget https://s3-eu-west-1.amazonaws.com/deb.robustperception.io/41EFC99D.gpg | sudo apt-key add -
Then, update the package list.
sudo apt-get update -y
Then you can install Prometheus.
sudo apt-get install prometheus
Once the installation process is complete, activate Prometheus by running the commands below:
sudo systemctl start prometheus
sudo systemctl enable prometheus
Confirm that the service has been created successfully and that it is in good state by running the following command:
sudo systemctl status prometheus
Below is the expected output.
- prometheus.service - Monitoring system and time series database
Loaded: loaded (/lib/systemd/system/prometheus.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2019-08-13 19:04:20 UTC; 22s ago
Docs: https://prometheus.io/docs/introduction/overview/
Main PID: 8976 (prometheus)
Tasks: 4 (limit: 2312)
CGroup: /system.slice/prometheus.service
??9078 /usr/bin/prometheus
Now that we have Prometheus installed, let's now move on to installing Blackbox Exporter.
Now you'll want to download Blackbox Exporter using the commands below.
cd ~
wget https://github.com/prometheus/blackbox_exporter/releases/download/v0.14.0/blackbox_exporter-0.14.0.linux-amd64.tar.gz
Extract the binaries with the following command.
tar -xvf blackbox_exporter-0.14.0.linux-amd64.tar.gz
Create a Blackbox Exporter user.
sudo useradd --no-create-home --shell /bin/false blackbox_exporter
Copy the extracted binaries to the new user's bin directory.
sudo mv blackbox_exporter-0.14.0.linux-amd64/blackbox_exporter /usr/local/bin/blackbox_exporter
Then remove the downloaded file.
sudo rm -r blackbox_exporter-0.14.0.linux-amd64.tar.gz
Next, run the command below to change directory ownership.
sudo chown blackbox_exporter:blackbox_exporter /usr/local/bin/blackbox_exporter
Now let's create a new configuration file for Blackbox Exporter. To do with the following commands:
sudo mkdir /etc/blackbox_exporter
sudo nano /etc/blackbox_exporter/blackbox.yml
Copy the configurations below to your file:
modules:
http_2xx:
prober: http
timeout: 5s
http:
valid_status_codes: []
method: GET
To proceed, you'll need to create a systemd service file. Create one with the following command:
sudo nano /etc/systemd/system/blackbox_exporter.service
Place the lines below into the file:
[Unit]
Description=Blackbox Exporter
Wants=network-online.target
After=network-online.target
[Service]
User=blackbox_exporter
Group=blackbox_exporter
Type=simple
ExecStart=/usr/local/bin/blackbox_exporter --config.file /etc/blackbox_exporter/blackbox.yml
[Install]
WantedBy=multi-user.target
Next, update Blackbox Exporter's configuration file permissions.
chown blackbox_exporter:blackbox_exporter /etc/blackbox_exporter/blackbox.yml
Reload the systemctl daemon.
sudo systemctl daemon-reload
Start the Blackbox Exporter service.
sudo systemctl start blackbox_exporter
Confirm that the service launched succesfully and is in a good status.
systemctl status blackbox_exporter
Below is the expected output.
blackbox_exporter.service - Blackbox Exporter
Loaded: loaded (/etc/systemd/system/blackbox_exporter.service; disabled; vendor preset: enabled)
Active: active (running) since Wed 2019-08-13 19:12:40 UTC; 18s ago
Main PID: 10084 (blackbox_export)
Tasks: 6
CGroup: /system.slice/blackbox_exporter.service
└─10084 /usr/local/bin/blackbox_exporter --config.file /etc/blackbox_exporter/blackbox.yml
Aug 13 00:02:40 ip-172-31-41-126 systemd[1]: Started Blackbox Exporter.
Aug 13 00:02:40 ip-92-101-100-68 blackbox_exporter[10084]: level=info ts=2019-08-13T19:12:40.5229204Z caller=main.go:213 msg="Starting blackbox_exporter" version="(version=0.14.0, branch=HEAD, revision=bb
Aug 13 00:02:40 ip-92-101-100-68 blackbox_exporter[10084]: level=info ts=2019-08-13T19:12:40.52553523Z caller=main.go:226 msg="Loaded config file"
Aug 13 00:02:40 ip-92-101-100-68 blackbox_exporter[10084]: level=info ts=2019-08-13T19:12:40.525695324Z caller=main.go:330 msg="Listening on address" address=:9115
Run the below command to start the service at system boot:
sudo systemctl enable blackbox_exporter
Now, we will monitor a simple Django application running on port 8000 using Prometheus and Blackbox Exporter. We will create an additional configuration in the Prometheus configuration file under the assumption that the app is already running on the respective port.
sudo nano /etc/prometheus/prometheus.yml
Add the lines below in the file.
- job_name: 'blackbox'
metrics_path: /probe
params:
module: [http_2xx]
static_configs:
- targets:
- http://localhost:8000
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: __address__
replacement: localhost:9115
Restart Prometheus with the following command.
sudo systemctl restart prometheus
Create the alert rules for Prometheus with the following command.
sudo nano /etc/prometheus/alert.rules.yml
Append the lines below in the file.
groups:
- name: alert.rules
rules:
- alert: EndpointDown
expr: probe_success == 0
for: 10s
labels:
severity: "critical"
annotations:
summary: "Endpoint down"
Update the permissions by running the command below.
sudo chown prometheus:prometheus /etc/prometheus/alert.rules.yml
Verify that the rules have been created successfully by running the command below.
sudo promtool check rules /etc/prometheus/alert.rules.yml
Below is the expected output.
Checking /etc/prometheus/alert.rules.yml
SUCCESS: 1 rules found
Now restart Prometheus with the following command.
sudo systemctl restart prometheus
In this section, we will install and configure Alertmanager to send alerts to an email based on Prometheus metrics to notify about endpoints that go down.
For this, to start off, create a new Alertmanager user and update directory permissions with the following commands:
sudo useradd --no-create-home --shell /bin/false alertmanager
sudo chown alertmanager:alertmanager /usr/local/bin/alertmanager
sudo chown alertmanager:alertmanager /usr/local/bin/amtool
Then, download Alertmanager.
sudo wget https://github.com/prometheus/alertmanager/releases/download/v0.17.0/alertmanager-0.17.0.linux-amd64.tar.gz
Extract the downloaded file. You can do this simply by running the command below.
sudo tar -xvf alertmanager-0.17.0.linux-amd64.tar.gz
Move the extracted binaries to the /usr/local/bin/
directory with the following command.
sudo mv alertmanager-0.17.0.linux-amd64/alertmanager /usr/local/bin/
sudo mv alertmanager-0.17.0.linux-amd64/amtool /usr/local/bin/
Remove downloaded file to save some disk space.
sudo rm -r alertmanager-0.17.0.linux-amd64.tar.gz
Next, create a configuration for Alertmanager.
sudo mkdir /etc/alertmanager
In your favorite editor, create a configuration file with the following command:
sudo nano /etc/alertmanager/alertmanager.yml
Append the following lines into the file.
global:
smtp_smarthost: 'smtp.domain.net:587'
smtp_from: 'AlertManager <mailer@localhost.com>'
smtp_require_tls: true
route:
group_by: ['instance', 'alert']
group_wait: 30s
group_interval: 5m
repeat_interval: 3h
receiver: team-1
receivers:
- name: 'team-1'
email_configs:
- to: 'alerts@email.com'
Note that for the above lines, if you don't want TLS or your email server does not support TLS, you may leave the value as false.
Now you'll want to update the file permissions by running the command below.
sudo chown alertmanager:alertmanager -R /etc/alertmanager
Now we need to create a systemd service for Alertmanager. Using your favorite editor, add a configuration file.
sudo nano /etc/systemd/system/alertmanager.service
Append the lines below in the file.
[Unit]
Description=Alertmanager
Wants=network-online.target
After=network-online.target
[Service]
User=alertmanager
Group=alertmanager
Type=simple
WorkingDirectory=/etc/alertmanager/
ExecStart=/usr/local/bin/alertmanager --config.file=/etc/alertmanager/alertmanager.yml --web.external-url http://0.0.0.0:9093
[Install]
WantedBy=multi-user.target
Integrate Prometheus using the configuration below.
sudo nano /etc/prometheus/prometheus.yml
Append the lines below in the file.
...
alerting:
alertmanagers:
- static_configs:
- targets:
- localhost:9093
...
Next, include the following lines in your /etc/systemd/system/prometheus.service
. In your favorite editor, run the command below:
sudo nano /etc/systemd/system/prometheus.service
Append and replace the lines below into the file:
[Unit]
Description=Alertmanager
Wants=network-online.target
After=network-online.target
[Service]
User=alertmanager
Group=alertmanager
Type=simple
WorkingDirectory=/etc/alertmanager/
ExecStart=/usr/local/bin/alertmanager --config.file=/etc/alertmanager/alertmanager.yml --web.external-url Error! Hyperlink reference not valid.
[Install]
WantedBy=multi-user.target
Update the systemd file by changing the lines below:
sudo nano /etc/systemd/system/prometheus.service
[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
--config.file /etc/prometheus/prometheus.yml \
--storage.tsdb.path /var/lib/prometheus/ \
--web.console.templates=/etc/prometheus/consoles \
--web.console.libraries=/etc/prometheus/console_libraries \
--web.external-url Error! Hyperlink reference not valid.
Reload daemon and restart the services with the following commands:
systemctl daemon-reload
systemctl restart prometheus
systemctl restart alertmanager
Next, confirm both Alertmanager and Prometheus are properly installed with the following commands.
systemctl status alertmanager
systemctl status prometheus
If all is well, enable alertmanager start at boot with this command.
sudo systemctl enable alertmanager
In this tutorial, we have covered how to configure Prometheus with Blackbox Exporter and Alertmanager to send email alerts. While in most cases Alibaba Cloud will rarely run down, sometimes services running on the platform may experience errors. Nonetheless by implementing the things we have done in this tutorial you can stay on top of things and keep your system up and running as quickly as possible in the event of outages.
Do you have an Alibaba Cloud account? Sign up for an account and try over 40 products for free. This deal is worth up to $1300. Get Started with Alibaba Cloud to learn more.
The views expressed herein are for reference only and don't necessarily represent the official views of Alibaba Cloud.
2,599 posts | 763 followers
FollowAlibaba Clouder - April 12, 2021
Alibaba Clouder - April 22, 2019
Alibaba Container Service - February 13, 2019
Alibaba Cloud Native - December 28, 2023
Alibaba Cloud Native - September 8, 2023
Alibaba Cloud Native - August 14, 2024
2,599 posts | 763 followers
FollowAutomate performance monitoring of all your web resources and applications in real-time
Learn MoreRespond to sudden traffic spikes and minimize response time with Server Load Balancer
Learn MoreExplore Web Hosting solutions that can power your personal website or empower your online business.
Learn MoreMulti-source metrics are aggregated to monitor the status of your business and services in real time.
Learn MoreMore Posts by Alibaba Clouder