This topic explains how to create a custom monitoring job with an example.
If you use ARMS for the first time, it captures logs for processing from the end of the log stream. Therefore, please make sure that your script or program can continuously output logs.
Currently, ARMS supports the StartAgent, Queqiao and MQ data sources. This topic is based on an example of using StarAgent data source. Please use any client to generate a log file stream in text format on the server.
In this example, the following log stream is generated in /home/admin/logs/arms/test.log on the server:
2016-07-27 23:37:23|c0a895e114526786450161001d1ed9|9|EADS|BIZ-MONITOR|0|Category=Women’s wear&District=Shanghai&eventType=3&Gender=0&Price=20|iZ28ql4lx29Z
2016-07-27 23:37:23|c0a895e114526786450161001d1ed9|9|EADS|BIZ-MONITOR|0|Category=Children’s wear&District=Shenzhen&eventType=3&Gender=1&Price=30|iZ28ql4lx29Z
This is a simplified log stream of a transaction system, and every line in the log represents a transaction record. Fields separated by “|” indicate the transaction time, transaction ID, and transaction details, respectively.
After reading this topic, you are expected to learn how to:
- Use the log file stream on the server as the data source of monitoring
- Configure the log cleansing mode, datasets, and alarms.
- Configure the interactive dashboard.
A. Add the data source
- In the left-side navigation pane of the console, choose Custom Monitoring > Jobs. The Instance List page is displayed.
- On the Instance List page, choose Create Monitoring Job > Custom Monitoring Job in the upper-right corner, and then enter the job name in the Create Custom Job dialog box and click Create and Configure. The Data Source Configuration page is displayed.
On the Data Source Configuration page, enter all required information.
- In the Log Source Type drop-down list, select StarAgent.
- In the Log Path textbox, enter the log path /home/admin/logs/arms/test.log.
- In the Collection Target textbox, enter the IP address of the target machine 10.36.25.98.
- In the Encoding drop-down list, select Auto-Detect.
Click Log Capturing Preview in the upper-right corner.
Note: ARMS captures part of data (up to 20 records) from the logs of specified machine. A temporary pre-capturing channel must be set up, and therefore it usually takes about 30 seconds to capture.
The log capturing result is displayed in the preview port. If the capturing doesn’t work, check the configured log path and collection target.
On the Data Source Configuration page, click Save and Next. The Log Cleansing page is displayed.B. Cleanse the log
On the Log Cleansing page, click Obtain Scheme on the Intelligent Splitting tab page.
Tip: Currently, ARMS supports intelligent splitting and custom splitting. In this example, the log format is simple. Therefore, you can use the intelligent splitting mode to obtain the key-value pairs.
The intelligent splitter correctly determines that the sample log is separated by “|” and accordingly splits the log automatically into _line_gen_0, _line_gen_1, and other fields. Among these fields, _line_gen_6 is a KV string and the fields in this string is also automatically split.
On the Log Cleansing page, click Save and Next. The Dataset and Alarm Configuration page is displayed.C. Configure the dataset
In the Dataset Settings area, click Add Dataset.
The Add Dataset dialog box is displayed.
In the Add Dataset dialog box, enter related information and click Save.
Example 1: count the number of orders per minute
Each log line represents an order record. Therefore, you only need to count the number of log lines, namely performing the COUNT operation on _line.
Example 2: Count the sales volume of each category.
Select the drill-down dimension “Category”, and perform SUM operation on “Price”. The commodity quantity of each order is 1.
The created dataset is displayed in the Dataset Settings area.
D. Add an alarm
In the Alarm Settings area, click Create Alarm.
The Create Alarm dialog box is displayed.
In the Create Alarm dialog box, enter related information and click Save.
Example: Set an alarm for when the number of orders in the last 10 minutes is less than a specified value
On the Dataset and Alarm Configuration page, click Save and Finish Configuration. The Start Job dialog box is displayed.
- In the Start Job dialog box, choose Consumption from the beginning or Consume from the latest location, and click OK. The monitoring job is started. One or two minutes later, you can go to the job details page to view dataset details, or query datasets on the Datasets page.