Cloud Monitor Log Generation from Sample Application

By Amit Maity, Alibaba Cloud Tech Share Author. Tech Share is Alibaba Cloud's incentive program to encourage the sharing of technical knowledge and best practices within the cloud community.

Overview

Alibaba Cloud Monitor service allows monitoring of cloud resource usage and health of the application/resources by collecting various monitoring metrics. In this tutorial, we will see how Cloud Monitor can help us notified in case of an exception in an application. Sample application, which is used here to demonstrate the cloud monitor use case, is a simple web application for uploading file into OSS bucket.

This application will write error logs into Alibaba Log Service and logs are being pulled into Cloud Monitor to monitor sample application exceptions.

There are few more use cases possible using the same idea.

Measure throughput or performance of the application from log entry timings.
Use Cloud Monitor Dashboard to monitor application health in real time

Architecture

In this below design, I have implemented a scenario where system administrator is notified by email if there is any failure while using file upload web application. The failure could be loss of connection between OSS and application or any other OSS technical/permission issue.

File upload web application is designed to write the error log message into "Log Service". Cloud Monitor monitors the log message written into Log Service. If Log Monitor identified number of error log exceeds the threshold count, then it will automatically send email alert to the system administrator stating that web application encountered an error.

Prerequisite

This tutorial can easily be understood and followed if you are familiar with the below technologies and concepts,

Alibaba Cloud Environment
If you do not have access, get 1 year free access with $10 credit using the below link
Alibaba Cloud Free Account
Alibaba Log Service
Click here to know basics of Cloud Message service
Alibaba Cloud Monitor
Click here to know about Batch Compute product
Python Programing Knowledge and Flask Web Framework

Server Configurations

To develop and run the web application you will need a host server (Alibaba Cloud ECS or any other windows/unix Server) with python 2.7 installed.

Download and install following Python packages and Alibaba Cloud Python SDKs in host server

Flask
Log Service Python SDK
OSS2
Download Sample Web Application from GitHub repository
Sample Application Source Code

Purchase and Activate following services in your cloud account,

Object Storage Service (OSS)
Cloud Monitor
Log Service

Configuring OSS, Log Service and Cloud Monitor

Please choose a single region for configuring the all the required Alibaba cloud services before you proceed. This will help you to avoid any technical challenges for integrating multi region services.

Create a new bucket in your OSS console to store the files, which you are going to upload using this sample application.
Keep a note of bucket name and OSS end point. Later you'll have to update the configuration file of the application with bucket name and OSS end point.
Create Log Service project in Log Service console. Choose region name same as OSS region.
Under Log Service project, create logstore with below screeshot.
Click on the Endpoints button as shown below to get the log service end points. This end point url needs to be updated in config.cfg file.
Open Cloud Service Monitor console for Log Service. You should be able to see log service project and associated LogStore, which are setup in previous steps.
Click on Alarm Rules link to create Alarm Rule to trigger email alert for error log entry in Log Service. This alarm is threshold-based alarm, which is executed if the monitoring criteria specified in rule description matches with the monitoring data.
Configure "Alarm Rules" to trigger the alert if there is at least 1 error log line is written into log service per minute.

Configure "Notification Method" to specify the email address of the alert email recipient. Create a contact group with the email address of the recipient if there is no contact group available.

Fill up the Email Subject and Email Remark fields with the email subject and email body content as per your requirement.

Sample Application Walkthrough

Here is the folder and file structure,


sampleWebApplication

│  common.py

│  config.cfg

|  config.py

│  LICENSE

│  Log.py

│  README.md

│  sampleweb.py

│

└─── app

│  routes.py

│  __init__.py

│

└─── templates

base.html

upload.html

a. config.cfg

This file keeps all required configuration values for running the application. Update the configuration parameters in this file according to your cloud account/service details, before running the application


AccessKeyId = <Cloud Account Access Key>

AccessKeySecret = < Cloud Account Access Secret Key >

LogEndpoint = <Log Service End points url>

i.e. ap-southeast-1.log.aliyuncs.com

LogProject = <Log Service Project Name>

i.e. sampleapplication

Logstore = <Log Service Project Logstore name>

i.e. sampleapplicationlogstore

OSSEndpoint = <OSS End Point url>

i.e. oss-cn-shenzhen.aliyuncs.com

OSSBucketName = <OSS Bucket Name>

i.e. samplewebapp

b. config.py

This is to hold configuration options for Flask application sessions.

c. common.py

This script is for parsing the required configuration parameter values from config.cfg and return the values to main python scripts for using in invoking cloud APIs.


# Read config.cfg and parse the configuration parameters

parser = ConfigParser.ConfigParser()

parser.read(cfg_fn)


# Assign each of the configuration parameters values to separate varaiables

accessKeyId = parser.get("Base", "AccessKeyId")

accessKeySecret = parser.get("Base", "AccessKeySecret")

logendpoint = parser.get("Base", "LogEndpoint")

.

.

.

return accessKeyId,accessKeySecret,logendpoint,logproject,logstore . . .

d. log.py

This script defines the reusable function write() to write log messages into logstore of the log service.

Import the common class from common.py to retrieve the access key id and secrect key and other required parameters for calling log service API,


from common import Common

Import log service modules


from aliyun.log.logitem import LogItem

from aliyun.log.putlogsrequest import PutLogsRequest

Defining write static method with below parameters to invoke log service APIs


@staticmethod

def write(client, project, logstore, topic, source, contents):

client: Mandatory, LogClient object

project: Mandatory, name of the Log Service project

logstore: Mandatory, name of the Logstore in Log Service Project

topic: Optional, to classify the log messages by topic

source: Optional, to update the source of the log messages

contents: Mandatory, actual content of the log message

Build an array of log item object with the log message content as Log service API expects collection of log item object.


logItem = LogItem()

logItem.set_time(int(time.time()))

logItem.set_contents(contents)

logitemList.append(logItem)

Get Log Request Object with all necessary request parameter values and pass the request object to put_logs method of Log Service API to write the log message into Log Service logstore.


req = PutLogsRequest(project, logstore, topic, source, logitemList)

res = client.put_logs(req)

Below scripts are written to develop web application using Flask web framework. To understand the basic of Flask web framework, please refer Flask Documentation.

e. samplewebapp.py

This script is for referring and pointing to flask application package developed for this demonstration.

Use below commands to run this flask application,


set FLASK_APP=samplewebapp.py

flask run

f. app/__init.py__

Import Flask package and instantiate Flask application


from flask import Flask

app = Flask(__name__)

g. app/templates/base.html

This is HTML template so that this can be rendered as web page by Flask engine. This template file contains basic HTML web page structure with url links in the navigation bar.

h. app/templates/upload.html

This is a template to display the file upload form and extension to base.html web page.


{% block content %}

<form action = "/upload" method = "POST"

enctype = "multipart/form-data">

<input type = "file" name = "file" />

<input type = "submit"/>

</form>

{% endblock %}

Using below syntax in base.html, upload.html form is appended to the main base.html web page.


{% block content %}{% endblock %}

i. app/routes.py

Web application forms view and controller logic is included in this script. This renders the html templates to display web form in the browser. This script is heart of this sample application.

upload_file() view function mapped to the /upload URL that creates a file upload web form, and passes it to the html templates for rendering


@app.route('/upload',methods = ['GET', 'POST'])

def upload_file():

if request.method == 'POST':

. . . . .

bucket.put_object(f.filename,'CustomUpload')

flash('File uploaded successfully!')

. . . . .

return render_template('upload.html')

Once file is uploaded by the browser, upload_file() function uses OSS2 API method put_object to upload the file into OSS bucket.

Additional logic for writing into Log Service is included in this script,

Import SDKs, common and log modules


from aliyun.log.logclient import LogClient

sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)) + "/..")

from common import Common

from log import Log

Retrieve the configuration parameter values from config.cfg


accessKeyId,accessKeySecret,logEndPoint,logProject,logStore,ossEndPoint,ossBucketName,securityToken = Common.LoadConfig()

Instantiate LogClient class with authentication details before using Log Service API


client = LogClient(logProject+'.'+logEndPoint, accessKeyId, accessKeySecret)

Use exception handling for OSS bucket write API. In case of API failure, update contents variable with error message and then use Log.write function to upload the error message into Log Service log store.


try:

bucket.put_object(f.filename,'CustomUpload')

. . . . .

except Exception, e:

contents = [('Msg', str(e) )]

Log.write(client, logProject, logStore, '', '', contents)

. . . . .

Run and Test

Now let's run the application.

In windows command prompt, change directory to sampleWebApplication folder and then import the flask application, by setting FLASK_APP environment variable as follows,

In Unix environment, use export command instead of set.

Run the flask application using "flask run" command,

Open the web application in any browser using the url http://127.0.0.1:5000/upload

Upload any file and click submit button.

Once the file is successfully uploaded, you'll see the uploaded file in OSS bucket

Now, to simulate the upload failure let's delete this bucket and again try uploading the same file.

This time application is not be able to upload and displaying message - "File upload failure" in web page.

Reason or log details of upload failure is available in sampleapplogstore console in Raw Logs tab as shown in below screenshot.

Above log entry will cause Cloud Monitor alarm threshold to exceed. Due to this, "Send Email Alert" alarm in Cloud Monitor is triggered which is visible in Cloud Monitor console.

As "Send Email Alert" alarm is triggered and you should receive alert email from cloud monitoring service. Below is the sample email, which is generated by Cloud Monitoring Alarm.

Community

Cloud Monitor Log Generation from Sample Application

Overview

Architecture

Prerequisite

Server Configurations

Configuring OSS, Log Service and Cloud Monitor

Sample Application Walkthrough

Run and Test

Read previous post:

Read next post:

Alibaba Clouder

You may also like

Comments

Alibaba Clouder

Related Products

CloudMonitor

Log Service

OSS(Object Storage Service)