Simple Log Service:Collect and analyze ECS text logs using LoongCollector

Prerequisites

Activate services and prepare an account

• Activate SLS: If this is your first time using SLS, log on to the Simple Log Service console and activate the service as prompted.

• Prepare an account:

  • Log on with an Alibaba Cloud account: This account has all permissions by default and can be used directly.

  • Log on with a RAM user: The Alibaba Cloud account must grant the required access policies to the RAM user:

    • AliyunLogFullAccess: Used to create and manage SLS resources such as projects and logstores.

    • AliyunECSFullAccess: Used to install the collection agent on an ECS instance.

    • AliyunOOSFullAccess: Used to automatically install the collection agent on an ECS instance through Alibaba Cloud Operation Orchestration Service (OOS).

    In a production environment, you can create custom permission policies for more fine-grained permission management of RAM users.

Prepare an ECS instance

Ensure that the security group of the ECS instance allows outbound traffic on port 80 (HTTP) and port 443 (HTTPS).

Generate mock logs

1. Log on to the ECS instance.

2. Create a script file named generate_nginx_logs.sh and paste the following content into the file. This script writes a standard NGINX access log entry to the /var/log/nginx/access.log file every 5 seconds.

   generate_nginx_logs.sh

   #!/bin/bash
   
   #==============================================================================
   # Script Name: generate_nginx_logs.sh
   # Script Description: Simulates an NGINX server and continuously writes logs to access.log.
   #==============================================================================
   
   # --- Configurable Parameters ---
   
   # Log file path
   LOG_FILE="/var/log/nginx/access.log"
   
   # --- Mock Data Pools ---
   
   # Random IP address pool
   IP_ADDRESSES=(
       "192.168.1.10" "10.0.0.5" "172.16.31.40" "203.0.113.15"
       "8.8.8.8" "1.1.X.X" "91.198.XXX.XXX" "114.114.114.114"
       "180.76.XX.XX" "223.5.5.5"
   )
   
   # HTTP request method pool
   HTTP_METHODS=("GET" "POST" "PUT" "DELETE" "HEAD")
   
   # Common request path pool
   REQUEST_PATHS=(
       "/index.html" "/api/v1/users" "/api/v1/products?id=123" "/images/logo.png"
       "/static/js/main.js" "/static/css/style.css" "/login" "/admin/dashboard"
       "/robots.txt" "/sitemap.xml" "/non_existent_page.html"
   )
   
   # HTTP status code pool (You can adjust the weights. For example, add more 200s to increase their probability.)
   HTTP_STATUSES=(200 200 200 200 201 301 404 404 500 502 403)
   
   # Common User-Agent pool
   USER_AGENTS=(
       "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36"
       "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36"
       "Mozilla/5.0 (iPhone; CPU iPhone OS 14_6 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.1.1 Mobile/15E148 Safari/604.1"
       "Mozilla/5.0 (Linux; Android 11; SM-G991U) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.120 Mobile Safari/537.36"
       "curl/7.68.0"
       "Googlebot/2.1 (+http://www.google.com/bot.html)"
   )
   
   # Common Referer pool
   REFERERS=(
       "https://www.google.com/"
       "https://www.bing.com/"
       "https://github.com/"
       "https://stackoverflow.com/"
       "-"
       "-"
       "-"
   )
   #  Check and create the log directory
   LOG_DIR=$(dirname "$LOG_FILE")
   if [ ! -d "$LOG_DIR" ]; then
       echo "Log directory '$LOG_DIR' does not exist. Attempting to create..."
       # Use sudo to create the directory because root permissions are usually required
       sudo mkdir -p "$LOG_DIR"
       if [ $? -ne 0 ]; then
           echo "Error: Failed to create directory '$LOG_DIR'. Check permissions or create it manually."
           exit 1
       fi
       echo "Directory created successfully."
   fi
   
   # Check write permissions for the log file
   trap 'echo -e "\n\nScript interrupted. Stopping log generation..."; exit 0;' SIGINT
   
   # --- Core Function ---
   
   # Define a function to randomly select an element from an array
   # Usage: random_element "array_name"
   function random_element() {
       local arr=("${!1}")
       echo "${arr[$((RANDOM % ${#arr[@]}))]}"
   }
   
   # Catch the Ctrl+C interrupt signal for a graceful exit
   trap 'echo -e "\n\nScript interrupted. Stopping log generation..."; exit 0;' SIGINT
   
   # --- Main Loop ---
   
   echo "Start generating mock NGINX logs to $LOG_FILE ..."
   echo "A log entry is generated every 5 seconds."
   echo "Press Ctrl+C to stop."
   sleep 2
   
   # Infinite loop to continuously generate logs
   while true; do
       # 1. Get the current time in the default NGINX format: [dd/Mon/YYYY:HH:MM:SS +ZZZZ]
       timestamp=$(date +'%d/%b/%Y:%H:%M:%S %z')
   
       # 2. Randomly select data from the pools
       ip=$(random_element IP_ADDRESSES[@])
       method=$(random_element HTTP_METHODS[@])
       path=$(random_element REQUEST_PATHS[@])
       status=$(random_element HTTP_STATUSES[@])
       user_agent=$(random_element USER_AGENTS[@])
       referer=$(random_element REFERERS[@])
   
       # 3. Generate a random response body size (in bytes)
       bytes_sent=$((RANDOM % 5000 + 100)) # A random number between 100 and 5100
   
       # 4. Concatenate into a complete NGINX combined format log entry
       # Format: $remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent"
       log_line="$ip - - [$timestamp] \"$method $path HTTP/1.1\" $status $bytes_sent \"$referer\" \"$user_agent\""
   
       # 5. Append the log line to the file
       # echo "$log_line" >> "$LOG_FILE"
       echo "$log_line" | sudo tee -a "$LOG_FILE" > /dev/null
       
       # 6. Wait for 5 seconds before the next loop
       sleep 5
   done

3. Grant execution permissions: chmod +x generate_nginx_logs.sh.

4. Run the script in the background: nohup ./generate_nginx_logs.sh &.

What to do next

You have now completed the process of log collection, query and analysis, dashboard visualization, and alert configuration. We recommend that you read the following documents to better understand the core concepts and plan your log resource system based on your business needs:

• Understand the concepts of projects and logstores.

• Familiarize yourself with data collection methods and choose the appropriate one for your business scenario.

• Understand the storage resource hierarchy, plan the resource lifecycle, and allocate a reasonable number of shards.

FAQ