All Products
Search
Document Center

Simple Log Service:Getting Started

Last Updated:Jan 25, 2024

This topic describes how to use Logtail to collect the NGINX access logs of an Alibaba Cloud Elastic Compute Service (ECS) instance. This topic also describes how to activate Simple Log Service, create a project and a Logstore, use Logtail to collect logs, and query and analyze the collected logs.

Prerequisites

  • An ECS instance is available. For more information, see ECS quick start.

  • The ECS instance continuously generates logs.

    Important

    Logtail collects only incremental logs. If a log file on a server is not updated after the applied Logtail configuration is delivered to the server, Logtail does not collect logs from the file. For more information, see Read log files.

Background information

In this example, the logs are stored in the /var/log/nginx/access.log file, and the sample log is 127.0.0.1 - - [10/Jun/2022:12:36:49 +0800] "GET /index.html HTTP/1.1" 200 612 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36". In this example, the full regex mode is used to collect logs based on the sample log. For more information, see Collect logs in full regex mode.

Step 1: Activate Simple Log Service

  1. Log on to the Simple Log Service console.

  2. Follow the on-screen instructions to activate Simple Log Service.

    For information about the billing of Simple Log Service, see Billing overview.

Step 2: Create a project and a Logstore

  1. Create a project.

    1. In the Projects section, click Create Project.

    2. In the Create Project panel, configure the following parameters. For other parameters, retain the default settings. For more information, see Create a project.

      Parameter

      Description

      Project Name

      The name of the project. The name must be unique within your Alibaba Cloud account. After the project is created, you cannot change the name of the project.

      Region

      The region where the data center of the project resides. We recommend that you select the region where the ECS instance resides. Then, you can use an internal network of Alibaba Cloud to accelerate log collection.

      After the project is created, you cannot change the region or migrate the project to another region.

    3. Click Create.

  2. Create a Logstore.

    After the project is created, you are prompted to create a Logstore.

    In the Create Logstore panel, configure the following parameters. For other parameters, retain the default settings. For more information, see Create a Logstore.

    Parameter

    Description

    Billing Mode

    The billing mode of the Logstore. Valid values: Pay-by-ingested-data and Pay-by-feature. For more information, see Billable items.

    Logstore Name

    The name of the Logstore. The name must be unique in the project to which the Logstore belongs.

    After the Logstore is created, you cannot change the name of the Logstore.

    Shards

    The number of shards. Simple Log Service provides shards that allow you to read and write data.

    Each shard supports a write capacity of 5 MB/s and 500 writes/s and a read capacity of 10 MB/s and 100 reads/s. If one shard can meet your business requirements, you can set Shards to 1.

    Automatic Sharding

    Specifies whether to enable the automatic sharding feature. If you turn on Automatic Sharding, Simple Log Service increases the number of shards when the existing shards cannot accommodate the data that is written.

    If the specified number of shards can meet your business requirements, you can turn off Automatic Sharding.

Step 3: Collect logs

After the Logstore is created, you are prompted to import data.

Important

By default, you can use only one Logtail configuration to collect logs from a log file. For information about how to use multiple Logtail configurations to collect logs from a log file, see What do I do if I want to use multiple Logtail configurations to collect logs from a log file?

  1. In the Created dialog box, click OK.

  2. In the left-side navigation section of the Import Data dialog box, click On-premises Open Source/Commercial Software. Then, click Integrate Now in the RegEx - Text Log card.

  3. Create a machine group.

    1. On the ECS Instances tab, select the ECS instance and click Create.

      For more information, see Install Logtail on ECS instances.

    2. In the Parameter Confirmation dialog box, click OK.

    3. Make sure that the value of the Execution Status parameter is Success. Then, click Complete Installation.

    4. In the Create Machine Group step, enter a machine group name and retain the default settings for other parameters. Then, click Next.

      For more information, see Create an IP address-based machine group.

  4. Confirm that the machine group is displayed in the Applied Server Groups section and click Next.

    Important

    If you apply a machine group immediately after you create the machine group, the heartbeat status of the machine group may be FAIL. This issue occurs because the machine group is not connected to Simple Log Service. To resolve this issue, you can click Automatic Retry. If the issue persists, see What do I do if no heartbeat connections are detected on Logtail?

  5. Create a Logtail configuration and click Next.

    Configure the following parameters and retain the default settings for other parameters. For more information, see Collect logs in full regex mode.

    Parameter

    Description

    Config Name

    The name of the Logtail configuration. The name must be unique in the project.

    After the Logtail configuration is created, you cannot change the name of the Logtail configuration.

    Log Path

    The directory and name of log files. The value varies based on the location of the logs on your server. In this example, specify /var/log/nginx/access.log.

    Log Sample

    A valid sample log that is collected from an actual scenario. In this example, enter the following sample log:

    127.0.0.1 - - [10/Jun/2022:12:36:49 +0800] "GET /index.html HTTP/1.1" 200 612 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"

    Extract Field

    If you turn on Extract Field, Simple Log Service can extract log content in key-value pairs by using a regular expression.

    RegEx

    The regular expression.

    • Automatic generation

      In the Log Sample field, select the content that you want to extract and click Generate Regular Expression. A regular expression is automatically generated. In this example, the following regular expression is generated: (\S+)\s-\s(\S+)\s\[([^]]+)]\s"(\w+)([^"]+)"\s(\d+)\s(\d+)[^-]+([^"]+)"\s"([^"]+).*.

    • Manual configuration

      Click Manual to specify a regular expression. Then, click Validate to check whether the regular expression can be used to parse the sample log and extract content from the sample log. For more information, see How do I test a regular expression?

    Extracted Content

    After the content of the sample log is extracted as values by using the regular expression, you must specify a key for each value. For example, if Value is 127.0.0.1, you can set Key to remote_addr.

    After you configure the parameters, click Next. Then, Simple Log Service starts to collect logs.

    Note
  6. Preview data, configure indexes, and then click Next.

    By default, full-text indexing is enabled for Simple Log Service. You can also configure field indexes based on collected logs in manual mode or automatic mode. To configure field indexes in automatic mode, click Automatic Index Generation. This way, Simple Log Service automatically creates field indexes. For more information, see Create indexes.

    Important

    If you want to query and analyze logs, you must enable full-text indexing or field indexing. If you enable both full-text indexing and field indexing, the system uses only field indexes.

Step 4: Query and analyze logs

After you create indexes, you can query and analyze logs.

  1. In the End step of the wizard, click Log Query.

    You must wait approximately 1 minute for the indexes to take effect. Then, you can view the collected logs on the Raw Logs tab. For more information, see Query and analyze logs.

  2. On the query and analysis page of the Logstore that you specify, enter a query statement and select a time range.

    For example, you can execute the following query statement to count the number of requests that correspond to each status code. The query and analysis results are displayed in a table.

FAQ

Am I charged if I only create projects and Logstores?

By default, shard resources are reserved when you create a Logstore. You are charged for active shards. For more information, see Why am I charged for active shards?

What do I do if logs fail to be collected?

When you use Logtail to collect logs, a failure may occur due to Logtail heartbeat failures, collection errors, or invalid Logtail configurations. For more information, see What do I do if errors occur when I use Logtail to collect logs?

What do I do if I can query logs but cannot analyze logs on the query and analysis page of a Logstore?

If you want to analyze logs, you must configure indexes for log fields and turn on Enable Analytics for the fields. For more information, see Create indexes.