All Products
Search
Document Center

Simple Log Service:Use Logtail to collect logs across Alibaba Cloud accounts

Last Updated:Aug 28, 2024

This topic describes how to collect logs from a server across Alibaba Cloud accounts.

Background information

If you want to use Logtail to collect logs from a server, you must install Logtail on the server and configure the ID of the Alibaba Cloud account to which Simple Log Service belongs as a user identifier for the server. This way, the Alibaba Cloud account can use Logtail to collect logs from the server. If you do not configure a user identifier for the server, Simple Log Service fails to receive the heartbeat of the server and Logtail fails to collect logs from the server to Simple Log Service.

For example, an e-commerce company has two e-commerce applications that are deployed in Elastic Compute Service (ECS) clusters in the China (Hangzhou) region. The company uses two Simple Log Service projects that reside in the China (Hangzhou) region to manage logs.

  • Application A is deployed in a Linux ECS cluster that belongs to Alibaba Cloud Account A, and a Simple Log Service project that belongs to Alibaba Cloud Account A is used to manage logs. The ID of Alibaba Cloud Account A is 12****456. The Linux ECS cluster that belongs to Alibaba Cloud Account A is referred to as ECS Cluster A.

  • Application B is deployed in a Linux ECS cluster that belongs to Alibaba Cloud Account B, and a Simple Log Service project that belongs to Alibaba Cloud Account B is used to manage logs. The ID of Alibaba Cloud Account B is 17****397. The Linux ECS cluster that belongs to Alibaba Cloud Account B is referred to as ECS Cluster B.

The company plans to collect the logs of the two applications to two Logstores of a project that belongs to Alibaba Cloud Account A to meet business development requirements. In this case, you must create a Logtail configuration, a machine group, and a Logstore to collect and store the logs of Application B. The Logtail configuration, machine group, and Logstore that are configured for Application A remain unchanged.

架构图

Step 1: Create a user identifier file

  1. Log on to an ECS instance that belongs to Alibaba Cloud Account B.

    Important

    You must create a user identifier file on each ECS instance of ECS Cluster B.

  2. Run the following command to create a user identifier file.

    In this example, a file that is named based on the ID of Alibaba Cloud Account A is created. The ID is used as the user identifier. For more information, see Configure a user identifier.

    touch /etc/ilogtail/users/12****456

Step 2: Create a custom identifier-based machine group

  1. Create a custom identifier file for the machine group that you want to create on an ECS instance.

    Important

    You must create the custom identifier file on each ECS instance of ECS Cluster B.

    1. Log on to an ECS instance that belongs to Alibaba Cloud Account B.

    2. Create a file named /etc/ilogtail/user_defined_id in the /etc/ilogtail/ directory and specify a custom identifier in the file.

      For example, if you want to set the custom identifier to application_b, specify application_b in the file and save the file. For more information about the file paths, see Create a custom identifier-based machine group.

  2. Create a machine group in the Simple Log Service console.

    1. Log on to the Simple Log Service console by using Alibaba Cloud Account A.

    2. In the Projects section, click the project that you want to manage.

    3. In the left-side navigation pane, choose Resources > Machine Groups.

    4. To the right of Machine Groups, choose 机器组 > Create Machine Group.

    5. In the Create Machine Group panel, configure the following parameters and click OK.

      In the Custom Identifier field, enter the custom identifier that you specified in 1. For more information about other parameters, see Create a custom identifier-based machine group.机器组

  3. Check whether the heartbeat status of each ECS instance in the machine group is OK.

    1. In the Machine Groups list, click the machine group that is created.

    2. On the Machine Group Configurations page, view the list of ECS instances that use the same custom identifier and the heartbeat status of each ECS instance.

      If Heartbeat is OK, the ECS instance is connected to Simple Log Service. If Heartbeat is FAIL, see How do I troubleshoot an error that is related to a Logtail machine group in a host environment?机器组状态

Step 3: Collect logs

  1. Log on to the Simple Log Service console by using Alibaba Cloud Account A.

  2. Click Import Data.image

  3. In the Import Data dialog box, click Regular Expression - Text Logs.

  4. In the Select Logstore step, select the project and Logstore that you want to manage and click Next.

  5. In the Machine Group Configurations step, configure the parameters.

    1. Select Servers for Scenario.

    2. Select ECS for Installation Environment.

    3. Select the machine group that is created in Step 2, move the machine group from the Source Machine Group section to the Applied Server Groups section, and then click Next.

  6. Create a Logtail configuration and click Next.

    For more information about the parameters, see Collect logs in full regex mode.

    Important
    • By default, you can use only one Logtail configuration to collect logs from a log file. The collection process of Logtail in Alibaba Cloud Account B is not stopped. In this case, the Logtail configuration of Alibaba Cloud Account A cannot take effect. To ensure that the Logtail configuration of Alibaba Cloud Account A takes effect, you can use one of the following methods:

    • After you create the Logtail configuration, delete the existing Logtail configuration of Alibaba Cloud Account B to prevent duplicate log collection. For more information, see Delete Logtail configurations.

    logtail采集配置

  7. Preview data, configure indexes, and then click Next.

    By default, full-text indexing is enabled in Simple Log Service. You can configure field indexes based on the collected logs in manual or automatic mode. For more information, see Create indexes.

Related operations

If you want to migrate historical data of Alibaba Cloud Account B to the current Logstore from a different Logstore, you can create a data transformation job in the different Logstore and replicate the data to the current Logstore. For more information, see Replicate data from a Logstore.

Important

If you create a data transformation job to transform data across Alibaba Cloud accounts, you must use a custom role or an AccessKey pair to grant the required permissions to the job. In this example, a custom role is used.