Simple Log Service:Collect logs from ECS instances in other Alibaba Cloud accounts to Simple Log Service

Scenarios

In a multi-account cloud environment, you may need to centralize logs for the following common reasons:

• Unified service monitoring: Store application logs and system logs from different business accounts in a central location. This simplifies unified monitoring, alerting, and troubleshooting.

• Security and compliance auditing: Aggregate all operation logs and security logs into a dedicated security or audit account to meet compliance requirements.

• Business migration: Transition log collection tasks to a new account during a business migration from one Alibaba Cloud account to another.

This topic uses the migration of log collection tasks from a source account to a destination account as an example.

Solution architecture

This solution uses a trust model based on file authorization. You can create specific authorization and identifier files on the ECS instances in the source account. This grants the destination account permission to collect logs from these instances using Logtail.

The workflow is as follows:

1. On each ECS instance in the source account (Account B), create an authorization file. The file is named after the UID of the destination account (Account A).

2. On the same ECS instances, create a file that contains a custom identifier. This identifier is used to group the servers in the destination account.

3. After the Logtail client on an ECS instance recognizes the authorization file, it sends heartbeats to Simple Log Service in the destination account (Account A) and reports its custom identifier.

4. In the Simple Log Service console of the destination account (Account A), create a machine group that uses this custom identifier. ECS instances that have normal heartbeats are automatically added to this machine group.

5. Apply a Logtail collection configuration to the machine group. Logtail then collects logs based on the configuration and sends them to the Logstore in the destination account (Account A).

Procedure

#!/bin/bash

# Replace with the 16-digit Alibaba Cloud account ID of the destination account (Account A).
TARGET_ACCOUNT_ID="{TARGET_ACCOUNT_ID}"

# Define a unique identifier for this group of machines.
# Use a combination of the business and environment names, such as "my-app-prod-servers".
MACHINE_GROUP_ID="{YOUR_MACHINE_GROUP_ID}"

# 1. Create the authorization file directory.
sudo mkdir -p /etc/ilogtail/users

# 2. Create the authorization file. The file name is the destination account ID. This grants collection permissions to that account.
sudo touch "/etc/ilogtail/users/${TARGET_ACCOUNT_ID}"

# 3. Create the custom identifier file. This file is used to identify and group the machines in the destination account.
echo "${MACHINE_GROUP_ID}" | sudo tee /etc/ilogtail/user_defined_id > /dev/null

# 4. Restart the Logtail service to apply the configuration.
# If your system does not use systemd, run 'sudo /etc/init.d/ilogtaild restart'.
sudo systemctl restart ilogtail

echo "Configuration complete. Check the heartbeat status of the machine group in the Simple Log Service console of the destination account."

Step 1: Create a user identity file

1. Log on to the ECS server that belongs to Alibaba Cloud account B.

   Important

   You must create the user identity file on each ECS server in ECS cluster B.

2. Run the following command to create the user identity file.

   Set the user identity to Alibaba Cloud account A. This creates a file named after the ID of Alibaba Cloud account A. For more information, see Configure a user identity.

   touch /etc/ilogtail/users/12****456

Step 2: Create a custom identifier-based machine group

1. Create a custom identifier file for the machine group on the ECS server.

   Important

   You must create the custom identifier file for the machine group on each ECS server in ECS cluster B.

   1. Log on to the ECS server that belongs to Alibaba Cloud account B.

   2. In the specified directory, create the /etc/ilogtail/user_defined_id file and add a custom identifier.

      For example, to set the custom identifier to application_b, enter application_b in the file and save it. For more information about the file path, see Create a custom identifier-based machine group.

2. Create a machine group in the Simple Log Service console.

   1. Using Alibaba Cloud account A, log on to the Simple Log Service console.

   2. In the Project List area, click the destination project.

   3. In the navigation pane, choose Resource > Machine Group.

   4. To the right of Machine Group, click > Create Machine Group.

   5. In the Create Machine Group dialog box, configure the following parameters and click OK.

      Set Custom Identifier to the custom identifier that you configured in step 1. For more information about the other parameters, see Create a custom identifier-based machine group.

3. Verify that the heartbeat status of the servers in the machine group is OK.

   1. In the machine group list, click the destination machine group.

   2. On the Machine Group Configuration page, view the ECS servers that use the same custom identifier and their heartbeat status.

      A Heartbeat status of OK indicates that the ECS server is connected to Simple Log Service. If the status is FAIL, see A Logtail machine group has no heartbeat.

Step 3: Collect logs

1. Log on to the Simple Log Service console with Alibaba Cloud account A.

2. Click Quick Integration.

   

3. In the Integrate Data dialog box, select Regex - Text Log.

4. In the Integrate Data wizard, select the destination project and Logstore, and then click Next.

5. Configure Machine Group Settings.

   1. Set Scenario to Host Scenario.

   2. Set Installation Environment to ECS.

   3. Select the machine group that you created in Step 2. Move the machine group from Source Machine Groups to Applied Machine Groups, and then click Next.

6. Create a Logtail collection configuration and click Next.

   For more information about the parameters, see Collect logs in full regex mode.

   Important

   • By default, only one Logtail collection configuration can match a file. Because the collection configuration for account B is still active, the new configuration for account A cannot take effect. You can use one of the following methods to activate the configuration for account A:

     • Stop the log collection for account B. To do this, log on to the Simple Log Service console with account B and remove the Logtail collection configuration from the destination machine group. For more information, see Apply a Logtail configuration to a specified machine group.

     • Add a forced collection configuration for account A. For more information, see How to collect logs from a file multiple times.

   • After you create the Logtail collection configuration, delete the original configuration for Alibaba Cloud account B to prevent duplicate log collection. For more information, see Delete a Logtail configuration.

   

7. Preview the data, configure indexes, and then click Next.

   Simple Log Service enables the full-text index feature by default. You can also manually or automatically configure field indexes based on the collected logs. For more information, see Create an index.

Related operations