Use Logtail to collect logs across regions - Simple Log Service

This topic describes how to use Logtail to collect logs from Linux servers that are in different regions but under the same Alibaba Cloud account.

Solution overview

For example, a company deploys its website application on an ECS instance in Region A and its Simple Log Service project in Region B. The company wants to use a Logtail configuration to collect logs from the ECS instance in Region A and send them to the Simple Log Service project in Region B. To do this, perform the following steps:

Step 1: Install Logtail on the ECS instance in Region A
Step 2: Configure a custom identifier for the ECS instance in Region A
Step 3: Create a custom identifier-based machine group in the Simple Log Service project in Region B
Step 4: Create a Logtail configuration in the Simple Log Service project in Region B

The following flowchart shows the configuration procedure.

Prerequisites

A project and a Logstore are required. For more information, see Manage a project and Create a Logstore.

Step 1: Install Logtail on the ECS instance in Region A

Network transmission

Transmission mode	Scenario
Internet	The Alibaba Cloud ECS instance and the Simple Log Service project reside in different regions. The server is provided by a third-party cloud service provider or is located in a data center.
Transfer acceleration	The server is provided by a third-party cloud service provider outside China or is located in a data center outside China. Data transfer over the Internet may cause high network latency and unstable connections. In this case, use the transfer acceleration feature. For more information, see Manage transfer acceleration.

Log on to the ECS instance in Region A. See Network transmission to select a Logtail installation script for your network environment. For information about the Linux operating systems that Logtail supports, see Limits.

Internet

Obtain the value of the Simple Log Service parameter for the region in which the Simple Log Service project resides. Replace the ${region_id} parameter with the obtained value and run the installation command.

Important

For more information about the value of the ${region_id} parameter for each region, see Supported regions. For example, the value of the ${region_id} parameter for the Singapore region is ap-southeast-1.

wget http://logtail-release-${region_id}.oss-${region_id}.aliyuncs.com/linux64/logtail.sh -O logtail.sh; chmod +x logtail.sh; ./logtail.sh install ${region_id}-internet

Transfer acceleration

Important

For more information about the value of the ${region_id} parameter for each region, see Supported regions. For example, the value of the ${region_id} parameter for the China (Hangzhou) region is cn-hangzhou.
After installing Logtial using transfer acceleration, you need to enable the transfer acceleration feature for it to take effect. For more information, see Enable the transfer acceleration feature for a project.

wget http://logtail-release-${region_id}.oss-${region_id}.aliyuncs.com/linux64/logtail.sh -O logtail.sh; chmod +x logtail.sh; ./logtail.sh install ${region_id}-acceleration

Step 2: Configure a custom identifier for the ECS instance in Region A

Create a file named user_defined_id in the specified directory and add a custom identifier to the file.
Important
- A Machine Group cannot contain both Linux and Windows servers, and the same Custom Identifier cannot be used for both.
- You can configure multiple Custom Identifiers for a server, separated by line feeds.

Linux environment

Log in to the Linux server where Logtail is installed and run the following command to configure a Custom Identifier.
Note
If the /etc/ilogtail/ directory does not exist, create the directory.
```
echo "user-defined-1" > /etc/ilogtail/user_defined_id
```
(Optional) Run the following command to check whether the Custom Identifier is configured. If user-defined-1 is returned, the custom identifier is configured.
```
cat /etc/ilogtail/user_defined_id
```
After you create, delete, or modify the user_defined_id file, the new settings take effect within 1 minute by default. To make the settings take effect immediately, you can run the following commands to restart Logtail.
```
/etc/init.d/ilogtaild stop
/etc/init.d/ilogtaild start
```

Windows environment

Log on to the Windows server where Logtail is installed. In the C:\LogtailData directory, create a file named user_defined_id that contains user-defined-1 and save the file.
Note
If the C:\LogtailData directory does not exist, create the directory.
After you create, delete, or modify the user_defined_id file, the new settings take effect within 1 minute by default. To make the settings take effect immediately, you can restart Logtail.
1. Choose Start > Control Panel > Administrative Tools > Services.
2. In the Services dialog box, select the service to manage.
3. For Logtail V0.x.x.x, select the LogtailWorker service.
4. For Logtail V1.0.0.0 or later, select the LogtailDaemon service.
5. Right-click the service and select Restart.

Container environment

The custom identifier is set in the ALIYUN_LOGTAIL_USER_DEFINED_ID environment variable of the Logtail container. You can run the docker inspect ${logtail_container_name} | grep ALIYUN_LOGTAIL_USER_DEFINED_ID command to view this setting.

Step 3: Create a custom identifier-based machine group in the Simple Log Service project in Region B

Log on to the Simple Log Service console. In the Projects section, click the one you want.
In the navigation pane on the left, choose Resources > Machine Groups. In the Machine Groups list, choose > Create Machine Group.

In the Create Machine Group panel, configure parameters and click OK. The following table describes the parameters.

Parameter	Description
Name	The name of the machine group must meet the following requirements: Contains only lowercase letters, digits, hyphens (-), and underscores (_). Starts and ends with a lowercase letter or a digit. Is 2 to 128 characters in length. Important After you create a machine group, you cannot change its name. Proceed with caution.
Machine Group Identifier	The identifier type of the machine group. Select IP Address.
Machine Group Topic	Optional. The topic is used to identify the logs generated by different servers.
IP Address	The IP address. Enter the IP address that is automatically obtained by Logtail. Important If you want to add multiple servers to a machine group, we recommend that you manually enter the IP addresses of the servers and separate the IP addresses with line feeds. A machine group cannot include both Linux and Windows servers. Do not add the IP addresses of Linux and Windows servers to the same machine group.

In the Machine Groups list, click the target machine group. On the Machine Group Configurations page, check the configuration of the machine group and the server status.

Step 4: Create a Logtail configuration in the Simple Log Service project in Region B

Important

Ports 80 (HTTP) and 443 (HTTPS) must be open for outbound traffic on the host where Logtail is installed. If the host is an ECS instance, the ports are controlled by security group rules. For more information about how to add a security group rule, see Add a security group rule.
Logtail collects only incremental logs from files that are continuously updated. If a log file is not updated after you apply a Logtail configuration, Logtail does not collect logs from that file. For more information, see Read log files.
To collect historical data, see Import historical log files.

References

If the preview page is blank or the query page displays no data when you use Logtail to collect logs, you can troubleshoot the issue by following the instructions provided in What do I do if errors occur when I use Logtail to collect logs? If you use Logtail to collect logs, errors may occur. For example, regular expressions may fail to be parsed, invalid file paths may exist, and traffic may exceed the processing capabilities of shards. You can view Logtail collection errors by following the instructions provided in How do I view Logtail collection errors? For more information about the common errors in data collection, see Common error types in Simple Log Service data collection.
By default, you can use only one Logtail configuration to collect logs from a log file. If you want to collect multiple copies of logs in a file, you can follow the instructions provided in How do I collect multiple copies of logs in a file?
For more information about how to collect logs from servers in a corporate intranet to Simple Log Service, see Collect logs from servers in a corporate intranet.