Use Logtail to collect logs across regions - Simple Log Service

This topic describes how to collect logs from a Linux server in a different region. This example assumes that both resources are under the same Alibaba Cloud account.

Solution overview

A company deploys its website application on an ECS instance in Region A and its Simple Log Service project in Region B. The company wants to use a Logtail configuration to send log data from the ECS instance in Region A to the Simple Log Service project in Region B. You can complete this configuration by performing the following steps:

Step 1: Install Logtail on the ECS instance in Region A
Step 2: Configure a custom identifier on the ECS instance in Region A
Step 3: Create a custom identifier-based machine group in the Simple Log Service project in Region B
Step 4: Create a Logtail collection configuration in the Simple Log Service project in Region B

The following flowchart shows the configuration process.

Prerequisites

A project and a Logstore are created. For more information, see Manage projects and Create a basic Logstore.

Step 1: Install Logtail on the ECS instance in Region A

Network transmission

Transmission mode	Scenario
Internet	The Alibaba Cloud ECS instance and the Simple Log Service project are in different regions. The server is from a third-party cloud provider or in an on-premises data center.
Transfer acceleration	Your servers are in on-premises data centers or from cloud providers outside China. Data transfer over the Internet may cause high network latency and unstable connections. In this case, use transfer acceleration. For more information, see Manage transfer acceleration.

Log on to the ECS instance in Region A and select a Logtail installation script based on your network environment. For more information, see Network transmission. For more information about the Linux operating systems that Logtail supports, see Limits.

Internet

Obtain the value of the Simple Log Service parameter for the region in which the Simple Log Service project resides. Replace the ${region_id} parameter with the obtained value and run the installation command.

Important

For more information about the value of the ${region_id} parameter for each region, see Supported regions. For example, the value of the ${region_id} parameter for the Singapore region is ap-southeast-1.

wget http://logtail-release-${region_id}.oss-${region_id}.aliyuncs.com/linux64/logtail.sh -O logtail.sh; chmod +x logtail.sh; ./logtail.sh install ${region_id}-internet

Transfer acceleration

Important

For more information about the value of the ${region_id} parameter for each region, see Supported regions. For example, the value of the ${region_id} parameter for the China (Hangzhou) region is cn-hangzhou.
After installing Logtial using transfer acceleration, you need to enable the transfer acceleration feature for it to take effect. For more information, see Enable the transfer acceleration feature for a project.

wget http://logtail-release-${region_id}.oss-${region_id}.aliyuncs.com/linux64/logtail.sh -O logtail.sh; chmod +x logtail.sh; ./logtail.sh install ${region_id}-acceleration

Step 2: Configure a custom identifier on the ECS instance in Region A

Create the custom identifier file user_defined_id in the specified directory and configure the custom identifier.
Important
- A machine group cannot contain both Linux and Windows servers. Do not configure the same custom identifier for Linux and Windows servers.
- You can configure multiple custom identifiers for a server. Separate the identifiers with line feeds.

Linux environment

Log on to the Linux server where Logtail is installed. Run the following command to configure the custom identifier.
Note
If the /etc/ilogtail/ folder does not exist, create it manually.
```
echo "user-defined-1" > /etc/ilogtail/user_defined_id
```
(Optional) Run the following command to check whether the custom identifier was written successfully. If user-defined-1 is returned, the write operation was successful.
```
cat /etc/ilogtail/user_defined_id
```
After you add, delete, or modify the user_defined_id file, the changes take effect within one minute by default. If you want the new configuration to take effect immediately, run the following commands to restart Logtail:
```
/etc/init.d/ilogtaild stop
/etc/init.d/ilogtaild start
```

Windows environment

Log on to the Windows server where Logtail is installed. In the C:\LogtailData folder, create a file named user_defined_id, write user-defined-1 to the file, and save it.
Note
If the C:\LogtailData folder does not exist, create it manually.
After you create, delete, or modify the user_defined_id file, the changes take effect within one minute by default. To make the changes take effect immediately, restart Logtail as follows:
1. Choose Start > Control Panel > Administrative Tools > Services.
2. In the Services dialog box, select the service.
3. For Logtail V0.x.x.x, select the LogtailWorker service.
4. For Logtail V1.0.0.0 or later, select the LogtailDaemon service.
5. Right-click Restart to apply the configuration.

Container environment

The custom identifier is configured in the ALIYUN_LOGTAIL_USER_DEFINED_ID environment variable of the Logtail container. You can run the docker inspect ${logtail_container_name} | grep ALIYUN_LOGTAIL_USER_DEFINED_ID command to view this setting.

Step 3: Create a custom identifier-based machine group in the Simple Log Service project in Region B

Log on to the Simple Log Service console. In the Projects section, click the one you want.
In the navigation pane on the left, choose Resources > Machine Groups. In the Machine Groups list, choose > Create Machine Group.

In the Create Machine Group panel, configure parameters and click OK. The following table describes the parameters.

Parameter	Description
Name	The name of the machine group must meet the following requirements: Contains only lowercase letters, digits, hyphens (-), and underscores (_). Starts and ends with a lowercase letter or a digit. Is 2 to 128 characters in length. Important After you create a machine group, you cannot change its name. Proceed with caution.
Machine Group Identifier	The identifier type of the machine group. Select IP Address.
Machine Group Topic	Optional. The topic is used to identify the logs generated by different servers.
IP Address	The IP address. Enter the IP address that is automatically obtained by Logtail. Important If you want to add multiple servers to a machine group, we recommend that you manually enter the IP addresses of the servers and separate the IP addresses with line feeds. A machine group cannot include both Linux and Windows servers. Do not add the IP addresses of Linux and Windows servers to the same machine group.

In the Machine Groups list, click the machine group that you created. On the Machine Group Configurations page, check the configuration and the server status.

Step 4: Create a Logtail collection configuration in the Simple Log Service project in Region B

Important

The host where Logtail is installed must have outbound access to port 80 (HTTP) and port 443 (HTTPS). For an ECS instance, security group rules control port access. For more information about how to add a security group rule, see Add a security group rule.
The log file on the server must be continuously updated. Logtail collects only incremental logs. If the log file is not updated after the Logtail configuration is applied, Logtail does not collect logs from the file. For more information, see Read log files.
To collect historical data, see Import historical log files.

References

If the preview page is blank or the query page displays no data when you use Logtail to collect logs, you can troubleshoot the issue by following the instructions provided in What do I do if errors occur when I use Logtail to collect logs? If you use Logtail to collect logs, errors may occur. For example, regular expressions may fail to be parsed, invalid file paths may exist, and traffic may exceed the processing capabilities of shards. You can view Logtail collection errors by following the instructions provided in How do I view Logtail collection errors? For more information about the common errors in data collection, see Common error types in Simple Log Service data collection.
By default, you can use only one Logtail configuration to collect logs from a log file. If you want to collect multiple copies of logs in a file, you can follow the instructions provided in How do I collect multiple copies of logs in a file?
For more information about how to collect logs from servers in a corporate intranet to Simple Log Service, see Collect logs from servers in a corporate intranet.