This topic describes how to create a Logtail configuration in Simple Log Service (SLS) using Terraform to automate log collection setup.
Run the sample code in this topic with a few clicks. For more information, visit Terraform Explorer.
Resource dependency chain
Setting up Logtail log collection requires five Terraform resources, created in this order:
alicloud_log_project
├── alicloud_log_store (Logstore)
├── alicloud_log_machine_group (machine group)
│
└── alicloud_logtail_config (Logtail configuration)
└── alicloud_logtail_attachment (binds config to machine group)
| Terraform resource | Description | Registry link |
|---|---|---|
alicloud_log_project |
SLS project | View |
alicloud_log_store |
Logstore within the project | View |
alicloud_log_machine_group |
Machine group for Logtail | View |
alicloud_logtail_config |
Logtail configuration | View |
alicloud_logtail_attachment |
Applies the Logtail configuration to the machine group | View |
Prerequisites
Before you begin, make sure that you have:
-
SLS activated. For more information, see Resource management overview
-
(Recommended) A Resource Access Management (RAM) user with the minimum required permissions to reduce the risk of leaking the AccessKey pair of your Alibaba Cloud account. For information about creating a RAM user and attaching policies, see Create a RAM user and Grant permissions to a RAM user
-
A Terraform runtime environment, set up using one of the following methods:
-
Terraform Explorer -- an online environment provided by Alibaba Cloud. No installation required.
-
Cloud Shell -- preinstalled with Terraform and preconfigured with your identity credentials.
-
Local installation -- suitable when network connections are unstable or a custom development environment is needed.
-
Minimum required permissions
Attach the following policy to the RAM user:
{
"Version": "1",
"Statement": [
{
"Effect": "Allow",
"Action": [
"log:GetProject",
"log:ListProject",
"log:CreateProject",
"log:DeleteProject",
"log:UpdateProject",
"log:GetAppliedMachineGroups",
"log:GetMachineGroup",
"log:ListMachineGroup",
"log:CreateMachineGroup",
"log:GetLogStore",
"log:GetLogStoreLogs",
"log:GetLogStoreMeteringMode",
"log:ListLogStores",
"log:CreateLogStore",
"log:PostLogStoreLogs",
"log:UpdateLogStore",
"log:GetLogtailPipelineConfig",
"log:UpdateLogtailPipelineConfig",
"log:ListTagResources",
"log:ListShards",
"log:ListSavedSearch",
"log:GetIndex",
"log:ListDashboard",
"log:ListConfig",
"log:CreateConfig",
"log:GetConfig",
"log:ApplyConfigToGroup",
"log:DeleteConfig",
"log:DeleteMachineGroup",
"log:GetProjectPolicy",
"log:DeleteLogStore"
],
"Resource": "*"
}
]
}
Resources created in this topic may incur charges. Release or unsubscribe from resources that you no longer need at the earliest opportunity.
Required resources
-
alicloud_log_project: the project.
-
alicloud_log_machine_group: the machine group.
-
alicloud_log_store: the Logstore.
-
alicloud_logtail_config: the Logtail configuration.
-
alicloud_logtail_attachment: apply the Logtail configuration to the machine group.
Step 1: Create a project
-
Create a working directory and a configuration file named main.tf in the directory. Add the following code to the file:
variable "region" { default = "cn-hangzhou" } variable "identify_list" { type = list(string) description = "IP addresses of machines included in the machine group" default = ["10.0.0.1", "10.0.0.2"] } provider "alicloud" { region = var.region } resource "random_integer" "default" { min = 10000 max = 99999 } # The project. resource "alicloud_log_project" "example" { project_name = "project-name-${random_integer.default.result}" description = "tf actiontrail example" } -
Initialize the Terraform runtime environment:
terraform initExpected output:
Terraform has been successfully initialized! You may now begin working with Terraform. Try running "terraform plan" to see any changes that are required for your infrastructure. All Terraform commands should now work. If you ever set or change modules or backend configuration for Terraform, rerun this command to reinitialize your working directory. If you forget, other commands will detect it and remind you to do so if necessary. -
Apply the execution plan:
terraform applyEnter
yeswhen prompted and press Enter. Expected output:Apply complete! Resources: 2 added, 0 changed, 0 destroyed. -
Verify the result.
Option A: Run
terraform showRun the following command in the working directory to view the project details:
terraform showOption B: Check the SLS console
Log on to the SLS console and verify that the project appears in the Projects section.
Step 2: Create a machine group and a Logstore
-
Add the following code to
main.tf:# The machine group. resource "alicloud_log_machine_group" "example" { project = alicloud_log_project.example.project_name name = "terraform-example-${random_integer.default.result}" identify_type = "ip" topic = "terraform" identify_list = var.identify_list } # The Logstore. resource "alicloud_log_store" "example" { project_name = alicloud_log_project.example.project_name logstore_name = "logstore_example_${random_integer.default.result}" retention_period = 3 } -
Preview the changes:
terraform plan -
Apply the execution plan:
terraform applyEnter
yeswhen prompted and press Enter. Expected output:Apply complete! Resources: 2 added, 0 changed, 0 destroyed. -
Verify the result.
Option A: Run
terraform showRun the following command in the working directory to view the machine group and Logstore details:
terraform showOption B: Check the SLS console
-
Log on to the SLS console. In the Projects section, find the project created in Step 1 and click the project name. In the left-side navigation pane, click the Log Storage icon. In the Logstores list, verify that the Logstore appears.
-
In the left-side navigation pane, move the pointer over the icon and select Machine Groups. In the Machine Groups list, verify that the machine group appears.
-
Step 3: Create a Logtail configuration
-
Add the following code to
main.tf:# The Logtail configuration. resource "alicloud_logtail_config" "example" { project = alicloud_log_project.example.project_name logstore = alicloud_log_store.example.logstore_name name = "config-sample-${random_integer.default.result}" input_type = "file" output_type = "LogService" input_detail = jsonencode( { "logPath": "/logPath", "filePattern": "access.log", "logType": "json_log", "topicFormat": "default", "discardUnmatch": false, "enableRawLog": true, "fileEncoding": "gbk", "maxDepth": 10 } ) } # Apply the Logtail configuration to the machine group. resource "alicloud_logtail_attachment" "example" { project = alicloud_log_project.example.project_name logtail_config_name = alicloud_logtail_config.example.name machine_group_name = alicloud_log_machine_group.example.name } -
Preview the changes:
terraform plan -
Apply the execution plan:
terraform applyEnter
yeswhen prompted and press Enter. Expected output:Apply complete! Resources: 2 added, 0 changed, 0 destroyed. -
Verify the result.
Option A: Run
terraform showRun the following command in the working directory to view the Logtail configuration details:
terraform showOption B: Check the SLS console
-
Log on to the SLS console. In the Projects section, find the project created in Step 1 and click the project name. In the left-side navigation pane, click the Log Storage icon. In the Logstores list, find the Logstore created in Step 2 and click the Logstore. Then, choose Data Collection > Logtail Configurations to view the Logtail configuration of the Logstore.
-
On the Logtail Configurations page, click the name of a Logtail configuration. Then, click the Manage Machine Groups tab. On this tab, verify that the machine group is applied with the Logtail configuration.
-
input_detail parameter reference
The input_detail field is a JSON object that defines how Logtail collects and parses log files.
| Field | Value in example | Description |
|---|---|---|
logPath |
"/logPath" |
The directory path where log files are stored. |
filePattern |
"access.log" |
The file name or pattern used to match log files. |
logType |
"json_log" |
The log format type. Set to json_log for JSON-formatted logs. |
topicFormat |
"default" |
The topic generation mode. default uses the log file path as the log topic. |
discardUnmatch |
false |
Whether to discard logs that fail to match the parsing rules. false retains unmatched logs. |
enableRawLog |
true |
Whether to upload raw log data. true uploads the original log content. |
fileEncoding |
"gbk" |
The character encoding of the log file. Common values: utf8, gbk. |
maxDepth |
10 |
The maximum depth of subdirectories to monitor. 0 monitors only the specified directory. 10 monitors up to 10 levels of subdirectories. |
Release resources
If the resources created in this topic are no longer needed, run the following command to release them:
terraform destroy
Enter yes when prompted and press Enter to confirm the deletion.
For more information about terraform destroy, see Common commands.
Complete example
Run this sample code with a few clicks using Terraform Explorer.
For more examples, visit the Log_Service(SLS) folder on GitHub.