This topic describes how to install and upgrade Logtail components in a self-managed Kubernetes cluster.
Install Logtail
Log on to the Simple Log Service console.
Create a project. For more information, see Create a project.
We recommend that you create a project whose name starts with
k8s-log-custom-
. Example: k8s-log-custom-sd89ehdq.Log on to your Kubernetes cluster.
Run the following commands to install Logtail and dependent components.
ImportantMake sure that the kubectl command-line tool is installed on the machine on which you want to run the commands.
alibaba-log-controller is available only in Kubernetes 1.6 or later.
If you no longer need to use custom resource definitions (CRDs), you can delete the
alibaba-cloud-log/templates/alicloud-log-config.yaml
file and rerun the following commands.
Download and decompress the installation package.
Regions in China
wget https://logtail-release-cn-hangzhou.oss-cn-hangzhou.aliyuncs.com/kubernetes/0.5.3/alibaba-cloud-log-all.tgz; tar xvf alibaba-cloud-log-all.tgz; chmod 744 ./alibaba-cloud-log-all/k8s-custom-install.sh
Regions outside China
wget https://logtail-release-ap-southeast-1.oss-ap-southeast-1.aliyuncs.com/kubernetes/0.5.3/alibaba-cloud-log-all.tgz; tar xvf alibaba-cloud-log-all.tgz; chmod 744 ./alibaba-cloud-log-all/k8s-custom-install.sh
Modify the
./alibaba-cloud-log-all/values.yaml
configuration file.# ===================== Required settings ===================== # The name of the project. SlsProjectName: # The region where the project resides. Region: # The ID of the Alibaba Cloud account to which the project belongs. You must enclose the ID in double quotation marks (""). AliUid: "11**99" # The AccessKey ID and AccessKey secret of the Alibaba Cloud account or a RAM user. The RAM user must have the AliyunLogFullAccess permission. AccessKeyID: AccessKeySercret: # The custom ID of the cluster. The ID can contain only letters, digits, and hyphens (-). ClusterID: # ========================================================== # Specifies whether to enable metric collection for the related components. Valid values: true and false. Default value: true. SlsMonitoring: true # The network type. Valid values: Internet and Intranet. Default value: Internet. Net: Internet # Specifies whether containerd is used as the container runtime. Valid values: true and false. Default value: false. SLS_CONTAINERD_USED: true
The following table describes the parameters that are included in the preceding command. You can configure the parameters based on your business scenario.
Parameter
Description
SlsProjectName
The name of the project that you created in Step 2.
Region
The ID of the region where your project resides. For example, the ID of the China (Hangzhou) region is
cn-hangzhou
. For more information, see Supported regions.AliUid
The ID of your Alibaba Cloud account. You must enclose the ID in double quotation marks (""). Example:
AliUid: "11**99"
. For information about how to obtain the ID of an Alibaba Cloud account, see Obtain the ID of the Alibaba Cloud account for which Simple Log Service is activated.AccessKeyID
The AccessKey ID of your Alibaba Cloud account. We recommend that you use the AccessKey pair of a RAM user and attach the AliyunLogFullAccess policy to the RAM user. For more information, see Create a RAM user and authorize the RAM user to access Simple Log Service.
AccessKeySercret
The AccessKey secret of your Alibaba Cloud account. We recommend that you use the AccessKey pair of a RAM user and attach the AliyunLogFullAccess policy to the RAM user. For more information, see Create a RAM user and authorize the RAM user to access Simple Log Service.
ClusterID
The custom ID of the cluster. The ID can contain only letters, digits, and hyphens (-). This parameter corresponds to the
${your_k8s_cluster_id}
variable in the following operations.ImportantDo not specify the same cluster ID for different Kubernetes clusters.
SlsMonitoring
Specifies whether to enable metric collection for the related components. Valid values:
true (default)
false
Net
The network type. Valid values:
Internet (default)
Intranet
SLS_CONTAINERD_USED
Specifies whether containerd is used as the container runtime. Valid values:
true: containerd is used.
false (default) : containerd is not used.
ImportantIn the self-managed Kubernetes cluster using containerd as the container runtime, this parameter must be set to true. Otherwise, Logtail may fail to collect logs.
Install Logtail and dependent components.
bash k8s-custom-install.sh; kubectl apply -R -f result
The following table describes the Simple Log Service resources that are automatically created after you install Logtail and dependent components.
Do not delete the
config-operation-log
Logstore.If you install Logtail components in a self-managed Kubernetes cluster, Logtail is granted the
privileged
permissions by default. This prevents thecontainer text file busy
error that may occur when other pods are deleted. For more information, see Bug 1468249, Bug 1441737, and Issue 34538.
Resource type | Resource name | Description | Example |
Machine group | k8s-group-${your_k8s_cluster_id} | The machine group of logtail-daemonset, which is used in log collection scenarios. | k8s-group-my-cluster-123 |
k8s-group-${your_k8s_cluster_id}-statefulset | The machine group of logtail-statefulset, which is used in metric collection scenarios. | k8s-group-my-cluster-123-statefulset | |
k8s-group-${your_k8s_cluster_id}-singleton | The machine group of a single instance, which is applicable to the Logtail configuration of the single instance. | k8s-group-my-cluster-123-singleton | |
Logstore | config-operation-log | The Logstore is used to collect and store log-controller logs. | None |
View the status, version number, and IP address of Logtail
View the status of Logtail
Run the following command to view the status of Logtail:
kubectl get po -n kube-system | grep logtail
The following output is returned:
NAME READY STATUS RESTARTS AGE
logtail-ds-gb92k 1/1 Running 0 2h
logtail-ds-wm7lw 1/1 Running 0 4d
View the version number and IP address of Logtail
Run the following command to view the version number and IP address of Logtail:
kubectl exec logtail-ds-gb92k -n kube-system cat /usr/local/ilogtail/app_info.json
The following output is returned:
{
"UUID" : "",
"hostname" : "logtail-ds-gb92k",
"instance_id" : "0EBB2B0E-0A3B-11E8-B0CE-0A58AC140402_172.20.4.2_1517810940",
"ip" : "192.0.2.0",
"logtail_version" : "0.16.2",
"os" : "Linux; 3.10.0-693.2.2.el7.x86_64; #1 SMP Tue Sep 12 22:26:13 UTC 2017; x86_64",
"update_time" : "2021-02-05 06:09:01"
}
Upgrade Logtail
Upgrade components
Upgrade components by installing the latest Logtail versions. If you upgrade only the image versions of specific components, such as logtail-ds or alibaba-log-controller, the upgrade may fail.
Back up the description files that are related to the Logtail components.
ImportantAn upgrade requires a few seconds to complete. During an upgrade, the Logtail container is restarted, which may cause a small amount of data to be collected again or lost during collection. Before you upgrade the Logtail components, we recommend that you back up the description files that are related to the Logtail components.
Sample command:
kubectl get ds -n kube-system logtail-ds -o yaml > logtail-ds.yaml kubectl get deployment -n kube-system alibaba-log-controller -o yaml > alibaba-log-controller.yaml kubectl get crd aliyunlogconfigs.log.alibabacloud.com -o yaml > aliyunlogconfigs-crd.yaml kubectl get cm -n kube-system alibaba-log-configuration -o yaml > alibaba-log-configuration.yaml kubectl get aliyunlogconfigs --all-namespaces -o yaml > aliyunlogconfigs-cr.yaml
Reinstall the Logtail components to complete the automatic upgrade. For more information, see Install Logtail.
Roll back Logtail
The following procedure shows how to roll back to a version.
The YAML files that are backed up before an upgrade contain redundant information. You must manually delete the redundant information before you can restore the configurations of Logtail. You can use the kubectl-neat tool to delete the redundant information. You must delete the following fields: metadata.creationTimestamp, metadata.generation, metadata.resourceVersion, metadata.uid, and status.
Check whether to retain the new configurations of Logtail after the upgrade based on your business requirements.
Delete redundant information from the backup files.
cat logtail-ds.yaml | kubectl-neat > neat-logtail-ds.yaml cat alibaba-log-controller.yaml | kubectl-neat > neat-alibaba-log-controller.yaml cat aliyunlogconfigs-crd.yaml | kubectl-neat > neat-aliyunlogconfigs-crd.yaml cat alibaba-log-configuration.yaml | kubectl-neat > neat-alibaba-log-configuration.yaml cat aliyunlogconfigs-cr.yaml | kubectl-neat > neat-aliyunlogconfigs-cr.yaml
Use the backup files after the deletion to restore the configurations of Logtail.
kubectl apply -f neat-logtail-ds.yaml kubectl apply -f neat-alibaba-log-controller.yaml kubectl apply -f neat-aliyunlogconfigs-crd.yaml kubectl apply -f neat-alibaba-log-configuration.yaml kubectl apply -f neat-aliyunlogconfigs-cr.yaml
What to do next
Create Logtail configurations to collect container logs.
DaemonSet mode
For information about how to collect container logs by using CRDs, see Use CRDs to collect container logs in DaemonSet mode.
For information about how to collect container standard output (stdout) and standard errors (stderr) in the Simple Log Service console, see Use the Simple Log Service console to collect container stdout and stderr in DaemonSet mode.
For information about how to collect container text logs in the Simple Log Service console, see Collect log data from containers by using Simple Log Service.
Sidecar mode
For information about how to collect container logs by using CRDs, see Use CRDs to collect container text logs in Sidecar mode.
For information about how to collect container logs in the Simple Log Service console, see Use the Simple Log Service console to collect container text logs in Sidecar mode.
FAQ
How do I view the version of a container image?
You can view the version of a container image by visiting one of the following image repositories:
How do I associate multiple self-managed Kubernetes clusters with the same Simple Log Service project?
If you want to collect container logs from multiple self-managed Kubernetes clusters to the same Simple Log Service project, you can set the SlsProjectName parameter to the same value for each self-managed Kubernetes cluster when you install Simple Log Service components.
How do I view the logs of Logtail?
The logs of Logtail are stored in the files named ilogtail.LOG
and logtail_plugin.LOG
in the /usr/local/ilogtail/
directory of a Logtail container.
The stdout of the container is irrelevant to this case. Ignore the following stdout:
start umount useless mount points, /shm$|/merged$|/mqueue$
umount: /logtail_host/var/lib/docker/overlay2/3fd0043af174cb0273c3c7869500fbe2bdb95d13b1e110172ef57fe840c82155/merged: must be superuser to unmount
umount: /logtail_host/var/lib/docker/overlay2/d5b10aa19399992755de1f85d25009528daa749c1bf8c16edff44beab6e69718/merged: must be superuser to unmount
umount: /logtail_host/var/lib/docker/overlay2/5c3125daddacedec29df72ad0c52fac800cd56c6e880dc4e8a640b1e16c22dbe/merged: must be superuser to unmount
......
xargs: umount: exited with status 255; aborting
umount done
start logtail
ilogtail is running
logtail status:
ilogtail is running
How do I view the status of Simple Log Service components in Kubernetes clusters?
Run the following commands:
kubectl get deploy alibaba-log-controller -n kube-system
kubectl get ds logtail-ds -n kube-system
What do I do if alibaba-log-controller fails to start?
Check whether alibaba-log-controller is installed by using the following method:
Run the installation command on the master node of your Kubernetes cluster.
Specify the ID of your Kubernetes cluster in the installation command.
If alibaba-log-controller is not installed by using the preceding method, run the kubectl delete -f deploy
command to delete the installation template that is generated. Then, rerun the installation command.
How do I view the status of the Logtail DaemonSet in a Kubernetes cluster?
Run the kubectl get ds -n kube-system
command to view the status of the Logtail DaemonSet.
The default namespace to which a Logtail container belongs is kube-system.
How do I view the oeprational logs of Logtail?
The operational logs of Logtail are stored in the ilogtail.LOG
file in the /usr/local/ilogtail/
directory. If the log file is rotated, the generated files are compressed and stored as ilogtail.LOG.x.gz
. Run the following command to view logs:
kubectl exec logtail-ds-gb92k -n kube-system tail /usr/local/ilogtail/ilogtail.LOG
The following output is returned:
[2018-02-05 06:09:02.168693] [INFO] [9] [build/release64/sls/ilogtail/LogtailPlugin.cpp:104] logtail plugin Resume:start
[2018-02-05 06:09:02.168807] [INFO] [9] [build/release64/sls/ilogtail/LogtailPlugin.cpp:106] logtail plugin Resume:success
[2018-02-05 06:09:02.168822] [INFO] [9] [build/release64/sls/ilogtail/EventDispatcher.cpp:369] start add existed check point events, size:0
[2018-02-05 06:09:02.168827] [INFO] [9] [build/release64/sls/ilogtail/EventDispatcher.cpp:511] add existed check point events, size:0 cache size:0 event size:0 success count:0
How do I restart Logtail for a pod?
Stop Logtail.
In the following command,
logtail-ds-gb92k
specifies the container, andkube-system
specifies the namespace. Configure the parameters based on your business scenario.kubectl exec logtail-ds-gb92k -n kube-system /etc/init.d/ilogtaild stop
If the following output is returned, Logtail is stopped.
kill process Name: ilogtail pid: 7 kill process Name: ilogtail pid: 9 stop success
Start Logtail.
In the following command,
logtail-ds-gb92k
specifies the container, andkube-system
specifies the namespace. Configure the parameters based on your business scenario.kubectl exec logtail-ds-gb92k -n kube-system /etc/init.d/ilogtaild start
If the following output is returned, Logtail is started.
ilogtail is running
How do I collect the logs of control plane components?
For information about how to collect the logs of control plane components from a self-managed Kubernetes cluster or a Container Service for Kubernetes (ACK) dedicated cluster, see Use the Simple Log Service console to collect container stdout and stderr in DaemonSet mode.
How do I uninstall the components such as logtail-ds and alibaba-log-controller?
Run the kubectl delete -R -f result
command to uninstall these components.