×
Community Blog Secure Docker Compose Stacks with CrowdSec

Secure Docker Compose Stacks with CrowdSec

This article explains how to make CrowdSec and Docker Compose work together to protect applications exposed in containers.

Introduction

CrowdSec is a massively multiplayer firewall designed to protect Linux servers, services, containers, or virtual machines exposed on the internet with a server-side agent. It was inspired by Fail2Ban. It aims to be a modernized, collaborative version of that intrusion-prevention tool.

CrowdSec is free and open-source (under an MIT License), and the source code is available on GitHub. It uses a behavior analysis system to assess whether someone is trying to hack you based on your logs. If your agent detects such aggression, the offending IP is dealt with and sent for curation. If the signal passes the curation process, the IP is redistributed to all users sharing a similar technological profile to "immunize" them against this IP.

The goal is to leverage the crowd power to create a real-time IP reputation database. As for the IP that aggressed your machine, you can choose to remedy the threat in any manner you feel appropriate. Ultimately, CrowdSec leverages the power of the crowd to create an extremely accurate IP reputation system that benefits all its users.

This article explains how to make CrowdSec and Docker Compose work together to protect applications exposed in containers. It should allow us to:
-- Automatically ban malevolent IPs from accessing our container services
-- Manually add/remove and inspect ban decisions
-- Monitor CrowdSec's behavior (via CLI and dashboards)

Target Architecture

The chart below shows a glimpse of how our target architecture should look:

1

Let's create a Docker Compose file that will set up the following:
-- A reverse-proxy that uses Nginx
-- A sample application that exposes an Apache2 "hello world"
-- A CrowdSec container that reads the reverse-proxy's logs to detect attacks on the HTTP service
-- A Metabase container that will generate fancy dashboards showing what has been happening

We have the simplest way to collect logs: by sharing volumes between containers. If you are in production, you are probably using a logging-driver to centralize logs with rsyslog or another driver. Don't forget to adapt the CrowdSec Docker Compose configuration to read your logs properly.

The docker-compose.yml file is shown below:

version: '3'

services:
 #the application itself : static html served by apache2.
 #the html can be found in ./app/
 app:
   image: httpd:alpine
   restart: always
   volumes:
     - ./app/:/usr/local/apache2/htdocs/
   networks:
     crowdsec_test:
       ipv4_address: 172.20.0.2

 #the reverse proxy that will serve the application
 #you can see nginx's config in ./reverse-proxy/nginx.conf
 reverse-proxy:
   image: nginx:alpine
   restart: always
   ports:
     - 8000:80
   depends_on:
     - 'app'
   volumes:
     - ./reverse-proxy/nginx.conf:/etc/nginx/nginx.conf
     - logs:/var/log/nginx
   networks:
     crowdsec_test:
       ipv4_address: 172.20.0.3
  #crowdsec : it will be fed nginx's logs
 #and later we're going to plug a firewall bouncer to it
 crowdsec:
   image: crowdsecurity/crowdsec:v1.0.8
   restart: always
   environment:
     #this is the list of collections we want to install
     #https://hub.crowdsec.net/author/crowdsecurity/collections/nginx
     COLLECTIONS: "crowdsecurity/nginx"
     GID: "${GID-1000}"
   depends_on:
     - 'reverse-proxy'
   volumes:
     - ./crowdsec/acquis.yaml:/etc/crowdsec/acquis.yaml
     - logs:/var/log/nginx
     - crowdsec-db:/var/lib/crowdsec/data/
     - crowdsec-config:/etc/crowdsec/
   networks:
     crowdsec_test:
       ipv4_address: 172.20.0.4
  #metabase, because security is cool, but dashboards are cooler
 dashboard:
   #we're using a custom Dockerfile so that metabase pops with pre-configured dashboards
   build: ./crowdsec/dashboard
   restart: always
   ports:
     - 3000:3000
   environment:
     MB_DB_FILE: /data/metabase.db
     MGID: "${GID-1000}"
   depends_on:
     - 'crowdsec'
   volumes:
     - crowdsec-db:/metabase-data/
   networks:
     crowdsec_test:
       ipv4_address: 172.20.0.5

volumes:
 logs:
 crowdsec-db:
 crowdsec-config:

networks:
 crowdsec_test:
   ipam:
     driver: default
     config:
       - subnet: 172.20.0.0/24

The reverse-proxy (nginx) container writes its logs to a logs volume mounted by the crowdsec container.

CrowdSec's SQLite database is in a crowdsec-db volume mounted by the dashboard (metabase) container

Initial Deployment

Prerequisites: Docker / Docker Compose

We put all the configuration files together on this repository, so you can clone it to deploy.

From the Docker Compose directory, you can deploy with docker-compose up –d and then check that everything is running with docker-compose ps.

▶ git clone https://github.com/crowdsecurity/example-docker-compose
▶ cd example-docker-compose
▶ sudo docker-compose up
▶ sudo docker-compose ps

2

Let's check and make sure everything is working!

Demo App Check

With the following command, we can check whether access to our demo app is working correctly.

▶ curl http://localhost:8000/
Hello world !%                                                 

CrowdSec Metrics Check

We need to validate whether our CrowdSec setup reads logs as expected.

docker-compose exec crowdsec cscli metrics

3

What happened? What is relevant here?

The command cscli metrics queries the Prometheus metrics exposed locally by CrowdSec and presents them in a user-friendly terminal output:
-- The "acquisition metrics" show us that our requests are generating logs that are being read ("LINES READ"), parsed ("LINES PARSED"), and matched with installed scenarios ("LINES POURED TO BUCKET")
-- The "buckets metrics" and "parser metrics" allow us to see which parsers and scenarios are being triggered.

CrowdSec Configuration Check

The command cscli hub list allows us to see which parsers and scenarios are deployed.

4

Metabase Check

Metabase is one of the components that has been deployed, which helps us generate dashboards for better observability. You can hop onto http://127.0.0.1:3000/ and log in with crowdsec@crowdsec.net and password !!Cr0wdS3c_M3t4b4s3??

Metabase comes with a default password because of how it is deployed. Remember to change the default password and restrict the metabase's access to relevant IP addresses or network ranges.

At first, dashboards will be empty since no attacks were detected yet. The main one should look like this:

5

If any of those checks have failed for you, take a look at container logs with docker-compose logs crowdsec (for example).

Detection Features

Note: In real-world setups, whitelists are deployed to prevent banning private IPs.

After checking to make sure everything is ready to go, let's try some detection features. As we work with an exposed HTTP service, let's fire a Nikto from another machine in the LAN nikto -host http://192.168.2.227:8000

6

Note: The IP is dependent on your LAN setup and addressing plan.

On the other hand, we kept an eye on CrowdSec's logs through the following command. docker-compose logs -f crowdsec:

7

Here, we can spot our client's IP (192.168.2.211) being flagged for triggering various scenarios:

-- crowdsecurity/http-bad-user-agent: The IP comes from a known bad user agent
-- crowdsecurity/http-probing: It tried to access a lot of distinct non-existent files
-- crowdsecurity/http-crawl-non_statics: It attempted to access a lot of distinct non-static resources
-- crowdsecurity/http-sensitive-files: The IP tried to access a lot of sensitive files
-- crowdsecurity/http-path-traversal-probing: The IP tried to perform a path traversal attack

We can see my IP is banned with docker-compose exec crowdsec cscli decisions list.

8

We can review and inspect alerts with docker-compose exec crowdsec cscli alerts list and docker-compose exec crowdsec cscli alerts inspect -d XX:

9

Note: cscli alerts list returns the list of all alerts triggered.

10

Note: cscli alerts inspect -d <ID> allows you to generate more details about a given alert.

Monitoring Activity with Dashboards

Now that we have triggered several scenarios, we can go back to our Metabase dashboards (http://127.0.0.1:3000 with the default setup) and check the activity.

11

If the traffic came from a public IP (rather than a private one, as in this example), crowdsecurity/geoip-enrich would have enriched events with geo-localization data and AS/range information.

Block Attacks with Bouncers

Now that we have a fully functional CrowdSec service, we can detect incoming attacks on our service based on installed collection scenarios.

After detecting these attacks, our objective is to block them. You will use the cs-firewall-bouncer to achieve this. Start by installing it on the host, and block malevolent traffic directly in the DOCKER-USER chain, the default chain created by Docker to filter traffic targeting the containers.

You can find the firewall bouncer on the CrowdSec Hub. As of today, the most up-to-date version is v0.10.

wget https://github.com/crowdsecurity/cs-firewall-bouncer/releases/download/v0.0.10/cs-firewall-bouncer.tgz
tar xvzf cs-firewall-bouncer.tgz
cd cs-firewall-bouncer-v0.0.10
sudo ./install.sh

The install is straightforward. Bouncers are not packaged yet, but the team is currently working on it. It might be achieved by the time you read this article. Please check the GitHub page for more information. It will deploy a systemd unit for the service and make sure you meet the requirements. Here, I didn't have ipset, and it installed it for me.

12

Here, we installed the bouncer on a host where CrowdSec isn't running. As a result, the service isn't happy.

Now, let's configure the bouncer to speak with the Local API running on our CrowdSec container. We start by creating an API token for our bouncer with cscli.

docker-compose exec crowdsec cscli bouncers add HostFirewallBouncer

13

Then, you need to configure the bouncer to use this token to authenticate with CrowdSec's Local API. In the /etc/crowdsec/cs-firewall-bouncer/cs-firewall-bouncer.yaml, edit api_url, api_key, and iptables_chains. In this case, IPv6 was also disabled with disable_ipv6:

mode: iptables
piddir: /var/run/
update_frequency: 10s
daemonize: true
log_mode: file
log_dir: /var/log/
log_level: info
api_url: http://172.20.0.4:8080/
api_key: aaebb3708fe67eeeccbb52a21e5e7862
disable_ipv6: true
#if present, insert rule in those chains
iptables_chains:
 - DOCKER-USER

Note: We edited the chains to only have DOCKER-USER, and we set api_url accordingly to our docker-compose.yml file, along with the newly generated api token.

You Shall Not Pass! CrowdSec in Action

Now, we can get our freshly configured bouncer started with sudo systemctl start cs-firewall-bouncer.service and take a look at our new firewall configuration:

▶ sudo iptables -L -n                                                                  
...
Chain DOCKER-USER (1 references)
target     prot opt source               destination         
DROP       all  --  0.0.0.0/0            0.0.0.0/0            match-set crowdsec-blacklists src
...

We can see that our DOCKER-USER chain has been populated with a rule to match incoming traffic against our ipset, and our ipset is filled with relevant information.

▶ sudo ipset -L crowdsec-blacklists            
Name: crowdsec-blacklists
Type: hash:net
Revision: 6
Header: family inet hashsize 1024 maxelem 65536 timeout 300
Size in memory: 6016
References: 1
Number of entries: 61
Members:
178.20.157.98 timeout 80103
52.184.35.59 timeout 80103
...

As you might have noticed by now, our ipset isn't only filled with our local decisions but also with ones from the community. However, we can see that our "local" decisions made their way to the ipset list.

▶ sudo ipset -L crowdsec-blacklists  | grep 192
192.168.2.211 timeout 12859

Now, we can check the attacker machine and see we are blocked and cannot access the application.

$ curl -vv 192.168.2.227:8000
* Rebuilt URL to: 192.168.2.227:8000/
*   Trying 192.168.2.227...
* TCP_NODELAY set
^C

The attacker is prevented from accessing all Docker Compose applications. We limited the decision to the DOCKER-USER chain on purpose, which means local applications exposed by the host would still be accessible. If we wish to extend the ban to all incoming traffic, we could have added the INPUT chain to the list, as is the case with the default setup.

This minimal but complete applicative stack that we just deployed using Docker Compose is now secured by adding CrowdSec.

Get Involved

Throughout this tutorial, we deployed a minimal yet complete applicative stack using Docker Compose. Then, we covered how to secure it with CrowdSec. While most people will use CrowdSec as a host-based defense mechanism, we can see that it's also suitable for Docker environments.

The CrowdSec team would love to hear your feedback about their tool. If you are interested in testing the software or would like to get in touch with the team, please check the following links:

-- Download CrowdSec v.1.0.x
-- CrowdSec website
-- GitHub Repository
-- Gitter Channel

0 0 0
Share on

Philippe-CrowdSec

1 posts | 0 followers

You may also like

Comments

Philippe-CrowdSec

1 posts | 0 followers

Related Products