How to Install Concourse CI on an ECS Instance and Encrypt All Traffic

By Hitesh Jethva, Alibaba Cloud Community Blog author. The Blog is a community-driven platform whose main aim is to demonstrate Alibaba Cloud's technical capabilities, brand message, and thought leadership through relevant and compelling content.

Concourse CI is a modern, flexible continuous integration platform that allows developers to merge modified code into a shared repository multiple times. After each merge, automatic builds and tests are performed to detect problems in the code that helps the developers to find and resolve the errors quickly.

In this tutorial, we will learn how to install and encrypt Concourse CI on an Alibaba Cloud Elastic Compute Service (ECS) instance installed with Ubuntu 16.04.

Requirements

A newly created ECS instance installed with Ubuntu 16.04.
The static IP address 192.168.43.193 is set up for your instance.
A root password is set up for your instance.

Procedure

To install and secure Concourse CI on an ECS instance, complete all of the following steps:

Launch Alibaba Cloud ECS Instance

First, log on to the Alibaba Cloud ECS Console. Then, create a new ECS instance, choose Ubuntu 16.04 as the operating system and make sure it is with at least 2GB RAM. Next, connect to your ECS instance and log on as the root user.

After you log on to your Ubuntu 16.04 instance, run the following command to update your base system with the latest available packages.

apt-get update -y

Install and Configure PostgreSQL

Concourse uses PostgreSQL to store its pipeline data. So you will need to install PostgreSQL server to your system. You can install it by using the following command:

apt-get install postgresql postgresql-contrib -y

When the installation is complete, log on to PostgreSQL user by running the following command:

su - postgres

Next, switch to the PostgreSQL shell and create a user and database for Concourse with the following command:

psql

postgres=# create user concourse;
postgres=# ALTER USER concourse WITH ENCRYPTED password 'password';
postgres=# CREATE DATABASE concourse OWNER concourse;

Exit from the PostgreSQL shell by running the following command:

postgres-# \q

Install Concourse CI

Download the latest version of the Concourse binary to the /usr/bin directory by using the following command:

wget https://github.com/concourse/concourse/releases/download/v3.10.0/concourse_linux_amd64 -O /usr/bin/concourse

Then, download the latest version of the fly binary to the /usr/bin directory with the following command:

wget https://github.com/concourse/concourse/releases/download/v3.10.0/fly_linux_amd64 -O /usr/bin/fly

Give execute permission to the downloaded binary:

chmod 755 /usr/bin/concourse 
chmod 755 /usr/bin/fly

Last, check the version of the Concourse and fly:

concourse -version

The following is output:

3.10.0

fly -version

The output of this last command is as follows:

3.10.0

Create a Concourse Configuration

Create a Concourse configuration directory to store all of the relevant files:

mkdir /opt/concourse

Next, you will need to create three separate keys: 1) Keys for the worker, 2) keys for the TSA, and 3) session signing keys to sign tokens so that each can communicate securely with one another.

You can generate the required keys with the following command:

ssh-keygen -t rsa -q -N '' -f /opt/concourse/session_signing_key
ssh-keygen -t rsa -q -N '' -f /opt/concourse/tsa_host_key
ssh-keygen -t rsa -q -N '' -f /opt/concourse/worker_key

Authorize the workers' public key by copying its contents to the authorized_worker_keys file.

cp /opt/concourse/worker_key.pub /opt/concourse/authorized_worker_keys

Configure the Concourse Environment

You will need to create an environment variable file for Concourse to read a value from configuration file natively. You can create a new environment file for Concourse web with the folloiwing command:

nano /opt/concourse/web.env

Add the following lines:

CONCOURSE_SESSION_SIGNING_KEY=/opt/concourse/session_signing_key
CONCOURSE_TSA_HOST_KEY=/opt/concourse/tsa_host_key
CONCOURSE_TSA_AUTHORIZED_KEYS=/opt/concourse/authorized_worker_keys

CONCOURSE_POSTGRES_USER=concourse
CONCOURSE_POSTGRES_PASSWORD=password
CONCOURSE_POSTGRES_DATABASE=concourse

CONCOURSE_BASIC_AUTH_USERNAME=admin
CONCOURSE_BASIC_AUTH_PASSWORD=password
CONCOURSE_EXTERNAL_URL=http://172.20.10.6:8080

Save and close the file. Then, create an environment file for the worker with the following command:

nano /opt/concourse/worker.env

Add the following lines:

CONCOURSE_WORK_DIR=/opt/concourse/worker
CONCOURSE_TSA_WORKER_PRIVATE_KEY=/opt/concourse/worker_key
CONCOURSE_TSA_PUBLIC_KEY=/opt/concourse/tsa_host_key.pub
CONCOURSE_TSA_HOST=127.0.0.1

Save and close the file. Then, give permissions to the environment file with the following command:

chmod 600 /opt/concourse/worker.env
chmod 600 /opt/concourse/web.env

Create a Systemd Unit File for Concourse

You will need to create a systemd file for concourse to manage concourse web and concourse worker service. First, create a new user to run web process. This user should match the PostgreSQL user which you have created earlier:

adduser --system --group concourse

Give this user ownership over /opt/concourse directory:

chown -R concourse:concourse /opt/concourse

Next, create a new systemd service file for the Concourse web service:

nano /etc/systemd/system/concourse-web.service

Add the following lines:

[Unit]
Description=Concourse CI web server

[Service]
Type=simple
User=concourse
Group=concourse
Restart=on-failure
EnvironmentFile=/opt/concourse/web.env
ExecStart=/usr/bin/concourse web
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=concourse_web

[Install]
WantedBy=multi-user.target

Save and close the file. Then, create a new service file for the Concourse worker service:

nano /etc/systemd/system/concourse-worker.service

Add the following lines:

[Unit]
Description=Concourse CI worker process

[Service]
Type=simple
Restart=on-failure
EnvironmentFile=/opt/concourse/worker.env
ExecStart=/usr/bin/concourse worker
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=concourse_worker

[Install]
WantedBy=multi-user.target

Save and close the file. Then, start the concourse web and worker service and enable them to start on boot time with the following command:

systemctl start concourse-web
systemctl enable concourse-web
systemctl start concourse-worker
systemctl enable concourse-worker

You can check the status of both service with the following command:

systemctl status concourse-web

The output is as follows:

● concourse-web.service - Concourse CI web server
   Loaded: loaded (/etc/systemd/system/concourse-web.service; disabled; vendor preset: enabled)
   Active: active (running) since Sun 2018-11-11 20:41:54 IST; 7s ago
 Main PID: 2527 (concourse)
   CGroup: /system.slice/concourse-web.service
           └─2527 /usr/bin/concourse web

Nov 11 20:41:54 Node2 systemd[1]: Started Concourse CI web server.

Check the status of Concourse Worker by running the following command:

systemctl status concourse-worker

The output is as follows:

● concourse-worker.service - Concourse CI worker process
   Loaded: loaded (/etc/systemd/system/concourse-worker.service; disabled; vendor preset: enabled)
   Active: active (running) since Sun 2018-11-11 20:41:30 IST; 56s ago
 Main PID: 2467 (concourse)
   CGroup: /system.slice/concourse-worker.service
           └─2467 /usr/bin/concourse worker

Nov 11 20:41:30 Node2 systemd[1]: Started Concourse CI worker process.
Nov 11 20:41:32 Node2 concourse_worker[2467]: {"timestamp":"1541949092.064973593","source":"worker","message":"worker.setup.unpacking","log_lev

Access Concourse Web UI

Concourse is now up and listening on port 8080. It's time to access Concourse through web browser and command line.

Open your web browser and type the URL http://172.20.10.6:8080. You will be redirected to the Concourse CI page:

Next, click on the login button. You will be asked to select your team in the following page:

Then, click on the main button. You will be redirected to the Concourse CI login page:

Provide your username and password which you have provided in the environment file. Then, click on the login button. You will be redirected to the Concourse CI dashboard:

You can also connect the Concourse CI using Fly with the following command:

fly -t test-ci login -c http://172.20.10.6:8080

You will be asked to provide username and password as shown below:

logging in to team 'main'

username: admin
password: 
target saved

You can also log out from the server with the following command:

fly -t test-ci logout

Secure Concourse CI

Concourse CI is now installed and configured. But, all the information sent through the web UI to the Concourse server are not secured and connection is not encrypted. We recommend to setup Nginx as a reverse proxy with a Let's Encrypt free SSL.

To do so, you will need to install Certbot to obtain an SSL certificate and Nginx to your system.

First, add the Certbot repository by running the following command:

add-apt-repository ppa:certbot/certbot

Next, update the repository and install Certbot, Certbot's Nginx package and Nginx with the following command:

apt-get update -y
apt-get install nginx certbot python-certbot-nginx -y

Next, you will need to obtain certificates from Let's Encrypt CA. You can generate the SSL certificates with the following command:

certbot --nginx -d alibabatest.com -d www.alibabatest.com

You will be asked to enter an email address and agree to the terms of service. After the process is complete, your certificates are downloaded and stored in the /etc/letsencrypt/live/alibabatest.com/ directory.

By default, Let's Encrypt certificates expire in 90 days, so you will need to set a cron job to renew all the certificates automatically.

You can do this by running the following command:

crontab -e

Add the following lines:

00 10 * * * /usr/bin/certbot renew --quiet

Save and close the file when you are finished.

Next, you will need to create an Nginx virtual host file for Concourse to redirect all the request coming on port 8080 to port 80. You can do this by running the following command:

nano /etc/nginx/sites-available/concourse

Add the following lines:

server {
    listen 80;
    server_name alibabatest.com;
    return 301 https://$host$request_uri;
}
server {

    listen 443;
    server_name alibabatest.com;

    ssl_certificate           /etc/letsencrypt/live/alibabatest.com/fullchain.pem;
    ssl_certificate_key       /etc/letsencrypt/live/alibabatest.com/privkey.pem;

    ssl on;
    ssl_session_cache  builtin:1000  shared:SSL:10m;
    ssl_protocols  TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers HIGH:!aNULL:!eNULL:!EXPORT:!CAMELLIA:!DES:!MD5:!PSK:!RC4;
    ssl_prefer_server_ciphers on;

    access_log    /var/log/nginx/concourse.access.log;

    location / {

      proxy_set_header        Host $host;
      proxy_set_header        X-Real-IP $remote_addr;
      proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header        X-Forwarded-Proto $scheme;
      proxy_pass          http://localhost:8080;
      proxy_read_timeout  90;

      proxy_redirect      http://localhost:8080 https://alibabatest.com;
    }
  }

Save and close the file. Then, enable Nginx virtual host by running the following command:

ln -s /etc/nginx/sites-available/concourse /etc/nginx/sites-enabled/concourse

Next, you will also need to modify Concourse web environment file:

nano /opt/concourse/web.env

Make the following changes:

CONCOURSE_EXTERNAL_URL=https://alibabatest.com
CONCOURSE_BIND_IP=127.0.0.1
CONCOURSE_BIND_PORT=8080

When finished, save and close the file.

Then, restart Nginx, Concourse web and Concourse worker service with the following command:

systemctl restart nginx
systemctl restart concourse-web
systemctl restart concourse-worker

All the traffic between Concourse CI and the browser are now secured with SSL encryption.

You can now access Concourse CI by visiting the address https://alibabatest.com

Community

How to Install Concourse CI on an ECS Instance and Encrypt All Traffic

Requirements

Procedure

Launch Alibaba Cloud ECS Instance

Install and Configure PostgreSQL

Install Concourse CI

Create a Concourse Configuration

Configure the Concourse Environment

Create a Systemd Unit File for Concourse

Access Concourse Web UI

Secure Concourse CI

Read previous post:

Read next post:

Alibaba Clouder

You may also like

Comments

Alibaba Clouder

Related Products

Certificate Management Service

Security Solution

Security Overview

ECS(Elastic Compute Service)