By Hitesh Jethva, Alibaba Cloud Community Blog author. The Blog is a community-driven platform whose main aim is to demonstrate Alibaba Cloud's technical capabilities, brand message, and thought leadership through relevant and compelling content.
Concourse CI is a modern, flexible continuous integration platform that allows developers to merge modified code into a shared repository multiple times. After each merge, automatic builds and tests are performed to detect problems in the code that helps the developers to find and resolve the errors quickly.
In this tutorial, we will learn how to install and encrypt Concourse CI on an Alibaba Cloud Elastic Compute Service (ECS) instance installed with Ubuntu 16.04.
To install and secure Concourse CI on an ECS instance, complete all of the following steps:
First, log on to the Alibaba Cloud ECS Console. Then, create a new ECS instance, choose Ubuntu 16.04 as the operating system and make sure it is with at least 2GB RAM. Next, connect to your ECS instance and log on as the root user.
After you log on to your Ubuntu 16.04 instance, run the following command to update your base system with the latest available packages.
apt-get update -y
Concourse uses PostgreSQL to store its pipeline data. So you will need to install PostgreSQL server to your system. You can install it by using the following command:
apt-get install postgresql postgresql-contrib -y
When the installation is complete, log on to PostgreSQL user by running the following command:
su - postgres
Next, switch to the PostgreSQL shell and create a user and database for Concourse with the following command:
psql
postgres=# create user concourse;
postgres=# ALTER USER concourse WITH ENCRYPTED password 'password';
postgres=# CREATE DATABASE concourse OWNER concourse;
Exit from the PostgreSQL shell by running the following command:
postgres-# \q
Download the latest version of the Concourse binary to the /usr/bin directory by using the following command:
wget https://github.com/concourse/concourse/releases/download/v3.10.0/concourse_linux_amd64 -O /usr/bin/concourse
Then, download the latest version of the fly binary to the /usr/bin directory with the following command:
wget https://github.com/concourse/concourse/releases/download/v3.10.0/fly_linux_amd64 -O /usr/bin/fly
Give execute permission to the downloaded binary:
chmod 755 /usr/bin/concourse
chmod 755 /usr/bin/fly
Last, check the version of the Concourse and fly:
concourse -version
The following is output:
3.10.0
fly -version
The output of this last command is as follows:
3.10.0
Create a Concourse configuration directory to store all of the relevant files:
mkdir /opt/concourse
Next, you will need to create three separate keys: 1) Keys for the worker, 2) keys for the TSA, and 3) session signing keys to sign tokens so that each can communicate securely with one another.
You can generate the required keys with the following command:
ssh-keygen -t rsa -q -N '' -f /opt/concourse/session_signing_key
ssh-keygen -t rsa -q -N '' -f /opt/concourse/tsa_host_key
ssh-keygen -t rsa -q -N '' -f /opt/concourse/worker_key
Authorize the workers' public key by copying its contents to the authorized_worker_keys file.
cp /opt/concourse/worker_key.pub /opt/concourse/authorized_worker_keys
You will need to create an environment variable file for Concourse to read a value from configuration file natively. You can create a new environment file for Concourse web with the folloiwing command:
nano /opt/concourse/web.env
Add the following lines:
CONCOURSE_SESSION_SIGNING_KEY=/opt/concourse/session_signing_key
CONCOURSE_TSA_HOST_KEY=/opt/concourse/tsa_host_key
CONCOURSE_TSA_AUTHORIZED_KEYS=/opt/concourse/authorized_worker_keys
CONCOURSE_POSTGRES_USER=concourse
CONCOURSE_POSTGRES_PASSWORD=password
CONCOURSE_POSTGRES_DATABASE=concourse
CONCOURSE_BASIC_AUTH_USERNAME=admin
CONCOURSE_BASIC_AUTH_PASSWORD=password
CONCOURSE_EXTERNAL_URL=http://172.20.10.6:8080
Save and close the file. Then, create an environment file for the worker with the following command:
nano /opt/concourse/worker.env
Add the following lines:
CONCOURSE_WORK_DIR=/opt/concourse/worker
CONCOURSE_TSA_WORKER_PRIVATE_KEY=/opt/concourse/worker_key
CONCOURSE_TSA_PUBLIC_KEY=/opt/concourse/tsa_host_key.pub
CONCOURSE_TSA_HOST=127.0.0.1
Save and close the file. Then, give permissions to the environment file with the following command:
chmod 600 /opt/concourse/worker.env
chmod 600 /opt/concourse/web.env
You will need to create a systemd file for concourse to manage concourse web and concourse worker service. First, create a new user to run web process. This user should match the PostgreSQL user which you have created earlier:
adduser --system --group concourse
Give this user ownership over /opt/concourse directory:
chown -R concourse:concourse /opt/concourse
Next, create a new systemd service file for the Concourse web service:
nano /etc/systemd/system/concourse-web.service
Add the following lines:
[Unit]
Description=Concourse CI web server
[Service]
Type=simple
User=concourse
Group=concourse
Restart=on-failure
EnvironmentFile=/opt/concourse/web.env
ExecStart=/usr/bin/concourse web
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=concourse_web
[Install]
WantedBy=multi-user.target
Save and close the file. Then, create a new service file for the Concourse worker service:
nano /etc/systemd/system/concourse-worker.service
Add the following lines:
[Unit]
Description=Concourse CI worker process
[Service]
Type=simple
Restart=on-failure
EnvironmentFile=/opt/concourse/worker.env
ExecStart=/usr/bin/concourse worker
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=concourse_worker
[Install]
WantedBy=multi-user.target
Save and close the file. Then, start the concourse web and worker service and enable them to start on boot time with the following command:
systemctl start concourse-web
systemctl enable concourse-web
systemctl start concourse-worker
systemctl enable concourse-worker
You can check the status of both service with the following command:
systemctl status concourse-web
The output is as follows:
● concourse-web.service - Concourse CI web server
Loaded: loaded (/etc/systemd/system/concourse-web.service; disabled; vendor preset: enabled)
Active: active (running) since Sun 2018-11-11 20:41:54 IST; 7s ago
Main PID: 2527 (concourse)
CGroup: /system.slice/concourse-web.service
└─2527 /usr/bin/concourse web
Nov 11 20:41:54 Node2 systemd[1]: Started Concourse CI web server.
Check the status of Concourse Worker by running the following command:
systemctl status concourse-worker
The output is as follows:
● concourse-worker.service - Concourse CI worker process
Loaded: loaded (/etc/systemd/system/concourse-worker.service; disabled; vendor preset: enabled)
Active: active (running) since Sun 2018-11-11 20:41:30 IST; 56s ago
Main PID: 2467 (concourse)
CGroup: /system.slice/concourse-worker.service
└─2467 /usr/bin/concourse worker
Nov 11 20:41:30 Node2 systemd[1]: Started Concourse CI worker process.
Nov 11 20:41:32 Node2 concourse_worker[2467]: {"timestamp":"1541949092.064973593","source":"worker","message":"worker.setup.unpacking","log_lev
Concourse is now up and listening on port 8080. It's time to access Concourse through web browser and command line.
Open your web browser and type the URL http://172.20.10.6:8080. You will be redirected to the Concourse CI page:
Next, click on the login button. You will be asked to select your team in the following page:
Then, click on the main button. You will be redirected to the Concourse CI login page:
Provide your username and password which you have provided in the environment file. Then, click on the login button. You will be redirected to the Concourse CI dashboard:
You can also connect the Concourse CI using Fly with the following command:
fly -t test-ci login -c http://172.20.10.6:8080
You will be asked to provide username and password as shown below:
logging in to team 'main'
username: admin
password:
target saved
You can also log out from the server with the following command:
fly -t test-ci logout
Concourse CI is now installed and configured. But, all the information sent through the web UI to the Concourse server are not secured and connection is not encrypted. We recommend to setup Nginx as a reverse proxy with a Let's Encrypt free SSL.
To do so, you will need to install Certbot to obtain an SSL certificate and Nginx to your system.
First, add the Certbot repository by running the following command:
add-apt-repository ppa:certbot/certbot
Next, update the repository and install Certbot, Certbot's Nginx package and Nginx with the following command:
apt-get update -y
apt-get install nginx certbot python-certbot-nginx -y
Next, you will need to obtain certificates from Let's Encrypt CA. You can generate the SSL certificates with the following command:
certbot --nginx -d alibabatest.com -d www.alibabatest.com
You will be asked to enter an email address and agree to the terms of service. After the process is complete, your certificates are downloaded and stored in the /etc/letsencrypt/live/alibabatest.com/ directory.
By default, Let's Encrypt certificates expire in 90 days, so you will need to set a cron job to renew all the certificates automatically.
You can do this by running the following command:
crontab -e
Add the following lines:
00 10 * * * /usr/bin/certbot renew --quiet
Save and close the file when you are finished.
Next, you will need to create an Nginx virtual host file for Concourse to redirect all the request coming on port 8080 to port 80. You can do this by running the following command:
nano /etc/nginx/sites-available/concourse
Add the following lines:
server {
listen 80;
server_name alibabatest.com;
return 301 https://$host$request_uri;
}
server {
listen 443;
server_name alibabatest.com;
ssl_certificate /etc/letsencrypt/live/alibabatest.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/alibabatest.com/privkey.pem;
ssl on;
ssl_session_cache builtin:1000 shared:SSL:10m;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers HIGH:!aNULL:!eNULL:!EXPORT:!CAMELLIA:!DES:!MD5:!PSK:!RC4;
ssl_prefer_server_ciphers on;
access_log /var/log/nginx/concourse.access.log;
location / {
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_pass http://localhost:8080;
proxy_read_timeout 90;
proxy_redirect http://localhost:8080 https://alibabatest.com;
}
}
Save and close the file. Then, enable Nginx virtual host by running the following command:
ln -s /etc/nginx/sites-available/concourse /etc/nginx/sites-enabled/concourse
Next, you will also need to modify Concourse web environment file:
nano /opt/concourse/web.env
Make the following changes:
CONCOURSE_EXTERNAL_URL=https://alibabatest.com
CONCOURSE_BIND_IP=127.0.0.1
CONCOURSE_BIND_PORT=8080
When finished, save and close the file.
Then, restart Nginx, Concourse web and Concourse worker service with the following command:
systemctl restart nginx
systemctl restart concourse-web
systemctl restart concourse-worker
All the traffic between Concourse CI and the browser are now secured with SSL encryption.
You can now access Concourse CI by visiting the address https://alibabatest.com
2,599 posts | 763 followers
FollowAlibaba Clouder - July 30, 2020
Alibaba Clouder - July 27, 2020
Alibaba Clouder - May 22, 2018
Alibaba Clouder - May 21, 2018
Alibaba Clouder - May 21, 2018
Alibaba Clouder - May 29, 2018
2,599 posts | 763 followers
FollowYou can use Certificate Management Service to issue, deploy, and manage public and private SSL/TLS certificates.
Learn MoreAlibaba Cloud is committed to safeguarding the cloud security for every business.
Learn MoreSimple, secure, and intelligent services.
Learn MoreProtect, backup, and restore your data assets on the cloud with Alibaba Cloud database services.
Learn MoreMore Posts by Alibaba Clouder