This topic introduces a simple Continuous Integration pipeline based on GitLab CI/CD. Although we keep it simple now, this pipeline will be extended in the next topic.
The application consists in a todo list where a user can add or remove items. The goal is to have a simple 3-tier architecture with enough features that allow us to explore important concepts:
- The file organization shows a way to combine backend and frontend code into a single module (to keep it simple).
- The backend is stateless, which means that it does not store any data (for example, no shared variable in the code). Instead, the data is saved in a database. This architecture is particularly useful for horizontal scaling.
- Because a relational database is involved, this project demonstrates how to use Flyway to help to upgrade the schema when the application evolves.
- The build process involves Npm, Babel, Webpack and Maven to compile and package the application for production.
- Code quality is achieved thanks to SonarQube, a tool that can detect bugs in the code and help us to maintain the project over time.
GitLab project creation
Let’s start by creating a project on GitLab:
- Open GitLab in your web browser (the URL must be like https://gitlab.my-sample-domain.xyz/);
- Click New… from the top (with a + icon) and select New project.
- Fill the new form with the following information:
- Project name = todolist
- Project slug = todolist
- Visibility Level = Private
- Click Create project.
We now have a project but we cannot download it on our computer yet, for that we need to generate and register a SSH key:
- In your GitLab web browser tab, click your avatar (top-right of the page) and select Settings.
- Click SSH Keys from the left.
- Open a terminal and type the following commands:
# Generate a SSH certificate (set the email address you set in your GitLab profile) ssh-keygen -o -t rsa -C "email@example.com" -b 4096 # Display the public key cat ~/.ssh/id_rsa.pub
- Copy the result of the
catcommand and paste in the Key field (in the GitLab web browser tab).
- The Title field should be automatically filled with your email address. The page looks like this:
- Click Add key to register your SSH key.
You can now configure git and clone the project on your computer. Enter the following commands in your terminal:
# Set your real name git config --global user.name "John Doe" # Set the same email address as the one you set in your GitLab profile git config --global user.email "firstname.lastname@example.org" # Create a directory for your projects mkdir ~/projects cd ~/projects # Clone the empty project on your computer (set your GitLab domain name and username) git clone email@example.com:johndoe/todolist.git # Change directory and check the ".git" folder is present cd todolist ls -la
Copy all the files from the folder sample-app/version1/* of this tutorial into ~/projects/todolist. You should have a directory with the following top files:
- .git: Folder containing information for git.
- .gitignore: List of files to ignore for Git.
- .gitlab-ci.yml: GitLab CI pipeline configuration (more information about this file later).
- package.json: Npm configuration for the frontend: it declares dependencies such as React, Babel and Webpack.
- pom.xml: Maven configuration for the backend: it declares dependencies, how to compile the code, how to run the tests, and how to package the complete application.
- src: Source code of the application.
The src folder is organized like this:
- src/main/java: Backend code in Java. The entry-point is
- src/main/js: Frontend code. The entry-point is
- src/main/resources/application.properties: Backend configuration (for example, database url).
- src/main/resources/db/migration: Database scripts for Flyway (more on this later).
- src/test/java: Backend tests.
- src/test/resources: Backend tests configuration.
Run the application locally
mvn clean package
This command should end with a BUILD SUCCESS message: it compiles and runs the tests and packages the application.
- The application source code organization is based on this tutorial. You can read this document if you are interested in HATEOAS, WebSockets and Spring Security.
- Although the application needs a database, the tests pass because they use H2, an in-memory database.
The next step is to setup a database locally:
- Download and install MySQL Community Server v5.7. Note that it will normally give you a temporary root password.
- MySQL should have installed the MySQL Command-Line Tool. You may need to configure your PATH environment variable if the
mysqlcommand is not available on your terminal. On Mac OSX you can do the following:
# Add the MySQL tools into the PATH variable echo 'export PATH=/usr/local/mysql/bin:$PATH' >> ~/.bash_profile # Reload .bash_profile . ~/.bash_profile
- Launch MySQL on your computer and connect to it with your terminal:
# Connect to the database (use the password you received during the installation) mysql -u root -p
- The command above should display a prompt. You can now configure your database:
-- Change the root password if you never did it before on this database ALTER USER 'root'@'localhost' IDENTIFIED BY 'YouNewRootPassword'; -- Create a database for our project CREATE DATABASE todolist; -- Create a user for our project and grant him the rights CREATE USER 'todolist'@'localhost' IDENTIFIED BY 'P@ssw0rd'; GRANT ALL PRIVILEGES ON todolist.* TO 'todolist'@'localhost'; -- Exit QUIT;
Now that we have a database up and running, we need to configure the application. Have a look at the backend configuration file “src/main/resources/application.properties” and check that the DB configuration corresponds to your installation:
spring.datasource.url=jdbc:mysql://localhost:3306/todolist?useSSL=false spring.datasource.username=todolist spring.datasource.password=P@ssw0rd
spring.datasource.urlproperty is in the format
- If you modified this file you need to re-run
mvn clean package.
You can now launch the application locally with the following command:
If everything went well, the application should print several lines of logs in the console. Look at the two last lines:
2018-11-02 13:56:18.139 INFO 87329 --- [main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 8080 (http) with context path '' 2018-11-02 13:56:18.145 INFO 87329 --- [main] com.alibaba.intl.todolist.Application : Started Application in 5.305 seconds (JVM running for 17.412)
Open a new tab in your web browser and open the URL
http://localhost:8080. You should normally get something like this:
Congratulation if you managed to get the application up and running! The source code has been written with the IntelliJ IDEA IDE (the ultimate edition is necessary for frontend development, you can evaluate it for free for 30 days).
Before we move on and create our first CI pipeline, there is still an important point to talk about: we didn’t create any table in the database, so how does the application work? Let’s have a look at our database with a terminal:
# Connect to the database (use your new root password) mysql -u root -p
The command above opens a prompt; please enter the following instructions:
-- Use our database USE todolist; -- Display the tables SHOW TABLES;
The last command should display something like this:
+-----------------------+ | Tables_in_todolist | +-----------------------+ | flyway_schema_history | | task | +-----------------------+ 2 rows in set (0.00 sec)
Now we can understand why the application works: because the database schema has been created. The task table corresponds to the Java class
src/main/java/com/alibaba/intl/todolist/model/Task.java. Let’s study
-- Look at the content of the flyway_schema_history table SELECT * FROM flyway_schema_history;
The result should look like this:
+----------------+---------+-------------------+------+-----------------------------+------------+--------------+---------------------+----------------+---------+ | installed_rank | version | description | type | script | checksum | installed_by | installed_on | execution_time | success | +----------------+---------+-------------------+------+-----------------------------+------------+--------------+---------------------+----------------+---------+ | 1 | 001 | Create task table | SQL | V001__Create_task_table.sql | -947603613 | todolist | 2018-10-31 17:57:51 | 24 | 1 | +----------------+---------+-------------------+------+-----------------------------+------------+--------------+---------------------+----------------+---------+ 1 row in set (0.00 sec)
The flyway_schema_history table has been created by Flyway, a tool that allows us to create and update our database schema. As you can see, the table contains the names of the scripts from src/main/resources/db/migration that have been successfully executed.
Working with Flyway requires us to follow this procedure:
- During the development of the application, when we want to upgrade our database schema, we need to add a new script in the src/main/resources/db/migration folder with a higher prefix number (we cannot modify existing scripts).
- When Flyway starts, it checks what are the scripts that have been already executed (thanks to the flyway_schema_history table), and run the new ones.
Flyway is automatically started when the applications starts, if you check the application logs, you can see that Spring calls Flyway during its initialization. For more information about this integration, please read the official documentation.
Commit and first CI pipeline
It is now time to save the project in the git repository. Please enter the following command in your terminal:
# Go to the project folder cd ~/projects/todolist # Check files to commit git status
The last command should print something like this:
On branch master No commits yet Untracked files: (use "git add <file>..." to include in what will be committed) .gitignore .gitlab-ci.yml package.json pom.xml src/ webpack.config.js
Add all these files and commit them:
# Add the files git add .gitignore .gitlab-ci.yml package.json pom.xml src/ webpack.config.js # Commit the files and write a comment git commit -m "Initial commit." # Push the commit to the GitLab server git push origin master
Pushing your code to GitLab triggers something interesting:
- Open GitLab in your web browser (the URL must be like https://gitlab.my-sample-domain.xyz/);
- Click Projects from the top and select Your projects.
- Click the todolist project to see your files.
- Click CI / CD from the left and select Pipelines.
You should see something like this:
Clicking Artifacts on the left allows you to download the generated .jar file containing your ready-for-production application.
Clicking the icon in the Stages column and then selecting build allows you to see the commands and logs used to compile and package the application.
This pipeline is triggered when somebody pushes code to the server. It is configured by the .gitlab-ci.yml file:
image: maven:3.6.0-jdk-8 variables: MAVEN_OPTS: "-Dmaven.repo.local=./.m2/repository" cache: paths: - ./.m2/repository stages: - build build: stage: build script: "mvn package" artifacts: paths: - target/*.jar
The first line
image: maven:3.6.0-jdk-8 defines the Docker image used to execute the build command (as you can see, using Docker relieves us to setup the JDK 8 and Maven on the GitLab runner manually).
MAVEN_OPTS variable and the
cache block are an optimization: because Maven takes a lot of time to download dependencies, these definitions allow us to re-use these dependencies among pipelines.
stages block defines only one stage
build, we will add new ones later in this tutorial.
build block is the most important one: it instructs the GitLab runner to execute
mvn package in order to compile and run the tests and package the application. The
artifacts block instructs GitLab to save the generated