HOW TO CREATE THE CI/CD USING GITLAB RUNNER WITH NODEJS ON THE EC2 INSTANCE

Posted: Wednesday October 7th, 2020/Under: News, IT technique/By: Nghia Nguyen Trung

Introduction

What is CI/CD

Continuous Integration is the process of automatically building, doing unit testing as source code is changed on the Version Control system (Git). The goal of CI is to quickly make sure a new change from a developer is “good” and suitable for further use in the codebase.

Continuous Delivery is a step beyond Continuous Integration. The application is not only built and tested with every code change in Version Control, but Continuous Delivery is also ensuring that the software can be reliably deployed at any time. However, deploying to the release environment needs to be done manually.

Continuous Deployment is also a further step beyond Continuous Integration, similar to Continuous Delivery. However, unlike Continuous Delivery, instead of deploying manually to the release environment, Continuous Deployment can perform deploy automatically and the user can use the latest source immediately.

What is Gitlab Runner

GitLab Runner is the open source project that is used to run your jobs (build, test, or deploy,…) and sends the results back to GitLab. A Runner can be a virtual machine, a VPS, a Docker container or even a cluster of containers. It is used in conjunction with GitLab CI/CD, the open-source continuous integration service included with GitLab that coordinates the jobs. GitLab and the Runners communicate through an API.

Runners classification:

Shared Runners: are runners provided by GitLab.
Specific Runners: are user-created runners for a certain GitLab project.
Group Runners: are user-created runners for a GitLab group consisting of many projects.

Selecting the executor:

GitLab Runner implements a number of Executors that serve a variety of purposes. Types of Executors that Gitlab Runner provides:

SSH
Shell
Parallels
VirtualBox
Docker
Docker Machine (auto-scaling)
Kubernetes
Custom

However, in this article we use 2 Executors:

Docker Executor for Test job. To save resources for the server, in the Test job we will use Shared Runner provided by Gitlab.
Shell Executor for Deploy job. We will use this Executor to run the command directly on our server.

To learn more about other types of Executors and their usage, see the following link: https://docs.gitlab.com/runner/executors/README.html

Structure development

When pushed code to GitLab develop branch, GitLab will create a Pipeline consisting of 2 Stage Test and Deploy. Since we are deploying a NodeJS application here, it only takes 2 stages, for a Typescript application, it may need to add a Build stage.

The Test stage will include the Test job, here as mentioned in the Executor section we will use the GitLab Shared Runner (with Executor Docker) provided by GitLab to save resources for the server. Once the Test job runs successfully and without any errors, GitLab moves to the next stage, the Deploy stage.

The Deploy stage will include the job Deploy, here we will use the GitLab Runner (with the Executor Shell) installed on the EC2 instance together with the Webserver. This Runner task is to detect the changes in the source code and copy those changes to the Project folder, then restart the NodeJS service.

To configure stages as well as jobs we need to create a .gitlab-ci.yml file and push it to GitLab with the source code.

Note: To exchange files between stages we can save files to GitLab Shared Storage. Normally we will save libraries /node_modules to Cache and for Typescript, files generated from .ts to .js build process will be saved in Artifacts (here we use NodeJS so we will not need Artifacts).

Steps taken

Setup the server

First we create a new EC2 instance running Amazon Linux AMI operating system. Open port 3000 (port to access NodeJS Server from browser). Then use PuTTY to access the server.

Install GitLab Runner with the following command:

$ curl -L https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.rpm.sh | sudo bash
$ sudo yum install gitlab-runner

Install NodeJS:

$ sudo yum install -y gcc-c++ make
$ curl -sL https://rpm.nodesource.com/setup_14.x | sudo -E bash –
$ sudo yum install -y nodejs

Install forever package (used to run the NodeJS project in background):

$ sudo npm install forever -g

Create a directory for the project:

$ sudo mkdir -p /var/www/project-ci

Set permissions so that gitlab-runner can read, write and execute in the project directory:

$ sudo setfacl -m user:gitlab-runner:rwx /var/www/project-ci

Create GitLab repository and NodeJS demo project

We create a new GitLab repository then clone to our computer.

Add the following files to the cloned directory:

package.json

{
  "name": "project-ci",
  "version": "1.0.0",
  "description": "project to research Gitlab Runner",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Test OK\" && exit 0",
    "start": "node index.js"
  },
  "keywords": [],
  "author": "nghia.nt",
  "license": "ISC",
  "dependencies": {
    "express": "^4.17.1"
  }
}

index.js

const express = require('express');
const app = express();
const port = process.env.NODE_PORT || 3000;
app.get('/', function (req, res) {
  res.send('Hello World From Briswell Vietnam!');
});
app.listen(port, function () {
  console.log('Listening on port ' + port);
});

.gitignore

node_modules/
iisnode/
npm-debug.log*
.vscode/
.idea/
.env*
**/*.log
dist/

Then push the three files above to GitLab develop branch.

Register GitLab Runner with GitLab

Go to GitLab Project under Settings > CI / CD. In the Runners section -> Set up a specific Runner manually, we need to note the URL and Token information.

Back at PuTTY we run the following command to register GitLab Runner:

$ sudo gitlab-runner register

After running the above command, the wizard will ask to enter the URL and Token information obtained above:

Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com/):
https://gitlab.com/
Please enter the gitlab-ci token for this runner:
k753PaGiosBxxxxxx
Please enter the gitlab-ci description for this runner:
my-shell-runner
Please enter the gitlab-ci tags for this runner (comma separated):
my-shell-runner
Registering runner... succeeded                     runner=ajgHxcNz
Please enter the executor: virtualbox, docker+machine, kubernetes, docker, shell, ssh, docker-ssh+machine, docker-ssh, parallels:
shell
Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!

After registration is complete we go back to the Settings page of GitLab to check. If the Runner information appears as shown, we have successfully registered.

Config stage, job for GitLab Project

In the source code of the project, create a new .gitlab-ci.yml file with the following content:

# We have 2 stage Test and Deploy
stages:
  - test
  - deploy

# Config cache
cache:
  paths:
    - node_modules/

# Test job
test:
  # Docker image
  image: node:latest
  # Attach Test job to Test stage
  stage: test
  # Config to use GitLab Shared Runner with Executor Docker
  tags:
    - docker
  script:
    - npm install
    - npm run test
  # Defines the names of branches and tags the job runs for
  only:
    - develop

# Deploy job
deploy:
  type: deploy
  # Attach Deploy job to Deploy stage
  stage: deploy
  # Config to use our Runner with Executor Shell
  tags:
    - my-shell-runner
  script:
    # Only copy changed files to project folder
    - cp -r -u * $PROJECT_DIR
    - cd $PROJECT_DIR
    - npm install
    # Restart NodeJS service
    - forever stop index.js || true
    - forever start index.js
  only:
    - develop

Then push the above file to the develop GitLab branch.

Now we have to setup Environment variables. On the GitLab page we access Settings > CI / CD > Variables then click Add Variable. Don’t forget un-check Protect variable.