Boost Docker CI Build Speed to ~10X times

As an software engineering developer, you know that automated CI testing is one of keys to improve software release life-cycle.

But sometimes reality is not as good as you think, CI testing speed is slow (3-10 minutes / build) and it slows the release cycle speed down. And you try to look into your build logs to find out what causes the problem. Then you got it, it’s mostly the DATABASE service (MySQL, Postgres, MongoDB, …)

I will summarize some stages of your database in a testing build:

  • First, it initializes the data, loads config and listens to the connections (takes around 10-45 seconds)
  • Second, that you import your testing database into the server (including schemas and initialized data) takes around 20-60 seconds
  • Then, on each test case, it needs to clear all data then re-imports fixture data (takes around 30-120 seconds)

So how to make these servers run as fast as possible like some Key-Value databases do? (Redis, Memcached). The main different point is the MEMORY! What if we put all data inside memory??

All of we know that RAM speed with 150 times lower latency is technically better than SSD and HDD speed. And as a matter of fact, Linux is a good OS that supports a lot of filesystems, specially tmpfs, which you can mount files into your RAM memory.

However, nothing is perfect and this is not an exception. Actually, it is not a good option for persistent data which is not necessary for testing database. What it really needs is speed only, so it fits in.

That’s my idea, now I will try to test it on my CI environment (I use DroneCI using Docker). In new version 0.8+ of DroneCI, they support us to run docker containers within tmpfs mount.

So I just add this line into my drone config

1
2
3
4
5
6
7
8
9
services:
testdatabase:
thumbnail: mysql:5.7
# Add this 2 lines below to boost your database container
tmpfs:
- /var/lib/mysql
environment:
- MYSQL_DATABASE=testdb
- MYSQL_ROOT_PASSWORD=passwd

Result:

  • MySQL service initializes in 3 seconds instead of 25 seconds
  • Import testing database using mysql client takes below 1 second instead of 17 seconds
  • My test cases run 20-30% faster (I have few testcases using database)

So, worth a shot !!


Ref:

#TIL : Fastly conflict detector script

Last month, I built a CI solution for our project and adding a conflict detector to our build commands. This script runned so slow because it will check all application files (and our application codebase has many of css, js files).

This was the script

1
2
3
4
5
6
7
8
9
#!/bin/bash

grep -rli --exclude=conflict_detector.sh --exclude-dir={.git,vendor,node_modules} "<<<<<<< HEAD" .

if [ $? -eq 0 ]; then
exit 1
else
exit 0
fi

Today, I think why don’t we just check recently updated files (in the latest commit) ??? Then I have this new script

1
2
3
4
5
6
7
8
9
10
11
12
13
14
#!/bin/bash

# New way :D
CHANGED_FILES=$(git log --pretty=format: --name-only HEAD^..HEAD | sort | uniq)

for f in $CHANGED_FILES
do
if grep --exclude=conflict_detector.sh -q "<<<<<<< HEAD" $f
then
exit 1
fi
done

exit 0

conflict_detector.sh is the filename of script, we exclude it from check to make sure changing this file doesn’t make it failed.

You can use this approach to check linter, coding standard or run preprocessor ;)

Result (in my context) :

  • Old script : 12 seconds
  • New script : ~ 50ms (200 times faster)

be Automated, be Fast, but be Careful !

#TIL : Reduce init time MySQL docker image

Original MySQL docker image uses a script to generate ssl certificates for service. Sometime we don’t really need it (connect via a docker network link or need a fast enough database service to build a automated test).

We can reduce init time by removing the script from original Docker image

1
2
3
4
FROM mysql:5.7

# Remove mysql_ssl_rsa_setup to ignore setup SSL certs
RUN rm -f /usr/bin/mysql_ssl_rsa_setup

FAST as a FEATURE !!! 🚀

Building Automated CI server with Drone and Docker

Introduction

Docker is great tool to management linux containers. It brings DevOps to next level, from development to production environment. And of course, before deploy anything to production, software should be tested carefully and automatically.

That’s why Drone, a new lightweight CI server built-on top Go lang and Docker, will help us to resolve the testing problems in simple and fast way.

Setup

This guide will assume you already have Docker and Docker Compose tool. And of course, root permission ;)

Step 1 : Clone my example docker-compose here : https://github.com/khanhicetea/drone-ci

1
2
3
$ git clone https://github.com/khanhicetea/drone-ci
$ cd drone-ci
$ cp .env.example .env

Step 2 : Update your setting in .env file

Step 3 : Run drone via docker-compose

1
2
$ source .env
$ sudo docker-compose up -d

Step 4 : Go to your Drone url (remember use https url), then authorize with Github provider.

Usage

In example repo, I created a sample .drone.sample.yml file so you can follow the structure to create own file.

I will explain some basics here

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
clone:
git:
thumbnail: plugins/git
depth: 5

pipeline:
phpunit:
thumbnail: php:7
commands:
- /bin/sh conflict_detector.sh
- /bin/sh phplinter.sh app lib test
- composer install -q --prefer-dist
- test/db/import testdatabase root passwd testdb test.sql
- php -d memory_limit=256M vendor/bin/phpunit --no-coverage --colors=never

notify:
thumbnail: plugins/slack
webhook: [your_slack_webhook_url]
channel: deployment
username: DroneCI
when:
status: success

notify-bug:
thumbnail: plugins/slack
webhook: [your_slack_webhook_url]
channel: bugs
username: DroneCI
when:
status: failure
branch: production

services:
testdatabase:
thumbnail: mysql:5.7
detach: true
environment:
- MYSQL_DATABASE=testdb
- MYSQL_ROOT_PASSWORD=passwd

This file consists 3 sections :

  • clone : To clone the source code and prepare for pipeline step. This section will be run first
  • services : Declare your docker services (databases, ip server) which source code connect to. This section will be run at sametime with pipeline (after clone)
  • pipeline : Testing pipe, where you put testing logic here.

In this pipeline, I made a example PHP testing through these steps :

  1. Check conflicts in code (grep for >>>> HEAD string)
  2. Run PHP linter in application codes
  3. Run Composer to install all dependencies
  4. Import testing database to mysql services (using testdatabase hostname to connect service)
  5. Run testing script via phpunit tool

Then, notify testing result via Slack channel ! ;)

A picture is worth a thousand words

Drone CI screenshot

Lets automate all the things !

#TIL : Using netcat to wait a TCP service

When doing a CI/CD testing, you would need to connect a external service (RDBMS, HTTP server or generic TCP server service). So you need waiting the service before running your test app.

One way to do right waiting instead of sleep for a specified time is using netcat tool

1
$ while ! echo -e '\x04' | nc [service_host] [service_port]; do sleep 1; done;

Examples

  • MySQL service on port 3306
1
2
$ while ! echo -e '\x04' | nc 127.0.0.1 3306; do sleep 1; done;
$ ./run_test.sh

Explanation :

echo -e '\x04' will send an EOT (End Of Transmission) to the TCP every second to check if it’s ready !