4 simple ways to reduce your (C)ontinuous(I)ntegration build cycle

May 10, 2014 - Marco Behler

Are you also dreading your slow CI jobs? A build+test cycle of a couple of minutes sounds like a far distant dream. Reality is, after each check-in you can grab a coffee and watch the first half of the Godfather until your test results trickle in from Teamcity or Jenkins.

This is of course a complex topic. While it could easily spiral into an endless stream of discussions, e.g. on "what is a unit/integration/functional test and why are they so slow", "my build tool sucks and takes ages to compile", "who needs CI anyway?", here are four quick and easy steps that will help you alleviate your pain - now. (Note: In the very near future we will follow up with a detailed tutorial and live examples, including benchmarks, of all following steps, so join our newsletter below that you don't miss out!)

1. Upgrade your hardware.

Yes, seriously, very easy but often overlooked: We have seen way too many organizations run their builds on (too) small hardware, or sharing a fairly packed build server with a ton of teams, which ultimately killed performance agqin. You can of course look into cloud offerings like the EC2 c-instances, rent a strong root server, or simply build one yourself. Do not try to save pennies here!

2. Tune your filesystem

Let's assume Ext4fs for now. Quick and painless:

noatime,data=writeback,nobh,barrier=0,commit=300

3. Use a Ramdisk (and smart rabbits combine that with a UnionFS)

Sure, you have upgraded all your mechanical drives in your build servers to SSDs already, or haven't you yet? Either way, to squeeze out maximum performance, think about using a ramdisk and moving most of your build infrastructure there.

mount -o size=16G -t tmpfs none /mnt/tmpfs

(that’s it!) Scared of loosing data and wanting to make data persistent? Combine your ramdisk with a normal, persistent filesystem. UnionFS is your friend. </p>

4. Parallelize your build and tests.

We're operating in the JVM world and most of the time use either Maven or Gradle. Both of them can be configured to use multiple threads quite easily. (Yup, executing tests in parallel, especially if it involves database access, can be a bit tricky at times, but that's the scope of another article).

mvn -T 1C clean install # Maven | 1 thread per cpu core

gradle clean build --parallel # Gradle | determines the optimal number of threads

Got some more tips yourself? Feel free to share them here!