Speed Up That Build, now! - Part 1: Ramdisk and Filesystem

May 19, 2014 - Andreas Eisele

Update: As reddit commenter flussence pointed out our numbers here aren't very reliable. So if you want to compare things some may suggest that running

sync; echo 3 > /proc/sys/vm/drop_caches

As mentioned before, CI builds can never be fast enough. Now it's time to get our hands dirty and speed that build up! (Note: Have no fear though at the end of this post we provide you with a simple bash script that applies all the necessary changes to your system automatically. You can also find it here)

One significant contribution to the overall build time is of course pure I/O. The sources have to transferred on to the build server, resources have to be copied or even compressed and of course the generation of the final artifacts happens. Our idea to get rid of as much of I/O latency as possible is through the usage of ram disks.

Our sample use case here will be the building (i.e. compiling and running all test cases) of the nice Java Open Source web framework Apache Wicket through the simple, yet powerful, CI server TeamCity. For that we use a default Ubuntu Server 14.04 installation.

After the initial installation of Ubuntu is finished we have modify the file system mount points. We want to add a ram disk of an appropriate size. Let's got with a size of 2GB for now. We will use the custom mountpoint /build for this.

# mkdir /build

Now we edit the /etc/fstab to add our ram disk partition of type tmpfs.

tmpfs    /build    tmpfs    noatime,nodev,nosuid,size=2G    0    0

As you might notice we also added some performance enhancing options to the tmpfs configuration. Worth mentioning is the option noatime which prevents the kernel from updating access time stamps on file access which obviously removes some extra latency.

To apply our changes we have to mount the partition.

# mount -a

Looks good:

tmpfs           2.0G     0  2.0G   0% /build

It is time to install the rest of the tools so that we can start a build already. We assume that TeamCity has already been setup on the machine as doing this properly is out of the scope of this small how-to.

# apt-get install openjdk-7-jdk
# echo JAVA_HOME='"/usr/lib/jvm/java-7-openjdk-amd64"' >> /etc/environment
# apt-get install maven

A first run of the complete build through TeamCity without any ram disk usage (i.e against conventional ext4) took 3:28 minutes. Not so bad at all. But lets compare this to our ram disk setup.

For this we have to modify the TeamCity agent configuration to use our /build mountpoint. But we want to make sure that only really temporary data ends up there. In this case we have two directories we can move into the ram disk namely the temp directory and the work directory of the agent. Note that still artifacts and other data will remain on the ordinary hard drive. The settings are located at $AGENT_HOME/conf/buildAgent.properties

# mkdir -p /build/agent/{work,temp}

## Container directory to create default checkout directories for the build configurations.
workDir=/build/agent/work
## Container directory for the temporary directories.
## Please note that the directory may be cleaned between the builds.
tempDir=/build/agent/temp

Our build against the ram disk only took a whooping 2:58 minutes. That is a difference of about a half a minute. If you do a few builds a day (like one build per team member push) calculate the time you would safe. And please mind that our tests were running against a very strong build server (32 logical cpu cores, ~60 GB of ram, SSD). If you happen to run on a more commodity setup I/O will matter even more for you.

The picture gets even more clearer if we make use of all those CPU cores. Let's run the maven build in parallel with one thread per core.

mvn -T 1C

In the non ram disk case the build is already down to 2:28 minutes for that. The ram disk case cuts it even further down to 2:26 minutes.

If you want to try running TeamCity with an agent living in a 2 GB ram disk at home you can use our handy startCI.sh bash script. This has been tested successfully on Ubuntu Server 14.04. Just download and run as root user.

# bash <(curl -s https://gist.githubusercontent.com/aeisele/f46a6849e7dac98643a0/raw/2146e0977919c543fcdab08c14becb59ca153042/startCI.sh)

"But what if I want to preserve build data?" you may ask. Don't worry in an upcoming post we will describe how you can sync your tmpfs to an optimized ext4 partition to asynchronously write changes back to a disk while still enjoying all the benefits of a ram disk.