How to speed up C++ compilation with a distributed compilation.
In the previous two posts, I covered caching and precompiled headers which help to bring down compilation times on a single machine. These techniques are sufficient for a developer working on small- to moderate-sized projects. However, for really big projects containing millions of SLOC, this may prove insufficient. Slow builds eventually lead to long queues in a team’s CI server. Because of that, changes get integrated slower and thus are more painful because of conflicts. If the team has at least a few computers at its disposal that can be used for compilation, then distributed compilation can help shorten those queues.
In this post I’m going to focus on an Icecream project, which is based on a widely-know distcc created by Martin Pool. Both are open-source and freely available. Icecream (IceCC) improves on distcc in a few areas, like toolchain distribution across slaves and a central scheduler.
How it works
Splitting a build between many computers is not absolutely trivial. When we take into account that those machines can run a variety of OSs and compilers on different processors, we have to make sure that a program built by a distributed compiler will be the same as the one built locally. IceCC does a few things to assure correctness. First, let’s take a look at a general IceCC setup:
There has to be at least one computer that plays a role of a scheduler (
$ ./icecc-scheduler -d), which distributes work among other computers running ordinary IceCC daemons (
$ ./iceccd -d). There’s nothing stopping the scheduler from running an ordinary IceCC daemon as well. Each daemon can start a build job and the scheduler takes care of an optimal work distribution between available daemons.
What’s nice about IceCC is that it checks for compiler compatibility between daemons and transparently prepares a tarball with build environment if necessary. The problem of heterogeneous systems is handled transparently. That’s really neat.
Compilation of a single source file resembles many characteristics to approaches described in previous posts. First of all, a source file is preprocessed in a local environment. Then, the preprocessed version is sent to one of the daemons, which in turn compiles it to an object file. When all of the object files are gathered from the daemons, the daemon that originated the build can carry on with the linking process and other work that cannot be distributed.
When to use
The main goal of a distributed compilation is to reduce compilation time of a complete rebuild, so it makes the most sense to use this technique on a build server. Nevertheless, a team of developers can use it to share processor power in cases where one of them needs a fresh build or needs to rebuild a significant portion of a large project. Since nobody likes see their computer slowed down by someone else’s work, jobs spawned by daemons are niced down by default . One of the problems in this scenario may be who should run the scheduler? Fortunately, IceCC doesn’t force you to single out one of the computers for that role. It’s perfectly fine to run a scheduler on all computers and let them elect the one that will be used by all of the daemons. Should the scheduler go down a new one will get chosen from those still working.
Another scenario is when a single developer has two computers and one of them is significantly more powerful than the other. Let’s say you have an ultrabook with a low-power CPU and a desktop with a 12-core CPU. Sitting at the desktop at all times when you code is an option but if you can work on a remarkably comfy sofa or in a nice garden then using your ultrabook is tempting. By running IceCC on both computers you can have the best of both worlds and work efficiently from the place of you choosing.
Distributed compilation with cache
It’s completely possible to combine distributed compilation with compiler caching. IceCC + ccache setup is perfectly doable and even recommended! The preferred way is to make ccache call IceCC when necessary and not the other way around. That’s why it’s really ccache matter to make this arrangement work. Again, this is fairly easy obtainable given the fact that ccache is designed to work well with other compiler wrappers. It suffice to set an enviroment variable:
and make sure that ccache symlinks are in the PATH before everything else. It’s that easy!
I won’t provide any benchmarks because the speedup you’ll get depends heavily on a project, a build script, a network, and available computers. It’s possible to observe nearly linear speedup with each computer or even an overhead when you’re on a slow or congested network. This post concludes the “Speed up C++ compilation” series, but if you have other ideas that were not covered please let me know in comments.
References and further reading
 distcc homepage
 local compilation with the daemon thread on icecream-users group