Tools - Distributed Compilers

You might have guessed by the amount that I talk about speeding up compile time, that compilation time becomes a big problem when working on big projects. A full rebuild of something could take 15 minutes or more depending on the project, machine speed etc (though most builds are a lot smaller). Multiply this by the number of programmers, and you start to have serious amounts of wasted time.

The techniques I’ve mentioned in the past help speed things up, but there’s also a tool solution. Commercial products like Incredibuild (for VC++ projects) and SN Systems DBS (SN make compilers and tools for console development) work by sharing the compilation across several machines. Let’s say there are 8 programmers. At any one moment, usually only one or two people are compiling. When a person compiles, it farms out individual cpp files to the different machines, and they compile in parallel.

There are eventually diminishing returns to the parallelism due to the amount of coordination that needs to happen, and the fact that linking is still done by one machine, but overall a good speedup can be attained.

How do these systems work? I’m not entirely sure about the internals, but my suspicion is that the PC which initiates the task does all the preprocessing of the cpp file, so that all the headers are already included. This means that one file can be moved to the client, and that the file doesn’t ‘need’ anything else.

The downsides are the cost of buying, and the reduced amount of time for juggling.