Improve collaborative build times with ccache
By Martin C. Brown2004-11-08
Combining ccache and distcc
|
|
You may already be aware of distcc, another tool from the Samba group, which allows you to spread the compilation across a number of machines. This effectively increases the number of simultaneous compilations that can occur, just as if you had used the multiple jobs option with make (with the -j command line option). The distcc system works by using on each host a daemon that accepts source files in their final pre-parsed form, and then compiles the files locally before returning the resulting object file.
When used properly, you can generally expect to get a slightly less than linear decrease in build times for each new identical node you add, although you'll only see the effects on projects where you have more than one source file, as distcc only distributes whole source files.
Because distcc distributes files in their parsed state, you can combine ccache, which speeds up the C preprocessing portion, with distcc to perform the actual compilation into object code. To use distcc and ccache in this way, configure distcc on your hosts and distcc and ccache on your main development machine.
Now set up your environment variables on the machine on which you want to build the project, as shown in Listing 4.
Listing 4. Environment variables for using ccache and distcc
export set DISTCC_HOSTS='localhost atuin nautilus pteppic kernel'
export set CCACHE_DIR=/Data/Cache/CCache
export set CCACHE_PREFIX=distcc
export set CCACHE_LOGFILE=/Data/Cache/CCache.log
export set CC='ccache gcc'
The variables are defined as follows:
• DISTCC_HOSTS specifies the hosts to distribute work to.
• CCACHE_DIR specifies the location of the ccache directory.
• CCACHE_PREFIX defines a prefix to use when ccache calls the true compiler to compile the source (after preprocessing).
• CC sets the name of the C compiler to use in the first instance (ccache).
Now when you run make with the -j option to specify the number of the simultaneous compiles to perform, the file will first be parsed with ccache (using the cache if necessary) before being distributed to one of the distcc hosts.
Although distcc speeds up the process of compiling, it doesn't alter the basic limitations of the environment. For example, you shouldn't set the number of simultaneous jobs operated by make to more than twice the number of available CPUs. For example, if you have four two-way machines, you won't notice much improvement in speed if you set the job value above 16.
Tutorial Pages:
» How to squeeze more speed out of your compilations
» Using ccache
» Combining ccache and distcc
» Statistics
» Summary
» Resources
First published by IBM DeveloperWorks
|
| Related Tutorials: » How to Install PHP 5 on Linux » How to Install Apache 2 on Linux » How to Install MySQL 5.0 on Linux » SMB Caching » Mound --Bind » Tar Wild Card Interpretation |