When compiling C/C++ code, header files get included. Headers often tend to include other headers. When a single c/cpp file is compiled, the compiler has to parse through all the headers recursively and process the contents. You may not realize this, but a huge amount of the build time is taken up with just processing the headers.
There's a concept called 'pre-compiled headers'. The idea is that you have one header which includes all the headers which don't change often (e.g. DirectX, OpenGL, third party rendering or sound APIs, STL etc.). These headers are processed and then stored as a processed binary blob. This pre-compiled header blob is then used by all your files, removing the need for them to parse the common files.
Whenever I've seen pre-compiled headers properly set up in a project, the improvement in build speed has been dramatic. Speed ups between 4x and 10x are normal.
The following web page does a great job of explaining this in more detail, and leading you through the simple (but un-intuitive) steps of setting up pre-compiled headers.
(Feel free to post a comment on how much this speeded your build time up)