Distcc adventures: Distributed cross-compiling with macOS and Windows/Linux

This post was originally written on April 20, and it uses LLVM 11.1

Recently I've had a sudden urge to do some distributed compilation to speed up the compilation times for my compact Vulkan engine (which is written in C++). It's not a big codebase by any means, but mostly being on an 8 year old MacBook (it's comfy), I thought it'd be nice to leverage the computational power of my Windows PC, which usually just plays some jazz or lo-fi streams in the background.

However, this innocent adventure turned into a slightly arduous journey. So I decided to turn this experience into a quick guide.

What we're going to do is to cross-compile and produce a macOS binary with distributed compilation on both macOS and Windows nodes (which is actually Linux via WSL).

Here's a condensed version of all the steps. Please note that, I'm on an x86 MacBook, but you should be able to replicate this for M1 as well.

WSL: The magic ingredient

We'll need WSL on Windows to have access to a Linux environment. This is of course not needed if you're running Linux natively.

I've gone with WSL1, since WSL2 seems to suffer from major memory issues.

Ubuntu works nicely, but others should work as well.

Upgrade Ubuntu to the non-LTS version

WSL Ubuntu is LTS by default. We'll turn that into a non-LTS version to have access to the latest packages. (Ubuntu 20.04 focal ⮕ 20.10 groovy)

Do a full system upgrade:

sudo apt update -y && sudo apt upgrade -y

Use the snippet below, or edit /etc/apt/sources.list, search & replace focal with groovy in order to use the latest repositories. (Names will be different depending on the Ubuntu versions in the future)
```
sudo sed -i 's/focal/groovy/g' /etc/apt/sources.list
```

Final full system upgrade:

sudo apt update -y && sudo apt upgrade -y

Clang

We need to install the same version of clang on all machines. This can be a bit of a manual work if package systems don't provide the same version.

macOS setup

I tend to use a manually installed clang on macOS. This is primarily because Apple's Clang lacks the leak sanitizer. Also, we need a specific version of llvm/clang in order to use it with distcc.

We can install it with the help of homebrew:

brew install llvm

Which version we install is important. At the time of this post, 11.1 is what is available on homebrew. So we'll stick with that.

As the brew package info mentions, we'll need to add llvm's bin folder to our path.

Install Xcode CommandLineTools

We'll be using the linker from CommandLineTools. This is because llvm's own linker on macOS does not seem to be fully functional yet. Therefore we're stuck with the latest and greatest linker Apple has ever provided.

If you have Xcode installed, you can install the command line tools from there.

If you don't have Xcode installed, you can grab CommandLineTools installer from Apple's developer downloads page (You'll need to login). Not having to install Xcode should save you a fair amount of disk space.

Linux setup (WSL)

Turns out, even the non-LTS Ubuntu does not have the clang version we need (11.1), so we'll need to install it manually.

We can simply grab one of the precompiled releases, and unpack it in our home folder.

curl -OL https://github.com/llvm/llvm-project/releases/download/llvmorg-11.1.0/clang+llvm-11.1.0-x86_64-linux-gnu-ubuntu-20.10.tar.xz
tar vfxJ clang+llvm-11.1.0-x86_64-linux-gnu-ubuntu-20.10.tar.xz

For convenience, we can either rename the folder to llvm or create a symlink to it.

mv clang+llvm-11.1.0-x86_64-linux-gnu-ubuntu-20.10 llvm

We'll need to update our path to include llvm/bin folder and we're nearly set.

Grab crt from the gcc package

Nearly, because llvm/clang on Linux is actually using crt from gcc by default. So we'll need to install the corresponding package.

sudo apt install libgcc-10-dev

This should give us the goods to proceed.

distcc

We can finally install distcc.

distcc on Linux (our only node in our compile farm)

I opted to go with the Ubuntu package, though it's also possible to compile manually. (I've tried this, but distcc had a few hard-coded paths in its symlinking script, and the uninstall target had issues at the time I tried)

sudo apt install distcc

We need to tell distcc which compilers are supported. This can be done automatically by firing up the script that is provided by the package:

sudo update-distcc-symlinks

This should create necessary symlinks under /usr/lib/distcc. It should look like this:

➜ /usr/lib/distcc ls -l
total 0
lrwxrwxrwx 1 root root 16 Apr  1 20:04 c++ -> ../../bin/distcc*
lrwxrwxrwx 1 root root 16 Apr  1 20:04 c89 -> ../../bin/distcc*
lrwxrwxrwx 1 root root 16 Apr  1 20:04 c99 -> ../../bin/distcc*
lrwxrwxrwx 1 root root 16 Apr  1 20:04 cc -> ../../bin/distcc*
lrwxrwxrwx 1 root root 16 Apr  1 20:04 clang -> ../../bin/distcc*
lrwxrwxrwx 1 root root 16 Apr  1 20:04 clang++ -> ../../bin/distcc*
lrwxrwxrwx 1 root root 16 Apr  1 20:04 clang++-11 -> ../../bin/distcc*
lrwxrwxrwx 1 root root 16 Apr  1 20:04 clang-11 -> ../../bin/distcc*

However, the script didn't seem to work in my case. So we can simply create them manually. Note that, these all point to the distcc binary, but I think distcc only checks the presence of the symlink.

Launch the daemons!

Now it is time for a test run, let's launch distcc:

distccd --no-detach --daemon --allow 127.0.0.1 --allow 192.168.1.0/24 --listen 192.168.1.42 --nice 10 --log-stderr

Now we're only allowing connections from our localhost and LAN, with a mask. We're also only listening on the LAN interface, where 192.168.1.42 is the IP of the server on our LAN.

nice parameter deprioritizes the processes, so it doesn't stall our jazz stream.

Finally, --log-stderr will print the log to stderr, we can additionally pass --verbose here to debug if something goes wrong.

If you have more computers, repeat this step to install distcc + llvm/clang so they're all available as nodes.

Open them firewalls

Don't forget to adjust your firewall(s) so your computers can communicate with each other. distcc uses tcp port 3632 by default. (It is also possible to use an ssh tunnel)

distcc on macOS (our main machine)

brew install distcc

That's it.

Using distcc as a compiler

Now, we just need to use distcc to compile on our main machine.

First, we need to define an environment variable for distcc to quickly find out which hosts are available as distcc nodes.

export DISTCC_HOSTS='localhost/4 bahamut/10'

Bahamut is the name of my PC, but you can also put IP addresses. The number after the slash is the number of cores distcc should utilize on the target node. Generally it's nice to increase this number by one or two, so we can fill out all the cores during stalls.

Note that it's possible to prevent compilation on localhost entirely. Simply remove localhost/X from DISTCC_HOSTS and set:

export DISTCC_FALLBACK=0

This prevents falling back to localhost if a remote compilation fails. This should allow you to compile everything on remote nodes at all times.

Actually compile

We need to set our compiler to distcc, and tell distcc to use clang (and avoid Apple's binaries).

The easiest way of doing this is to set CC and CXX environment variables, however you can also make adjustments in your build system.

export CC='distcc clang'
export CXX='distcc clang++'

Here distcc acts as a compiler driver.

The target triplet

Now, in order for cross-compiling to work, we need to adjust our build system to always set -target parameter. This tells clang to produce code for the right architecture and feature set.

You can find the target triplet on the host computer from clang:

➜ clang --version
clang version 11.1.0
Target: x86_64-apple-darwin20.3.0
Thread model: posix
InstalledDir: /usr/local/opt/llvm/bin

In this case, x86_64-apple-darwin20.3.0 is the triplet I need to use in order to produce binaries for my macOS.

When using CMake, we can simply use something like this:

SET(TARGET x86_64-apple-darwin20.3.0)

However, we can automate this completely by getting the triplet with clang -dumpmachine.

So instead of the manual SET statement above, we can set TARGET with the output of this command:

execute_process(COMMAND clang -dumpmachine OUTPUT_VARIABLE TARGET OUTPUT_STRIP_TRAILING_WHITESPACE)

Accommodate for the number of (virtual) cores

Make sure to pass an adequate number of cores to your build system, so it will parallelize the compilation and accommodate for the number of cores in our compile farm.

It is possible to derive this number from distcc itself: distcc -j

For example, if you're using Make on Bash, you can use:

make -j`distcc -j`

If you're using CMake on Fish:

cmake --build build --config Debug -j (distcc -j)

If you want some extra oomph:

Make / Bash:

make -j$((`distcc -j` + 4))

CMake / Fish:

cmake --build build --config Debug -j (math (distcc -j) + 4)

We're done!

Now, we can ~~finally~~ simply compile stuff with our regular build systems, and it should distribute it to the remote nodes.

We're not using distcc's pump mode, as I didn't find it necessary in my setup. If you have a massive compile farm and huge codebases, you might want to consider using it.

Debugging & monitoring

If something goes wrong, it's possible to launch your build system with DISTCC_VERBOSE set.

DISTCC_VERBOSE=1 cmake --build ...

You can also monitor distcc using its command line tool:

distccmon-text 1

Where 1 is the number of seconds between updates, adjust if necessary.

There is also a GUI version of this tool, written for gnome. If your system supports it, you might want to check it out.

But wait, there's more

Like ccache, but we can take a look at that in another blog post.

That's all, for now!

Questions or comments? Hit me up on Twitter.