This site is now 100% read-only, and retired.

Securing distcc with chroot and OpenVPN

Posted by thomas on Mon 6 Jun 2005 at 23:13

This is a short guide on how to set up distcc to run in chroot and send/receive traffic over an encrypted tunnel using the OpenVPN software.

We've mentioned speeding up compilation with distcc previously - if you've never used it before that will serve as a good introduction to the tool.

The OpenVPN software is a versatile VPN solution which allows anything from joining networks as a VPN to allowing road-warriors to connect to a system remotely.

There are several benefits of this approach:

  • Both distcc and OpenVPN runs as unprivilaged users in a chroot, which restricts the commands available to the client
  • distcc accepts jobs only from clients that has a certificate issued by the OpenVPN server
  • distcc listens for job on a private address space (10.8.0.0/24)
  • There is no need to create additional user accounts on the server, which you had to do if you were using SSH

Of course there are some drawbacks too:

  • Since the connections are made over OpenVPN it's slower.
  • You need to manually update the chroot when a new version of distccd or gcc is available, this is however made easier with the available script.
  • You need to issue certificates and distribute them to the client(s).
Setting up OpenVPN

Since openvpn.net/howto.html#pki covers all the steps in a detailed manner, we're only going to touch this subject briefly.

To install openvpn use:

apt-get install openvpn

Copy the /usr/share/doc/openvpn/examples/easy-rsa directory to /etc/openvpn and change its permissions:

cd /etc/openvpn
cp -R /usr/share/doc/openvpn/examples/easy-rsa/ ./
chmod 0700 easy-rsa/

Edit /etc/openvpn/easy-rsa/vars, and uncompress the openssl.cnf.gz file by running :

cd /etc/openvpn/easy-rsa
gunzip openssl.cnf.gz

Create the CA key:

. ./vars
./clean-all
./build-ca

Generate a certificate and private key for the server:

./build-key-server server

Generate client certificate(s):

./build-key client1
(Where "client1" is the identifier for the client - Always use a unique common name for each client.)

Diffie Hellman parameters must be generated for the OpenVPN server. This can be achieved by running:

./build-dh

Copy the necessary keys to /etc/openvpn:

cp easy-rsa/keys/{ca.crt,server.crt,server.key,dh1024.pem} ./

Now you need to adjust the server configuration:

tls-server

port 1194
proto udp 
dev tun

ca ca.crt
cert server.crt
key server.key 
dh dh1024.pem

server 10.8.0.0 255.255.255.0
ifconfig-pool-persist ipp.txt

keepalive 10 120

comp-lzo

chroot /etc/openvpn
user nobody
group nogroup

persist-key
persist-tun

status openvpn-status.log
log openvpn.log

verb 3
mute 20

The client configuration should now be very straightforward. For each client you'll need to setup something similar to:

dev tun
proto udp
remote distcc.example.com
ns-cert-type server
client
ca ca.crt
cert client1.crt
key client1.key
chroot /etc/openvpn
user nobody
group nogroup
comp-lzo
persist-tun
persist-key
status openvpn-status.log
log openvpn.log
verb 3
mute 20
Setting up distcc

Install distcc if you've not installed it previously:

apt-get install distcc

Change the "ALLOWEDNETS" setting in /etc/default/distcc to read:

ALLOWEDNETS="127.0.0.1 10.8.0.0/24"

(This allows the connections to be accepted from our VPN network 10.8.0.0/24).

Run the chroot script available at www.northernsecurity.net/download/distccd-chroot and copy the init.d script (www.northernsecurity.net/download/distccd-chroot.init) file to /etc/init.d/distcc.

(These files are both mirrored here)

If everything went well /var/chroot/distccd/ should be created containing all necessary files.

Start distccd with "/etc/init.d/distcc start" and verify that it runs in the chroot.

# ps -fe|grep distccd|awk '{print $2}'
29302
# ls -l /proc/29302/root 
lrwxrwxrwx  1 distccd nogroup 0 2005-06-01 01:41 /proc/22628/root -> /var/chroot/distccd

From the the distccd logfile you should see something to indicate it is successfully listening upon the private network:

distccd[29301] (dcc_listen_by_addr) listening on 10.8.0.1:3632

The client should then be able to run distcc just like normal, except that the server is located at 10.8.0.1.

$ export DISTCC_VERBOSE=1       # We want details
$ export DISTCC_HOSTS='10.8.0.1'

Watching with jobs with distccmon-text you should see connection attempts as follows:

18929  Preprocess  util_ebcdic.c   10.8.0.1[0]
11311  Compile     util_time.c     10.8.0.1[0]

Log on the client and if things work correctly you'll see something like this:

distcc[191] exec on 10.8.0.1: cc -g -O2 -pthread -c modules.c -o modules.o
distcc[191] (dcc_note_state) note state 2, file "modules.c", host "10.8.0.1"
distcc[191] (dcc_connect_by_name) connecting to 10.8.0.1 port 3632
distcc[191] (dcc_connect_by_addr) started connecting to 10.8.0.1:3632