A transient /var/log

So you're afraid of wearing out your SSD. Or you don't want to spin up a sleeping hard disk just to write a line of log. Or maybe you're just looking for some use for the disproportionate amount of RAM you've got. Whatever your case is, what you need is a way to keep the log files in memory as long as the system is up, and write them down to the regular storage just before shutdown.

Starting

The concept itself is pretty simple: mount a tmpfs on top of /var/log.

mount -t tmpfs -o nosuid,noexec,nodev,mode=0755,size=16M \
      transientlog /var/log

The problem of mounting a new file system on top of a non-empty directory, is that it hides the previous contents of such directory: they are still there, but you cannot access them any more. To overcome this drawback, you have to relocate /var/log to another place before mounting the tmpfs. The best way to do this is via a "bind mount".

mount --bind /var/log /var/my_true_log

The next thing to do is to populate the newly created directory with the contents from the old one so that the logging facilities can continue to work as if nothing happened.

cp -rfp /var/my_true_log -T /var/log

If you want to create a script to automate all this, you must be prepared to handle the exceptions: in case the copy should fail, just unmount everything in the reverse order. And before anything else, ensure that you are root, that the directories actually exist, and that the transient log is not already running. The latter can be handled with a lock file. So this will be the preamble to your script.

[ -f /var/lock/transientlog.lock ] && return 1
[ `id -u` -eq 0 ] || return 2
[ -d /var/log ] || return 2
[ -d /var/my_true_log ] || mkdir -p /var/my_true_log

And this will be the complete function.

do_start()
{
        # Return
        #   0 if transient log has been started
        #   1 if transient log was already running
        #   2 if transient log could not be started

        [ -f /var/lock/transientlog.lock ] && return 1
        [ `id -u` -eq 0 ] || return 2
        [ -d /var/log ] || return 2
        [ -d /var/my_true_log ] || mkdir -p /var/my_true_log
        mount --bind /var/log /var/my_true_log
        mount -t tmpfs -o nosuid,noexec,nodev,mode=0755,size=16M \
              transientlog /var/log
        if [ $? -eq 0 ]; then
                if cp -rfp /var/my_true_log -T /var/log; then
                        touch /var/lock/transientlog.lock
                        return 0
                fi
                # Rollback the mount
                umount /var/log
        fi
        # Rollback the directory mangling
        umount /var/my_true_log
        return 2
}

Stopping

At shut down, the log files need to be copied back to the permanent storage before unmounting the tmpfs. Using cp with the option -u is an effective way to restrict the copy to only those files that were actually changed, thus minimizing the writes to the disk. The option -l to umount, known as lazy unmount, tells the kernel to delay the execution as long there are processes with an open file on that file system.

cp -rfup /var/log -T /var/my_true_log
umount -l /var/log
umount -l /var/my_true_log

Be aware that a race condition can happen here, because a process may decide to start writing to /var/log right between the copy and the unmount. There is little you can do about it, except trying to stop the transient log after everything else. Be prepared to risk losing the last messages of those processes that are still running!

After adding the usual sanity checks, the final look of the stop routine is the following.

do_stop() {
        # Return
        #   0 if transient log has been stopped
        #   1 if transient log was already stopped
        #   2 if transient log could not be stopped
        #   other if a failure occurred

        [ -f /var/lock/transientlog.lock ] || return 1

        # Check if I am root
        [ `id -u` -eq 0 ] || return 2

        # Merge back to permanent storage
        cp -rfup /var/log -T /var/my_true_log

        umount -l /var/log
        umount -l /var/my_true_log
        rm -r /var/lock/transientlog.lock
        return 0
}

When a refresh is needed

Now you have two independent log directories, one static and one dynamic, that exchange data only at boot and shutdown. It would be a nice feature to allow the two directories to be synchronized at will. A wise administrator could then add a cron rule to periodically ensure consistency between the transient and permanent directories.

The idea is simple: just duplicate the concept seen for the stopping phase, leaving out the unmounts. Without any further delay, this is your whole routine (called do_reload() for reasons you will soon understand).

do_reload() {
        # Return
        #   0 if transient log has been reloaded
        #   1 if transient log was not running
        #   2 if transient log could not be reloaded

        [ -f /var/lock/transientlog.lock ] || return 1
        
        # Check if I am root
        [ `id -u` -eq 0 ] || return 2

        # Merge back to permanent storage
        cp -rfup /var/log -T /var/my_true_log
        touch /var/lock/transientlog.lock
        return 0
}

The initscript

In order to have the transient log automatically started at boot time and stopped at shutdown, you need to shape everything into an initscript. You can use the skeleton in /etc/init.d as a starting point and merge the three functions defined above, namely do_start(), do_stop() and do_reload().

What you do have to change, though, are the so-called LSB headers. These are nothing but comments at the beginning of an initscript, which define the relative dependences with other initscripts. Setting up the right headers saves you from the heavy task of finding the correct order of initscripts, because insserv will do it for you (it is, in fact, the only way to handle the init sequence from Debian Squeeze on).

Transient log must be started before and stopped after any service that writes to /var/log, and you can reasonably assume that the first of such services is syslog. The following lines express these relationships.

# X-Start-Before:  $syslog
# X-Stop-After:    $syslog

Now, this sounds reasonable, and is indeed the best approximation you can have. But for sake of completeness, you must be aware that, while sysolg is for sure the first program to access /var/log, there are so many processes writing log files that you can hardly tell which will be still running at shutdown. This issue has no easy solution, not under Debian's runlevels model. Things may change if Debian moves to other boot schemes, but for the moment you have to live with it. At least, the lazy unmount seen above prevents the system from hanging forever if some process keeps /var/log busy at shutdown.

Another LSB header that you might want to deploy is this Debian-specific keyword:

# X-Interactive:   yes

There should be no doubts in which runlevels transient log is supposed to run and in which not:

# Default-Start:    2 3 4 5
# Default-Stop:     0 1 6

Add some descriptions and your whole LSB header will look like this:

### BEGIN INIT INFO
# Provides: transientlog
# X-Start-Before:       $syslog
# X-Stop-After:         $syslog
# X-Interactive:        yes
# Default-Start:        2 3 4 5
# Default-Stop:         0 1 6
# Short-Description:    Keeps /var/log in RAM
# Description: Moves the contents of /var/log to RAM during boot
#              and keeps it there until shutdown/reboot, when it
#              copies the contents back to permanent storage.
### END INIT INFO

The rest of the script can be left mostly untouched. A finished version can be found here (Local Mirror). Save it as /etc/init.d/transientlog, make it executable, then let insserv create the symbolic links in the relevant runlevels.

# chmod a+x /etc/init.d/transientlog
# insserv transientlog

If you have an older Debian without insserv, you must find out yourself reasonable sequence numbers for the start and stop actions. Assuming 00 for start and 99 for stop, the following command will create the symbolic links that you need.

# update-rc.d transientlog start 00 2 3 4 5 . stop 99 0 1 6 . 

The cron job

The initscript seen in the previous section can be called with the reload argument in order to force the synchronization of the transient and permanent directories. You might want to invoke it periodically, for example once a day: just write a script like the following and put it in /etc/cron.daily

#!/bin/sh

[ -x /etc/init.d/transientlog ] || exit 0
[ -f /var/lock/transientlog.lock ] || exit 0
/etc/init.d/transientlog reload