This site is now 100% read-only, and retired.

XML logo

Ensuring system updates are actually in use
Posted by dkg on Sun 10 Sep 2006 at 15:30
Tags: none.
I keep my machines all patched with the latest updates from security.debian.org. But sometimes, a simple apt-get update && apt-get dist-upgrade is not enough, particularly when system libraries are being upgraded (e.g. the latest openssl vulnerability, DSA 1173). In this situation, running processes could have loaded copies of the old libraries, so they wouldn't get the benefits of the old version. as DSA 1173 says:
Note that services linking against the openssl shared libraries will need to be restarted. Common examples of such services include most Mail Transport Agents, SSH servers, and web servers.

So after an upgrade, it's important to check that the affected services are really restarted. The best way i've found so far is to check what deleted files are currently being held open by any process. I use lsof to look at this:

root@serverX:~# lsof | grep 'path inode'
apache2    9255    www-data  mem       REG        3,3              725883 /usr/lib/i686/cmov/libssl.so.0.9.7 (path inode=725841)
apache2    9255    www-data  mem       REG        3,3              725770 /usr/lib/i686/cmov/libcrypto.so.0.9.7 (path inode=725839)
root@serverX:~# 
(note: i've excised some lines of the actual output for brevity)

I think the files that lsof lists with the (path inode=NNNN) designation are files being held open by the system after having been removed or replaced. So seeing the above output lets me know that i need to restart apache2 on this particular server. Otherwise, apache2 is still vulnerable to the openssl vulnerability, despite having the updated version installed on the system.

Try running this on one of your own systems. You might be surprised at the services in want of an upgrade, especially if you tracked the latest r3 of sarge, with its glibc upgrade.

i'm not convinced that my lsof|grep filter is the best way to look up this information, but it seems to work for me. I'd be interested to know if other people have better ways to check on the same situation.

 

Comments on this Entry

Re: Ensuring system updates are actually in use
Posted by Anonymous (62.6.xx.xx) on Mon 11 Sep 2006 at 14:30
Thank you very much. This is a real eye-opener.

[ Parent ]

Re: Ensuring system updates are actually in use
Posted by mcphail (62.56.xx.xx) on Tue 12 Sep 2006 at 00:03
You may be interested in a script called "checkrestart" in debian-goodies. It would seem to do the job you require, and could be run after each apt-get update and upgrade.

Neil McPhail

[ Parent ]

Re: Ensuring system updates are actually in use
Posted by dkg (216.254.xx.xx) on Tue 12 Sep 2006 at 01:04
[ View Weblogs ]
Hey, thanks for pointing out this tool! I'd never heard of it before, and it sounds like it is supposed to do exactly what i want. Unfortunately, it doesn't work.

On a sarge system running apache2 and fetchmail (both linked against libssl), i applied the latest version of libssl from the DSA. I then restarted fetchmail, but not apache2.

root@foo:~# lsof | head -n1
COMMAND     PID        USER   FD      TYPE     DEVICE     SIZE       NODE NAME
root@foo:~# lsof | grep libssl
apache2    5073    www-data  mem       REG        3,3              725883 /usr/lib/i686/cmov/libssl.so.0.9.7 (path inode=725841)
apache2   10131    www-data  mem       REG        3,3              725883 /usr/lib/i686/cmov/libssl.so.0.9.7 (path inode=725841)
fetchmail 10275   fetchmail  mem       REG        3,3   198576     725841 /usr/lib/i686/cmov/libssl.so.0.9.7
apache2   13576    www-data  mem       REG        3,3              725883 /usr/lib/i686/cmov/libssl.so.0.9.7 (path inode=725841)
apache2   13658    www-data  mem       REG        3,3              725883 /usr/lib/i686/cmov/libssl.so.0.9.7 (path inode=725841)
apache2   13661    www-data  mem       REG        3,3              725883 /usr/lib/i686/cmov/libssl.so.0.9.7 (path inode=725841)
apache2   13663    www-data  mem       REG        3,3              725883 /usr/lib/i686/cmov/libssl.so.0.9.7 (path inode=725841)
apache2   13666    www-data  mem       REG        3,3              725883 /usr/lib/i686/cmov/libssl.so.0.9.7 (path inode=725841)
apache2   16249    www-data  mem       REG        3,3              725883 /usr/lib/i686/cmov/libssl.so.0.9.7 (path inode=725841)
apache2   16250    www-data  mem       REG        3,3              725883 /usr/lib/i686/cmov/libssl.so.0.9.7 (path inode=725841)
apache2   17177        root  mem       REG        3,3              725883 /usr/lib/i686/cmov/libssl.so.0.9.7 (path inode=725841)
apache2   25777    www-data  mem       REG        3,3              725883 /usr/lib/i686/cmov/libssl.so.0.9.7 (path inode=725841)
root@foo:~# checkrestart 
Found 0 processes using old versions of upgraded files
root@foo:~# 
As you can see, checkrestart failed to notice that apache2 was using the old version of libssl.

I'd say this means that bug 264985 actually warrants consideration as a security concern.

[ Parent ]

Re: Ensuring system updates are actually in use
Posted by dkg (216.254.xx.xx) on Tue 12 Sep 2006 at 03:17
[ View Weblogs ]
This is still broken on etch, as well. I just did a big pull from etch on a desktop system, and here's the evidence:
[0 root@squeak ~]# dpkg -l debian-goodies lsof
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Installed/Config-files/Unpacked/Failed-config/Half-ins talled
|/ Err?=(none)/Hold/Reinst-required/X=both-problems (Status,Err: uppercase=bad)
||/ Name           Version        Description
+++-==============-==============-=============================== =============
ii  debian-goodies 0.24           Small toolbox-style utilities for Debian sys
ii  lsof           4.77.dfsg.1-3  List open files
[0 root@squeak ~]# lsof | grep inode | awk '{print $2}' | sort -u | wc -l
29
[0 root@squeak ~]# checkrestart 
Found 0 processes using old versions of upgraded files
[0 root@squeak ~]# 
29 != 0!

[ Parent ]

Re: Ensuring system updates are actually in use
Posted by dkg (216.254.xx.xx) on Fri 15 Sep 2006 at 23:04
[ View Weblogs ]
a friend just pointed me to a recent discussion on debian-security about this same topic.

[ Parent ]