This site is now 100% read-only, and retired.

Rescuing a system with massively broken filesystem permissions

Posted by Steve on Thu 21 Dec 2006 at 09:30

There are times when people make mistakes, and manage to do crazy things to working systems. It wasn't so long ago that a hasty deletion caused me all kinds of problems. Recently I read of an unfortunate sysadmin who managed to recursively change permissions on their root filesystem - and here is my attempted solution.

The problematic command was:

(none):~# chmod -R 777 /

Here the entire filesystem, and all mounted partitions, were modified to be world-writable. One immediate consequence of this is that ssh logins fail:

skx@lappy:~$ ssh -l root
ssh_exchange_identification: Connection closed by remote host

Looking in /var/log/auth.log upon the host we can see the cause of this failure:

error: Could not load host key: /etc/ssh/ssh_host_rsa_key
error: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
error: @         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
error: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
error: Permissions 0777 for '/etc/ssh/ssh_host_dsa_key' are too open.
error: It is recommended that your private key files are NOT accessible by others.
error: This private key will be ignored.
error: bad permissions: ignore key: /etc/ssh/ssh_host_dsa_key
error: Could not load host key: /etc/ssh/ssh_host_dsa_key
fatal: /var/run/sshd must be owned by root and not group or world-writable.

These errors are fixable fairly easily since we still have an open root session:

(none):~# chmod 600 /var/run/sshd
(none):~# chmod 600 /etc/ssh/ssh_host_*

Since we have root access we can now fix things up, and the availability of remote ssh access is a nice bonus. However these remote logins cannot become root themselves just yet - because su and sudo have lost their setuid flags:

skx@(none):/$ su
setgid: Operation not permitted

We can fix this fairly easily, again using our existing root login:

(none):~# chmod 4755 /bin/su /usr/bin/sudo

Now we can login remotely and become root. So the system is 90% fixed!

But what if we didn't have root access? What if we could only login via a normal user account, via the console perhaps?

Lets try that now.

(none):~# chmod -R 777 / 
(none):~# exit

Remote logins via ssh will have become broken again - how can we fix things now? Suddenly we can't fixup permissions as we're not root, and we can't become root since the binaries are no longer setuid.

It seems that our first job is to gain root access. Whilst there are many things we could try (such as emptying the password fields in /etc/passwd - possible now all files are world-writable) first solution to occur to me was to abuse a daemon which is already running and which has permissions to do root-type jobs. In a word: cron.

Since the system has various php4 packages installed we can modify the cronjob which has been installed:

echo "*/1 * * * * root cp /bin/sh /tmp/sh && chmod 4755 /tmp/sh" > /etc/cron.d/php4

Waiting a minute finds us a setuid shell in /tmp and we can use that to become root and fix things as we did previously:

skx@uml:~$ /tmp/sh                                  
sh-3.1# id
uid=1000(skx) gid=1000(skx) euid=0(root) groups=1000(skx)

Having writable files will allow a lot of different attacks against a system, cron is just the most obvious. The important thing is to not panic, and do anything hasty such as rebooting or exiting your root shell!



Re: Rescuing a system with massively broken filesystem permissions
Posted by spiney (85.127.xx.xx) on Thu 21 Dec 2006 at 14:09
Damn, the moment I thought the interesting parts come, the article was over. Not that I was ever in the situation (and I hope I never will), but what would be the quickest way to get the permissions set up properly again? Reinstalling packages fixes a lot, but I'd be interested to know how people would handle this.

Kind regards
Wolfgang Karall
Debian GNU/Linux on an IBM Thinkpad T43p

[ Parent ]

Re: Rescuing a system with massively broken filesystem permissions
Posted by Steve (80.68.xx.xx) on Thu 21 Dec 2006 at 15:02
[ View Weblogs ]

I guess it did just tail off a little, but the hard parts were done. Once you can login as root remotely then you're easily able to do the site-specific fixups from the comfort of your desk.

If it were me I'd just restore from backups which would incidentally fix the permissions!

Still if you have another working system I guess the obvious thing to do is to write a simple script to output a file containing the path + permissions of every file on the system. Then another to read in that file and apply the fixups.

I remember doing that once in the past. Something like this simple script would do the job:

#!/usr/bin/perl -w

use strict;
use File::Find;

find(\&wanted, "/");

sub wanted
    my $file = $File::Find::name;
    my $mode = (stat($file))[2];
    printf "%04o, %s\n", $mode & 077777, $file;

This will give output such as this:

40755, /
0600, /.bash_history
0600, /.viminfo
0600, /.lesshst
0600, /initrd.img.old
0644, /vmlinuz.old
40755, /lost+found
40755, /home

The script to restore those permissions is left as an excercise ;)


[ Parent ]

Re: Rescuing a system with massively broken filesystem permissions
Posted by Anonymous (84.45.xx.xx) on Thu 21 Dec 2006 at 18:02
"It seems that our first job is to gain root access."

Boot from rescue CD, pick unused console, "chroot /mnt/rootfs ; passwd".

Root doesn't have to give the old password to set a new one.

I'm not sure one should rely on having insecure temporary files around the place to become root.

I'm not sure leaving a system with warped file permissions up is a good idea either, but hey Linux and Unix are remarkably robust to such cock-ups.

Perhaps the next article should be on the U-Kuang program, and how to spot when you've left stupid file permissions laying around.

[ Parent ]

Good article.
Posted by elivs (202.78.xx.xx) on Fri 22 Dec 2006 at 17:54
Great article. We all break our systems from time to time !

What I'd like to know is once you have root access is there a simple way to get apt/aptitude to refresh all the permissions in /usr /sbin etc?

The same question applies to the situation where you accedently delete part of /usr. Is there a way to get apt to verify every install package?

<disclaimer> I probably should just use google to answer this question, but if you could post and answer here it would be a good resource for people.<disclaimer>


[ Parent ]

Re: Good article.
Posted by Steve (62.30.xx.xx) on Fri 22 Dec 2006 at 18:03
[ View Weblogs ]

Not directly; although you could run "apt-get install --reinstall [package1] [package2] .." to do it.

Failing that you could install the acl package and use getfacl + setfacl to create/restore from a backup as suggested by mvanbaak already.

In the case of missing files then the "--reinstall" trick might work, but the obvious answer is "restore from backups". If you don't currently have backups you'll have learnt how important they are for the future ;)


[ Parent ]

Re: Rescuing a system with massively broken filesystem permissions
Posted by Anonymous (199.82.xx.xx) on Tue 9 Jan 2007 at 15:02
I did something far far worse then this once, and it was a life lesson on why to use relative paths. I typed "rm -rf /" and then sneezed. I hit control C with a quickness. However, you know what the first things to go where? /usr/bin, /bin, and /sbin. Needless to say I was humped and had to simply rebuild by my box. Luckily the delete did not make it to the partition that kept all of my personal stuff on. My friends still give me grief to this day about that one.

[ Parent ]

Re: Rescuing a system with massively broken filesystem permissions
Posted by tarunbhatia (64.185.xx.xx) on Wed 24 Oct 2007 at 23:14
Thank God for this post otherwise I would have to reinstall the entire OS on my linux box again. However in my case, I had already rebooted my machine and was no longer able to login as root. Fortunately, I had knoppix on a CD. When I booted in to that for some reason knoppix mounted my Hard Drive as a read only. To change the permissions on that, all I had to do was unmount then remount the drive. After that, I just followed the instructions and it has been working perfectly fine so far ever since.

[ Parent ]

Re: extra cleanup afterward ... short of restoring/reinstalling
Posted by Anonymous (71.146.xx.xx) on Thu 21 Feb 2008 at 19:51
I was taking care of a sizable number of Unix systems at work, a few years back, and one of my desktop users who used to do things in strange ways.One day he totally messed up his SunOS 5.x system by a bad chmod -fR
command on the root filesystem.
I wanted to just put the modes back, by duplicating them from a similar
desktop system ... just the items that matched name and location.
I couldn't figure out how to do that using rsync at the time, not sure if
it was feasible.So I wrote a shell script that accomplished it, I don't remember if it was based on cpio or something else. Now I'm thinking that I may have written some perl code that traversed the system, operating on each directory that existed on both sides.Anyhow, I think that it's possible to use rsync to derive just the modes, if you have a second system that you want to copy modes from (assuming that you've fixed the broken system enough to make the basic commands and networking run, and you just want to clean up the rest of the system areas). Larry

[ Parent ]

Re: extra cleanup afterward ... short of restoring/reinstalling
Posted by Anonymous (203.89.xx.xx) on Fri 7 May 2010 at 04:43
Reinstalling: As root
apt-get --reinstall install $(dpkg -S /usr/bin/* | awk ' { print $1 } ' | sort -u | sed -e 's/:$//')

Should do the package reinstall.
I had a remote login with ssh-keygen ability from my master server. Otherwise there would have been no ability to change to root user.

[ Parent ]

Alternative method to (re)gain root access.
Posted by Anonymous (24.132.xx.xx) on Sat 15 Oct 2011 at 22:10
When searching for a solution I also found another trick to re-gain root access at least on Ubuntu.

The command issued was to allow group read / write permissions. So this did really destroy my system, but not so much that I could easily regain root access via the methods mentioned in the post (I had no root shell).

Someone on another forum mentioned pkexec. Apparently pkexec is not that choosy about file permissions and will let me get a root shell as long as I am logged in as a user in the admin group. This allowed me to get root access with the running ssh session.

Thanks for the original post, it helped!

[ Parent ]

Re: Rescuing a system with massively broken filesystem permissions
Posted by Anonymous (92.249.xx.xx) on Sat 23 Jun 2012 at 14:54
find / -type d -print0 2>/dev/null |xargs -0 ls -ald|awk -F " " {'print $9" "$3" "$4" "$1'} >all_perms_dirs
find / -type f -print0 2>/dev/null |xargs -0 ls -al |awk -F " " {'print $9" "$3" "$4" "$1'} >all_perms_files

+ perl script(which make bash script for recovery permissions).

[ Parent ]