This site is now 100% read-only, and retired.

How many hours did you spend updating systems made vulnerable by the Debian OpenSSL bug(DSA-1571)?

Submitted by emeitner on Tue 3 Jun 2008

Tags: none.
None  <-> 17% 389 votes
less than 1 hour  <-> 36% 815 votes
1-5 hours  <-> 25% 555 votes
6-10 hours  <-> 8% 182 votes
11-20 hours  <-> 3% 82 votes
21-30 hours  <-> 1% 38 votes
31-40 hours  <-> 0% 17 votes
more than 40 hours  <-> 6% 138 votes
Total 2218 votes

 

 

 

Re: How many hours did you spend updating systems made vulnerable by the De
Posted by debianite (193.137.xx.xx) on Tue 3 Jun 2008 at 16:56
Fortunately i only had 2 severs and 5 workstations suffering from that vulnerability.

[ Parent ]

Re: How many hours did you spend updating systems made vulnerable by the De
Posted by Anonymous (78.102.xx.xx) on Tue 3 Jun 2008 at 20:14
cfengine rules :-)

[ Parent ]

Re: How many hours did you spend updating systems made vulnerable by the De
Posted by Steve (82.41.xx.xx) on Wed 4 Jun 2008 at 11:47
[ View Weblogs ]

I can only concur.

But despite having CFEngine setup on 100-150 machines there were still many hours spent testing things, and the ongoing time spent validating and accepting new SSH keys when re-connecting to updated machines.

Steve

[ Parent ]

Re: How many hours did you spend updating systems made vulnerable by the De
Posted by Anonymous (84.105.xx.xx) on Tue 3 Jun 2008 at 20:43
This really was a bad one. Updating the systems wasn't our biggest problem. That took like 10 minutes for all of our servers. But finding and replacing all of our ssh-keys proved to be a bigger problem.

This all had one positive side tho... While replacing all the keys and testing the scripts again I also found some bugs which I then fixed ;-)

Roedie

[ Parent ]

Re: How many hours did you spend updating systems made vulnerable by the De
Posted by endecotp (86.6.xx.xx) on Wed 4 Jun 2008 at 00:12
[ View Weblogs ]
An excellent idea for a poll. I look forward to all the extrapolations... how many Debian systems are there in total?

My main worry was to need to promptly advise users, and also former users who may still have vulnerable keys forgotten about in authorized_keys files on their servers, that they needed to take action.

The next most painful bit was dealing with SSL certificates. This is one of those subjects that I have to re-learn every time I deal with it, and it probably took me half a day or so to be certain that I had made the right changes. You might also like to ask people how much they had to pay for new certificates.

Worryingly, as far as I can see only about one user in three has actually acted on my email telling them that their keys were vulnerable. If this is generally true, then there are a hell of a lot of vulnerable systems still out there. I have not yet seem any attacks attempting to exploit this - has anyone? - which surprises me, since I see a lot of ssh password attacks. If such attacks do start, I think many machines will be compromised.

I do also worry that all those people who've answered "less than 1 hour" here may have not thoroughly understood the implications of this situation. Do those people just have a single machine, not internet-facing?

[ Parent ]

Re: How many hours did you spend updating systems made vulnerable by the De
Posted by Anonymous (81.193.xx.xx) on Wed 4 Jun 2008 at 00:41
I have 6 machines, two of them servers, affected by this problem. Still took me less than 1 hour and I'm sure I understood the problem.

I do,however, wonder about the 40+ hours. Are we talking about hundreds of machines?

[ Parent ]

Re: How many hours did you spend updating systems made vulnerable by the De
Posted by Anonymous (87.194.xx.xx) on Tue 24 Jun 2008 at 12:38
We are talking about updating the openSSL packages PLUS replacing/updating all authorized_keys, known_hosts, reissuing SSL certificates, testing.

[ Parent ]

Re: How many hours did you spend updating systems made vulnerable by the De
Posted by Anonymous (12.41.xx.xx) on Wed 4 Jun 2008 at 22:25
I own three machines (only two of them can be reached over the Internet), and it took me (just barely) less than one hour, total. I'm sure I understand the issues.

[ Parent ]

Re: How many hours did you spend updating systems made vulnerable by the De
Posted by emeitner (216.170.xx.xx) on Wed 4 Jun 2008 at 03:33
[ View Weblogs ]
My guess was 10-15 hours. I had 7 servers affected(2 clusters of two nodes, plus three others). Four other Sarge servers had public keys that needed replacing. Lots of SSH keys used for automated systems needed replacing and the systems needed to be tested to ensure nothing was missed. Two SSL certificates for public web servers needed to be reissued. Fortunately I am the only person who actively uses SSH in our organization so there were few interruptions for users.

[ Parent ]

Re: How many hours did you spend updating systems made vulnerable by the De
Posted by Anonymous (67.88.xx.xx) on Thu 5 Jun 2008 at 17:45
Sarge was not effected. Any keys you made with Sarge and before should be fine. Not sure why you had to change those? Because we are so slow to upgrade we pretty much got lucky since the majority of what we run is Sarge.

http://www.us.debian.org/security/2008/dsa-1571
"The first vulnerable version, 0.9.8c-1, was uploaded to the unstable distribution on 2006-09-17, and has since that date propagated to the testing and current stable (etch) distributions. The old stable distribution (sarge) is not affected."

[ Parent ]

Re: How many hours did you spend updating systems made vulnerable by the De
Posted by emeitner (69.129.xx.xx) on Fri 6 Jun 2008 at 04:17
[ View Weblogs ]
Sure, but the sarge servers had accounts with compromised public keys in ~/.ssh/authorized_keys. These needded to be replaced.

[ Parent ]

Re: How many hours did you spend updating systems made vulnerable by the De
Posted by Anonymous (217.199.xx.xx) on Sat 28 Jun 2008 at 21:31
Also from a security POV you need to consider any key compromised that ever connected to a box that had the openSSL vulnerability.

[ Parent ]

Re: How many hours did you spend updating systems made vulnerable by the De
Posted by ajt (204.193.xx.xx) on Fri 6 Jun 2008 at 13:35
[ View Weblogs ]
I had to upgrade three desktops, a laptop and two servers. The upgrade was almost automatic and didn't take any effort. My SSH keys were actually generated by PuTTY not OpenSSH so in theory they were okay - turns out a lot of people in my LUG were in the same situation.

I did take the opportunity to replace all my keys with strong 2048-bit keys, so except the machines that were switched off at the time, it was all done within a few hours.

--
"It's Not Magic, It's Work"
Adam

[ Parent ]

Re: How many hours did you spend updating systems made vulnerable by the De
Posted by atrixnet (69.152.xx.xx) on Thu 12 Jun 2008 at 19:02
[ View Weblogs ]
Oh geez, hundreds of servers on dozens of networks... this was a killer. Good thing I could script much of it.

[ Parent ]

Re: How many hours did you spend updating systems made vulnerable by the De
Posted by Anonymous (125.236.xx.xx) on Thu 19 Jun 2008 at 11:15
I only spent a couple of minutes on it. I only have one (affected) Debian machine (Ubuntu) which is largely just used as a workstation for myself.
I wasn't thorough and I likely missed one or two of the things (yeah I am very slack). SSH is fine though, and even that can't be reached from the internet. I'll finish the job one of these days.

[ Parent ]

Re: How many hours did you spend updating systems made vulnerable by the De
Posted by Anonymous (80.94.xx.xx) on Wed 25 Jun 2008 at 20:17
Between 6 and 10 hours . Too much servers...

[ Parent ]

Re: How many hours did you spend updating systems made vulnerable by the De
Posted by Anonymous (220.233.xx.xx) on Fri 27 Jun 2008 at 08:39
So glad I only had a total of 5 servers and 2 or 3 desktops effected. And of those, 0 critical issues relating to it require testing. :-)

The only problem I hit was I had to regenerate keys for logging into my home server from my home and work desktops... Total time of about 30 minutes because I'd forgotten how to do it.

[ Parent ]