logo

 

This site is now 100% read-only, and retired.

How do you manage your SSH host keys?

Posted by Steve on Fri 2 Mar 2007 at 11:28

Tags: ,

When connecting to a new OpenSSH server for the first time you'll be prompted to accept its host key - but how do you know if it is valid? How do you manage SSH keys for multiple machines?

As part of the package installation the openssh, and openssh-server will generate "host key" which is used when communicating with clients.

The first time you connect to a server you'll see a message similar to this:

steve@steve:~$ ssh localhost
The authenticity of host 'localhost (127.0.0.1)' can't be established.
RSA key fingerprint is 2d:d3:29:bd:4d:e2:7d:a3:b0:15:96:26:d4:60:13:34.
Are you sure you want to continue connecting (yes/no)?

This allows you to test that the key is valid, by showing you the a fingerprint (which is a hash of the full key) and prompting you to confirm its validity.

Once accepted this fingerprint will be stored in the file ~/.ssh/knownhosts either in plain text, or if you're running a recent version of OpenSSH it will be stored in a hashed format. (Set "HashKnownHosts no" if you want to disable this behaviour - either in ~/.ssh/options or /etc/ssh/ssh_config.)

The real question now is how do you know whether the key is valid?

Some organisations store OpenSSH fingerprints online so that you may compare what is presented with what is expected, others assume that 99% of people will merely type "yes" when prompted to accept a key - which is pretty dangerous thing to do.

Build up a global fingerprint list

In the same way that details are stored in ~/.ssh/known_hosts, once accepted, the OpenSSH client program may be configured to read a system-wide list of keys and hostnames.

By default the Debian packages are configured to look at the file /etc/ssh/ssh_known_hosts if it exists.

If you were to build up a list of fingerprints used within your LAN, using the ssh-keyscan command in this file then all your users would avoid prompts connecting to your internal hosts.

As a simple example you could run something like this:

ssh-keyscan -t rsa,dsa cfmaster.my.flat >> /etc/ssh/ssh_known_hosts
ssh-keyscan -t rsa,dsa cfmaster         >> /etc/ssh/ssh_known_hosts
ssh-keyscan -t rsa,dsa mine.my.flat     >> /etc/ssh/ssh_known_hosts
ssh-keyscan -t rsa,dsa mine             >> /etc/ssh/ssh_known_hosts
ssh-keyscan -t rsa,dsa yours.my.flat    >> /etc/ssh/ssh_known_hosts
ssh-keyscan -t rsa,dsa yours           >> /etc/ssh/ssh_known_hosts

(Note the use of both "short" and "long" hostnames here; since the machine names must match exactly what the user typed to connect we add both versions. However the users will still be prompted to accept a key if they connect by IP address.)

Store fingerprints in DNS

If you're using a recent version of OpenSSH then it can be configured to check fingerprints against records stored in DNS. This is obviously only an option if you're running your own nameserver, but with the correct options you can see something like this:

steve@steve:~$ ssh localhost -o "VerifyHostKeyDNS=yes"
yes authenticity of host 'localhost (127.0.0.1)' can't be established.
RSA key fingerprint is 2d:d3:29:bd:4d:e2:7d:a3:b0:15:96:26:d4:60:13:34.
Matching host key fingerprint found in DNS.
Are you sure you want to continue connecting (yes/no)?

Here you've not been told that the key is valid and trusted, but you've been told that does match what the DNS server-admin has inserted. By contrast a missing key would look like this:

RSA key fingerprint is 2d:d3:29:bd:4d:e2:7d:a3:b0:15:96:26:d4:60:13:34.
No matching host key fingerprint found in DNS.
Are you sure you want to continue connecting (yes/no)?

Adding the keys to DNS in the first place is a little outside the scope of this introduction/question but you can use software such as sshfp to do the job.

Now that the simple introduction is out of the way here come the questions:

  • How do you manage your SSH fingerprints?
    • Blindly assume that all users will type yes and ignore it?
    • Centralised lists?
    • DNS records?
    • Something else?

 

 

 

#
Re: How do you manage your SSH host keys?
Posted by lee (193.82.xx.xx) on Fri 2 Mar 2007 at 12:50
[ View Weblogs ]
ssh localhost -o "VerifyHostKeyDNS=The"

The? Shouldn't that be ssh localhost -o "VerifyHostKeyDNS ask"

[ Parent ]

#
Re: How do you manage your SSH host keys?
Posted by Steve (80.68.xx.xx) on Fri 2 Mar 2007 at 13:05
[ View Weblogs ]

It should have been "yes" - and I've updated the text now.

Either you set it to "no" in which case DNS isn't consulted and you're prompted to accept/reject as normal or you set it to "yes" and you're still prompted - but you get a message such as:

  • Matching host key fingerprint found in DNS.
  • No matching host key fingerprint found in DNS.

So there is no need for an "ask" setting.

Steve

[ Parent ]

#
Re: How do you manage your SSH host keys?
Posted by Anonymous (12.41.xx.xx) on Fri 2 Mar 2007 at 15:51
I blindly assume that I am not the target of a man-in-the-middle attack and click yes. Of course, the only thing I ever ssh to is my home computer. I wouldn't take this risk if I were using SSH for work.

[ Parent ]

#
Re: How do you manage your SSH host keys?
Posted by JulienV (86.204.xx.xx) on Fri 2 Mar 2007 at 17:41
[ View Weblogs ]
I always click on 'y' for machines in my local network, but try and check the keys when connecting to a remote host (like alioth of SourceForge).

I hadn't heard about SSHFP before, and I wanted to try and set it up for my local network, but just noticed bind9 package in Sarge is too old to support it (it was included in 9.3.0beta1 only). Only too bad - will have to wait for etch ;-)

Cheers,
Julien

[ Parent ]

#
Re: How do you manage your SSH host keys?
Posted by ajt (195.112.xx.xx) on Fri 2 Mar 2007 at 18:41
[ View Weblogs ]
Ditto, SSHFP looks like a useful way of controlling and distributing keys. At the same time it also looks like a way of causing havoc if someone were to tamper with DNS...

--
"It's Not Magic, It's Work"
Adam

[ Parent ]

#
Re: How do you manage your SSH host keys?
Posted by jeroen (62.45.xx.xx) on Fri 2 Mar 2007 at 22:06
With older bind versions you can use numeric types. So instead of "IN SSHFP 1 1" you write "IN TYPE44 \# 22 01 01". ssh-keygen will give you the latter if you pass it the -g option, e.g.

jeroen@dijkstra:~$ ssh-keygen -r dijkstra -f /etc/ssh/ssh_host_rsa_key
dijkstra IN SSHFP 1 1 29ebc6beac2b4c8d3cc4f6088a46348d8376f89d
jeroen@dijkstra:~$ ssh-keygen -r dijkstra -f /etc/ssh/ssh_host_rsa_key -g
dijkstra IN TYPE44 \# 22 01 01 29ebc6beac2b4c8d3cc4f6088a46348d8376f89d

As for creating havoc with DNS tempering, DNSSEC should solve that eventually. But until DNSSEC is deployed, publishing keys in DNS doesn't really add that much security. If somebody is able to do a man-in-the-middle attack on your ssh session there is a big chance that he is also able to give you other DNS records.

[ Parent ]

#
Re: How do you manage your SSH host keys?
Posted by ajt (195.112.xx.xx) on Sun 4 Mar 2007 at 12:38
[ View Weblogs ]
Both Etch and Sarge ssh-keygen seem to generate both kinds of record okay.

The BIND9 in Sarge won't start with the SSHFP records in, but it seems okay with the older TYPE44 option.

However, OpenSSH in Etch/Sid claims that there are no SSH finger prints in DNS even though host -a clearly shows the record I added. Am I missing something?

--
"It's Not Magic, It's Work"
Adam

[ Parent ]

#
Re: How do you manage your SSH host keys?
Posted by JulienV (90.6.xx.xx) on Sun 4 Mar 2007 at 20:14
[ View Weblogs ]
>Both Etch and Sarge ssh-keygen seem to generate both kinds of record okay.

I am just trying to understand everything...
ssh-keygen is limited as it can only read entries from a key file, am I right?

sshfp can read keys from a known_hosts file or use ssh-keyscan to retrieve public keys. But this seems quite unsecure imho, as there could be a man-in-the-middle when retrieving the key.

I have opened an ITP for sshfp, the package is ready, but now, I am not sure it is a good idea to include it into Debian, as it could be dangerous, and ssh-keygen is an alternative (for "secure" features only).

What do you think?

Cheers,
Julien

[ Parent ]

#
Re: How do you manage your SSH host keys?
Posted by ajt (195.112.xx.xx) on Sun 4 Mar 2007 at 21:32
[ View Weblogs ]
On my home boxes I ran ssh-keygen locally per box and then pasted that directly into the DNS record of my server. Being able to generate the correct DNS record remotely would be nice but I don't think it's a must have.

However my problem was that once I got the record into BIND9 (9.2.4), my SSH clients didn't seem to see it, even when instructed to verify signatures against DNS. A quick scan of
/usr/share/doc/openssh-client/README.dns says that the you need to add not only the server key you are interested in to your zone file, but the zone file needs signing also. Serves me right for not reading all the instructions first.

--
"It's Not Magic, It's Work"
Adam

[ Parent ]

#
you could GPG keys instead of raw RSA or DSA
Posted by dkg (216.254.xx.xx) on Fri 2 Mar 2007 at 18:20
[ View Weblogs ]
One proposal advanced a while ago was to use GPG keys instead of raw RSA or DSA keys.

Then each host would have its own key (e.g. SSH Host Key For host.domain.com <ssh@host.domain.com>), which would be signed by the relevant administrators (and/or a domain-wide key made specifically for this purpose). If you have an account on that machine, you would presumably already know the operator (because you've arranged with them to get an account), so your ssh client could verify the the signatures on the key using GPG directly.

Other methods of verifying keys programmatically include X509 integration and GSSAPI support (GSSAPI is currently patched and enabled in the etch builds of openssh).

I think the GPG integration provides the best method theoretically for a number of reasons:

  • you get programmatic verification of the keys, as you also do with X509 and GSSAPI
  • There is a well-defined method for key expiration and renewal. There is also a clear method for key revocation in the event of a compromise, disk failure, or other reason for key change: publish the key's revocation certificate to public keyservers. GSSAPI and X509 also have good, established methods for revocation, though it's not clear to me how many folks actually use CRLs (let alone OCSP) for X509.
  • No one needs to run a KDC (GSSAPI) or an official Certificate Authority (X509) in this model, though having publicly-accessible keyservers is important for functional revocation.
  • GPG provides multiple angles for authentication. If a server has multiple administrators (or committee members, or resellers, or whatever, depending on your political approach to management), each admin can sign the server's public key. Then your arrangement for an account on the machine with one admin will work fine independent of the other admins, so long as you trust that admin to identify the machine properly (which you'll need to do to start using it anyway).
  • It might encourage people to start thinking more about gpg and actually using it (OK, that's a side benefit unrelated to ssh)
Sadly, the existing openssh-gpg patchset is out of date (last applied to 3.6p1). The architecture proposed is a nice one, though: it looks like the author has cleanly separated the tools across process boundaries, so there should be no trouble with the incompatible licensing schemes.

One final addition that would be good would be to do GPG verification the other way around: hosts verifying users. That is, provide sshd with its own keyring, and a clean, small set of trusted users (say, the admins on the machine), and then replace ~/.ssh/authorized_keys with ~/.ssh/authorized_gpg_uids. Then each admin wouldn't need to provide a password to the end user at all. Through their pre-existing communication, they would have verified and signed the GPG key of the user. That signature would be presented to the sshd who could use it to verify the uid and match it against a line in ~/.ssh/authorized_gpg_uids

If only i had the time to work on coaxing the patchset into compatibility with modern versions of GPG and OpenSSH, or re-implementing it myself...

[ Parent ]

#
Re: How do you manage your SSH host keys?
Posted by cparker (204.136.xx.xx) on Fri 2 Mar 2007 at 18:45
[ View Weblogs ]
When connecting to a new host, I always get the key via some means before connecting for the first name and I add it to my known_hosts.

[ Parent ]

#
Re: How do you manage your SSH host keys?
Posted by dkg (216.254.xx.xx) on Fri 2 Mar 2007 at 19:03
[ View Weblogs ]
If you're using a recent version of OpenSSH then it can be configured to check fingerprints against TXT records stored in DNS.
Is it really TXT records? I've never set this up, but i'd like to, and the only relevant RFC i found seem to claim that this is a separate DNS type, SSHFP, not a TXT record.

Do you have a tool of choice for converting these records into tinydns custom format?

[ Parent ]

#
Re: How do you manage your SSH host keys?
Posted by Steve (62.30.xx.xx) on Fri 2 Mar 2007 at 19:46
[ View Weblogs ]

Sorry you're correct. It isn't TXT records, I was misremembering SPF.

Unfortunately I don't have a great link handy. I swear I bookmarked an online site for pasting in keys which would generate the magical entries - but I can't find it at the moment.

I'll followup once I stumble upon it again.

Steve

[ Parent ]

#
Re: How do you manage your SSH host keys?
Posted by dkg (216.254.xx.xx) on Fri 2 Mar 2007 at 20:07
[ View Weblogs ]
On etch, at least, it looks like you can generate a bind-compatible RR with: 
ssh-keygen -r $(hostname --fqdn) -f /etc/ssh/ssh_host_rsa_key.pub
and a generic RR just by adding the -g flag.

And I just found this, which relies on gawk, but it doesn't quite work for me. After playing with it a bit more, the following gawk script works OK:

#!/usr/bin/gawk -f
# taken from: http://msgs.securepoint.com/cgi-bin/get/djbdns-0702/5.html
# modified by Daniel Kahn Gillmor <dkg-debian.org@fifthhorseman.net>

# example: ssh_host_key_to_sshfp $(hostname --fqdn) </etc/ssh/ssh_host_rsa_key.pub
BEGIN {
   hostname=ARGV[1]
   keygen="ssh-keygen -gr " hostname " -f /dev/stdin"
   while(keygen | getline) {
      if($8) {
         rr=sprintf(":%s:44:\\%03.3o\\%03.3o", hostname, $6, $7)
         for(i=1; i<length($8); i+=2) {
            rr=sprintf("%s\\%03.3o", rr, strtonum("0x" substr($8, i, 2)))
         }
         print "# " $0
         print rr ":::"
      }
   }
}
As the comment says, run this script like this: 
ssh_host_key_to_sshfp $(hostname --fqdn) </etc/ssh/ssh_host_rsa_key.pub

[ Parent ]

#
Re: How do you manage your SSH host keys?
Posted by JulienV (86.204.xx.xx) on Fri 2 Mar 2007 at 21:03
[ View Weblogs ]
sshfp linked in the article seems to be very good, reading entries from known_hosts files, or using ssh-keyscan to gather public keys directly from hosts. It sends its output to stdout (or, as an option, to a file), eg: 
./sshfp -s www.debian-administration.org 
www.debian-administration.org IN SSHFP 1 1 6fe7cfbc539c2488193dee45ce661126a33d01c
www.debian-administration.org IN SSHFP 2 1 e58a8ccd12c06997ed1b68f8f1b0fedcf72375b
Directly usable in Bind if I am right. Cheers, Julien

[ Parent ]

#
Re: How do you manage your SSH host keys?
Posted by Tolaris (82.163.xx.xx) on Sat 3 Mar 2007 at 12:20
I have packaged sshfp. I've tested it in Ubuntu dapper and edgy, without problems. It should work in any Debian-based distribution, on any machine with python (package has correct dependencies for it).

Either add one of these lines to your sources.list and "apt-get update; apt-get install sshfp", or fetch the package directly and use dpkg -i on it. I haven't signed it.

deb http://www.tolaris.com ubuntu edgy
deb http://www.tolaris.com ubuntu dapper

http://www.tolaris.com/dists/ubuntu/edgy/binary-i386/sshfp-1.1.1. deb

Thanks to the SSHFP guys for making this tool.

[ Parent ]

#
Re: How do you manage your SSH host keys?
Posted by JulienV (90.6.xx.xx) on Sat 3 Mar 2007 at 12:26
[ View Weblogs ]
I wanted to do this job this afternoon for Debian ;-)

Not that the installation of the upstream tarball causes issues...

Would you please provide your source package so that I can have a look at them? I will take them as a basis to build packages for Debian and upload them to my unofficial repository.

Cheers,
Julien

[ Parent ]

#
Re: How do you manage your SSH host keys?
Posted by Tolaris (82.163.xx.xx) on Sat 3 Mar 2007 at 13:15
This is when I reveal myself to be a complete amateur. :)

There is no source. It's python, so no compilation needed, and the only file you need to make the .deb is the control file. I simply have a tree that I compiled the .deb from. You can recreate it with these commands:

dpkg -X sshfp-1.1.1.deb /tmp/sshfp
dpkg -e sshfp-1.1.1.deb /tmp/sshfp/DEBIAN

Make your changes within /tmp/sshfp, then rebuild:

dpkg -b /tmp/sshfp sshfp-1.1.1.deb

I have only made a few .debs myself. I am aware that there are more advanced tools to do it, but I have not learned how to use them. Any recommendations you could make would be appreciated.

If you feel the urge to become the maintainer of the package, please do so. I need no credit. :)

[ Parent ]

#
Re: How do you manage your SSH host keys?
Posted by Anonymous (63.249.xx.xx) on Sun 4 Mar 2007 at 21:18
Almost directly usable. I added the output from the script to my zone file - but queries failed. Looked and looked, then I finally noticed that it left off the trailing . at the end of the hostnames.

[ Parent ]

#
Re: How do you manage your SSH host keys?
Posted by barbarossa (89.245.xx.xx) on Mon 19 Mar 2007 at 00:04
Hi!

I have written some pice of software for this purpose:

It is called ssh-keysync and is divided into two parts:
1. A ssh-keysync server, which has all the ssh-{d,r}sa.pub key files of all machines and merges them all together to a global ssh_known_hosts file.
(It can handle multiple domains and is kind of client-capable).
This process is run every hour by default. The gathering of the client keys could be included in the installation process (think of FAI).

The ssh_known_hosts file is placed in a public folder which could be shared via an webserver, all client have access too.

2. A ssh-keysync client does just pull the "compiled" ssh_known_hosts file to /etc/ssh_known_hosts on the client.

Debian packages are available.

See http://www.rfc2324.org/projects/ssh-keysync for APT information and more details

Comments welcome.

--
Follow the white penguin.

[ Parent ]

#
Re: How do you manage your SSH host keys?
Posted by Anonymous (212.15.xx.xx) on Sun 6 May 2007 at 13:32
Puppet configuration management system (www.reductivelabs.com)
also has support for generating and distributing keys.

[ Parent ]

#
Re: How do you manage your SSH host keys?
Posted by Anonymous (166.84.xx.xx) on Tue 7 May 2013 at 14:06
Shouldn't SSH host key fingerprints be delivered by SSL or other secure medium?

[ Parent ]

#
Re: How do you manage your SSH host keys?
Posted by Anonymous (82.68.xx.xx) on Mon 3 Mar 2014 at 23:00
Nop, it's very public and should be.

[ Parent ]