Weblog entry #103 for dkg

Posted by dkg on Wed 30 Oct 2013 at 17:00
Many protocols today allow you to upgrade to TLS from within a cleartext version of the protocol. This often falls under the rubric of "STARTTLS", though different protocols have different ways of doing it.

I often forget the exact steps, and when i'm debugging a TLS connection (e.g. with tools like gnutls-cli) i need to poke a remote peer into being ready for a TLS handshake. So i'm noting the different mechanisms here. lines starting with C: are from the client, lines starting with S: are from the server.

many of these are (roughly) built into openssl s_client, using the -starttls option. Sometimes this doesn't work because the handshake needs tuning for a given server; other times you want to do this with a different TLS library. To use the techniques below with gnutls-cli from the gnutls-bin package, just provide the --starttls argument (and the appropriate --port XXX argument), and then hit Ctrl+D when you think it's ok to start the TLS negotiation.


The polite SMTP handshake (on port 25 or port 587) that negotiates a TLS upgrade looks like:
C: EHLO myhostname.example
S: [...]
S: [...]
S: 250 [somefeature]
S: 220 2.0.0 Ready to start TLS
<Client can begin TLS handshake>


The polite IMAP handshake (on port 143) that negotiates a TLS upgrade looks like:
S: OK [CAPABILITY IMAP4rev1 [...] STARTTLS [...]] [...]
S: A OK Begin TLS negotiation now
<Client can begin TLS handshake>


The polite POP handshake (on port 110) that negotiates a TLS upgrade looks like:
S: +OK POP3 ready
S: +OK Begin TLS 
<Client can begin TLS handshake>


The polite XMPP handshake (on port 5222 for client-to-server, or port 5269 for server-to-server) that negiotiates a TLS upgrade looks something like (note that the domain requested needs to be the right one):
C: <?xml version="1.0"?><stream:stream to="example.net"
C:  xmlns="jabber:client" xmlns:stream="http://etherx.jabber.org/streams" version="1.0">
S: <?xml version='1.0'?>
S: <stream:stream
S:  xmlns:db='jabber:server:dialback'
S:  xmlns:stream='http://etherx.jabber.org/streams'
S:  version='1.0'
S:  from='example.net'
S:  id='d34edc7c-22bd-44b3-9dba-8162da5b5e72'
S:  xml:lang='en'
S:  xmlns='jabber:server'>
S: <stream:features>
S: <dialback xmlns='urn:xmpp:features:dialback'/>
S: <starttls xmlns='urn:ietf:params:xml:ns:xmpp-tls'/>
S: </stream:features>
C: <starttls xmlns="urn:ietf:params:xml:ns:xmpp-tls" id="1"/>
S: <proceed xmlns='urn:ietf:params:xml:ns:xmpp-tls'/>
<Client can begin TLS handshake>


RogerBW (in the comments below) points out that NNTP has TLS support:
S: [...]
S: [...]
S: .
S: 382 Continue with TLS negotiation
<Client can begin TLS handshake>


I got mail from James Cloos suggesting how to negotiate an upgrade to TLS over the PostgreSQL RDBMS. He points to the protocol docs, and in particular, to multiple protocol flow documents, and SSLRequest and StartupMessage chunks of the protocol spec (and clarification that data is sent in network byte order). It won't work in a text-mode communication, but it's worth noting here anyway:

The client starts by sending these eight octets:

0x00 0x00 0x00 0x08 0x04 0xD2 0x16 0x2F
and the server replies with 'S' for secure or 'N' for not. If the reply is S, TLS negotiation follows.

The message represents int32(8) specifying that there are 8 octets and int16(1234),int16(5678). All sent in network order.

(The non-TLS case starts with a similar message with int16(3),int16(0) for protocol version 3.0. Starttls is essentially pg protocol version 1234.5678.)

what else?

I don't know (but would like to) how to do:

  • mysql TLS negotiation
  • other reasonable network protocols capable of upgrade
  • other free TLS wrapping tools like openssl s_client or gnutls-cli that can start off in the clear and negotiate to TLS. I am trying to get libNSS's tstclnt into the libnss3-tools package, but that hasn't happened yet.
If you know other mechanisms, or see bugs with the simple handshakes i've posted above, please let me know either by e-mail or on the comments here.

Other interesting notes: RFC 2817, a not-widely-supported mechanism for upgrading to TLS in the middle of a normal HTTP session.


Comments on this Entry

Posted by Anonymous (209.144.xx.xx) on Wed 30 Oct 2013 at 19:30
LDAP is a personal favorite of mine that needs TLS capability. Also, SIP (possibly via DTLS).

LDAP does it as part of a ldap operation (""). I don't know about SIP, sorry.

[ Parent | Reply to this comment ]

Posted by Anonymous (70.83.xx.xx) on Mon 13 Apr 2015 at 18:41
My findings get me exactly there, too. The OID "" (LDAP_START_TLS_OID) is requested by the client in an extended request.

See this RFC for an in-detail description of this operation:

Then, the server should send an extended response with resultcode == 0, and empty matchedDN and errorMessage fields. This means you can then start the TLS handshake.

Since the LDAP protocol is not a text protocol, you can't easily use gnutls-cli's trick to make it start the handshake at the right moment, which is too bad.
I'm wondering, though, if it could still be achieved by hooking up a script to gnutls-cli's stdin/out that blurts out the expected numerical packet structure, then verifies that it gets the correct answer and sends a signal to the gnutls-cli's process.

[ Parent | Reply to this comment ]

Posted by Anonymous (81.187.xx.xx) on Thu 31 Oct 2013 at 22:41
NNTP/TLS isn't widely supported, but RFC4642 has details. In short:

S: [...]
S: [...]
S: .
S: 382 Continue with TLS negotiation

-- RogerBW

[ Parent | Reply to this comment ]

Posted by dkg (212.110.xx.xx) on Thu 31 Oct 2013 at 22:51
[ View dkg's Scratchpad | View Weblogs ]
great, thanks, Roger!

[ Parent | Reply to this comment ]

Posted by LeLutin (96.127.xx.xx) on Tue 2 Jun 2015 at 04:50
Mysql TLS connections (as unreliable as they are -- because of their implementation in both server and client), are initiated by sending a specially crafted packet as first response to the server.

The MySQL protocol is not ASCII-based, however, so if you're using gnutls-cli for establishing a crypted connection, you'll have to find a way to send the correct binary data before htting Ctrl-D.

Documentation that's most useful for understanding how to establish a TLS communication is:


and an example of packets here:


[ Parent | Reply to this comment ]

Posted by LeLutin (96.127.xx.xx) on Tue 2 Jun 2015 at 05:18

PostgreSQL is another candidate for StartTLS-like connections. The protocol is not ASCII-based, but starts out in plain text and client can request for a TLS handshake to happen.

The most comprehensive description I've found of how this is done was this:

https://github.com/brianc/node-postgres/issues/25#issuecomment-75 42143

So, client sends an SSLRequest packet instead of a StartupMessage, then server responds with one byte:

  • an 'S' means that the server is ready for the SSL handshake
  • an 'N' means that it's not able to initiate a handshake.

The above post mentions documentation from Postgres. This documentation shows that the description in the post is accurate (see section "SSL Session Encryption" at the end of the page):


[ Parent | Reply to this comment ]

Posted by LeLutin (96.127.xx.xx) on Fri 30 Oct 2015 at 08:30

Sieve is another text protocol that uses STARTTLS for initiating encrypted connexions.

You can use gnutls-cli --starttls -p 4190 your.mailserver.tld to connect to the sieve service, then type STARTTLS and the server should answer OK "Begin TLS negotiation now." and then you can hit Ctrl-D to start the TLS handshake.

For further testing the service, you can start by logging in with:


Here the string before it is encoded to base64 should be \0username\0password. You can obtain the encoded string that you'll use in the command above by issuing this command:

printf "\0joebob@example.com\0ilovemycat33" | base64

Once logged in you can try to list scripts for the account and get their contents:

"main" ACTIVE
OK "Listscripts completed."
{<some number that I believe represents the script length>}
contents are shown here...

Once you're done testing you can issue a command to log out:

OK "Logout completed."
- Peer has closed the GnuTLS connection

This info comes mostly from the following documentation page: http://wiki2.dovecot.org/Pigeonhole/ManageSieve/Troubleshooting

[ Parent | Reply to this comment ]

Posted by dkg (118.243.xx.xx) on Fri 30 Oct 2015 at 10:12
[ View dkg's Scratchpad | View Weblogs ]
Sweet, thanks for the added info, LeLutin!

[ Parent | Reply to this comment ]