December 2014

Please note that republishing this article in full or in part is only allowed under the conditions described here.

Quantifying the Quality of SMTP TLS Support

This study analyzes, how many mail server offer a TLS upgrade and examines the quality of the STARTTLS implementation or setup. It checks the quality of protocol version and ciphers, but also determines how widespread are some setup or implementation errors I've seen in practice. To make the results somewhat more relevant it uses the top 1 million domains based on the studies by Alexa. It also provides the necessary tools to do such study with your own domain list.

Some background to mail delivery

Mail is transferred from sender to recipient with multiple hops. The initial hop is usually from senders MUA (Mail User Agent) to the MTA (Mail Transfer Agent) of the senders domain using the SMTP (Simple Mail Transfer Protocol) protocol. Then there is at least on hop from there to the MTA of the recipients domain, but in more complex setups like in companies there can be multiple hops or there can be some middleboxes like firewalls in-between.

While the MTA for the sender is usually configured within the senders MUA, the MTA for the recipients domain is determined by looking at the MX (Mail Exchange) records in the DNS (Domain Name System). Per definition the recipients MTA is contacted by plain, unencrypted SMTP on port 25. Only after issuing a STARTTLS command the connection can be upgraded to TLS.

What this study is not about

This study does not check if the recipients MTA provides a valid certificate, which matches the recipients domain and is signed by a trusted certificate agency. Such certificates are needed with do detect active man-in-the-middle attacks. Unfortunatly, due to the overall design of the STARTTLS process an active attacker can just simply strip the STARTTLS support from the communication instead of mounting the much harder man-in-the-middle attack against TLS. Thus STARTTLS like currently used is only able to protect the connection against passive sniffing and not active attacks.

What happens if the initial TLS upgrade fails?

Contrary to most desktop browsers MTA will usually not retry with a downgraded TLS protocol or without TLS later, but will simply retry the same way again at some later time. Thus the delivery will fail again and usually after some days the mail is finally considered undeliverable. Unfortunatly, all one will get as a feedback is usually a "TLS handshake failed" message which most system administrators will either ignore or will need to explicitly add an delivery exception for this specific domain.

Summary of results

Problems seen in practice

Even if the large majority of the mail servers are not broken, there are still enough servers out there which cause real problems. Only about 30% of the servers I had problem reports for were even in the top 1M list and even then at the lower end of the list, where the most problems (apart from MD5 client certificates) can be seen.

Croaks on client certificate signed with MD5

Lots of sites (9% of the top 5k, 4% of all) simply close the connection if the client provides an client certificate signed with MD5 within a TLS1.2 connection. This problem does not happen if the certificate is signed with SHA-1 or others signature algorithms, or if a TLS protocol lower than TLS1.2 is used. Interestingly it also works if the client does not provide a certificate at all, because the server only requests but not requires client certificates. And while MD5 signatures should not be used anymore it is not uncommon that they are just kept if older systems get upgraded.

This problem shows up with sites where *.mail.protection.outlook.com is configured as MX. It can also be seen with other sites and often a hint to some "AntiSpamProxy" can be seen inside the servers certificate. Because of the strange symptoms the underlying cause of delivery errors is hard to find unless one specifically looks out for this problem.

Only TLS1.2 ciphers

Due to a misunderstanding of the POODLE attack some sites disabled all SSL3.0 ciphers instead of only disabling the SSL3.0 protocol version. Since SSL3.0 ciphers are also needed for TLS1.0 and TLS1.1 and only TLS1.2 defines some new ciphers, this effectivly makes these server TLS1.2 only and thus inaccesible for about 30..40% of the mail server which can do only TLS1.0. While not very common this error is much more prominent among the less used sites.

TLS1.1+ required

Some sites only allow TLS1.1+ for some yet unknown reason. One reason is the disabling of SSL3.0 ciphers as described in the previous issue, but there seem to be other causes too. The impact is about the same as with the previous problems, that is it makes the server inaccessible for 30..40% of the MTAs's worldwide.

Detailed results

The test was done with the mx_starttls_bulk Perl program and the summary was created with another Perl program and some editing.

----------------------------------------------------------------------------------------------------
                          |                |                  |                  | 
Total domains:            |       5000     |      20000       |     100000       |     998549
                          |                |                  |                  | 
----------------------------------------------------------------------------------------------------
                          |                |                  |                  | 
No MX record:             |   647 (12.9%)  |   2762 (13.8%)   |  16793 (16.8%)   |  161555 (16.2%)
No STARTTLS:              |  1214 (24.3%)  |   4295 (21.5%)   |  21124 (21.1%)   |  199902 (20%)
TLS available:            |  3058 (61.2%)  |  12338 (61.7%)   |  58101 (58.1%)   |  601647 (60.3%)
                          |                |                  |                  | 
----------------------------------------------------------------------------------------------------
Relative to TLS sites:    |                |                  |                  | 
TLS initial success:      |  3058 (100%)   |  12335 (100%)    |  58080 (100%)    |  601451 (100%)
TLS after downgrade:      |     0 (0%)     |      3 (0.024%)  |     16 (0.028%)  |     157 (0.026%)
TLS total fail:           |     0 (0%)     |      0 (0%)      |      4 (0.0069%) |      32 (0.0053%)
                          |                |                  |                  | 
----------------------------------------------------------------------------------------------------
TLS common problems:      |                |                  |                  |           
TLS1.1+ required:         |     1 (0.033%) |      1 (0.0081%) |      4 (0.0069%) |      83 (0.014%)
Only TLS1.2 ciphers:      |     0 (0%)     |      0 (0%)      |      4 (0.0069%) |      84 (0.014%)
Croak on MD5 client cert: |   261 (8.53%)  |    792 (6.42%)   |   3120 (5.37%)   |   22810 (3.79%)
                          |                |                  |                  | 
----------------------------------------------------------------------------------------------------
TLS protocols used:       |                |                  |                  |           
TLS1.2:                   |  1798 (58.8%)  |   7847 (63.6%)   |  38401 (66.1%)   |  410073 (68.2%)
TLS1.1:                   |     4 (0.13%)  |     17 (0.14%)   |    106 (0.18%)   |    1245 (0.21%)
TLS1.0:                   |  1255 (41%)    |   4473 (36.3%)   |  19586 (33.7%)   |  190261 (31.6%)
SSL3.0:                   |     1 (0.033%) |      1 (0.0081%) |      3 (0.0052%) |      29 (0.0048%)
                          |                |                  |                  | 
----------------------------------------------------------------------------------------------------
Ciphers used:             |                |                  |                  |           
ECDHE-* (FS)              |  1408 (46%)    |   5836 (47.3%)   |  24879 (42.8%)   |  197756 (32.9%)
DHE-* (FS)                |  1036 (33.9%)  |   4270 (34.6%)   |  23426 (40.3%)   |  310913 (51.7%)
RC4 (bad)                 |    19 (0.62%)  |    100 (0.81%)   |    510 (0.88%)   |    8150 (1.35%)