#4598 fedorahosted https handshake problem
Closed: Fixed None Opened 9 years ago by jasonnet.

= bug description =
From certain machines curl, wget and FireFox all complain when trying to access https urls like
https://admin.fedoraproject.org, https://fedorahosted.org
Yet https://koji.fedorahosted.org and https://kojipkgs.fedoraproject.org work or get further in the handshake process. So far I've found
no other sites that these clients fail on in this way. Visiting sites
like https://yahoo.com, https://google.com, etc all work fine.
Firefox provides a bit more diagnostics than the others and mentions
"ssl_error_no_cypher_overlap".
I've tested about 15 diverse client machines . The only common
distinguishing feature in common with the failing clients is
that they are in a certain data center. They use different
versions of CentOS, but all are up to date.

= bug analysis =

= fix recommendation =


Replying to [ticket:4598 jasonnet]:

This thread might be helpful. It documents recent changes to some of the mentioned sites:

http://comments.gmane.org/gmane.linux.redhat.fedora.devel/200274

Comments by Kevin there might suggest why it behaves differently on the koji sites.

Hi,

Could you tell me which versions of CentOS you're running, and for one server on each version send the output of the "openssl ciphers" command?

Thanks,
Patrick

Also, please give the output of the following command from one of the machines that fail: "openssl s_client -showcerts -debug -msg -state -connect fedoraproject.org:443"

Replying to [comment:2 puiterwijk]:

Hi,

Could you tell me which versions of CentOS you're running, and for one server on each version send the output of the "openssl ciphers" command?

Thanks,
Patrick

  • (1bad) One of the failing machines is CentOS Linux release 7.0.1406 fresh install. So is one (1good) of the working VM's in a different data center.
  • (2bad) Another failing machine is CentOS 6.6 (Final). So are two (2goodA, 2goodB) of the working VM's in a third and fourth data center and another working one at home (2goodHome).
  • (3bad) Another failing machine is CentOS 6.5 (Final). I don't have another one of those to test outside the failing data center.
  • (4) I can test more machines if you like.

openssl cipher Output:
(1bad) ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-DSS-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA256:DHE-RSA-AES256-SHA:DHE-DSS-AES256-SHA:DHE-RSA-CAMELLIA256-SHA:DHE-DSS-CAMELLIA256-SHA:ECDH-RSA-AES256-GCM-SHA384:ECDH-ECDSA-AES256-GCM-SHA384:ECDH-RSA-AES256-SHA384:ECDH-ECDSA-AES256-SHA384:ECDH-RSA-AES256-SHA:ECDH-ECDSA-AES256-SHA:AES256-GCM-SHA384:AES256-SHA256:AES256-SHA:CAMELLIA256-SHA:PSK-AES256-CBC-SHA:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:DHE-DSS-AES128-GCM-SHA256:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES128-SHA256:DHE-DSS-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA:ECDHE-RSA-DES-CBC3-SHA:ECDHE-ECDSA-DES-CBC3-SHA:DHE-RSA-SEED-SHA:DHE-DSS-SEED-SHA:DHE-RSA-CAMELLIA128-SHA:DHE-DSS-CAMELLIA128-SHA:EDH-RSA-DES-CBC3-SHA:EDH-DSS-DES-CBC3-SHA:ECDH-RSA-AES128-GCM-SHA256:ECDH-ECDSA-AES128-GCM-SHA256:ECDH-RSA-AES128-SHA256:ECDH-ECDSA-AES128-SHA256:ECDH-RSA-AES128-SHA:ECDH-ECDSA-AES128-SHA:ECDH-RSA-DES-CBC3-SHA:ECDH-ECDSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES128-SHA256:AES128-SHA:SEED-SHA:CAMELLIA128-SHA:DES-CBC3-SHA:IDEA-CBC-SHA:PSK-AES128-CBC-SHA:PSK-3DES-EDE-CBC-SHA:KRB5-IDEA-CBC-SHA:KRB5-DES-CBC3-SHA:KRB5-IDEA-CBC-MD5:KRB5-DES-CBC3-MD5:ECDHE-RSA-RC4-SHA:ECDHE-ECDSA-RC4-SHA:ECDH-RSA-RC4-SHA:ECDH-ECDSA-RC4-SHA:RC4-SHA:RC4-MD5:PSK-RC4-SHA:KRB5-RC4-SHA:KRB5-RC4-MD5

(2bad) ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-DSS-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA256:DHE-RSA-AES256-SHA:DHE-DSS-AES256-SHA:DHE-RSA-CAMELLIA256-SHA:DHE-DSS-CAMELLIA256-SHA:ECDH-RSA-AES256-GCM-SHA384:ECDH-ECDSA-AES256-GCM-SHA384:ECDH-RSA-AES256-SHA384:ECDH-ECDSA-AES256-SHA384:ECDH-RSA-AES256-SHA:ECDH-ECDSA-AES256-SHA:AES256-GCM-SHA384:AES256-SHA256:AES256-SHA:CAMELLIA256-SHA:PSK-AES256-CBC-SHA:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:DHE-DSS-AES128-GCM-SHA256:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES128-SHA256:DHE-DSS-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA:ECDHE-RSA-DES-CBC3-SHA:ECDHE-ECDSA-DES-CBC3-SHA:DHE-RSA-SEED-SHA:DHE-DSS-SEED-SHA:DHE-RSA-CAMELLIA128-SHA:DHE-DSS-CAMELLIA128-SHA:EDH-RSA-DES-CBC3-SHA:EDH-DSS-DES-CBC3-SHA:ECDH-RSA-AES128-GCM-SHA256:ECDH-ECDSA-AES128-GCM-SHA256:ECDH-RSA-AES128-SHA256:ECDH-ECDSA-AES128-SHA256:ECDH-RSA-AES128-SHA:ECDH-ECDSA-AES128-SHA:ECDH-RSA-DES-CBC3-SHA:ECDH-ECDSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES128-SHA256:AES128-SHA:SEED-SHA:CAMELLIA128-SHA:DES-CBC3-SHA:IDEA-CBC-SHA:PSK-AES128-CBC-SHA:PSK-3DES-EDE-CBC-SHA:KRB5-IDEA-CBC-SHA:KRB5-DES-CBC3-SHA:KRB5-IDEA-CBC-MD5:KRB5-DES-CBC3-MD5:ECDHE-RSA-RC4-SHA:ECDHE-ECDSA-RC4-SHA:ECDH-RSA-RC4-SHA:ECDH-ECDSA-RC4-SHA:RC4-SHA:RC4-MD5:PSK-RC4-SHA:KRB5-RC4-SHA:KRB5-RC4-MD5

(3bad) ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-DSS-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA256:DHE-RSA-AES256-SHA:DHE-DSS-AES256-SHA:DHE-RSA-CAMELLIA256-SHA:DHE-DSS-CAMELLIA256-SHA:ECDH-RSA-AES256-GCM-SHA384:ECDH-ECDSA-AES256-GCM-SHA384:ECDH-RSA-AES256-SHA384:ECDH-ECDSA-AES256-SHA384:ECDH-RSA-AES256-SHA:ECDH-ECDSA-AES256-SHA:AES256-GCM-SHA384:AES256-SHA256:AES256-SHA:CAMELLIA256-SHA:PSK-AES256-CBC-SHA:ECDHE-RSA-DES-CBC3-SHA:ECDHE-ECDSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:EDH-DSS-DES-CBC3-SHA:ECDH-RSA-DES-CBC3-SHA:ECDH-ECDSA-DES-CBC3-SHA:DES-CBC3-SHA:PSK-3DES-EDE-CBC-SHA:KRB5-DES-CBC3-SHA:KRB5-DES-CBC3-MD5:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:DHE-DSS-AES128-GCM-SHA256:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES128-SHA256:DHE-DSS-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA:DHE-RSA-SEED-SHA:DHE-DSS-SEED-SHA:DHE-RSA-CAMELLIA128-SHA:DHE-DSS-CAMELLIA128-SHA:ECDH-RSA-AES128-GCM-SHA256:ECDH-ECDSA-AES128-GCM-SHA256:ECDH-RSA-AES128-SHA256:ECDH-ECDSA-AES128-SHA256:ECDH-RSA-AES128-SHA:ECDH-ECDSA-AES128-SHA:AES128-GCM-SHA256:AES128-SHA256:AES128-SHA:SEED-SHA:CAMELLIA128-SHA:IDEA-CBC-SHA:PSK-AES128-CBC-SHA:KRB5-IDEA-CBC-SHA:KRB5-IDEA-CBC-MD5:ECDHE-RSA-RC4-SHA:ECDHE-ECDSA-RC4-SHA:ECDH-RSA-RC4-SHA:ECDH-ECDSA-RC4-SHA:RC4-SHA:RC4-MD5:PSK-RC4-SHA:KRB5-RC4-SHA:KRB5-RC4-MD5:EDH-RSA-DES-CBC-SHA:EDH-DSS-DES-CBC-SHA:DES-CBC-SHA:KRB5-DES-CBC-SHA:KRB5-DES-CBC-MD5:EXP-EDH-RSA-DES-CBC-SHA:EXP-EDH-DSS-DES-CBC-SHA:EXP-DES-CBC-SHA:EXP-RC2-CBC-MD5:EXP-KRB5-RC2-CBC-SHA:EXP-KRB5-DES-CBC-SHA:EXP-KRB5-RC2-CBC-MD5:EXP-KRB5-DES-CBC-MD5:EXP-RC4-MD5:EXP-KRB5-RC4-SHA:EXP-KRB5-RC4-MD5

As an example of a working one: (1good) ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-DSS-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA256:DHE-RSA-AES256-SHA:DHE-DSS-AES256-SHA:DHE-RSA-CAMELLIA256-SHA:DHE-DSS-CAMELLIA256-SHA:ECDH-RSA-AES256-GCM-SHA384:ECDH-ECDSA-AES256-GCM-SHA384:ECDH-RSA-AES256-SHA384:ECDH-ECDSA-AES256-SHA384:ECDH-RSA-AES256-SHA:ECDH-ECDSA-AES256-SHA:AES256-GCM-SHA384:AES256-SHA256:AES256-SHA:CAMELLIA256-SHA:PSK-AES256-CBC-SHA:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:DHE-DSS-AES128-GCM-SHA256:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES128-SHA256:DHE-DSS-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA:ECDHE-RSA-DES-CBC3-SHA:ECDHE-ECDSA-DES-CBC3-SHA:DHE-RSA-SEED-SHA:DHE-DSS-SEED-SHA:DHE-RSA-CAMELLIA128-SHA:DHE-DSS-CAMELLIA128-SHA:EDH-RSA-DES-CBC3-SHA:EDH-DSS-DES-CBC3-SHA:ECDH-RSA-AES128-GCM-SHA256:ECDH-ECDSA-AES128-GCM-SHA256:ECDH-RSA-AES128-SHA256:ECDH-ECDSA-AES128-SHA256:ECDH-RSA-AES128-SHA:ECDH-ECDSA-AES128-SHA:ECDH-RSA-DES-CBC3-SHA:ECDH-ECDSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES128-SHA256:AES128-SHA:SEED-SHA:CAMELLIA128-SHA:DES-CBC3-SHA:IDEA-CBC-SHA:PSK-AES128-CBC-SHA:PSK-3DES-EDE-CBC-SHA:KRB5-IDEA-CBC-SHA:KRB5-DES-CBC3-SHA:KRB5-IDEA-CBC-MD5:KRB5-DES-CBC3-MD5:ECDHE-RSA-RC4-SHA:ECDHE-ECDSA-RC4-SHA:ECDH-RSA-RC4-SHA:ECDH-ECDSA-RC4-SHA:RC4-SHA:RC4-MD5:PSK-RC4-SHA:KRB5-RC4-SHA:KRB5-RC4-MD5

Given the apparent coincidences of the failing all happen in one data center. I would imagine the possibility that something in the "bad" data center is altering the selection of ciphers in the first packet. (I am assuming the problem REALLY IS a matter of non-overlapping ciphers when I suggest that.)

Replying to [comment:3 puiterwijk]:

Also, please give the output of the following command from one of the machines that fail: "openssl s_client -showcerts -debug -msg -state -connect fedoraproject.org:443"

Sure. From (1bad):
{{{
[root@gitlab01 ~]# openssl s_client -showcerts -debug -msg -state -connect fedoraproject.org:443
CONNECTED(00000003)
SSL_connect:before/connect initialization
write to 0x1fc9120 [0x1ff06b0] (249 bytes => 249 (0xF9))
0000 - 16 03 01 00 f4 01 00 00-f0 03 03 54 6a 73 4a eb ...........TjsJ.
0010 - 7f 4d f1 37 56 56 e4 04-8a 3e 54 f1 76 f1 22 7d .M.7VV...>T.v."}
0020 - 32 a1 11 6a dc 24 61 8d-e0 7f 33 00 00 84 c0 30 2..j.$a...3....0
0030 - c0 2c c0 28 c0 24 c0 14-c0 0a 00 a3 00 9f 00 6b .,.(.$.........k
0040 - 00 6a 00 39 00 38 00 88-00 87 c0 32 c0 2e c0 2a .j.9.8.....2...*
0050 - c0 26 c0 0f c0 05 00 9d-00 3d 00 35 00 84 c0 2f .&.......=.5.../
0060 - c0 2b c0 27 c0 23 c0 13-c0 09 00 a2 00 9e 00 67 .+.'.#.........g
0070 - 00 40 00 33 00 32 c0 12-c0 08 00 9a 00 99 00 45 .@.3.2.........E
0080 - 00 44 00 16 00 13 c0 31-c0 2d c0 29 c0 25 c0 0e .D.....1.-.).%..
0090 - c0 04 c0 0d c0 03 00 9c-00 3c 00 2f 00 96 00 41 .........<./...A
00a0 - 00 0a 00 07 c0 11 c0 07-c0 0c c0 02 00 05 00 04 ................
00b0 - 00 ff 01 00 00 43 00 0b-00 04 03 00 01 02 00 0a .....C..........
00c0 - 00 08 00 06 00 19 00 18-00 17 00 23 00 00 00 0d ...........#....
00d0 - 00 22 00 20 06 01 06 02-06 03 05 01 05 02 05 03 .". ............
00e0 - 04 01 04 02 04 03 03 01-03 02 03 03 02 01 02 02 ................
00f0 - 02 03 01 01 00 0f 00 01-01 .........

TLS 1.2 Handshake [length 00f4], ClientHello
01 00 00 f0 03 03 54 6a 73 4a eb 7f 4d f1 37 56
56 e4 04 8a 3e 54 f1 76 f1 22 7d 32 a1 11 6a dc
24 61 8d e0 7f 33 00 00 84 c0 30 c0 2c c0 28 c0
24 c0 14 c0 0a 00 a3 00 9f 00 6b 00 6a 00 39 00
38 00 88 00 87 c0 32 c0 2e c0 2a c0 26 c0 0f c0
05 00 9d 00 3d 00 35 00 84 c0 2f c0 2b c0 27 c0
23 c0 13 c0 09 00 a2 00 9e 00 67 00 40 00 33 00
32 c0 12 c0 08 00 9a 00 99 00 45 00 44 00 16 00
13 c0 31 c0 2d c0 29 c0 25 c0 0e c0 04 c0 0d c0
03 00 9c 00 3c 00 2f 00 96 00 41 00 0a 00 07 c0
11 c0 07 c0 0c c0 02 00 05 00 04 00 ff 01 00 00
43 00 0b 00 04 03 00 01 02 00 0a 00 08 00 06 00
19 00 18 00 17 00 23 00 00 00 0d 00 22 00 20 06
01 06 02 06 03 05 01 05 02 05 03 04 01 04 02 04
03 03 01 03 02 03 03 02 01 02 02 02 03 01 01 00
0f 00 01 01
SSL_connect:SSLv2/v3 write client hello A
read from 0x1fc9120 [0x1ff5c10] (7 bytes => -1 (0xFFFFFFFFFFFFFFFF))
SSL_connect:error in SSLv2/v3 read server hello A
write:errno=104


no peer certificate available

No client certificate CA names sent

SSL handshake has read 0 bytes and written 249 bytes

New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE


}}}

Okay, thanks.
Could you try this command and give me the output: "openssl s_client -showcerts -debug -msg -state -cipher ECDHE-RSA-AES128-GCM-SHA256 -tls1_2 -connect fedoraproject.org:443"?

Replying to [comment:6 puiterwijk]:

Okay, thanks.
Could you try this command and give me the output: "openssl s_client -showcerts -debug -msg -state -cipher ECDHE-RSA-AES128-GCM-SHA256 -tls1_2 -connect fedoraproject.org:443"?

Sure. It's long. I've attached it as openssl.1bad.listing3.txt

Thanks for the outputs.
I'm going to run some checks on them, and will come back to you tomorrow.

In the meantime, could you give me the output of "rpm -qi openssl", so I know the exact version of openssl you're running this with?

Sure:
{{{
[root ~]# rpm -qi openssl
Name : openssl
Epoch : 1
Version : 1.0.1e
Release : 34.el7_0.6
Architecture: x86_64
Install Date: Fri 07 Nov 2014 05:08:26 PM EST
Group : System Environment/Libraries
Size : 1610001
License : OpenSSL
Signature : RSA/SHA256, Thu 16 Oct 2014 12:06:46 PM EDT, Key ID 24c6a8a7f4a80eb5
Source RPM : openssl-1.0.1e-34.el7_0.6.src.rpm
Build Date : Thu 16 Oct 2014 11:42:42 AM EDT
Build Host : worker1.bsys.centos.org
Relocations : (not relocatable)
Packager : CentOS BuildSystem http://bugs.centos.org
Vendor : CentOS
URL : http://www.openssl.org/
Summary : Utilities from the general purpose cryptography library with TLS implementation
Description :
The OpenSSL toolkit provides support for secure communications between
machines. OpenSSL includes a certificate management tool and shared
libraries which provide various cryptographic algorithms and
protocols.
}}}

It's been a year here... can you please reopen if you are still seeing this?

Login to comment on this ticket.

Metadata