pkg_add fails with error "TLS connect failure: failed to set session"

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

pkg_add fails with error "TLS connect failure: failed to set session"

Sven Wolf-3
Hi,

I run current. After I run sysupgrade today (GENERIC.MP #626 build Jan
30) it's not possible to run pkd_add. I always get the error
TLS connect failure: failed to set session
signify: gzheader truncated

The error is reproducible on two machines and didn't occur until build
#616 (Jan 21).

/etc/installurl points to an internal mirror server. This mirror server
runs on Debian/Apache and has a letsencrypt certificate. Maybe the
letsencrypt certificate is the root cause.
When I switch /etc/installurl to an official OpenBSD mirror (e.g.
https://artfiles.org/openbsd/) the error doesn't occur.
Also when /etc/installurl points to the internal mirror server using the
http instead of the https protocol then there is also no error.

sysupgrade runs without errors against the internal mirror server via
https. Also an wget of a package (e.g atk-2.34.1p1) via the https
protocol shows no error.

I compared the atk-2.34.1p1 package against an official mirror. There is
no difference in the md5sum.

Maybe the pkg_add error has something in common with
https://marc.info/?t=157996435100001&r=1&w=2

If there is something I should test/change, please let me know.

Thanks and best regards,
Sven

Reply | Threaded
Open this post in threaded view
|

Re: pkg_add fails with error "TLS connect failure: failed to set session"

Stuart Henderson
On 2020/01/31 21:24, Sven Wolf wrote:
> Hi,
>
> I run current. After I run sysupgrade today (GENERIC.MP #626 build Jan 30)
> it's not possible to run pkd_add. I always get the error
> TLS connect failure: failed to set session
> signify: gzheader truncated

pkg_add runs ftp many times and tries to resume TLS sessions between calls
to reduce setup overhead. The failure is connected with this but is only
seen with some sites.

> The error is reproducible on two machines and didn't occur until build #616
> (Jan 21).
>
> /etc/installurl points to an internal mirror server. This mirror server runs
> on Debian/Apache and has a letsencrypt certificate. Maybe the letsencrypt
> certificate is the root cause.
> When I switch /etc/installurl to an official OpenBSD mirror (e.g.
> https://artfiles.org/openbsd/) the error doesn't occur.

There is no general problem with letsencrypt certificates, probably most of
the official mirrors use them (artfiles.org certainly does).

> Also when /etc/installurl points to the internal mirror server using the
> http instead of the https protocol then there is also no error.
>
> sysupgrade runs without errors against the internal mirror server via https.
> Also an wget of a package (e.g atk-2.34.1p1) via the https protocol shows no
> error.
>
> I compared the atk-2.34.1p1 package against an official mirror. There is no
> difference in the md5sum.
>
> Maybe the pkg_add error has something in common with
> https://marc.info/?t=157996435100001&r=1&w=2
>
> If there is something I should test/change, please let me know.
>
> Thanks and best regards,
> Sven
>

Generally it is hard to debug these without access to the server (at
least to make an HTTPS connection if not actually fetch files) so it being
an internal server makes that hard. However I have found some other hosts
which also have the same symptom so hopefully this will help libressl
developers track it down.

https://cloudflare.cdn.openbsd.org/pub/OpenBSD/
https://mirrors.ucr.ac.cr/pub/OpenBSD/
https://mirrors.dotsrc.org/pub/OpenBSD/
https://mirror.one.com/pub/OpenBSD/
https://openbsd.c3sl.ufpr.br/pub/OpenBSD/

And there's a bonus "SSL_internal:unknown failure occurred" at

https://mirror.vdms.com/pub/OpenBSD/

Reply | Threaded
Open this post in threaded view
|

Re: pkg_add fails with error "TLS connect failure: failed to set session"

Stuart Henderson
On 2020/01/31 23:03, Stuart Henderson wrote:

> On 2020/01/31 21:24, Sven Wolf wrote:
> > Hi,
> >
> > I run current. After I run sysupgrade today (GENERIC.MP #626 build Jan 30)
> > it's not possible to run pkd_add. I always get the error
> > TLS connect failure: failed to set session
> > signify: gzheader truncated
>
> pkg_add runs ftp many times and tries to resume TLS sessions between calls
> to reduce setup overhead. The failure is connected with this but is only
> seen with some sites.
<snip>

> Generally it is hard to debug these without access to the server (at
> least to make an HTTPS connection if not actually fetch files) so it being
> an internal server makes that hard. However I have found some other hosts
> which also have the same symptom so hopefully this will help libressl
> developers track it down.
>
> https://cloudflare.cdn.openbsd.org/pub/OpenBSD/
> https://mirrors.ucr.ac.cr/pub/OpenBSD/
> https://mirrors.dotsrc.org/pub/OpenBSD/
> https://mirror.one.com/pub/OpenBSD/
> https://openbsd.c3sl.ufpr.br/pub/OpenBSD/

These all run TLS 1.3. The ones I ran through ssllabs checker all support
session resumption as well (I didn't do all of them because it's takes forever :).


> And there's a bonus "SSL_internal:unknown failure occurred" at
>
> https://mirror.vdms.com/pub/OpenBSD/
>

Reply | Threaded
Open this post in threaded view
|

Re: pkg_add fails with error "TLS connect failure: failed to set session"

Marc Espie-2
In reply to this post by Sven Wolf-3
On Fri, Jan 31, 2020 at 09:24:08PM +0100, Sven Wolf wrote:

> Hi,
>
> I run current. After I run sysupgrade today (GENERIC.MP #626 build Jan 30)
> it's not possible to run pkd_add. I always get the error
> TLS connect failure: failed to set session
> signify: gzheader truncated
>
> The error is reproducible on two machines and didn't occur until build #616
> (Jan 21).
>
> /etc/installurl points to an internal mirror server. This mirror server runs
> on Debian/Apache and has a letsencrypt certificate. Maybe the letsencrypt
> certificate is the root cause.
> When I switch /etc/installurl to an official OpenBSD mirror (e.g.
> https://artfiles.org/openbsd/) the error doesn't occur.
> Also when /etc/installurl points to the internal mirror server using the
> http instead of the https protocol then there is also no error.

pkg_add(1) does not deal directly with network connections.
Any TLS bug  is related to ftp(1).

If you sign your packages, https is not event needed.

Reply | Threaded
Open this post in threaded view
|

Re: pkg_add fails with error "TLS connect failure: failed to set session"

Sven Wolf-3
In reply to this post by Sven Wolf-3
Hi,

I did some debugging on the server side.
Even with loglevel trace5 and also different TLS versions (I tested 1.1,
1.2 and 1.3) I didn't find the root cause.

In the attachment you'll find the export of the Apache error log with
loglevel trace5. Maybe it's helpfull for the libressl developers.

On the client side I just did an pkg_add -v bash

Best regards,
Sven

On 1/31/20 9:24 PM, Sven Wolf wrote:

> Hi,
>
> I run current. After I run sysupgrade today (GENERIC.MP #626 build Jan
> 30) it's not possible to run pkd_add. I always get the error
> TLS connect failure: failed to set session
> signify: gzheader truncated
>
> The error is reproducible on two machines and didn't occur until build
> #616 (Jan 21).
>
> /etc/installurl points to an internal mirror server. This mirror server
> runs on Debian/Apache and has a letsencrypt certificate. Maybe the
> letsencrypt certificate is the root cause.
> When I switch /etc/installurl to an official OpenBSD mirror (e.g.
> https://artfiles.org/openbsd/) the error doesn't occur.
> Also when /etc/installurl points to the internal mirror server using the
> http instead of the https protocol then there is also no error.
>
> sysupgrade runs without errors against the internal mirror server via
> https. Also an wget of a package (e.g atk-2.34.1p1) via the https
> protocol shows no error.
>
> I compared the atk-2.34.1p1 package against an official mirror. There is
> no difference in the md5sum.
>
> Maybe the pkg_add error has something in common with
> https://marc.info/?t=157996435100001&r=1&w=2
>
> If there is something I should test/change, please let me know.
>
> Thanks and best regards,
> Sven
>

apache.log (9K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: pkg_add fails with error "TLS connect failure: failed to set session"

Marc Espie-2
On Sat, Feb 01, 2020 at 12:48:40PM +0100, Sven Wolf wrote:

> Hi,
>
> I did some debugging on the server side.
> Even with loglevel trace5 and also different TLS versions (I tested 1.1, 1.2
> and 1.3) I didn't find the root cause.
>
> In the attachment you'll find the export of the Apache error log with
> loglevel trace5. Maybe it's helpfull for the libressl developers.
>
> On the client side I just did an pkg_add -v bash
>
> Best regards,
> Sven
If you can expose that server to the outside world, try

https://www.ssllabs.com/

what does the report say, especially wrt session resumption ?

Reply | Threaded
Open this post in threaded view
|

Re: pkg_add fails with error "TLS connect failure: failed to set session"

Stuart Henderson
On 2020/02/01 13:36, Marc Espie wrote:

> On Sat, Feb 01, 2020 at 12:48:40PM +0100, Sven Wolf wrote:
> > Hi,
> >
> > I did some debugging on the server side.
> > Even with loglevel trace5 and also different TLS versions (I tested 1.1, 1.2
> > and 1.3) I didn't find the root cause.
> >
> > In the attachment you'll find the export of the Apache error log with
> > loglevel trace5. Maybe it's helpfull for the libressl developers.
> >
> > On the client side I just did an pkg_add -v bash
> >
> > Best regards,
> > Sven
> If you can expose that server to the outside world, try
>
> https://www.ssllabs.com/
>
> what does the report say, especially wrt session resumption ?
>

jsing@ has committed a workaround to src/lib/libssl for now.

Reply | Threaded
Open this post in threaded view
|

Re: pkg_add fails with error "TLS connect failure: failed to set session"

Sven Wolf-3
In reply to this post by Marc Espie-2
Hi Marc,

here are the protocol details for my internal mirror.

Protocol Details
DROWN No, server keys and hostname not seen elsewhere with SSLv2
(1) For a better understanding of this test, please read this longer
explanation
(2) Key usage data kindly provided by the Censys network search engine;
original DROWN website here
(3) Censys data is only indicative of possible key and certificate
reuse; possibly out-of-date and not complete
Secure Renegotiation Supported
Secure Client-Initiated Renegotiation No
Insecure Client-Initiated Renegotiation No
BEAST attack Not mitigated server-side (more info)   TLS 1.0: 0x2f
POODLE (SSLv3) No, SSL 3 not supported (more info)
POODLE (TLS) No (more info)
Zombie POODLE No (more info)   TLS 1.2 : 0x002f
GOLDENDOODLE No (more info)   TLS 1.2 : 0x002f
OpenSSL 0-Length No (more info)   TLS 1.2 : 0x002f
Sleeping POODLE No (more info)   TLS 1.2 : 0x002f
Downgrade attack prevention Yes, TLS_FALLBACK_SCSV supported (more info)
SSL/TLS compression No
RC4 No
Heartbeat (extension) No
Heartbleed (vulnerability) No (more info)
Ticketbleed (vulnerability) No (more info)
OpenSSL CCS vuln. (CVE-2014-0224) No (more info)
OpenSSL Padding Oracle vuln.
(CVE-2016-2107) No (more info)
ROBOT (vulnerability) No (more info)
Forward Secrecy With some browsers (more info)
ALPN Yes   http/1.1
NPN No
Session resumption (caching) Yes
Session resumption (tickets) Yes
OCSP stapling No
Strict Transport Security (HSTS) Yes
max-age=15768000
HSTS Preloading Not in: Chrome  Edge  Firefox  IE
Public Key Pinning (HPKP) No (more info)
Public Key Pinning Report-Only No
Public Key Pinning (Static) No (more info)
Long handshake intolerance No
TLS extension intolerance No
TLS version intolerance No
Incorrect SNI alerts No
Uses common DH primes No
DH public server param (Ys) reuse No
ECDH public server param reuse No
Supported Named Groups secp256r1, secp384r1, secp521r1, x25519, x448
(Server has no preference)
SSL 2 handshake compatibility Yes
0-RTT enabled No

Here is the diff of the protocol details for my not working internal
server and the artfiles openbsd mirror.

1d0
< Protocol Details
9c8
< BEAST attack Not mitigated server-side (more info)   TLS 1.0: 0x2f
---
 > BEAST attack Not mitigated server-side (more info)   TLS 1.0: 0xc013
12,15c11,14
< Zombie POODLE No (more info)   TLS 1.2 : 0x002f
< GOLDENDOODLE No (more info)   TLS 1.2 : 0x002f
< OpenSSL 0-Length No (more info)   TLS 1.2 : 0x002f
< Sleeping POODLE No (more info)   TLS 1.2 : 0x002f
---
 > Zombie POODLE No (more info)   TLS 1.2 : 0xc027
 > GOLDENDOODLE No (more info)   TLS 1.2 : 0xc027
 > OpenSSL 0-Length No (more info)   TLS 1.2 : 0xc027
 > Sleeping POODLE No (more info)   TLS 1.2 : 0xc027
26c25
< Forward Secrecy With some browsers (more info)
---
 > Forward Secrecy Yes (with most browsers)   ROBUST (more info)
32,33c31
< Strict Transport Security (HSTS) Yes
< max-age=15768000
---
 > Strict Transport Security (HSTS) No
45c43
< Supported Named Groups secp256r1, secp384r1, secp521r1, x25519, x448
(Server has no preference)
---
 > Supported Named Groups x25519, secp256r1, x448, secp521r1, secp384r1
(server preferred order)
47d44
< 0-RTT enabled No



Best regards,
Sven

On 2/1/20 1:36 PM, Marc Espie wrote:

> On Sat, Feb 01, 2020 at 12:48:40PM +0100, Sven Wolf wrote:
>> Hi,
>>
>> I did some debugging on the server side.
>> Even with loglevel trace5 and also different TLS versions (I tested 1.1, 1.2
>> and 1.3) I didn't find the root cause.
>>
>> In the attachment you'll find the export of the Apache error log with
>> loglevel trace5. Maybe it's helpfull for the libressl developers.
>>
>> On the client side I just did an pkg_add -v bash
>>
>> Best regards,
>> Sven
> If you can expose that server to the outside world, try
>
> https://www.ssllabs.com/
>
> what does the report say, especially wrt session resumption ?
>

Reply | Threaded
Open this post in threaded view
|

Re: pkg_add fails with error "TLS connect failure: failed to set session"

Sven Wolf-3
In reply to this post by Sven Wolf-3
Hi,

the pkg_add problem with an https installurl is solved (tested with
snapshot #639/2020-02-07).

Thanks and best regards,
Sven

On 1/31/20 9:24 PM, Sven Wolf wrote:

> Hi,
>
> I run current. After I run sysupgrade today (GENERIC.MP #626 build Jan
> 30) it's not possible to run pkd_add. I always get the error
> TLS connect failure: failed to set session
> signify: gzheader truncated
>
> The error is reproducible on two machines and didn't occur until build
> #616 (Jan 21).
>
> /etc/installurl points to an internal mirror server. This mirror server
> runs on Debian/Apache and has a letsencrypt certificate. Maybe the
> letsencrypt certificate is the root cause.
> When I switch /etc/installurl to an official OpenBSD mirror (e.g.
> https://artfiles.org/openbsd/) the error doesn't occur.
> Also when /etc/installurl points to the internal mirror server using the
> http instead of the https protocol then there is also no error.
>
> sysupgrade runs without errors against the internal mirror server via
> https. Also an wget of a package (e.g atk-2.34.1p1) via the https
> protocol shows no error.
>
> I compared the atk-2.34.1p1 package against an official mirror. There is
> no difference in the md5sum.
>
> Maybe the pkg_add error has something in common with
> https://marc.info/?t=157996435100001&r=1&w=2
>
> If there is something I should test/change, please let me know.
>
> Thanks and best regards,
> Sven
>

Reply | Threaded
Open this post in threaded view
|

Re: pkg_add fails with error "TLS connect failure: failed to set session"

Theo Buehler-3
On Sat, Feb 08, 2020 at 12:51:24PM +0100, Sven Wolf wrote:
> Hi,
>
> the pkg_add problem with an https installurl is solved (tested with snapshot
> #639/2020-02-07).

Thanks for letting us know. This issue was first resolved by disabling
the TLSv1.3 client last Saturday.  A workaround was committed in
revision 1.12 of ssl_methods.c:

https://cvsweb.openbsd.org/src/lib/libssl/ssl_methods.c

The client has since been re-enabled.