Cloudflare mirror link broken & more

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Cloudflare mirror link broken & more

Anatoli
Hi All,

I see for some time that the link to Cloudflare CDN is broken.
https://www.openbsd.org/ftp.html says it is
https://cloudflare.cdn.openbsd.org/pub/OpenBSD/ but it gives 404.

It looks like Cloudflare removed /pub/ and renamed to lowercase OpenBSD
so the link that works is https://cloudflare.cdn.openbsd.org/openbsd/.

Also, the Fastly (CDN) mirror frequently (like half the times) gives
connection errors, at least using it from Latin America. The IPs I get
from different LA countries are 151.101.2.217 (Brazil) & 151.101.218.217
(Argentina). ftp.openbsd.org works always so when I get errors with
Fastly, I switch to it and it works well (but slowly), or to Cloudflare
which works well too and it's fast (at the modified URL).

The Fastly errors are of the form "connection closed at byte xxx", "ftp:
connect: operation timed out \n signify: gzheader truncated", something
like "no valid ip address found" and similar. Probably it's a faulty or
overloaded server serving some LA countries?


And right now I'm getting an invalid cert error for
https://firmware.openbsd.org. It resolves to 145.238.209.46
(pond.obspm.bsdfrog.org) and 94.142.244.34. The certificate is only
valid for the following names: distfiles.bsdfrog.org, emma-en-quete.com,
ftp.fr.openbsd.org, pond.obspm.bsdfrog.org, pond.stats.bsdfrog.org,
portroach.openbsd.org, www.emma-en-quete.com. Not sure if it's a
configuration error of some mirror server or something else.

I know that the firmware as well as all other files are checked with
signify so https is not strictly required for authenticity (though it
does for privacy) and I don't remember if this domain has ever worked
via https before, anyway just in case there's really some misconfiguration.

Regards,
Anatoli

Reply | Threaded
Open this post in threaded view
|

Re: Cloudflare mirror link broken & more

Theo de Raadt-2
The fastly/cloudflare issues will be looked at by the people who
handle that.

But I can answer this piece:

firmware.openbsd.org is not available via HTTPS.  The tools only use
them it via http, so your testing for https is a mistake.  You are
noticing that some of these machines are multiple-purpose and happen
to have https, but not related to firmware downloads.

> And right now I'm getting an invalid cert error for
> https://firmware.openbsd.org. It resolves to 145.238.209.46
> (pond.obspm.bsdfrog.org) and 94.142.244.34. The certificate is only
> valid for the following names: distfiles.bsdfrog.org, emma-en-quete.com,
> ftp.fr.openbsd.org, pond.obspm.bsdfrog.org, pond.stats.bsdfrog.org,
> portroach.openbsd.org, www.emma-en-quete.com. Not sure if it's a
> configuration error of some mirror server or something else.
>
> I know that the firmware as well as all other files are checked with
> signify so https is not strictly required for authenticity (though it
> does for privacy) and I don't remember if this domain has ever worked
> via https before, anyway just in case there's really some misconfiguration.
>
> Regards,
> Anatoli
>

Reply | Threaded
Open this post in threaded view
|

Re: Cloudflare mirror link broken & more

Stuart Henderson
In reply to this post by Anatoli
On 2019-09-24, Anatoli <[hidden email]> wrote:
> Hi All,
>
> I see for some time that the link to Cloudflare CDN is broken.
> https://www.openbsd.org/ftp.html says it is
> https://cloudflare.cdn.openbsd.org/pub/OpenBSD/ but it gives 404.
>
> It looks like Cloudflare removed /pub/ and renamed to lowercase OpenBSD
> so the link that works is https://cloudflare.cdn.openbsd.org/openbsd/.

That would be due to the origin server which the cloudflare CDN is pointed at.
(The CDNs aren't "real" content servers, they are just caching proxies).
If this is still happening, please show the output from
ftp -o- https://cloudflare.cdn.openbsd.org/pub/OpenBSD/ and
ftp -o- https://cloudflare.cdn.openbsd.org/openbsd/ so we can get
a better idea which origin server it's using etc.

> Also, the Fastly (CDN) mirror frequently (like half the times) gives
> connection errors, at least using it from Latin America. The IPs I get
> from different LA countries are 151.101.2.217 (Brazil) & 151.101.218.217
> (Argentina). ftp.openbsd.org works always so when I get errors with
> Fastly, I switch to it and it works well (but slowly), or to Cloudflare
> which works well too and it's fast (at the modified URL).

Is https://openbsd.c3sl.ufpr.br/pub/OpenBSD/ any better for you?

> The Fastly errors are of the form "connection closed at byte xxx", "ftp:
> connect: operation timed out \n signify: gzheader truncated", something
> like "no valid ip address found" and similar. Probably it's a faulty or
> overloaded server serving some LA countries?

Or a slow link between the CDN and the origin server, or maybe some other
reasons. Personally I would normally only regard the CDNs as a fallback
option if other ways to fetch the files are not working well ..

> And right now I'm getting an invalid cert error for
> https://firmware.openbsd.org. It resolves to 145.238.209.46
> (pond.obspm.bsdfrog.org) and 94.142.244.34. The certificate is only
> valid for the following names: distfiles.bsdfrog.org, emma-en-quete.com,
> ftp.fr.openbsd.org, pond.obspm.bsdfrog.org, pond.stats.bsdfrog.org,
> portroach.openbsd.org, www.emma-en-quete.com. Not sure if it's a
> configuration error of some mirror server or something else.
>
> I know that the firmware as well as all other files are checked with
> signify so https is not strictly required for authenticity (though it
> does for privacy) and I don't remember if this domain has ever worked
> via https before, anyway just in case there's really some misconfiguration.

It's tricky to arrange certificates for this (firmware.openbsd.org is
served from multiple completely separate places, run by different people,
so each instance would need a different key+certificate, and CA challenges
can't be done over HTTP for this). There is a way it can be done via
letsencrypt with DNS-01 challenges and distributing updated certificates
to the various mirrors, but it's fiddly to set this up, and it can't be
done with OpenBSD's acme-client, and doesn't really solve any problems.


Reply | Threaded
Open this post in threaded view
|

Re: Cloudflare mirror link broken & more

Anatoli
Hi Stuart,

Sorry for late reply.

Upon Theo's request I provided job@ with the needed info and the issues
were triaged and fixed. cdn.openbsd.org now works fine. And the location
of files at cloudflare.cdn.openbsd.org is correct again too.

BTW,

> Is https://openbsd.c3sl.ufpr.br/pub/OpenBSD/ any better for you?

This mirror works well from Brazil, but very slow from Argentina as the
route goes via Miami when it already reaches Brazil (hop 8 is at Brazil,
then it goes to Miami, then back to Brazil :)

Telecom Italia Sparkle (aka seabone.net) is the main backbone provider
for Argentina but they have (there are) some issues with intl routing in
Brazil.

traceroute to sagres.c3sl.ufpr.br (200.236.31.1), 64 hops max, 40 byte
packets
 1  192.168.0.1 (192.168.0.1)  1.585 ms  6.74 ms  10.435 ms
 2  * * *
 3  * * *
 4  * * *
 5  * * 132-208-88-200.fibertel.com.ar (200.88.208.132)  107.303 ms
 6  185.70.203.32 (185.70.203.32)  77.977 ms
host63.181-96-120.telecom.net.ar (181.96.120.63)  112.473 ms
185.70.203.32 (185.70.203.32)  59.782 ms
 7  185.70.203.32 (185.70.203.32)  99.471 ms * *
 8  ntt-verio.sanpaolo8.spa.seabone.net (149.3.181.65)  41.224 ms *
40.239 ms
 9  unknown.r20.miamfl02.us.bb.gin.ntt.net (129.250.2.196)  170.192 ms
194.816 ms ntt-verio.sanpaolo8.spa.seabone.net (149.3.181.65)  49.554 ms
10  unknown.r20.miamfl02.us.bb.gin.ntt.net (129.250.2.196)  175.984 ms
176.301 ms  174.46 ms
11  ae-8.r05.miamfl02.us.bb.gin.ntt.net (129.250.3.150)  183.177 ms
ae-2.a01.miamfl02.us.bb.gin.ntt.net (129.250.3.167)  182.203 ms
ae-3.a01.miamfl02.us.bb.gin.ntt.net (129.250.3.208)  181.342 ms
12  xe-0-0-26-2.a01.miamfl02.us.ce.gin.ntt.net (129.250.202.94)  181.301
ms ae-3.a01.miamfl02.us.bb.gin.ntt.net (129.250.3.208)  177.877 ms
ae-2.a01.miamfl02.us.bb.gin.ntt.net (129.250.3.167)  185.902 ms
13  xe-0-0-26-2.a01.miamfl02.us.ce.gin.ntt.net (129.250.202.94)  181.56
ms  201.18 ms  181.97 ms
14  * * *
15  * * *
16  p2-v103-araucaria-lapa.pop-pr.rnp.br (200.238.139.10)  257.613 ms *
323.722 ms
17  p2-v103-araucaria-lapa.pop-pr.rnp.br (200.238.139.10)  343.196 ms
474.198 ms 200.17.202.62 (200.17.202.62)  974.067 ms
18  200.17.202.62 (200.17.202.62)  259.173 ms sagres.c3sl.ufpr.br
(200.236.31.1)  257.664 ms 200.17.202.62 (200.17.202.62)  256.431 ms

And thank you for your detailed explanation about the certs for firmware
sub-domain. Just wanted to say that IMO there's actually one thing that
it would solve: the privacy of the requests, i.e. we wouldn't be leaking
info about our devices with proprietary fw to anyone listening on the
wires. But I see it's a considerable effort to set it up. I already know
whom to contact to collaborate with the infrastructure.

Regards,
Anatoli

On 25/9/19 15:26, Stuart Henderson wrote:

> On 2019-09-24, Anatoli <[hidden email]> wrote:
>> Hi All,
>>
>> I see for some time that the link to Cloudflare CDN is broken.
>> https://www.openbsd.org/ftp.html says it is
>> https://cloudflare.cdn.openbsd.org/pub/OpenBSD/ but it gives 404.
>>
>> It looks like Cloudflare removed /pub/ and renamed to lowercase OpenBSD
>> so the link that works is https://cloudflare.cdn.openbsd.org/openbsd/.
>
> That would be due to the origin server which the cloudflare CDN is pointed at.
> (The CDNs aren't "real" content servers, they are just caching proxies).
> If this is still happening, please show the output from
> ftp -o- https://cloudflare.cdn.openbsd.org/pub/OpenBSD/ and
> ftp -o- https://cloudflare.cdn.openbsd.org/openbsd/ so we can get
> a better idea which origin server it's using etc.
>
>> Also, the Fastly (CDN) mirror frequently (like half the times) gives
>> connection errors, at least using it from Latin America. The IPs I get
>> from different LA countries are 151.101.2.217 (Brazil) & 151.101.218.217
>> (Argentina). ftp.openbsd.org works always so when I get errors with
>> Fastly, I switch to it and it works well (but slowly), or to Cloudflare
>> which works well too and it's fast (at the modified URL).
>
> Is https://openbsd.c3sl.ufpr.br/pub/OpenBSD/ any better for you?
>
>> The Fastly errors are of the form "connection closed at byte xxx", "ftp:
>> connect: operation timed out \n signify: gzheader truncated", something
>> like "no valid ip address found" and similar. Probably it's a faulty or
>> overloaded server serving some LA countries?
>
> Or a slow link between the CDN and the origin server, or maybe some other
> reasons. Personally I would normally only regard the CDNs as a fallback
> option if other ways to fetch the files are not working well ..
>
>> And right now I'm getting an invalid cert error for
>> https://firmware.openbsd.org. It resolves to 145.238.209.46
>> (pond.obspm.bsdfrog.org) and 94.142.244.34. The certificate is only
>> valid for the following names: distfiles.bsdfrog.org, emma-en-quete.com,
>> ftp.fr.openbsd.org, pond.obspm.bsdfrog.org, pond.stats.bsdfrog.org,
>> portroach.openbsd.org, www.emma-en-quete.com. Not sure if it's a
>> configuration error of some mirror server or something else.
>>
>> I know that the firmware as well as all other files are checked with
>> signify so https is not strictly required for authenticity (though it
>> does for privacy) and I don't remember if this domain has ever worked
>> via https before, anyway just in case there's really some misconfiguration.
>
> It's tricky to arrange certificates for this (firmware.openbsd.org is
> served from multiple completely separate places, run by different people,
> so each instance would need a different key+certificate, and CA challenges
> can't be done over HTTP for this). There is a way it can be done via
> letsencrypt with DNS-01 challenges and distributing updated certificates
> to the various mirrors, but it's fiddly to set this up, and it can't be
> done with OpenBSD's acme-client, and doesn't really solve any problems.
>
>

Reply | Threaded
Open this post in threaded view
|

Re: Cloudflare mirror link broken & more

Theo de Raadt-2
Anatoli <[hidden email]> wrote:

> And thank you for your detailed explanation about the certs for firmware
> sub-domain. Just wanted to say that IMO there's actually one thing that
> it would solve: the privacy of the requests, i.e. we wouldn't be leaking
> info about our devices with proprietary fw to anyone listening on the
> wires. But I see it's a considerable effort to set it up. I already know
> whom to contact to collaborate with the infrastructure.

oh really, https solves that??

Sorry to burst your bubble, but looking at the number of bytes moved in
the sessions is sufficient to determine which firmwares were selected
and downloaded.

Reply | Threaded
Open this post in threaded view
|

Re: Cloudflare mirror link broken & more

Anatoli
> looking at the number of bytes moved in the sessions is sufficient to
> determine which firmwares were selected and downloaded.

Theo, I may be completely wrong here (please excuse my ignorance if it
is the case), but I see it this way:

On a shared server (or one fronted by a CDN) on the same pool of IPs
there are lots of domains hosted (at cdn.openbsd.org right now there are
140 domains of which 63 are wildcards and they are shuffled all the
time), they could have infinite amount of files.

With ESNI there's no way to know which domain we are requesting, so we
could be downloading/requesting anything (files and dynamic content,
RTC, streaming) from hundreds of unrelated domains.

On top of this, if we use HTTP/2 multiplexing and request all the
firmware binaries over the same connection, the exact size wouldn't be
known either. You can add additional obfuscations if needed, like
randomly mix-querying small files over the same multiplexed connection.

I know tls1.3 is not there yet in LibreSSL and ESNI is at draft-04 at
this moment, but I'm not talking about an immediate fully-DPI-resistant
deployment. All CloudFlare hosted domains are with ESNI already for a
year [1] and ff has it in nightly. OpenSSL, Fastly, Apple and Google are
also working on it, there's a fairly good interop testing ground.

My question was about why not to cloud-front-with-https (like
cdn.openbsd.org is) the firmware sub-domain too (or
cdn.firmware.openbsd.org). Just my 2-cents-IMO :)

Regards,
Anatoli

[1] https://blog.cloudflare.com/encrypted-sni/


On 7/10/19 15:38, Theo de Raadt wrote:

> Anatoli <[hidden email]> wrote:
>
>> And thank you for your detailed explanation about the certs for firmware
>> sub-domain. Just wanted to say that IMO there's actually one thing that
>> it would solve: the privacy of the requests, i.e. we wouldn't be leaking
>> info about our devices with proprietary fw to anyone listening on the
>> wires. But I see it's a considerable effort to set it up. I already know
>> whom to contact to collaborate with the infrastructure.
>
> oh really, https solves that??
>
> Sorry to burst your bubble, but looking at the number of bytes moved in
> the sessions is sufficient to determine which firmwares were selected
> and downloaded.
>

Reply | Threaded
Open this post in threaded view
|

Re: Cloudflare mirror link broken & more

Theo de Raadt-2
Anatoli <[hidden email]> wrote:

> > looking at the number of bytes moved in the sessions is sufficient to
> > determine which firmwares were selected and downloaded.
>
> Theo, I may be completely wrong here (please excuse my ignorance if it
> is the case), but I see it this way:
>
> On a shared server (or one fronted by a CDN)

firmware.openbsd.org is not a CDN, it is a DNS name pointing a handful
of hosts, so everything else you wrote is irrelevant.  I'm not sure
why you bothered.

> My question was about why not to cloud-front-with-https (like
> cdn.openbsd.org is) the firmware sub-domain too (or
> cdn.firmware.openbsd.org). Just my 2-cents-IMO :)

Because we don't.

End of story.

Reply | Threaded
Open this post in threaded view
|

Re: Cloudflare mirror link broken & more

Theo de Raadt-2
In reply to this post by Anatoli
Anatoli <[hidden email]> wrote:

> > looking at the number of bytes moved in the sessions is sufficient to
> > determine which firmwares were selected and downloaded.
>
> Theo, I may be completely wrong here (please excuse my ignorance if it
> is the case), but I see it this way:
>
> On a shared server (or one fronted by a CDN) on the same pool of IPs
> there are lots of domains hosted (at cdn.openbsd.org right now there are
> 140 domains of which 63 are wildcards and they are shuffled all the
> time), they could have infinite amount of files.
>
> With ESNI there's no way to know which domain we are requesting, so we
> could be downloading/requesting anything (files and dynamic content,
> RTC, streaming) from hundreds of unrelated domains.
>
> On top of this, if we use HTTP/2 multiplexing and request all the
> firmware binaries over the same connection, the exact size wouldn't be
> known either. You can add additional obfuscations if needed, like
> randomly mix-querying small files over the same multiplexed connection.
>
> I know tls1.3 is not there yet in LibreSSL and ESNI is at draft-04 at
> this moment, but I'm not talking about an immediate fully-DPI-resistant
> deployment. All CloudFlare hosted domains are with ESNI already for a
> year [1] and ff has it in nightly. OpenSSL, Fastly, Apple and Google are
> also working on it, there's a fairly good interop testing ground.

The amazing thing about all those security buzzwords is they decrypt
inside the servers of one company which operates under US legal
doctrine.

You are a very trustful believer.  The internet is full of snakes, but
the endpoint is paradise, there are no snakes at the endpoints.