Re: fetch problem with //pypi.io/packages/source/

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: fetch problem with //pypi.io/packages/source/

Stuart Henderson
Moved ports@ -> bugs@


On 2018/11/05 18:41, [hidden email] wrote:
> Just for the heck of it I set up an amd64 snapshot system to compare the builds.  The problem does not occur on the amd64 system. I expected this or else it would have been fixed long ago.

Is the amd64 system in the same place on your network as the arm
system (going through same routers / NAT / etc)?

> So what does FETCH do that is different on arm from amd64 and why only to the pypi.io website?

Nothing. It's nothing specific to fetch. As you showed earlier the
problem was occurring with any use of ftp(1) connecting to this site
and didn't occur with curl/wget.

What happens if you try "nc -Dvc pypi.io 443" on the arm and amd64
systems?

What is the full output from "ftp -d -o /dev/null https://pypi.io/" on both?

>
> From amd64 system:
>
> ===>  Verifying install for py-alabaster-* in textproc/py-alabaster
> ===>  Checking files for py-alabaster-0.7.10
> >> Fetch https://pypi.io/packages/source/a/alabaster/alabaster-0.7.10.tar.gz
> alabaster-0.7.10.tar.gz 100% |**************************************************| 10486       00:00
> >> (SHA256) alabaster-0.7.10.tar.gz: OK
> ===> py-alabaster-0.7.10 depends on: python->=2.7,<2.8 -> python-2.7.15p1
> ===> py-alabaster-0.7.10 depends on: py-setuptools->=39.0.1v0 -> py-setuptools-40.0.0v0
>
>
> From arm system:
>
> -----Original Message-----
> From: [hidden email] <[hidden email]> On Behalf Of [hidden email]
> Sent: November 5, 2018 10:17 AM
> To: [hidden email]
> Subject: fetch problem with //pypi.io/packages/source/
>
> I still get this when as part of a php build.
>
> ===>  Verifying install for py-alabaster-* in textproc/py-alabaster ===>  Checking files for py-alabaster-0.7.10
> >> Fetch
> >> https://pypi.io/packages/source/a/alabaster/alabaster-0.7.10.tar.gz
> ftp: SSL write error: handshake failed: Operation timed out
> >> Fetch
> https://ftp.openbsd.org/pub/OpenBSD/distfiles/alabaster-0.7.10.tar.gz
> alabaster-0.7.10.tar.gz 100% |**************************| 10486       00:00
> >> (SHA256) alabaster-0.7.10.tar.gz: OK
> ===> py-alabaster-0.7.10 depends on: python->=2.7,<2.8 -> python-2.7.15p1 ===> py-alabaster-0.7.10 depends on: py-setuptools->=39.0.1v0 ->
> py-setuptools-40.0.0v0
> ===>  Extracting for py-alabaster-0.7.10
>
> I can take the url and paste it into a Win web browser and it downloads without a problem.  This problem only occurs on this site.
>
> Even it this cannot be resolved, is there a way to change the timeout to something reasonable, like about 20 seconds instead of the 5 minutes? Trying to build on arm is slow enough without these delays, and there are quite a number of packages on this site
>
>

Reply | Threaded
Open this post in threaded view
|

Re: fetch problem with //pypi.io/packages/source/

s_graf
Both system are on the same network, hub, router etc.  The amd system is on
Virtualbox on my PC and the arm system is an orangepi one.

From the arm system:

op1bsdsnap1102# nc -Dvc pypi.io 443
Connection to pypi.io 443 port [tcp/https] succeeded!
TLS handshake negotiated TLSv1.2/ECDHE-RSA-AES128-GCM-SHA256 with host
pypi.io
Peer name: pypi.io
Subject: /businessCategory=Private
Organization/jurisdictionCountryName=US/jurisdictionStateOrProvinceName=Dela
ware/serialNumber=3359300/C=US/ST=New Hampshire/L=Wolfeboro/O=Python
Software Foundation/CN=www.python.org
Issuer: /C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert SHA2 Extended
Validation Server CA
Valid From: Mon Sep 17 17:00:00 2018
Valid Until: Wed Oct 14 05:00:00 2020
Cert Hash:
SHA256:6ef70b832fa021a18e4ba948b8f646beddac321e73a34d4c1e7835559e65396b
OCSP URL: http://ocsp.digicert.com
OCSP Stapling: good
  response_status=0 cert_status=0 crl_reason=0
  this update: Fri Nov  2 20:02:59 2018
  next update: Fri Nov  9 18:17:59 2018
  revocation:

*** At this the system seemed to be waiting for input.  After about 10
minutes I hit the enter key to continue.

op1bsdsnap1102#
op1bsdsnap1102# ftp -d -o /dev/null https://pypi.io/
host pypi.io, port https, path , save as /dev/null, auth none.
Trying 151.101.0.223...
Requesting https://pypi.io/

*** After about 5 minutes, without any intervention:

ftp: SSL write error: handshake failed: Operation timed out
op1bsdsnap1102#

From the amd system:

obsdamdsnap# nc -Dvc pypi.io 443
Connection to pypi.io 443 port [tcp/https] succeeded!
TLS handshake negotiated TLSv1.2/ECDHE-RSA-AES128-GCM-SHA256 with host
pypi.io
Peer name: pypi.io
Subject: /businessCategory=Private
Organization/jurisdictionCountryName=US/jurisdictionStateOrProvinceName=Dela
ware/serialNumber=3359300/C=US/ST=New Hampshire/L=Wolfeboro/O=Python
Software Foundation/CN=www.python.org
Issuer: /C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert SHA2 Extended
Validation Server CA
Valid From: Mon Sep 17 17:00:00 2018
Valid Until: Wed Oct 14 05:00:00 2020
Cert Hash:
SHA256:6ef70b832fa021a18e4ba948b8f646beddac321e73a34d4c1e7835559e65396b
OCSP URL: http://ocsp.digicert.com
OCSP Stapling: good
  response_status=0 cert_status=0 crl_reason=0
  this update: Fri Nov  2 20:02:59 2018
  next update: Fri Nov  9 18:17:59 2018
  revocation:

*** At this the system seemed to be waiting for input.  After about 10
minutes I hit the enter key to continue.

obsdamdsnap#
obsdamdsnap# ftp -d -o /dev/null https://pypi.io/
host pypi.io, port https, path , save as /dev/null, auth none.
Trying 151.101.128.223...
Requesting https://pypi.io/
GET / HTTP/1.0
Host: pypi.io
User-Agent: OpenBSD ftp

received 'HTTP/1.1 301 Redirect to Primary Domain'
received 'Server: Varnish'
received 'Retry-After: 0'
received 'Location: https://pypi.org/'
Redirected to https://pypi.org/
host pypi.org, port https, path , save as /dev/null, auth none.
Trying 151.101.192.223...
Requesting https://pypi.org/
GET / HTTP/1.0
Host: pypi.org
User-Agent: OpenBSD ftp

received 'HTTP/1.1 200 OK'
received 'Content-Security-Policy: base-uri 'self'; block-all-mixed-content;
connect-src 'self' https://api.github.com/repos/ *.fastly-insights.com
sentry.io https://2p66nmmycsj3.statuspage.io; default-src 'none'; font-src
'self' fonts.gstatic.com; form-action 'self'; frame-ancestors 'none';
frame-src 'none'; img-src 'self' https://warehouse-camo.cmh1.psfhosted.org/
www.google-analytics.com *.fastly-insights.com; script-src 'self'
www.googletagmanager.com www.google-analytics.com *.fastly-insights.com
https://cdn.ravenjs.com; style-src 'self' fonts.googleapis.com; worker-src
*.fastly-insights.com'
received 'Content-Type: text/html; charset=UTF-8'
received 'ETag: "gSRI8fFiG1RCiXGvdpDkVw"'
received 'Referrer-Policy: origin-when-cross-origin'
received 'Server: nginx/1.13.9'
received 'Content-Length: 17655'
received 'Accept-Ranges: bytes'
received 'Date: Tue, 06 Nov 2018 18:35:39 GMT'
received 'Age: 183'
received 'Connection: close'
received 'X-Served-By: cache-iad2125-IAD, cache-sea1043-SEA'
received 'X-Cache: HIT, HIT'
received 'X-Cache-Hits: 1, 1'
received 'X-Timer: S1541529340.841572,VS0,VE1'
received 'Vary: Accept-Encoding, Accept-Encoding'
received 'Strict-Transport-Security: max-age=31536000; includeSubDomains;
preload'
received 'X-Frame-Options: deny'
received 'X-XSS-Protection: 1; mode=block'
received 'X-Content-Type-Options: nosniff'
received 'X-Permitted-Cross-Domain-Policies: none'
100%
|**************************************************************************|
17655       00:00
17655 bytes received in 0.01 seconds (3.34 MB/s)
obsdamdsnap#





-----Original Message-----
From: Stuart Henderson <[hidden email]>
Sent: November 6, 2018 5:13 AM
To: [hidden email]
Cc: [hidden email]
Subject: Re: fetch problem with //pypi.io/packages/source/

Moved ports@ -> bugs@


On 2018/11/05 18:41, [hidden email] wrote:
> Just for the heck of it I set up an amd64 snapshot system to compare the
builds.  The problem does not occur on the amd64 system. I expected this or
else it would have been fixed long ago.

Is the amd64 system in the same place on your network as the arm system
(going through same routers / NAT / etc)?

> So what does FETCH do that is different on arm from amd64 and why only to
the pypi.io website?

Nothing. It's nothing specific to fetch. As you showed earlier the problem
was occurring with any use of ftp(1) connecting to this site and didn't
occur with curl/wget.

What happens if you try "nc -Dvc pypi.io 443" on the arm and amd64 systems?

What is the full output from "ftp -d -o /dev/null https://pypi.io/" on both?

>
> From amd64 system:
>
> ===>  Verifying install for py-alabaster-* in textproc/py-alabaster
> ===>  Checking files for py-alabaster-0.7.10
> >> Fetch
> >> https://pypi.io/packages/source/a/alabaster/alabaster-0.7.10.tar.gz
> alabaster-0.7.10.tar.gz 100%
|**************************************************| 10486       00:00

> >> (SHA256) alabaster-0.7.10.tar.gz: OK
> ===> py-alabaster-0.7.10 depends on: python->=2.7,<2.8 ->
> python-2.7.15p1 ===> py-alabaster-0.7.10 depends on:
> py-setuptools->=39.0.1v0 -> py-setuptools-40.0.0v0
>
>
> From arm system:
>
> -----Original Message-----
> From: [hidden email] <[hidden email]> On Behalf Of
> [hidden email]
> Sent: November 5, 2018 10:17 AM
> To: [hidden email]
> Subject: fetch problem with //pypi.io/packages/source/
>
> I still get this when as part of a php build.
>
> ===>  Verifying install for py-alabaster-* in textproc/py-alabaster
> ===>  Checking files for py-alabaster-0.7.10
> >> Fetch
> >> https://pypi.io/packages/source/a/alabaster/alabaster-0.7.10.tar.gz
> ftp: SSL write error: handshake failed: Operation timed out
> >> Fetch
> https://ftp.openbsd.org/pub/OpenBSD/distfiles/alabaster-0.7.10.tar.gz
> alabaster-0.7.10.tar.gz 100% |**************************| 10486
00:00
> >> (SHA256) alabaster-0.7.10.tar.gz: OK
> ===> py-alabaster-0.7.10 depends on: python->=2.7,<2.8 ->
> python-2.7.15p1 ===> py-alabaster-0.7.10 depends on:
> py-setuptools->=39.0.1v0 ->
> py-setuptools-40.0.0v0
> ===>  Extracting for py-alabaster-0.7.10
>
> I can take the url and paste it into a Win web browser and it downloads
without a problem.  This problem only occurs on this site.
>
> Even it this cannot be resolved, is there a way to change the timeout
> to something reasonable, like about 20 seconds instead of the 5
> minutes? Trying to build on arm is slow enough without these delays,
> and there are quite a number of packages on this site
>
>

Reply | Threaded
Open this post in threaded view
|

Re: fetch problem with //pypi.io/packages/source/

Mark Kettenis
> From: <[hidden email]>
> Date: Tue, 6 Nov 2018 10:50:20 -0800
>
> Both system are on the same network, hub, router etc.  The amd system is on
> Virtualbox on my PC and the arm system is an orangepi one.
>
> >From the arm system:
>
> op1bsdsnap1102#
> op1bsdsnap1102# ftp -d -o /dev/null https://pypi.io/
> host pypi.io, port https, path , save as /dev/null, auth none.
> Trying 151.101.0.223...
> Requesting https://pypi.io/

I can reproduce this on my Allwinner A20 system (Banana Pi) but on on
NXP i.MX6 (Cubox-i4).  That suggests this is a network driver bug.
However, my device is using dwge(4), whereas yours is using dwxe(4)
isn't it?  It isn't inconceivable that both drivers have the same bug
though, since the hardware is very similar.

Reply | Threaded
Open this post in threaded view
|

Re: fetch problem with //pypi.io/packages/source/

s_graf
In reply to this post by s_graf
Yes, the orangepi one uses dwxe (allwinner H3).

-----Original Message-----
From: Mark Kettenis <[hidden email]>
Sent: November 6, 2018 11:19 AM
To: [hidden email]
Cc: [hidden email]; [hidden email]
Subject: Re: fetch problem with //pypi.io/packages/source/

> From: <[hidden email]>
> Date: Tue, 6 Nov 2018 10:50:20 -0800
>
> Both system are on the same network, hub, router etc.  The amd system
> is on Virtualbox on my PC and the arm system is an orangepi one.
>
> >From the arm system:
>
> op1bsdsnap1102#
> op1bsdsnap1102# ftp -d -o /dev/null https://pypi.io/ host pypi.io,
> port https, path , save as /dev/null, auth none.
> Trying 151.101.0.223...
> Requesting https://pypi.io/

I can reproduce this on my Allwinner A20 system (Banana Pi) but on on NXP
i.MX6 (Cubox-i4).  That suggests this is a network driver bug.
However, my device is using dwge(4), whereas yours is using dwxe(4) isn't
it?  It isn't inconceivable that both drivers have the same bug though,
since the hardware is very similar.

Reply | Threaded
Open this post in threaded view
|

Re: fetch problem with //pypi.io/packages/source/

Artturi Alm
In reply to this post by Mark Kettenis
On Tue, Nov 06, 2018 at 08:19:07PM +0100, Mark Kettenis wrote:

> > From: <[hidden email]>
> > Date: Tue, 6 Nov 2018 10:50:20 -0800
> >
> > Both system are on the same network, hub, router etc.  The amd system is on
> > Virtualbox on my PC and the arm system is an orangepi one.
> >
> > >From the arm system:
> >
> > op1bsdsnap1102#
> > op1bsdsnap1102# ftp -d -o /dev/null https://pypi.io/
> > host pypi.io, port https, path , save as /dev/null, auth none.
> > Trying 151.101.0.223...
> > Requesting https://pypi.io/
>
> I can reproduce this on my Allwinner A20 system (Banana Pi) but on on
> NXP i.MX6 (Cubox-i4).  That suggests this is a network driver bug.
> However, my device is using dwge(4), whereas yours is using dwxe(4)
> isn't it?  It isn't inconceivable that both drivers have the same bug
> though, since the hardware is very similar.
>

I was able to reproduce this on A20/dwge(4)/armv7 too, but not on
A64/dwxe(4)/arm64; just in case this does help narrowing down for the bug.

-Artturi

Reply | Threaded
Open this post in threaded view
|

Re: fetch problem with //pypi.io/packages/source/

Theo de Raadt-2
In reply to this post by s_graf
I have observed this before also.  A packet gets lost.

You can also visually observe this a different way.  Over a ssh session,
type ls in a large directory.  On all armv7 systems have a "pause" at
the end, getting caught up.

> The attached tcpdump traces are from armv7 and amd64 systems.  The problem
> on the armv7 system is very reproducible even though the pypi.io site uses
> different hosts at random.
> It has been many years since I analysed tcp traces and then it was with
> wireshark and so my abilities are lacking.  However I am suspicious about
> the sequence of packets:
>
> 21:49:47.217013 op1bsdtest.graf.lan.39119 > 151.101.64.223.https: S
> 891608003:891608003(0) win 16384 <mss 1460,nop,nop,sackOK,nop,wscale
> 6,nop,nop,timestamp 3208818117 0> (DF)
> 21:49:47.226246 151.101.64.223.https > op1bsdtest.graf.lan.39119: S
> 3712909619:3712909619(0) ack 891608004 win 28960 <mss 1460,sackOK,timestamp
> 447678784 3208818117,nop,wscale 9> (DF)
> 21:49:47.226339 op1bsdtest.graf.lan.39119 > 151.101.64.223.https: . ack 1
> win 256 <nop,nop,timestamp 3208818117 447678784> (DF)
> 21:49:47.312947 op1bsdtest.graf.lan.39119 > 151.101.64.223.https: P
> 1:210(209) ack 1 win 256 <nop,nop,timestamp 3208818117 447678784> (DF)
> 21:49:48.254031 151.101.64.223.https > op1bsdtest.graf.lan.39119: S
> 3712909619:3712909619(0) ack 891608004 win 28960 <mss 1460,sackOK,timestamp
> 44767904
>
> It looks like far end missed an ack from the arm system.
>
>
> -----Original Message-----
> From: Mark Kettenis <[hidden email]>
> Sent: November 6, 2018 11:19 AM
> To: [hidden email]
> Cc: [hidden email]; [hidden email]
> Subject: Re: fetch problem with //pypi.io/packages/source/
>
> > From: <[hidden email]>
> > Date: Tue, 6 Nov 2018 10:50:20 -0800
> >
> > Both system are on the same network, hub, router etc.  The amd system
> > is on Virtualbox on my PC and the arm system is an orangepi one.
> >
> > >From the arm system:
> >
> > op1bsdsnap1102#
> > op1bsdsnap1102# ftp -d -o /dev/null https://pypi.io/ host pypi.io,
> > port https, path , save as /dev/null, auth none.
> > Trying 151.101.0.223...
> > Requesting https://pypi.io/
>
> I can reproduce this on my Allwinner A20 system (Banana Pi) but on on NXP
> i.MX6 (Cubox-i4).  That suggests this is a network driver bug.
> However, my device is using dwge(4), whereas yours is using dwxe(4) isn't
> it?  It isn't inconceivable that both drivers have the same bug though,
> since the hardware is very similar.