I have a mirror testing program for you.

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

I have a mirror testing program for you.

Luke Small
I have a 500 line program I wrote that reads openbsd.org.ftp.html and
scraps off the html and ftp mirrors, records them all without redundancies
as http mirrors in memory and downloads the appropriate version and machine
architecture's SHA256 in the package folder. It tests all the mirrors for
time, one at a time and uses kqueue to kill any laggy ftp calls. It uses
ftp() calls for all its networking, so it shouldn't be too much of a
security issue I'd guess. It writes the top 8 mirrors into /etc/pkg.conf it
erases all the installpath entries while leaving everything else in the
file. It can run as an unprivileged user, but of course it won't rewrite
/etc/pkg.conf
-Luke N Small

pkg_ping.c (15K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: I have a mirror testing program for you. - Mirror down

Benjamin Baier
Important thing first, the mirror http://openbsd.cs.fau.de/pub/OpenBSD/
seems to be down.

On Tue, 19 Jan 2016 22:19:42 -0600
Luke Small <[hidden email]> wrote:

> I have a 500 line program I wrote that reads openbsd.org.ftp.html and
> scraps off the html and ftp mirrors, records them all without redundancies
> as http mirrors in memory and downloads the appropriate version and machine
> architecture's SHA256 in the package folder. It tests all the mirrors for
> time, one at a time and uses kqueue to kill any laggy ftp calls. It uses
> ftp() calls for all its networking, so it shouldn't be too much of a
> security issue I'd guess. It writes the top 8 mirrors into /etc/pkg.conf it
> erases all the installpath entries while leaving everything else in the
> file. It can run as an unprivileged user, but of course it won't rewrite
> /etc/pkg.conf
> -Luke N Small
My quick-n-dirty script[1] tells me as long as I use
a mirror geographically close to me, I'll be fine.
- time diff in same country between different mirrors <=10%
- time diff same server tested multiple times <=10%
- time diff local vs. halfway around the earth >100% to <300%


[1] pkg_ping.sh
#!/bin/sh
#usage: ./pkg_ping.sh uk
#to test all the uk servers.
TMPFILE=/tmp/pkg_ping.tmp
SHATMP=/tmp/pkg_ping.sha

if [ ! -f ${TMPFILE} ]
then
        ftp -o - http://www.openbsd.org/build/mirrors.dat |\
        grep -e ^GZ -e ^UH |\
        grep -B1 -e ^UH |\
        grep -e ^GZ -e ^UH |\
        cut -f 2- > ${TMPFILE}
fi

grep -A1 -e "^$1" ${TMPFILE} | grep http |\
while read line
do
        echo trying ${line}
        time ftp -o ${SHATMP} ${line}/snapshots/packages/amd64/SHA256
done

Reply | Threaded
Open this post in threaded view
|

Re: I have a mirror testing program for you. - Mirror down

Stuart Henderson-6
On 2016/01/20 10:38, Benjamin Baier wrote:
> Important thing first, the mirror http://openbsd.cs.fau.de/pub/OpenBSD/
> seems to be down.

+cc maintainer, could you take a look please Simon? Down for v4+v6,
traceroute stops at informatik.gate.uni-erlangen.de (131.188.20.38 /
2001:638:a000::3341:41) with !A on v6.

> On Tue, 19 Jan 2016 22:19:42 -0600
> Luke Small <[hidden email]> wrote:
>
> > I have a 500 line program I wrote that reads openbsd.org.ftp.html and

Here's a simple alternative that will often be good enough.

ftp -o- -V http://www.openbsd.org/cgi-bin/ftplist.cgi |
    sed -e 's, .*,/%m/,' -e 's,^,pkgpath = ,' -e q

The C program is too trusting with its fixed-size buffers and unchecked
mallocs etc, it's not something to run as root as-is.

Reply | Threaded
Open this post in threaded view
|

Re: I have a mirror testing program for you.

Luke Small
In reply to this post by Luke Small
< The C program is too trusting with its fixed-size buffers and unchecked
< mallocs etc, it's not something to run as root as-is.

I realize I got a little lazy with no checking the mallocs, but that is
fixed.

I wrote this to be resource-light and thorough. No half-ass bullshit. If
somebody wants to not update their system for over a year, it will find the
remaining mirrors, no matter where they are.I got rid of the fixed size
buffers with the "totalLength" integer.

I also made the mirrors that instantly give an error have a larger diff
than the ones that merely took too long, just in case those are the only
ones available. Maybe I could enable an argument to increase the timeout
beyond 7 seconds.

pkg_ping.c (16K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: I have a mirror testing program for you. - Mirror down

Luke Small
In reply to this post by Stuart Henderson-6
OK, there, I put in error checks, so that the index used to write into the
arrays can't get too big.

-Luke

On Wed, Jan 20, 2016 at 4:27 AM, Stuart Henderson <[hidden email]> wrote:

> On 2016/01/20 10:38, Benjamin Baier wrote:
> > Important thing first, the mirror http://openbsd.cs.fau.de/pub/OpenBSD/
> > seems to be down.
>
> +cc maintainer, could you take a look please Simon? Down for v4+v6,
> traceroute stops at informatik.gate.uni-erlangen.de (131.188.20.38 /
> 2001:638:a000::3341:41) with !A on v6.
>
> > On Tue, 19 Jan 2016 22:19:42 -0600
> > Luke Small <[hidden email]> wrote:
> >
> > > I have a 500 line program I wrote that reads openbsd.org.ftp.html and
>
> Here's a simple alternative that will often be good enough.
>
> ftp -o- -V http://www.openbsd.org/cgi-bin/ftplist.cgi |
>     sed -e 's, .*,/%m/,' -e 's,^,pkgpath = ,' -e q
>
> The C program is too trusting with its fixed-size buffers and unchecked
> mallocs etc, it's not something to run as root as-is.
>

pkg_ping.c (16K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: I have a mirror testing program for you. - Mirror down

Luke Small
The real reason I wrote this is to have an automated way to set up the
pkg_add mirrors especially for folks that don't care to set them up
manually (Afterall, that's what computers are for!). Before I wrote this, I
had a PKG_PATH mirror go down and I didn't know what was going on. At least
this could get some failover that would work for everyone running the
release or older at least. I put in a minor edit that kills ftp and sed if
the buffer gets too full, as well as exiting.

>
>
>
>> > > I have a 500 line program I wrote that reads openbsd.org.ftp.html and
>>
>> Here's a simple alternative that will often be good enough.
>>
>> ftp -o- -V http://www.openbsd.org/cgi-bin/ftplist.cgi |
>>     sed -e 's, .*,/%m/,' -e 's,^,pkgpath = ,' -e q
>>
>> The C program is too trusting with its fixed-size buffers and unchecked
>> mallocs etc, it's not something to run as root as-is.
>>
>
>

pkg_ping.c (16K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: I have a mirror testing program for you. - Mirror down

Simon Kuhnle-2
In reply to this post by Stuart Henderson-6
Hi all,

On Wed, Jan 20, 2016 at 10:27:44AM +0000, Stuart Henderson wrote:
> On 2016/01/20 10:38, Benjamin Baier wrote:
> > Important thing first, the mirror http://openbsd.cs.fau.de/pub/OpenBSD/
> > seems to be down.
>
> +cc maintainer, could you take a look please Simon? Down for v4+v6,
> traceroute stops at informatik.gate.uni-erlangen.de (131.188.20.38 /
> 2001:638:a000::3341:41) with !A on v6.

I was about to write a mail to mirrors-discuss@ since Monday, sorry...

Seems like one of the disks gave up and the hardware RAID controller
messed up. I need to look at this on Saturday, when I have physical
access to the machine.

Stuart, can you maybe remove it from the list for now until I know how
bad it really is?

Regards,

Simon

Reply | Threaded
Open this post in threaded view
|

Re: I have a mirror testing program for you. - Mirror down

Luke Small
In reply to this post by Luke Small
Ok. I added a lot of security fixes added a feature to put in a custom
floating point timeout as an argument and got rid of the 8 mirror limit. It
puts in all the mirrors that didn't either exceed the timeout period or
have a download error. It should be safe to run as root I guess. There is
nothing that can be done to make it core dump. The only thing, I suspect,
that can go wrong is a man in the middle attack downloading ftp.html. Is
there even a hash value for ftp.html ?

-Luke

On Thu, Jan 21, 2016 at 1:18 AM, Luke Small <[hidden email]> wrote:

> The real reason I wrote this is to have an automated way to set up the
> pkg_add mirrors especially for folks that don't care to set them up
> manually (Afterall, that's what computers are for!). Before I wrote this, I
> had a PKG_PATH mirror go down and I didn't know what was going on. At least
> this could get some failover that would work for everyone running the
> release or older at least. I put in a minor edit that kills ftp and sed if
> the buffer gets too full, as well as exiting.
>
>>
>>
>>
>>> > > I have a 500 line program I wrote that reads openbsd.org.ftp.html and
>>>
>>> Here's a simple alternative that will often be good enough.
>>>
>>> ftp -o- -V http://www.openbsd.org/cgi-bin/ftplist.cgi |
>>>     sed -e 's, .*,/%m/,' -e 's,^,pkgpath = ,' -e q
>>>
>>> The C program is too trusting with its fixed-size buffers and unchecked
>>> mallocs etc, it's not something to run as root as-is.
>>>
>>
>>
>

pkg_ping.c (19K) Download Attachment