pkg_add and authentication

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

pkg_add and authentication

Antti Harri

Hello,

what's the correct method to make pkg_add understand
PKG_PATH=ftp://[hidden email]/path/to/packages/ ?

In my case scp is also available (why isn't sftp:// working
btw?), but it's *terribly* slow so I'd rather just use ftp.

--
Antti Harri

Reply | Threaded
Open this post in threaded view
|

Re: pkg_add and authentication

Ingo Schwarze
Hi Antti,

your questions can be answered from
  /usr/src/usr.sbin/pkg_add/OpenBSD/PackageRepository.pm:

Antti Harri wrote on Mon, May 28, 2007 at 10:19:59AM +0300:
> what's the correct method to make pkg_add understand
> PKG_PATH=ftp://[hidden email]/path/to/packages/ ?

OpenBSD::PackageRepository::HTTPorFTP::grab_object()
simply issues
  ftp -o - $self->{baseurl}.$object->{name}.".tgz", i.e.
  ftp -o - ftp://[hidden email]/path/to/packages/mydistro.tgz
which is NOT supported by ftp(1).

Instead, the syntax specified in ftp(1) is
  ftp [-o output] ftp://[user:password@]host[:port]/file[/]
so you need to specify the password, i.e.
  PKG_PATH=ftp://ftp:[hidden email]/path/to/packages/

Note using this for anything in any context is almost certainly
a bad idea.  It sends cleartext passwords unencrypted over the
wire and includes them in arguments passed to exec(3), exposing
them to ps(1).  In case neither is an issue in your context, why
do you need authentication at all?  By the way, if your packages
are secret, are you sure you want to install them?  On the other
hand, in case you just use wierd non-standard usernames for
anonymous ftp, you are successfully implementing obscurity by
obscurity which is probably a bad idea, too.

Thus, i suggest to just use standard anonymous ftp instead.

> In my case scp is also available (why isn't sftp:// working
> btw?), but it's *terribly* slow so I'd rather just use ftp.

sftp:// simply isn't implemented, see the list in
OpenBSD::PackageRepository::new(): ftp, http, scp, src.
By the way, i have no idea why sftp should be quicker than scp.

Still, sftp:// might be useful for servers offering SFTP access
without offering shell access.  Try submitting a patch to
OpenBSD/PackageRepository* if you need it.  The pkg_add part
will probably be easy.  What might be a bother is that sftp(1),
contrary to ftp(1), doesn't appear to support output to stdout(4).

Yours,
  Ingo

Reply | Threaded
Open this post in threaded view
|

Re: pkg_add and authentication

Antti Harri

On Mon, 28 May 2007, Ingo Schwarze wrote:

> Hi Antti,

Hi and thanks for your reply.

[snip]
> Instead, the syntax specified in ftp(1) is
>  ftp [-o output] ftp://[user:password@]host[:port]/file[/]
> so you need to specify the password, i.e.
>  PKG_PATH=ftp://ftp:[hidden email]/path/to/packages/

Yeah I read the manual before posting but didn't realize
it needs password too and I was uncomfortable providing
it from the command line.

> [snip] By the way, if your packages
> are secret, are you sure you want to install them?  On the other
> hand, in case you just use wierd non-standard usernames for
> anonymous ftp, you are successfully implementing obscurity by
> obscurity which is probably a bad idea, too.

For snapshot installation I fetch the whole tree to a server
near by in case I have to install something later on. I
don't want to mix packages from newer snapshots.

> Thus, i suggest to just use standard anonymous ftp instead.

Earlier the server in question was in fact anonymous, not anymore.
The source I'm using is and it's official OpenBSD mirror.

> sftp:// simply isn't implemented, see the list in
> OpenBSD::PackageRepository::new(): ftp, http, scp, src.
> By the way, i have no idea why sftp should be quicker than scp.

The reason why I was asking this is that usually sftp:// just works
so I tried it first, took a while to notice I have to specify
"scp".

> Still, sftp:// might be useful for servers offering SFTP access
> without offering shell access.  Try submitting a patch to
> OpenBSD/PackageRepository* if you need it.  The pkg_add part
> will probably be easy.  What might be a bother is that sftp(1),
> contrary to ftp(1), doesn't appear to support output to stdout(4).

I'm not that much of perl coder 8-)

PS. Isn't there some library/layer/whatever for perl that could be used to
enable protocol independent package fetching? Something like
kioslave in kde? I don't see a reason why one would have to implement all
url schemes from scratch..

--
Antti Harri

Reply | Threaded
Open this post in threaded view
|

Re: pkg_add and authentication

Stuart Henderson
On 2007/05/28 15:00, Antti Harri wrote:
>  PS. Isn't there some library/layer/whatever for perl that could be used to
>  enable protocol independent package fetching?

scp:// in pkg_add is a bit special, it implements it's own protocol
which is run by a program that's piped to Perl on the machine holding
the packages.

/usr/libdata/perl5/OpenBSD/PackageRepository/SCP.pm

Reply | Threaded
Open this post in threaded view
|

Re: pkg_add and authentication

Marc Espie-2
In reply to this post by Antti Harri
On Mon, May 28, 2007 at 03:00:51PM +0300, Antti Harri wrote:
> PS. Isn't there some library/layer/whatever for perl that could be used to
> enable protocol independent package fetching? Something like
> kioslave in kde? I don't see a reason why one would have to implement all
> url schemes from scratch..

pkg_add shares as much code as it can... The fact is, the scp: part is
actually more efficient, because it establishes exactly one connection
and talks with the other side.

Reply | Threaded
Open this post in threaded view
|

Re: pkg_add and authentication

Joachim Schipper
In reply to this post by Antti Harri
On Mon, May 28, 2007 at 03:00:51PM +0300, Antti Harri wrote:

>
> On Mon, 28 May 2007, Ingo Schwarze wrote:
>
> >Hi Antti,
>
> Hi and thanks for your reply.
>
> [snip]
> >Instead, the syntax specified in ftp(1) is
> > ftp [-o output] ftp://[user:password@]host[:port]/file[/]
> >so you need to specify the password, i.e.
> > PKG_PATH=ftp://ftp:[hidden email]/path/to/packages/
>
> Yeah I read the manual before posting but didn't realize
> it needs password too and I was uncomfortable providing
> it from the command line.
>
> >[snip] By the way, if your packages
> >are secret, are you sure you want to install them?  On the other
> >hand, in case you just use wierd non-standard usernames for
> >anonymous ftp, you are successfully implementing obscurity by
> >obscurity which is probably a bad idea, too.
>
> For snapshot installation I fetch the whole tree to a server
> near by in case I have to install something later on. I
> don't want to mix packages from newer snapshots.

Does 'the whole tree' mean
ftp://ftp.openbsd.org/pub/OpenBSD/snapshots/packages/i386/* in this
case? In that case, don't do that - it puts far too much strain on the
mirrors.

                Joachim

--
TFMotD: kgmon (8) - generate a dump of the operating system's profile
buffers

Reply | Threaded
Open this post in threaded view
|

Re: pkg_add and authentication

Antti Harri

On Tue, 29 May 2007, Joachim Schipper wrote:

> Does 'the whole tree' mean
> ftp://ftp.openbsd.org/pub/OpenBSD/snapshots/packages/i386/* in this
> case? In that case, don't do that - it puts far too much strain on the
> mirrors.

What do you suggest I should do if I want to install stuff later on? New
snapshot overwrites stuff on FTP about once in a week (i386). So about in
a week there are no more packages that have been build against the base
I'm running. I'd have to install newer base snapshot if the libraries
mismatch, wouldn't I?

I doubt it puts *that* much strain on any of the mirrors if I
fetch the tree once in a two months that I use for a handful of
installations. Other mirrors (official and unofficial) download
snapshots/* way more often.

--
Antti Harri

Reply | Threaded
Open this post in threaded view
|

Re: pkg_add and authentication

Marc Espie-2
On Tue, May 29, 2007 at 06:21:22PM +0300, Antti Harri wrote:

>
> On Tue, 29 May 2007, Joachim Schipper wrote:
>
> >Does 'the whole tree' mean
> >ftp://ftp.openbsd.org/pub/OpenBSD/snapshots/packages/i386/* in this
> >case? In that case, don't do that - it puts far too much strain on the
> >mirrors.
>
> What do you suggest I should do if I want to install stuff later on? New
> snapshot overwrites stuff on FTP about once in a week (i386). So about in
> a week there are no more packages that have been build against the base
> I'm running. I'd have to install newer base snapshot if the libraries
> mismatch, wouldn't I?
>

I would tend to side with you...

This is an actual problem with the way mirroring works: there is so much
stuff to copy that mirrors only have one single snapshot, and the new
one overrides the old.

This is an issue if you happen to run into a mirror that is in the middle
of updating, because it will have old packages and new packages.

As it stands, it works most of the time because stuff is more less upwards
compatible from one snapshot to the next.

The correct solution would be to have date tags on snapshots, and have
mirrors be much smarter about what they do... this also means they would
need twice as much room to store stuff, so this is totally impractical.

With the way stuff currently works, you have no real choice if you run current
from binaries and you want reliability but to mirror whatever you need
locally.

Reply | Threaded
Open this post in threaded view
|

Re: pkg_add and authentication

Tom Cosgrove-2
In reply to this post by Antti Harri
>>> Antti Harri 29-May-07 16:21 >>>
>
> On Tue, 29 May 2007, Joachim Schipper wrote:
>
> > Does 'the whole tree' mean
> > ftp://ftp.openbsd.org/pub/OpenBSD/snapshots/packages/i386/* in this
> > case? In that case, don't do that - it puts far too much strain on
> > the mirrors.
>
> What do you suggest I should do if I want to install stuff later
> on? New snapshot overwrites stuff on FTP about once in a week
> (i386). So about in a week there are no more packages that have been
> build against the base I'm running. I'd have to install newer base
> snapshot if the libraries mismatch, wouldn't I?

Do what many of us do: install the stuff you know you need/think you
will need when you first install, then if you need something later
try the snapshot package; if it doesn't work either update to a more
recent snapshot (and update packages) or use ports.

If you're using snapshots, you are expected to use a bit of brain
power rather than hog the bandwidth :)

> I doubt it puts *that* much strain on any of the mirrors if I
> fetch the tree once in a two months that I use for a handful of
> installations. Other mirrors (official and unofficial) download
> snapshots/* way more often.

Please don't do this unless you want to offer the files to others
yourself (i.e. host a mirror).

Tom

Reply | Threaded
Open this post in threaded view
|

Re: pkg_add and authentication

Joachim Schipper
In reply to this post by Antti Harri
On Tue, May 29, 2007 at 06:21:22PM +0300, Antti Harri wrote:

>
> On Tue, 29 May 2007, Joachim Schipper wrote:
>
> >Does 'the whole tree' mean
> >ftp://ftp.openbsd.org/pub/OpenBSD/snapshots/packages/i386/* in this
> >case? In that case, don't do that - it puts far too much strain on the
> >mirrors.
>
> What do you suggest I should do if I want to install stuff later on? New
> snapshot overwrites stuff on FTP about once in a week (i386). So about in
> a week there are no more packages that have been build against the base
> I'm running. I'd have to install newer base snapshot if the libraries
> mismatch, wouldn't I?
>
> I doubt it puts *that* much strain on any of the mirrors if I
> fetch the tree once in a two months that I use for a handful of
> installations. Other mirrors (official and unofficial) download
> snapshots/* way more often.

Well, it seems Espie agrees, so who am I?

Still, this is not really necessary. Updating to a newer -current isn't
too bad, and pretty much needed if you want to keep getting/testing the
newest features, anyway; plus, you can always install most of what you
need and build the one or two packages you missed from the ports tree.
If you have room to mirror the entire FTP tree, you certainly have room
to keep a ports tree around, after all.

Mirroring a select subset of packages is a very good idea, of course,
but I don't really see why you'd need to keep everything around.

                Joachim

(Disclaimer: you'll not find too much help if something fails to build
from a ports tree three months old. Then again, the same problem is
true for packages.)
--
TFMotD: ttys (5) - terminal initialization information