go ports.

classic Classic list List threaded Threaded
17 messages Options
Reply | Threaded
Open this post in threaded view
|

go ports.

Edd Barrett
Hey,

I just tried porting a go application and ran into trouble. Looks like
there are no go ports in-tree, and I think I know why.

Go has one of these build systems that downloads (moving target)
dependencies. It reminds me of node.

Has anyone else tried porting go stuff?

I guess what we would need to do is port each individual go dependency we
need, then add a systrace policy that blocks the go build system from
downloading dependencies. Any comments on that strategy?

The application I was porting was pulse, a distributed file sync tool:
https://github.com/jasperla/openbsd-wip/tree/master/net/pulse

If this turns out to be useful, and if I find some time, I may have a go at
porting all the deps and making a go.port.mk to automate much of the path
jiggery pokery the go build system expects.

--
Best Regards
Edd Barrett

http://www.theunixzoo.co.uk

Reply | Threaded
Open this post in threaded view
|

Re: go ports.

Kenneth Westerback
On 19 October 2014 13:32, Edd Barrett <[hidden email]> wrote:
> Hey,
>
> I just tried porting a go application and ran into trouble. Looks like
> there are no go ports in-tree, and I think I know why.
>
> Go has one of these build systems that downloads (moving target)
> dependencies. It reminds me of node.
>
> Has anyone else tried porting go stuff?

This sounds similar to the challenges facing chrisz@ over Ocaml ports
vs opam. Perhaps the two of you should discuss a Grand Unified Theory
of Dependencies.

.... Ken

>
> I guess what we would need to do is port each individual go dependency we
> need, then add a systrace policy that blocks the go build system from
> downloading dependencies. Any comments on that strategy?
>
> The application I was porting was pulse, a distributed file sync tool:
> https://github.com/jasperla/openbsd-wip/tree/master/net/pulse
>
> If this turns out to be useful, and if I find some time, I may have a go at
> porting all the deps and making a go.port.mk to automate much of the path
> jiggery pokery the go build system expects.
>
> --
> Best Regards
> Edd Barrett
>
> http://www.theunixzoo.co.uk
>

Reply | Threaded
Open this post in threaded view
|

Re: go ports.

Stuart Henderson-6
In reply to this post by Edd Barrett
On 2014/10/19 18:32, Edd Barrett wrote:
> I guess what we would need to do is port each individual go dependency we
> need, then add a systrace policy that blocks the go build system from
> downloading dependencies. Any comments on that strategy?

systrace isn't the thing to use for this..

(Actually, now ports can build/fake/package as non-root, there isn't really
much need for systrace in ports any more.)

Reply | Threaded
Open this post in threaded view
|

Re: go ports.

Edd Barrett
On Sun, Oct 19, 2014 at 09:20:07PM +0100, Stuart Henderson wrote:
> On 2014/10/19 18:32, Edd Barrett wrote:
> > I guess what we would need to do is port each individual go dependency we
> > need, then add a systrace policy that blocks the go build system from
> > downloading dependencies. Any comments on that strategy?
>
> systrace isn't the thing to use for this..
>
> (Actually, now ports can build/fake/package as non-root, there isn't really
> much need for systrace in ports any more.)

Actually I wasn't quite right about pulse's build system. The build
itself doesn't download things. It builds pre-fetched dependencies
distributed in the tarball (the test target *does* however download things).

Now I am reminded of the way that ffmpeg is often statically embedded
into projects. Out of the frying pan and in to the fire.

That's not to say no go build will ever try to download things. Stuart, how
would you recommend preventing the go build system from downloading?

--
Best Regards
Edd Barrett

http://www.theunixzoo.co.uk

Reply | Threaded
Open this post in threaded view
|

Re: go ports.

Stuart Henderson-6
On 2014/10/20 09:44, Edd Barrett wrote:

> On Sun, Oct 19, 2014 at 09:20:07PM +0100, Stuart Henderson wrote:
> > On 2014/10/19 18:32, Edd Barrett wrote:
> > > I guess what we would need to do is port each individual go dependency we
> > > need, then add a systrace policy that blocks the go build system from
> > > downloading dependencies. Any comments on that strategy?
> >
> > systrace isn't the thing to use for this..
> >
> > (Actually, now ports can build/fake/package as non-root, there isn't really
> > much need for systrace in ports any more.)
>
> Actually I wasn't quite right about pulse's build system. The build
> itself doesn't download things. It builds pre-fetched dependencies
> distributed in the tarball (the test target *does* however download things).
>
> Now I am reminded of the way that ffmpeg is often statically embedded
> into projects. Out of the frying pan and in to the fire.
>
> That's not to say no go build will ever try to download things. Stuart, how
> would you recommend preventing the go build system from downloading?

I think we'd need to have something a bit like --skipdeps in perl's
Module::AutoInstall(3).

Reply | Threaded
Open this post in threaded view
|

Time to kill systrace? (was: Re: go ports.)

Christian Weisgerber
In reply to this post by Stuart Henderson-6
On 2014-10-19, Stuart Henderson <[hidden email]> wrote:

> (Actually, now ports can build/fake/package as non-root, there isn't really
> much need for systrace in ports any more.)

What remaining use is there?  Can't we just kill it?

--
Christian "naddy" Weisgerber                          [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Time to kill systrace? (was: Re: go ports.)

Landry Breuil-6
On Tue, Oct 21, 2014 at 02:58:52PM +0000, Christian Weisgerber wrote:
> On 2014-10-19, Stuart Henderson <[hidden email]> wrote:
>
> > (Actually, now ports can build/fake/package as non-root, there isn't really
> > much need for systrace in ports any more.)
>
> What remaining use is there?  Can't we just kill it?

Preventing network access from the build steps ?

Landry

Reply | Threaded
Open this post in threaded view
|

Re: Time to kill systrace? (was: Re: go ports.)

Stuart Henderson-6
On 2014/10/21 17:03, Landry Breuil wrote:
> On Tue, Oct 21, 2014 at 02:58:52PM +0000, Christian Weisgerber wrote:
> > On 2014-10-19, Stuart Henderson <[hidden email]> wrote:
> >
> > > (Actually, now ports can build/fake/package as non-root, there isn't really
> > > much need for systrace in ports any more.)
> >
> > What remaining use is there?  Can't we just kill it?

The only use I see at the moment is as a placeholder showing where we
might want to add some "sudo -u $PORTBUILDUSER" steps if we wanted to do
automated user switching via Makefiles, though that can be retrieved
from the attic.

> Preventing network access from the build steps ?

I'm fetching distfiles as my normal uid, then doing builds as pbuild.
pf.conf:

"block quick log proto {tcp udp} user pbuild"

Reply | Threaded
Open this post in threaded view
|

Re: Time to kill systrace? (was: Re: go ports.)

Theo de Raadt
In reply to this post by Christian Weisgerber
What if in the future you were to get a stronger better replacement
for systrace.  Would it slot into the same places in the scaffold?

Reply | Threaded
Open this post in threaded view
|

Re: Time to kill systrace? (was: Re: go ports.)

Amit Kulkarni-5
In reply to this post by Stuart Henderson-6
On Tue, Oct 21, 2014 at 10:28 AM, Stuart Henderson <[hidden email]>
wrote:

> On 2014/10/21 17:03, Landry Breuil wrote:
> > On Tue, Oct 21, 2014 at 02:58:52PM +0000, Christian Weisgerber wrote:
> > > On 2014-10-19, Stuart Henderson <[hidden email]> wrote:
> > >
> > > > (Actually, now ports can build/fake/package as non-root, there isn't
> really
> > > > much need for systrace in ports any more.)
> > >
> > > What remaining use is there?  Can't we just kill it?
>
> The only use I see at the moment is as a placeholder showing where we
> might want to add some "sudo -u $PORTBUILDUSER" steps if we wanted to do
> automated user switching via Makefiles, though that can be retrieved
> from the attic.
>
> > Preventing network access from the build steps ?
>
> I'm fetching distfiles as my normal uid, then doing builds as pbuild.
> pf.conf:
>
> "block quick log proto {tcp udp} user pbuild"
>
>
This can be disabled by user and bypassed, you can't bypass systrace during
ports build. Also, it would be possible to place files in FAKE /etc i.e in
places other than /usr/local?
Reply | Threaded
Open this post in threaded view
|

Re: Time to kill systrace? (was: Re: go ports.)

Stuart Henderson-6
On 2014/10/21 10:58, Amit Kulkarni wrote:
> On Tue, Oct 21, 2014 at 10:28 AM, Stuart Henderson <[hidden email]>
> > I'm fetching distfiles as my normal uid, then doing builds as pbuild.
> > pf.conf:
> >
> > "block quick log proto {tcp udp} user pbuild"
> >
> >
> This can be disabled by user and bypassed,

If you're aware of a way in which an unprivileged user can change PF
rules, it's probably best if you let me (or security@) know in private
mail.

> you can't bypass systrace during ports build. Also, it would be
> possible to place files in FAKE /etc i.e in places other than /usr/local?

I'm confused. It's ok if the port build puts things in directories
writable by the user doing port builds, because that user only has
filesystem permissions to write to a limited number of places
(mostly the build dir).

Reply | Threaded
Open this post in threaded view
|

Re: Time to kill systrace? (was: Re: go ports.)

patrick keshishian
On 10/21/14, Stuart Henderson <[hidden email]> wrote:

> On 2014/10/21 10:58, Amit Kulkarni wrote:
>> On Tue, Oct 21, 2014 at 10:28 AM, Stuart Henderson <[hidden email]>
>> > I'm fetching distfiles as my normal uid, then doing builds as pbuild.
>> > pf.conf:
>> >
>> > "block quick log proto {tcp udp} user pbuild"
>> >
>> >
>> This can be disabled by user and bypassed,
>
> If you're aware of a way in which an unprivileged user can change PF
> rules, it's probably best if you let me (or security@) know in private
> mail.

I read that comment as: the system admin, may not
(forgets to?) enable such a rule. Also, the pf rule route
seems a bit "clunky" and disjointed from the ports process.

>> you can't bypass systrace during ports build. Also, it would be
>> possible to place files in FAKE /etc i.e in places other than /usr/local?
>
> I'm confused. It's ok if the port build puts things in directories
> writable by the user doing port builds, because that user only has
> filesystem permissions to write to a limited number of places
> (mostly the build dir).

Consider a wip port, which may write files in $HOME, or
worse yet, delete files or directories from $HOME.

I always felt more at ease, knowing systrace would "slap"
the hand that attempted that, whether maliciously or
erroneously.

--patrick

Reply | Threaded
Open this post in threaded view
|

Re: Time to kill systrace? (was: Re: go ports.)

Amit Kulkarni-5
On Tue, Oct 21, 2014 at 12:18 PM, patrick keshishian <[hidden email]>
wrote:

> On 10/21/14, Stuart Henderson <[hidden email]> wrote:
> > On 2014/10/21 10:58, Amit Kulkarni wrote:
> >> On Tue, Oct 21, 2014 at 10:28 AM, Stuart Henderson <[hidden email]>
> >> > I'm fetching distfiles as my normal uid, then doing builds as pbuild.
> >> > pf.conf:
> >> >
> >> > "block quick log proto {tcp udp} user pbuild"
> >> >
> >> >
> >> This can be disabled by user and bypassed,
> >
> > If you're aware of a way in which an unprivileged user can change PF
> > rules, it's probably best if you let me (or security@) know in private
> > mail.
>
> I read that comment as: the system admin, may not
> (forgets to?) enable such a rule. Also, the pf rule route
> seems a bit "clunky" and disjointed from the ports process.
>


Somebody might disable the default PF rules and overwrite with their own,
and forget about it. If it isn't caught by anybody else port might get
committed. In that case, it will be caught in a bulk build by someone.
Generally, people don't touch systrace enable/disable, but they usually
fiddle with PF rules. But yes, this is immaterial. Patrick got the drift of
this, sorry for not explaining clearly in the initial email.


>
> >> you can't bypass systrace during ports build. Also, it would be
> >> possible to place files in FAKE /etc i.e in places other than
> /usr/local?
> >
> > I'm confused. It's ok if the port build puts things in directories
> > writable by the user doing port builds, because that user only has
> > filesystem permissions to write to a limited number of places
> > (mostly the build dir).
>
> Consider a wip port, which may write files in $HOME, or
> worse yet, delete files or directories from $HOME.
>
> I always felt more at ease, knowing systrace would "slap"
> the hand that attempted that, whether maliciously or
> erroneously.
>
> --patrick
>

+1

I am asking if user can create /etc/sysctl.conf in a port, that port
overwrites the real /etc/sysctl.conf because a port has superuser
privileges during install. If systrace is not there to catch it, would it
get installed? Or as Patrick says in another email: what about add/delete
in $HOME? Systrace protects us here. Is there any way to solve this problem
and the arbitrary net download problem during port building?
Reply | Threaded
Open this post in threaded view
|

Re: Time to kill systrace? (was: Re: go ports.)

Stuart Henderson-6
On 2014/10/21 13:32, Amit Kulkarni wrote:

> On Tue, Oct 21, 2014 at 12:18 PM, patrick keshishian <[hidden email]>
> wrote:
>
> > On 10/21/14, Stuart Henderson <[hidden email]> wrote:
> > > On 2014/10/21 10:58, Amit Kulkarni wrote:
> > >> On Tue, Oct 21, 2014 at 10:28 AM, Stuart Henderson <[hidden email]>
> > >> > I'm fetching distfiles as my normal uid, then doing builds as pbuild.
> > >> > pf.conf:
> > >> >
> > >> > "block quick log proto {tcp udp} user pbuild"
> > >> >
> > >> >
> > >> This can be disabled by user and bypassed,
> > >
> > > If you're aware of a way in which an unprivileged user can change PF
> > > rules, it's probably best if you let me (or security@) know in private
> > > mail.
> >
> > I read that comment as: the system admin, may not
> > (forgets to?) enable such a rule. Also, the pf rule route
> > seems a bit "clunky" and disjointed from the ports process.
> >
>
>
> Somebody might disable the default PF rules and overwrite with their own,
> and forget about it. If it isn't caught by anybody else port might get
> committed. In that case, it will be caught in a bulk build by someone.
> Generally, people don't touch systrace enable/disable, but they usually
> fiddle with PF rules. But yes, this is immaterial. Patrick got the drift of
> this, sorry for not explaining clearly in the initial email.

Exact opposite on most machines I use for ports work I hardly ever
touch PF rules there, but before we had build-as-nonroot I was often
doing 'make USE_SYSTRACE=Yes' when working on a new port or testing
a submission.

systrace is not without problems, it can make some syscalls fail which
people don't expect or check for, so can have an impact on how the build
works (another issue is that it's very noticably slower).

> > >> you can't bypass systrace during ports build. Also, it would be
> > >> possible to place files in FAKE /etc i.e in places other than
> > /usr/local?
> > >
> > > I'm confused. It's ok if the port build puts things in directories
> > > writable by the user doing port builds, because that user only has
> > > filesystem permissions to write to a limited number of places
> > > (mostly the build dir).
> >
> > Consider a wip port, which may write files in $HOME, or
> > worse yet, delete files or directories from $HOME.
> >
> > I always felt more at ease, knowing systrace would "slap"
> > the hand that attempted that, whether maliciously or
> > erroneously.

Yes that is a problem if you do builds as your normal userid.

> > --patrick
> >
>
> +1
>
> I am asking if user can create /etc/sysctl.conf in a port, that port
> overwrites the real /etc/sysctl.conf because a port has superuser
> privileges during install. If systrace is not there to catch it, would it
> get installed?

It does not, not any more - this whole mail thread probably makes
little sense if you missed that change.
http://undeadly.org/cgi?action=article&sid=20141003132339


>                Or as Patrick says in another email: what about add/delete
> in $HOME? Systrace protects us here. Is there any way to solve this problem
> and the arbitrary net download problem during port building?

Using a separate UID for builds protects against this too.

Reply | Threaded
Open this post in threaded view
|

Re: Time to kill systrace? (was: Re: go ports.)

patrick keshishian
On 10/21/14, Stuart Henderson <[hidden email]> wrote:

> On 2014/10/21 13:32, Amit Kulkarni wrote:
>> On Tue, Oct 21, 2014 at 12:18 PM, patrick keshishian <[hidden email]>
>> wrote:
>>
>> > On 10/21/14, Stuart Henderson <[hidden email]> wrote:
>> > > On 2014/10/21 10:58, Amit Kulkarni wrote:
>> > >> On Tue, Oct 21, 2014 at 10:28 AM, Stuart Henderson
>> > >> <[hidden email]>
>> > >> > I'm fetching distfiles as my normal uid, then doing builds as
>> > >> > pbuild.
>> > >> > pf.conf:
>> > >> >
>> > >> > "block quick log proto {tcp udp} user pbuild"
>> > >> >
>> > >> >
>> > >> This can be disabled by user and bypassed,
>> > >
>> > > If you're aware of a way in which an unprivileged user can change PF
>> > > rules, it's probably best if you let me (or security@) know in
>> > > private
>> > > mail.
>> >
>> > I read that comment as: the system admin, may not
>> > (forgets to?) enable such a rule. Also, the pf rule route
>> > seems a bit "clunky" and disjointed from the ports process.
>> >
>>
>>
>> Somebody might disable the default PF rules and overwrite with their own,
>> and forget about it. If it isn't caught by anybody else port might get
>> committed. In that case, it will be caught in a bulk build by someone.
>> Generally, people don't touch systrace enable/disable, but they usually
>> fiddle with PF rules. But yes, this is immaterial. Patrick got the drift
>> of
>> this, sorry for not explaining clearly in the initial email.
>
> Exact opposite on most machines I use for ports work I hardly ever
> touch PF rules there, but before we had build-as-nonroot I was often
> doing 'make USE_SYSTRACE=Yes' when working on a new port or testing
> a submission.
>
> systrace is not without problems, it can make some syscalls fail which
> people don't expect or check for, so can have an impact on how the build
> works (another issue is that it's very noticably slower).
>
>> > >> you can't bypass systrace during ports build. Also, it would be
>> > >> possible to place files in FAKE /etc i.e in places other than
>> > /usr/local?
>> > >
>> > > I'm confused. It's ok if the port build puts things in directories
>> > > writable by the user doing port builds, because that user only has
>> > > filesystem permissions to write to a limited number of places
>> > > (mostly the build dir).
>> >
>> > Consider a wip port, which may write files in $HOME, or
>> > worse yet, delete files or directories from $HOME.
>> >
>> > I always felt more at ease, knowing systrace would "slap"
>> > the hand that attempted that, whether maliciously or
>> > erroneously.
>
> Yes that is a problem if you do builds as your normal userid.

I don't mean to split hairs on this. You guys know better.
But, are you suggesting a $PORTUSER that the ports's
infrastructure will automatically sudo to for all make
commands? This I like!

Otherwise, as an example, I maintain a notes.txt for
ports I work on, contents of which includes a fair amount
of output from `make SHOW=this-and-that'. It would be a
bit of PITA to maintain that file as my everyday user.

OK, said my piece, thanks for considering it.

--patrick


>
>> > --patrick
>> >
>>
>> +1
>>
>> I am asking if user can create /etc/sysctl.conf in a port, that port
>> overwrites the real /etc/sysctl.conf because a port has superuser
>> privileges during install. If systrace is not there to catch it, would it
>> get installed?
>
> It does not, not any more - this whole mail thread probably makes
> little sense if you missed that change.
> http://undeadly.org/cgi?action=article&sid=20141003132339
>
>
>>                Or as Patrick says in another email: what about add/delete
>> in $HOME? Systrace protects us here. Is there any way to solve this
>> problem
>> and the arbitrary net download problem during port building?
>
> Using a separate UID for builds protects against this too.
>
>

Reply | Threaded
Open this post in threaded view
|

Re: Time to kill systrace? (was: Re: go ports.)

Stuart Henderson-6
In reply to this post by Theo de Raadt
On 2014/10/21 09:30, Theo de Raadt wrote:
> What if in the future you were to get a stronger better replacement
> for systrace.  Would it slot into the same places in the scaffold?

I think not. IMHO dpb would be a far better place to control privs.

The current scaffold calls systrace for individual build stages that
have reduced privileges. A diff to remove it gets rid of calls to
_SYSTRACE_CMD or to variables including _SYSTRACE_CMD in 29 places.
Coverage is variable and it's a pain to have different policies for
different things so we have things like "make patch" is protected
but "make fetch" isn't.

Now that we can build and package as non-root there are two parts
needing elevated privs:

pkg_add for dependencies: installs new files, modifies pkg database,
execute programs (makewhatis, mkfontdir/mkfontscale, rebuild icon
caches, update-desktop-databases, xml databases, and a few other less
common ones).

fetch: network access, write to distfiles directory

These two are exactly things which dpb already has a high level of
control over.

From this point until "make package" is done, privs aren't needed.
Dropping separately for all the various stages from this point on
doesn't make a lot of sense, just wrap the whole lot in one big
privdrop instead.

Then privs are needed again if you want to install the package you
just built.

Another problem with USE_SYSTRACE, it's under control of the ports
Makefiles themselves - it gives no protection to somebody who tries to
build a port they've downloaded from somewhere without reviewing it.
Proper protection against this when you build a port by cd'ing to the
directory and running "make" is hard. Handling privs in dpb would be
safer as it can force a low-priv user to run "make" as well.

One problem: dpb's UI isn't really suited to normal ports dev work.
I can think of a few specific changes that would probably be enough
to let it replace a direct "make" call as the main front-end without
getting in the way though.

Reply | Threaded
Open this post in threaded view
|

Re: Time to kill systrace? (was: Re: go ports.)

Antoine Jacoutot-7
> I think not. IMHO dpb would be a far better place to control privs.
>
> The current scaffold calls systrace for individual build stages that
> have reduced privileges. A diff to remove it gets rid of calls to
> _SYSTRACE_CMD or to variables including _SYSTRACE_CMD in 29 places.
> Coverage is variable and it's a pain to have different policies for
> different things so we have things like "make patch" is protected
> but "make fetch" isn't.

And it gets worse when ports redefine some targets.

--
Antoine