waf woes

classic Classic list List threaded Threaded
20 messages Options
Reply | Threaded
Open this post in threaded view
|

waf woes

frantisek holop

no waf maintainer so i am putting this out here for
discussion.

right, so mpv in their next release switched from the
mplayer inherited custom configure/gmake build system
to waf.  good move? at first glance yes, as probably
anything is better then gnu autotools.

unfortunately waf is a python program the kind of i
have never seen before. the "waf" file contains a bzip2
stream containing the python library files that are
extracted upon running into a hidden directory where
the waf executable resides (so having a single
/usr/local/bin/waf and running that in the project
directory is a no go, as it will try to extract to
/usr/local/bin/.waf-* and not $PWD)

furthermore waf is hostile to packaging (there is in an
ancient version in the ports), su much so, that the
install target has been removed and the "waf book"
clearly states that the preferred modus operandi is to
put the waf file into the project directory.

this wouldnt be such an issue if all the projects using
it would simply have it in their release archives. this
is not the case. mpv provides a "bootstrap.py" file
that downloads their preferred version.

some possible solutions:

1. post-extract runs bootsrap.py (python is already a
   dependency anyway)

2. post-extract copies files/waf to ${WRKDIST}

3. patch waf so it tries to unpack into $PWD
   (${WRKDIST}) instead of where the waf file resides.

4. update the waf port, but this is a can of worms as
   a) it "does not want" to be packaged
   b) probably breaks all other projects using waf that
      do not contain their own copy, in other words
      the port would have to contain every single waf
      version and BUILD_DEPENDS would pull in the
      correct version the project has been tested with.

no 3 seems like an elegant solution (intead of trying
to shoehorn waf into site-packages) and the problem of
multiple versions for multiple ports could be solved by
making a waf package that contains every single waf
version (that are used by ports atm) and pulling in the
good version somehow in the makefile.  so the package
would install:

/usr/local/bin/waf-a.b.c
/usr/local/bin/waf-d.e.f
...

and with that frankenstein idea i wish everyone a happy
new bsd year.

-f
--
"give me two personal pronouns." "who, me?" -- benny hill

Reply | Threaded
Open this post in threaded view
|

Re: waf woes

Landry Breuil-6
On Tue, Dec 31, 2013 at 01:26:03PM +0100, frantisek holop wrote:
>
> no waf maintainer so i am putting this out here for
> discussion.
>
> right, so mpv in their next release switched from the
> mplayer inherited custom configure/gmake build system
> to waf.  good move? at first glance yes, as probably
> anything is better then gnu autotools.

Hell no. Waf is far worse than autohell, and is in the same boat as gyp,
scons & jam in my book.

Landry

Reply | Threaded
Open this post in threaded view
|

Re: waf woes

frantisek holop
hmm, on Tue, Dec 31, 2013 at 01:30:33PM +0100, Landry Breuil said that

> On Tue, Dec 31, 2013 at 01:26:03PM +0100, frantisek holop wrote:
> >
> > no waf maintainer so i am putting this out here for
> > discussion.
> >
> > right, so mpv in their next release switched from the
> > mplayer inherited custom configure/gmake build system
> > to waf.  good move? at first glance yes, as probably
> > anything is better then gnu autotools.
>
> Hell no. Waf is far worse than autohell, and is in the same boat as gyp,
> scons & jam in my book.

i thought autohell is considered the rock bottom :]
waf is at least not in m4 :]

-f
--
what, me ambivalent?  well, yes and no.

Reply | Threaded
Open this post in threaded view
|

Re: waf woes

Stuart Henderson-6
On 2013/12/31 13:34, frantisek holop wrote:

> hmm, on Tue, Dec 31, 2013 at 01:30:33PM +0100, Landry Breuil said that
> > On Tue, Dec 31, 2013 at 01:26:03PM +0100, frantisek holop wrote:
> > >
> > > no waf maintainer so i am putting this out here for
> > > discussion.
> > >
> > > right, so mpv in their next release switched from the
> > > mplayer inherited custom configure/gmake build system
> > > to waf.  good move? at first glance yes, as probably
> > > anything is better then gnu autotools.
> >
> > Hell no. Waf is far worse than autohell, and is in the same boat as gyp,
> > scons & jam in my book.
>
> i thought autohell is considered the rock bottom :]

It's a huge mess, but at least there's a decent amount of knowledge
about how to work with it, both amongst packagers, and amongst (some ;)
upstream developers. And it's already been modified as needed to
handle things for OpenBSD.

> waf is at least not in m4 :]

waf seems at something like the stage of autoconf 2.13 or so, where
many people using it are patching it and want you to use their own
version, making it impossible to centralise any os-specific changes
that are needed.

If people are looking for a non-autoconf build system, cmake seems
about the best choice.. it's not perfect, but CMakefiles are fairly
readable, and it's nicely cross-platform..

Reply | Threaded
Open this post in threaded view
|

Re: waf woes

frantisek holop
hmm, on Tue, Dec 31, 2013 at 01:06:47PM +0000, Stuart Henderson said that

> On 2013/12/31 13:34, frantisek holop wrote:
> > hmm, on Tue, Dec 31, 2013 at 01:30:33PM +0100, Landry Breuil said that
> > > On Tue, Dec 31, 2013 at 01:26:03PM +0100, frantisek holop wrote:
> > > >
> > > > no waf maintainer so i am putting this out here for
> > > > discussion.
> > > >
> > > > right, so mpv in their next release switched from the
> > > > mplayer inherited custom configure/gmake build system
> > > > to waf.  good move? at first glance yes, as probably
> > > > anything is better then gnu autotools.
> > >
> > > Hell no. Waf is far worse than autohell, and is in the same boat as gyp,
> > > scons & jam in my book.
> >
> > i thought autohell is considered the rock bottom :]
>
> It's a huge mess, but at least there's a decent amount of knowledge
> about how to work with it, both amongst packagers, and amongst (some ;)
> upstream developers. And it's already been modified as needed to
> handle things for OpenBSD.

waf.port.mk is very nice, but as it stands, it will
probably work with the only 2(?) ports that rely on it.

it seems like, just like autoconf, it will need
multiple versions in ports.

i have thrown together the frankenstein port that under
the latest upstream version installs 3 versions (current
in port, mpv needed and latest) but i am stuck as the
patch to extract the stream into ${WRKDIST} strips away
the bzip2 stream :(  how can i patch "binary files"?

-f
--
in an atomic war, all men will be cremated equal.

Reply | Threaded
Open this post in threaded view
|

Re: waf woes

Brad Smith-14
In reply to this post by frantisek holop
On 31/12/13 7:34 AM, frantisek holop wrote:

> hmm, on Tue, Dec 31, 2013 at 01:30:33PM +0100, Landry Breuil said that
>> On Tue, Dec 31, 2013 at 01:26:03PM +0100, frantisek holop wrote:
>>>
>>> no waf maintainer so i am putting this out here for
>>> discussion.
>>>
>>> right, so mpv in their next release switched from the
>>> mplayer inherited custom configure/gmake build system
>>> to waf.  good move? at first glance yes, as probably
>>> anything is better then gnu autotools.
>>
>> Hell no. Waf is far worse than autohell, and is in the same boat as gyp,
>> scons & jam in my book.
>
> i thought autohell is considered the rock bottom :]
> waf is at least not in m4 :]

Sadly enough autohell is the suck least of build infrastructure and
there is a lot of documentation and knowledge regarding its inner
workings. IMO not something that can be said about the other build
infrastructure whether it is relatively common or not.

It might not be m4, but it is python, that's a pretty heavy dependency
for build infrastructure.

--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.

Reply | Threaded
Open this post in threaded view
|

Re: waf woes

Brad Smith-14
In reply to this post by Stuart Henderson-6
On 31/12/13 8:06 AM, Stuart Henderson wrote:
> It's a huge mess, but at least there's a decent amount of knowledge
> about how to work with it, both amongst packagers, and amongst (some ;)
> upstream developers. And it's already been modified as needed to
> handle things for OpenBSD.

Exactly, core autoconf and more of the related macros and so forth are
well supported on OpenBSD.

>> waf is at least not in m4 :]
>
> waf seems at something like the stage of autoconf 2.13 or so, where
> many people using it are patching it and want you to use their own
> version, making it impossible to centralise any os-specific changes
> that are needed.

IMO it is worse, at least with autoconf at the macro level it tended to
be relatively easy to copy things around and fix issues. With most
projects using WAF 1.5 and upstream being at WAF 1.7 and with the
differences between them it can be a bigger issue having issues fixed
between upstream and then back porting to projects local copies
of WAF 1.5.

> If people are looking for a non-autoconf build system, cmake seems
> about the best choice.. it's not perfect, but CMakefiles are fairly
> readable, and it's nicely cross-platform..

Yes, CMake isn't perfect but its a fairly good start and going in
the right direction. Quite a few projects are using CMake or are
switching.

--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.

Reply | Threaded
Open this post in threaded view
|

Re: waf woes

frantisek holop
In reply to this post by Brad Smith-14
hmm, on Tue, Dec 31, 2013 at 03:00:40PM -0500, Brad Smith said that
> >i thought autohell is considered the rock bottom :]
> >waf is at least not in m4 :]
>
> Sadly enough autohell is the suck least of build infrastructure and
> there is a lot of documentation and knowledge regarding its inner
> workings. IMO not something that can be said about the other build
> infrastructure whether it is relatively common or not.

i have no experience with "alternative" build systems,
but i am no fan of autoconf. having lots of knowledge
about a crazy design/implementation does not make it
not crazy, only predictably crazy.

> It might not be m4, but it is python, that's a pretty heavy dependency
> for build infrastructure.

i dont perceive python a heavier dependency than e.g.
perl, but i am biased as it is my favourite scripting
language.  it sure seems more sexy to me to write
python wscripts than to mess with m4, shell and autoconf.
but i am not a developer of a (huge) project that needs
a sophisticated build system, so i dont know which one
is the least pain.

right now i am just trying to make waf work for
everybody.

ps. could someone upload somewhere a libc.so.72.0?
(or send me in mail) i deleted it by mistake. sigh.

-f
--
want to forget all your troubles?  wear tight shoes.

Reply | Threaded
Open this post in threaded view
|

Re: waf woes

Brad Smith-14
On 31/12/13 3:35 PM, frantisek holop wrote:
> having lots of knowledge about a crazy design/implementation does not make it
> not crazy, only predictably crazy.

No one is denying that it isn't crazy, but as you said its predictable
crazy and for the most part works; it has also been around for a long
time so it generally works well across a wide range of operating
systems. IMO a lot of the other build infrastructure is crazier to
use.

>> It might not be m4, but it is python, that's a pretty heavy dependency
>> for build infrastructure.
>
> i dont perceive python a heavier dependency than e.g.
> perl, but i am biased as it is my favourite scripting
> language.  it sure seems more sexy to me to write
> python wscripts than to mess with m4, shell and autoconf.
> but i am not a developer of a (huge) project that needs
> a sophisticated build system, so i dont know which one
> is the least pain.

autoconf scripts are shell scripts and then some generated
Makefile versus Python.. for me dealing with autoconf is less
pain than dealing with WAF.

> right now i am just trying to make waf work for
> everybody.

Well, like autoconf, CMake or anything else there is the core
functionality and then the bits for each respective project
that uses the build infrastructure. Upstream 1.7 at least knows
how to deal with OpenBSD for shared libs; though I'm sure WAF
itself probably needs some more work for OpenBSD.

--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.

Reply | Threaded
Open this post in threaded view
|

Re: waf woes

Ian McWilliam-2
In reply to this post by Brad Smith-14
On 1/01/2014 7:00 AM, Brad Smith wrote:

<snip>
> Sadly enough autohell is the suck least of build infrastructure and
> there is a lot of documentation and knowledge regarding its inner
> workings. IMO not something that can be said about the other build
> infrastructure whether it is relatively common or not.
>
> It might not be m4, but it is python, that's a pretty heavy dependency
> for build infrastructure.
>

Yup and any new samba + external samba deps is riddled with it. It's
what is stopping me from moving forward atm. Need to learn python. At
least I have a few weeks to pull out some python books.

A newer in tree waf, or multiple versions would help ease the situation
rather than patch the hell out of samba build infrastructure.

Ian McWilliam

Reply | Threaded
Open this post in threaded view
|

Re: waf woes

Landry Breuil-6
On Thu, Jan 02, 2014 at 11:40:40PM +1100, Ian McWilliam wrote:

> On 1/01/2014 7:00 AM, Brad Smith wrote:
>
> <snip>
> >Sadly enough autohell is the suck least of build infrastructure and
> >there is a lot of documentation and knowledge regarding its inner
> >workings. IMO not something that can be said about the other build
> >infrastructure whether it is relatively common or not.
> >
> >It might not be m4, but it is python, that's a pretty heavy dependency
> >for build infrastructure.
> >
>
> Yup and any new samba + external samba deps is riddled with it. It's what is
> stopping me from moving forward atm. Need to learn python. At least I have a
> few weeks to pull out some python books.
>
> A newer in tree waf, or multiple versions would help ease the situation
> rather than patch the hell out of samba build infrastructure.

I've moved away all waf users from it (ie x11/gigolo & www/vteplugin)
but i'd rather remove waf from the tree instead of having someone losing
time on that crap. Given that it's clearly hostile to any packaging
effort, why bother with it ? Just do the minimum and patch all the
bundled copies.

Landry

Reply | Threaded
Open this post in threaded view
|

Re: waf woes

Antoine Jacoutot-7
On Thu, Jan 02, 2014 at 03:19:16PM +0100, Landry Breuil wrote:

> On Thu, Jan 02, 2014 at 11:40:40PM +1100, Ian McWilliam wrote:
> > On 1/01/2014 7:00 AM, Brad Smith wrote:
> >
> > <snip>
> > >Sadly enough autohell is the suck least of build infrastructure and
> > >there is a lot of documentation and knowledge regarding its inner
> > >workings. IMO not something that can be said about the other build
> > >infrastructure whether it is relatively common or not.
> > >
> > >It might not be m4, but it is python, that's a pretty heavy dependency
> > >for build infrastructure.
> > >
> >
> > Yup and any new samba + external samba deps is riddled with it. It's what is
> > stopping me from moving forward atm. Need to learn python. At least I have a
> > few weeks to pull out some python books.
> >
> > A newer in tree waf, or multiple versions would help ease the situation
> > rather than patch the hell out of samba build infrastructure.
>
> I've moved away all waf users from it (ie x11/gigolo & www/vteplugin)
> but i'd rather remove waf from the tree instead of having someone losing
> time on that crap. Given that it's clearly hostile to any packaging
> effort, why bother with it ? Just do the minimum and patch all the
> bundled copies.

I agree.

--
Antoine

Reply | Threaded
Open this post in threaded view
|

Re: waf woes

Vadim Zhukov
In reply to this post by Landry Breuil-6
2014/1/2 Landry Breuil <[hidden email]>:

> On Thu, Jan 02, 2014 at 11:40:40PM +1100, Ian McWilliam wrote:
>> On 1/01/2014 7:00 AM, Brad Smith wrote:
>>
>> <snip>
>> >Sadly enough autohell is the suck least of build infrastructure and
>> >there is a lot of documentation and knowledge regarding its inner
>> >workings. IMO not something that can be said about the other build
>> >infrastructure whether it is relatively common or not.
>> >
>> >It might not be m4, but it is python, that's a pretty heavy dependency
>> >for build infrastructure.
>> >
>>
>> Yup and any new samba + external samba deps is riddled with it. It's what is
>> stopping me from moving forward atm. Need to learn python. At least I have a
>> few weeks to pull out some python books.
>>
>> A newer in tree waf, or multiple versions would help ease the situation
>> rather than patch the hell out of samba build infrastructure.
>
> I've moved away all waf users from it (ie x11/gigolo & www/vteplugin)
> but i'd rather remove waf from the tree instead of having someone losing
> time on that crap. Given that it's clearly hostile to any packaging
> effort, why bother with it ? Just do the minimum and patch all the
> bundled copies.

I'll learn jig to dance it on the WAF grave.

--
  WBR,
  Vadim Zhukov

Reply | Threaded
Open this post in threaded view
|

Re: waf woes

Antoine Jacoutot-7
> I'll learn jig to dance it on the WAF grave.

Can't wait to see that ;-)

--
Antoine

Reply | Threaded
Open this post in threaded view
|

Re: waf woes

frantisek holop
In reply to this post by Landry Breuil-6
hmm, on Thu, Jan 02, 2014 at 03:19:16PM +0100, Landry Breuil said that
> I've moved away all waf users from it (ie x11/gigolo & www/vteplugin)
> but i'd rather remove waf from the tree instead of having someone losing
> time on that crap. Given that it's clearly hostile to any packaging
> effort, why bother with it ? Just do the minimum and patch all the
> bundled copies.

i would bother with it because more and more projects
will go with it.  it would be nice if all of them
bundled it (as its author advises) but that is not the
case.  that is what my question was about.

should it be downloaded at compile time? i am no fan of
hidden online dependencies as i am offline quite a lot.

should it be included under files/? cvs will go crazy
with this hybrid file..

so i think a package that contains multiple needed
versions (patched so they would unpack their library
into ${WRKDIST}) is the least painful way to go.  i am
working on that now, but it is not finished.

just by hating it it won't go away.  none of the gnu
stuff did..

-f
--
how much can i get away with and still go to heaven?

Reply | Threaded
Open this post in threaded view
|

Re: waf woes

Marc Espie-2
On Thu, Jan 02, 2014 at 06:59:02PM +0100, frantisek holop wrote:

> hmm, on Thu, Jan 02, 2014 at 03:19:16PM +0100, Landry Breuil said that
> > I've moved away all waf users from it (ie x11/gigolo & www/vteplugin)
> > but i'd rather remove waf from the tree instead of having someone losing
> > time on that crap. Given that it's clearly hostile to any packaging
> > effort, why bother with it ? Just do the minimum and patch all the
> > bundled copies.
>
> i would bother with it because more and more projects
> will go with it.  it would be nice if all of them
> bundled it (as its author advises) but that is not the
> case.  that is what my question was about.

How about you talk to those projects and tell them waf is a bad idea ?

or do you want to be part of the problem ?

Reply | Threaded
Open this post in threaded view
|

Re: waf woes

frantisek holop
hmm, on Thu, Jan 02, 2014 at 07:32:19PM +0100, Marc Espie said that

> On Thu, Jan 02, 2014 at 06:59:02PM +0100, frantisek holop wrote:
> > hmm, on Thu, Jan 02, 2014 at 03:19:16PM +0100, Landry Breuil said that
> > > I've moved away all waf users from it (ie x11/gigolo & www/vteplugin)
> > > but i'd rather remove waf from the tree instead of having someone losing
> > > time on that crap. Given that it's clearly hostile to any packaging
> > > effort, why bother with it ? Just do the minimum and patch all the
> > > bundled copies.
> >
> > i would bother with it because more and more projects
> > will go with it.  it would be nice if all of them
> > bundled it (as its author advises) but that is not the
> > case.  that is what my question was about.
>
> How about you talk to those projects and tell them waf is a bad idea ?
>
> or do you want to be part of the problem ?


i dont *know* if waf is a bad idea.  have you tried
using it? or is it just because it's a python script
that prefers to be copied into the project dir?

why is it so bad? ports can cope with it. in the end it
just takes ${CONFIGURE_ARGS} just like autohell.  and
it's not like it is alone in the package hostile club.
clearly it has some merit if some big projects switched
to it.

do you go around telling projects stop using <insert
your favourite build system to hate>?  i can imagine
your answer if someone did the same for openbsd and/or
ports.  it is nobody elses business what a project
uses. i am surprised you dont share this view.

and what do i tell them?  please switch to autohell?
i wouldnt in my project.

as faithless sings "if loving you is wrong, i dont want
to be right", if "to be part of the solution" is to
ignore/hate waf then i dont want to be part of it and
can be hardly called a solution in my eyes.

frankly, this is a storm in a teacup issue. i will send
the port shortly.

-f
--
i'm feeling rather blonde today.

Reply | Threaded
Open this post in threaded view
|

Re: waf woes

Stuart Henderson-6
In reply to this post by frantisek holop
On 2014/01/02 18:59, frantisek holop wrote:
> i would bother with it because more and more projects
> will go with it.  it would be nice if all of them
> bundled it (as its author advises) but that is not the
> case.  that is what my question was about.
>
> should it be downloaded at compile time?

Add it as a SUPDISTFILE instead, so it is hash-checked, and so that
the port can build for people using USE_SYSTRACE=Yes.

Reply | Threaded
Open this post in threaded view
|

Re: waf woes

Marc Espie-2
In reply to this post by frantisek holop
On Fri, Jan 03, 2014 at 05:41:39AM +0100, frantisek holop wrote:
> do you go around telling projects stop using <insert
> your favourite build system to hate>?  i can imagine
> your answer if someone did the same for openbsd and/or
> ports.  it is nobody elses business what a project
> uses. i am surprised you dont share this view.

Actually, no, you probably can't imagine my answer.

If the view had technical merits, I would do my best to fix
things.

The hard part with ports and packages is not the implementation,
it's the correct design.  The very great thing with what we
currently have is that it's fluid enough that it can evolve.

as far as a autohell replacement goes, there currently is nothing worth
saving. The cmake + ninja combination is probably the best thing currently
in existence for parallel builds of big projects.

As for discovering OS properties, nothing is currently really good.

Reply | Threaded
Open this post in threaded view
|

Re: waf woes

Dmitrij D. Czarkoff-2
In reply to this post by Stuart Henderson-6
On Tuesday, December 31, 2013 01:06:47 PM Stuart Henderson wrote:
> If people are looking for a non-autoconf build system, cmake seems
> about the best choice.. it's not perfect, but CMakefiles are fairly
> readable, and it's nicely cross-platform..

In my opinion plain Makefiles are the sanest build system. Look at how
suckless.org people do their stuff.

--
Dmitrij D. Czarkoff