boost 1.52.0 status

classic Classic list List threaded Threaded
64 messages Options
1234
Reply | Threaded
Open this post in threaded view
|

Re: Parallel cmake (diff to review)

Landry Breuil-6
On Wed, Dec 12, 2012 at 12:48:09PM +0100, David Coppa wrote:

> On Tue, 11 Dec 2012, Amit Kulkarni wrote:
>
> > >> > > I tried with USE_GMAKE = Yes and it drives all cores....
> > >> > >
> > >> > > so I think please add CONFIGURE_ARGS = --parallel=${MAKE_JOBS} and
> > >> > > USE_GMAKE = Yes and then both configure and build will go parallel.
> > >> > >
>
> ...
>
> > > The use of GNU make like this seems bogus. I take it CMake
> > > isn't doing something right.
> >
> > as far back as 2008 I seem to remember cmake having implicit
> > dependency on gmake (i tried it on on Solaris,FreeBSD and then
> > OpenBSD). my builds would always parallel build with gmake but not
> > with plain os make. i still haven't figured out why. maybe david can
> > chime in with more...
> >
> > perhaps
> > http://cmake.3232098.n2.nabble.com/parallel-make-j-N-td7193296.html
> >
> > which leads to
> > http://public.kitware.com/Bug/view.php?id=12882
>
> Hi again,
>
> Since I've tried all the weapons in my arsenal and found nothing
> useful, I think I can be ok with Amit's approach.
>
> I did the usual tests and it doesn't seem to introduce any regressions
> after all...
>
> Summarizing:
>
> I've added "--parallel=${MAKE_JOBS}".
>
> Added "--verbose" that prints some nice informations, like
>
> System:           OpenBSD
> Doing parallel make: 2
>
> Removed USE_GROFF which is not needed.
>
> Added a "XXX" to the comment.
>
> Thoughts? OKs?

USE_GMAKE is _not_ the way to go. What next, add USE_GMAKE to all ports
using cmake ? Our make -j itself works, one just need to find why cmake
doesnt take benefit from it.

Landry

Reply | Threaded
Open this post in threaded view
|

Re: Parallel cmake (diff to review)

Vadim Zhukov
In reply to this post by David Coppa
12.12.2012 15:55 пользователь "David Coppa" <[hidden email]> написал:
>
> On Tue, 11 Dec 2012, Amit Kulkarni wrote:
>
> > >> > > I tried with USE_GMAKE = Yes and it drives all cores....
> > >> > >
> > >> > > so I think please add CONFIGURE_ARGS = --parallel=${MAKE_JOBS}
and
> > >> > > USE_GMAKE = Yes and then both configure and build will go
parallel.

> > >> > >
>
> ...
>
> > > The use of GNU make like this seems bogus. I take it CMake
> > > isn't doing something right.
> >
> > as far back as 2008 I seem to remember cmake having implicit
> > dependency on gmake (i tried it on on Solaris,FreeBSD and then
> > OpenBSD). my builds would always parallel build with gmake but not
> > with plain os make. i still haven't figured out why. maybe david can
> > chime in with more...
> >
> > perhaps
> > http://cmake.3232098.n2.nabble.com/parallel-make-j-N-td7193296.html
> >
> > which leads to
> > http://public.kitware.com/Bug/view.php?id=12882
>
> Hi again,
>
> Since I've tried all the weapons in my arsenal and found nothing
> useful, I think I can be ok with Amit's approach.
>
> I did the usual tests and it doesn't seem to introduce any regressions
> after all...
>
> Summarizing:
>
> I've added "--parallel=${MAKE_JOBS}".
>
> Added "--verbose" that prints some nice informations, like
>
> System:           OpenBSD
> Doing parallel make: 2
>
> Removed USE_GROFF which is not needed.
>
> Added a "XXX" to the comment.
>
> Thoughts? OKs?

Until this go in (if ever), could please anyone make clear a few things?

1. USE_GMAKE=Yes in CMake's port enables parallel build of CMake itself
only, or does it equate to setting USE_GMAKE=Yes to all CMake-based ports
too? IIRC, CMake does not care, thus we can use GMake to build CMake itself
and our make(1) for building CMake-based ports, but I could be wrong.

2. Do we have to set USE_GMAKE=Yes for each CMake-based port where we want
to achieve parallel building (at least until problem with our make and
CMake is fixed)?

> Index: Makefile
> ===================================================================
> RCS file: /cvs/ports/devel/cmake/Makefile,v
> retrieving revision 1.73
> diff -u -p -r1.73 Makefile
> --- Makefile    11 Dec 2012 11:51:39 -0000      1.73
> +++ Makefile    12 Dec 2012 11:42:10 -0000
> @@ -8,7 +8,7 @@ HOMEPAGE =      http://www.cmake.org/
>  CATEGORIES =   devel
>  COMMENT =      portable build system
>  DISTNAME =     cmake-2.8.10.2
> -REVISION =     0
> +REVISION =     1
>  MASTER_SITES = ${HOMEPAGE}files/v2.8/
>
>  MAINTAINER =   David Coppa <[hidden email]>
> @@ -29,12 +29,16 @@ CONFIGURE_STYLE =   simple
>  CONFIGURE_ARGS =       --prefix=${PREFIX} \
>                         --datadir=/share/cmake \
>                         --docdir=/share/doc/cmake \
> -                       --system-libs
> +                       --parallel=${MAKE_JOBS} \
> +                       --system-libs \
> +                       --verbose
>
>  PORTHOME =             ${WRKDIR}
>
>  SEPARATE_BUILD =       Yes
> -USE_GROFF =            Yes
> +
> +# XXX: GNU make enables dpb parallel builds
> +USE_GMAKE =            Yes
>
>  REGRESS_TARGET =       test
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Parallel cmake (diff to review)

David Coppa
On Wed, Dec 12, 2012 at 1:09 PM, Vadim Zhukov <[hidden email]> wrote:

> Until this go in (if ever), could please anyone make clear a few things?
>
> 1. USE_GMAKE=Yes in CMake's port enables parallel build of CMake itself
> only, or does it equate to setting USE_GMAKE=Yes to all CMake-based ports
> too? IIRC, CMake does not care, thus we can use GMake to build CMake itself
> and our make(1) for building CMake-based ports, but I could be wrong.

CMake itself only. No need to set USE_GMAKE=Yes for CMake-based ports.

> 2. Do we have to set USE_GMAKE=Yes for each CMake-based port where we want
> to achieve parallel building (at least until problem with our make and CMake
> is fixed)?

I don't know but I suspect the answer is yes.

Reply | Threaded
Open this post in threaded view
|

Re: Parallel cmake (diff to review)

Stuart Henderson
In reply to this post by Vadim Zhukov
On 2012/12/12 15:09, Vadim Zhukov wrote:
> 2. Do we have to set USE_GMAKE=Yes for each CMake-based port where we want
> to achieve parallel building (at least until problem with our make and
> CMake is fixed)?

in testing, it seems so.

weird though, I was pretty sure I'd seen an improvement from doing this
in the past, so either I was mistaken (not unlikely ;) or something changed.

Reply | Threaded
Open this post in threaded view
|

Re: Parallel cmake (diff to review)

Amit Kulkarni-5
>> 2. Do we have to set USE_GMAKE=Yes for each CMake-based port where we want
>> to achieve parallel building (at least until problem with our make and
>> CMake is fixed)?
>
> in testing, it seems so.
>
> weird though, I was pretty sure I'd seen an improvement from doing this
> in the past, so either I was mistaken (not unlikely ;) or something changed.
>

i asked [hidden email] for help in figuring out which area
of code to look at.

Reply | Threaded
Open this post in threaded view
|

Re: Parallel cmake (diff to review)

David Coppa
On Wed, Dec 12, 2012 at 2:44 PM, Amit Kulkarni <[hidden email]> wrote:

>>> 2. Do we have to set USE_GMAKE=Yes for each CMake-based port where we want
>>> to achieve parallel building (at least until problem with our make and
>>> CMake is fixed)?
>>
>> in testing, it seems so.
>>
>> weird though, I was pretty sure I'd seen an improvement from doing this
>> in the past, so either I was mistaken (not unlikely ;) or something changed.
>>
>
> i asked [hidden email] for help in figuring out which area
> of code to look at.

Thanks.
I'll wait for useful pointers...

Reply | Threaded
Open this post in threaded view
|

using cmake in parallel builds, Re: Parallel cmake (diff to review)

Marc Espie-2
In reply to this post by Vadim Zhukov
Very bad subject line, I almost did not read the thread, because
I thought you guys were talking about building cmake proper in parallel.

Anyways, there are TWO distinct points:
- problems with make -j.
- cmake not writing correct makefiles for parallel building without gmake.

I haven't looked too closely, I have NO IDEA which of the two issues we're
talking about (besides that, it might be a good idea to have a generator
that caters a bit more to our make, specifically, include subfile is now
part of posix.

As far as make -j is concerned, the main issue left is matching file paths
to targets. So if you see make saying "hey, I don't know how to build that"
or "target foo unaccounted for", that's probably it.

Reply | Threaded
Open this post in threaded view
|

Re: Parallel cmake (diff to review)

Kent R. Spillner-2
In reply to this post by David Coppa
Hey, dude-

On Dec 12, 2012, at 7:51, David Coppa <[hidden email]> wrote:
> Thanks.
> I'll wait for useful pointers...

I don't think CMake does anything specifically to handle parallel recursive builds.  It works with GNU make because when invoked with -jX the top level gmake sets some environment variables (it adds -j to MAKEFLAGS, but i think it uses some others for job control, too) so sub-makes pick that up in parallel builds and CMake doesn't need to explicitly write -j$((X - 1)) in targets invoking sub-makes.

How does our make handle parallel job control between recursive invocations?

Best,
Kent


Reply | Threaded
Open this post in threaded view
|

Re: using cmake in parallel builds, Re: Parallel cmake (diff to review)

David Coppa
In reply to this post by Marc Espie-2
On Wed, 12 Dec 2012, Marc Espie wrote:

> Very bad subject line, I almost did not read the thread, because
> I thought you guys were talking about building cmake proper in parallel.
>
> Anyways, there are TWO distinct points:
> - problems with make -j.
> - cmake not writing correct makefiles for parallel building without gmake.

I think this is the issue (cmake not writing correct makefiles for
parallel building)

> I haven't looked too closely, I have NO IDEA which of the two issues we're
> talking about (besides that, it might be a good idea to have a generator
> that caters a bit more to our make, specifically, include subfile is now
> part of posix.
>
> As far as make -j is concerned, the main issue left is matching file paths
> to targets. So if you see make saying "hey, I don't know how to build that"
> or "target foo unaccounted for", that's probably it.

Attached you have one of the generated Makefiles
(/usr/ports/pobj/cmake-2.8.10.2/build-amd64/Makefile)

cheers,
David

Makefile (31K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: using cmake in parallel builds, Re: Parallel cmake (diff to review)

Kent R. Spillner-2
In reply to this post by Marc Espie-2
Hey, dude-

On Dec 12, 2012, at 9:30, Marc Espie <[hidden email]> wrote:

> Anyways, there are TWO distinct points:
> - problems with make -j.
> - cmake not writing correct makefiles for parallel building without gmake.


The second one is the problem here.  CMake doesn't do anything special for parallel builds, it just leans on gmake's environment variable fu for parallel job control.


> (besides that, it might be a good idea to have a generator
> that caters a bit more to our make, specifically, include subfile is now
> part of posix.


I think that's the right way to fix this problem.

Best,
Kent

Reply | Threaded
Open this post in threaded view
|

Re: using cmake in parallel builds, Re: Parallel cmake (diff to review)

Christian Weisgerber
In reply to this post by Marc Espie-2
Marc Espie:

> I thought you guys were talking about building cmake proper in parallel.

We did.  cmake proper first builds a minimal bootstrap cmake, then
rebuilds itself with it, so getting cmake proper to build in parallel
*is* the same problem as getting any other cmake-using port to build
in parallel.

> Anyways, there are TWO distinct points:
> - problems with make -j.
> - cmake not writing correct makefiles for parallel building without gmake.

The problem isn't that make -j fails with cmake.  The build succeeds
just fine.  The problem is that with our make there is no parallelism.
It's as if the -j was ignored.

--
Christian "naddy" Weisgerber                          [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: using cmake in parallel builds, Re: Parallel cmake (diff to review)

Marc Espie-2
On Wed, Dec 12, 2012 at 04:52:17PM +0100, Christian Weisgerber wrote:

> Marc Espie:
>
> > I thought you guys were talking about building cmake proper in parallel.
>
> We did.  cmake proper first builds a minimal bootstrap cmake, then
> rebuilds itself with it, so getting cmake proper to build in parallel
> *is* the same problem as getting any other cmake-using port to build
> in parallel.
>
> > Anyways, there are TWO distinct points:
> > - problems with make -j.
> > - cmake not writing correct makefiles for parallel building without gmake.
>
> The problem isn't that make -j fails with cmake.  The build succeeds
> just fine.  The problem is that with our make there is no parallelism.
> It's as if the -j was ignored.

It's likely that cmake decides (arbitrarily) things don't work without gmake.
Since there is some recursive makefiles involved, it probably strips the
extra stuff early on...

Reply | Threaded
Open this post in threaded view
|

Re: using cmake in parallel builds, Re: Parallel cmake (diff to review)

Amit Kulkarni-5
In reply to this post by Kent R. Spillner-2
On Wed, Dec 12, 2012 at 9:48 AM, Kent R. Spillner <[hidden email]> wrote:

> Hey, dude-
>
> On Dec 12, 2012, at 9:30, Marc Espie <[hidden email]> wrote:
>
>> Anyways, there are TWO distinct points:
>> - problems with make -j.
>> - cmake not writing correct makefiles for parallel building without gmake.
>
>
> The second one is the problem here.  CMake doesn't do anything special for parallel builds, it just leans on gmake's environment variable fu for parallel job control.
>
>
>> (besides that, it might be a good idea to have a generator
>> that caters a bit more to our make, specifically, include subfile is now
>> part of posix.
>
>
> I think that's the right way to fix this problem.
>
> Best,
> Kent
>

here is the mail from Brad King

On 12/12/2012 08:41 AM, Amit Kulkarni wrote:

> On OpenBSD, we want to build cmake and some critical cmake based ports
> in parallel using our dpb (distributed ports builder). We pass j=N in
> a MAKE_JOBS environment variable when building cmake and cmake based
> ports using our Makefile hierarchy. It seems that when we use gmake,
> cmake builds in parallel, i.e with -j option. Plain bsd make fails
> here...
>
> http://www.mail-archive.com/ports@.../msg44397.html
>
> Our porters want to track down this issue and have it built in
> parallel, so is the environment variable to pass j=N flag called
> MAKEFLAGS? A pointer would be appreciated before digging into the
> source.

CMake does not require GNU make on any platform.  The Makefile we
generate will work with any make tool, even for parallel builds.

For projects other than CMake itself, CMake simply generates the
build files.  Then it is up to you to run "make -j $N" if you want
to build the result in parallel.

For CMake itself there is the "bootstrap" script (and a "configure"
alias for it).  This script's only job is to build a minimal part
of CMake with just enough to run on CMake itself to generate build
files like any other CMake-based project.  After bootstrap is done
it is still up to you to run "make -j $N" to build CMake in parallel.

What remains is to get the bootstrap build to work in parallel too.
That is what the "--parallel" option to the bootstrap script requests.
That script tries to detect whether the make tool supports parallel
builds.  Only if that detection fails would the script choose not to
pass "-j $N" to make.

Reply | Threaded
Open this post in threaded view
|

Re: using cmake in parallel builds, Re: Parallel cmake (diff to review)

Amit Kulkarni-5
In reply to this post by Marc Espie-2
>> > I thought you guys were talking about building cmake proper in parallel.
>>
>> We did.  cmake proper first builds a minimal bootstrap cmake, then
>> rebuilds itself with it, so getting cmake proper to build in parallel
>> *is* the same problem as getting any other cmake-using port to build
>> in parallel.
>>
>> > Anyways, there are TWO distinct points:
>> > - problems with make -j.
>> > - cmake not writing correct makefiles for parallel building without gmake.
>>
>> The problem isn't that make -j fails with cmake.  The build succeeds
>> just fine.  The problem is that with our make there is no parallelism.
>> It's as if the -j was ignored.
>
> It's likely that cmake decides (arbitrarily) things don't work without gmake.
> Since there is some recursive makefiles involved, it probably strips the
> extra stuff early on...

i could not see any gmake specific code when i grepped in the cmake codebase.

i can confirm what naddy@ sees, when i cd ${WRKBUILD} && make -j4, i
see only 1 core being used. but if i use gmake -j4 all cores are used.
our make is ignoring -j but what is confusing is that: just before
building, in bootstrapping with --parallel, it uses -j successfully.

Reply | Threaded
Open this post in threaded view
|

Re: using cmake in parallel builds, Re: Parallel cmake (diff to review)

Vadim Zhukov
13.12.2012 4:22 пользователь "Amit Kulkarni" <[hidden email]> написал:
>
> >> > I thought you guys were talking about building cmake proper in
parallel.
> >>
> >> We did.  cmake proper first builds a minimal bootstrap cmake, then
> >> rebuilds itself with it, so getting cmake proper to build in parallel
> >> *is* the same problem as getting any other cmake-using port to build
> >> in parallel.
> >>
> >> > Anyways, there are TWO distinct points:
> >> > - problems with make -j.
> >> > - cmake not writing correct makefiles for parallel building without
gmake.
> >>
> >> The problem isn't that make -j fails with cmake.  The build succeeds
> >> just fine.  The problem is that with our make there is no parallelism.
> >> It's as if the -j was ignored.
> >
> > It's likely that cmake decides (arbitrarily) things don't work without
gmake.
> > Since there is some recursive makefiles involved, it probably strips the
> > extra stuff early on...
>
> i could not see any gmake specific code when i grepped in the cmake
codebase.
>
> i can confirm what naddy@ sees, when i cd ${WRKBUILD} && make -j4, i
> see only 1 core being used. but if i use gmake -j4 all cores are used.
> our make is ignoring -j but what is confusing is that: just before
> building, in bootstrapping with --parallel, it uses -j successfully.

The problem was already made clear: GNU Make propagates "-j" to subcalls,
our - does not. The latter is by design, IIRC (if I'm wrong here, then,
probably, espie@ will use his cluestick to teach me not to write about
things I understand badly), to avoid extra subprocesses being run: suppose
that "make -j 4" runs four "make -j 3", then each runs three "make -j 2"...
That's GNU Make's way, IIRC, and it's broken by design. Unfortunately,
three is no easy way to fix this. The best option I see (and it's probably
wrong) is to create socket in /tmp on the initial make(1) invocation, and
pass its path to subprocesses through environment variable. Through this
socket, each sub-make could request the right to start one or more jobs,
and wait until "master" make process answers. But I'm not ready to prepare
any patches implementing such functionality now: too much time is being
spent on fixing KDE breakage (due to upstream lazyness, my own stupidity
and a lots of inaccuracy from both sides).
Reply | Threaded
Open this post in threaded view
|

Re: using cmake in parallel builds, Re: Parallel cmake (diff to review)

David Coppa
On Thu, Dec 13, 2012 at 7:01 AM, Vadim Zhukov <[hidden email]> wrote:

> 13.12.2012 4:22 пользователь "Amit Kulkarni" <[hidden email]> написал:
>
>
>>
>> >> > I thought you guys were talking about building cmake proper in
>> >> > parallel.
>> >>
>> >> We did.  cmake proper first builds a minimal bootstrap cmake, then
>> >> rebuilds itself with it, so getting cmake proper to build in parallel
>> >> *is* the same problem as getting any other cmake-using port to build
>> >> in parallel.
>> >>
>> >> > Anyways, there are TWO distinct points:
>> >> > - problems with make -j.
>> >> > - cmake not writing correct makefiles for parallel building without
>> >> > gmake.
>> >>
>> >> The problem isn't that make -j fails with cmake.  The build succeeds
>> >> just fine.  The problem is that with our make there is no parallelism.
>> >> It's as if the -j was ignored.
>> >
>> > It's likely that cmake decides (arbitrarily) things don't work without
>> > gmake.
>> > Since there is some recursive makefiles involved, it probably strips the
>> > extra stuff early on...
>>
>> i could not see any gmake specific code when i grepped in the cmake
>> codebase.
>>
>> i can confirm what naddy@ sees, when i cd ${WRKBUILD} && make -j4, i
>> see only 1 core being used. but if i use gmake -j4 all cores are used.
>> our make is ignoring -j but what is confusing is that: just before
>> building, in bootstrapping with --parallel, it uses -j successfully.
>
> The problem was already made clear: GNU Make propagates "-j" to subcalls,
> our - does not. The latter is by design, IIRC (if I'm wrong here, then,
> probably, espie@ will use his cluestick to teach me not to write about
> things I understand badly), to avoid extra subprocesses being run: suppose
> that "make -j 4" runs four "make -j 3", then each runs three "make -j 2"...
> That's GNU Make's way, IIRC, and it's broken by design. Unfortunately, three
> is no easy way to fix this. The best option I see (and it's probably wrong)
> is to create socket in /tmp on the initial make(1) invocation, and pass its
> path to subprocesses through environment variable. Through this socket, each
> sub-make could request the right to start one or more jobs, and wait until
> "master" make process answers. But I'm not ready to prepare any patches
> implementing such functionality now: too much time is being spent on fixing
> KDE breakage (due to upstream lazyness, my own stupidity and a lots of
> inaccuracy from both sides).

Btw, I still think it's not so bad to use gmake *only* for cmake (not
for all the cmake-based ports!) to speed up dpb builds...

Reply | Threaded
Open this post in threaded view
|

Re: using cmake in parallel builds, Re: Parallel cmake (diff to review)

Stuart Henderson
I think we now have the critical path to libreoffice building quickly enough that it's not going to reduce the overall dpb build time (the problem DPB_PROPERTIES=parallel intends to solve is waiting for libreoffice to finish when everything else is done. Moving to gmake may actually increase the overall time (gmake uses crazy amounts of cpu at times).

David Coppa <[hidden email]> wrote:

>On Thu, Dec 13, 2012 at 7:01 AM, Vadim Zhukov <[hidden email]>
>wrote:
>> 13.12.2012 4:22 пользователь "Amit Kulkarni" <[hidden email]>
>написал:
>>
>>
>>>
>>> >> > I thought you guys were talking about building cmake proper in
>>> >> > parallel.
>>> >>
>>> >> We did.  cmake proper first builds a minimal bootstrap cmake,
>then
>>> >> rebuilds itself with it, so getting cmake proper to build in
>parallel
>>> >> *is* the same problem as getting any other cmake-using port to
>build
>>> >> in parallel.
>>> >>
>>> >> > Anyways, there are TWO distinct points:
>>> >> > - problems with make -j.
>>> >> > - cmake not writing correct makefiles for parallel building
>without
>>> >> > gmake.
>>> >>
>>> >> The problem isn't that make -j fails with cmake.  The build
>succeeds
>>> >> just fine.  The problem is that with our make there is no
>parallelism.
>>> >> It's as if the -j was ignored.
>>> >
>>> > It's likely that cmake decides (arbitrarily) things don't work
>without
>>> > gmake.
>>> > Since there is some recursive makefiles involved, it probably
>strips the
>>> > extra stuff early on...
>>>
>>> i could not see any gmake specific code when i grepped in the cmake
>>> codebase.
>>>
>>> i can confirm what naddy@ sees, when i cd ${WRKBUILD} && make -j4, i
>>> see only 1 core being used. but if i use gmake -j4 all cores are
>used.
>>> our make is ignoring -j but what is confusing is that: just before
>>> building, in bootstrapping with --parallel, it uses -j successfully.
>>
>> The problem was already made clear: GNU Make propagates "-j" to
>subcalls,
>> our - does not. The latter is by design, IIRC (if I'm wrong here,
>then,
>> probably, espie@ will use his cluestick to teach me not to write
>about
>> things I understand badly), to avoid extra subprocesses being run:
>suppose
>> that "make -j 4" runs four "make -j 3", then each runs three "make -j
>2"...
>> That's GNU Make's way, IIRC, and it's broken by design.
>Unfortunately, three
>> is no easy way to fix this. The best option I see (and it's probably
>wrong)
>> is to create socket in /tmp on the initial make(1) invocation, and
>pass its
>> path to subprocesses through environment variable. Through this
>socket, each
>> sub-make could request the right to start one or more jobs, and wait
>until
>> "master" make process answers. But I'm not ready to prepare any
>patches
>> implementing such functionality now: too much time is being spent on
>fixing
>> KDE breakage (due to upstream lazyness, my own stupidity and a lots
>of
>> inaccuracy from both sides).
>
>Btw, I still think it's not so bad to use gmake *only* for cmake (not
>for all the cmake-based ports!) to speed up dpb builds...


Reply | Threaded
Open this post in threaded view
|

Re: using cmake in parallel builds, Re: Parallel cmake (diff to review)

Marc Espie-2
In reply to this post by Amit Kulkarni-5
On Wed, Dec 12, 2012 at 06:21:43PM -0600, Amit Kulkarni wrote:

> >> > I thought you guys were talking about building cmake proper in parallel.
> >>
> >> We did.  cmake proper first builds a minimal bootstrap cmake, then
> >> rebuilds itself with it, so getting cmake proper to build in parallel
> >> *is* the same problem as getting any other cmake-using port to build
> >> in parallel.
> >>
> >> > Anyways, there are TWO distinct points:
> >> > - problems with make -j.
> >> > - cmake not writing correct makefiles for parallel building without gmake.
> >>
> >> The problem isn't that make -j fails with cmake.  The build succeeds
> >> just fine.  The problem is that with our make there is no parallelism.
> >> It's as if the -j was ignored.
> >
> > It's likely that cmake decides (arbitrarily) things don't work without gmake.
> > Since there is some recursive makefiles involved, it probably strips the
> > extra stuff early on...
>
> i could not see any gmake specific code when i grepped in the cmake codebase.
>
> i can confirm what naddy@ sees, when i cd ${WRKBUILD} && make -j4, i
> see only 1 core being used. but if i use gmake -j4 all cores are used.
> our make is ignoring -j but what is confusing is that: just before
> building, in bootstrapping with --parallel, it uses -j successfully.

No, our make is not ignoring -j, and it's passed to submakes. Using standard
posix mechanisms. That is, it's passed through the environment, using
MAKEFLAGS.

There are two ways to defeat that mechanism: either by explicitly wiping
out the environment, or by passing another -j somewhere.

cmake is probably doing either of those.

If you think make does not pass -j to submakes by default, you're confused,
again, or you explain yourself poorly.

Reply | Threaded
Open this post in threaded view
|

Re: Parallel cmake (diff to review)

Marc Espie-2
In reply to this post by Kent R. Spillner-2
On Wed, Dec 12, 2012 at 09:32:23AM -0600, Kent R. Spillner wrote:
> Hey, dude-
>
> On Dec 12, 2012, at 7:51, David Coppa <[hidden email]> wrote:
> > Thanks.
> > I'll wait for useful pointers...
>
> I don't think CMake does anything specifically to handle parallel recursive builds.  It works with GNU make because when invoked with -jX the top level gmake sets some environment variables (it adds -j to MAKEFLAGS, but i think it uses some others for job control, too) so sub-makes pick that up in parallel builds and CMake doesn't need to explicitly write -j$((X - 1)) in targets invoking sub-makes.
>
> How does our make handle parallel job control between recursive invocations?

Our make passes -j through MAKEFLAGS.

The way it handles recursive invocation is documented. Read the end of
make(1):

     In parallel mode, -j n only limits the number of direct children of make.
     During recursive invocations, each level may multiply the total number of
     processes by n.  However, make includes some heuristics to try to prevent
     catastrophic behavior: if a command is marked as expensive, or preceded
     by `+', or seems to invoke a program that looks sufficiently like `make',
     make will assume recursive invocation, and not start any new process
     until said command has finished running.  Thus the number of processes
     run directly or indirectly by make will increase linearly with each level
     of recursion instead of exponentially.

Reply | Threaded
Open this post in threaded view
|

Re: using cmake in parallel builds, Re: Parallel cmake (diff to review)

Amit Kulkarni-5
In reply to this post by Marc Espie-2
> No, our make is not ignoring -j, and it's passed to submakes. Using standard
> posix mechanisms. That is, it's passed through the environment, using
> MAKEFLAGS.
>
> There are two ways to defeat that mechanism: either by explicitly wiping
> out the environment, or by passing another -j somewhere.
>
> cmake is probably doing either of those.
>
> If you think make does not pass -j to submakes by default, you're confused,
> again, or you explain yourself poorly.

cmake is destroying the env by setting MAKEFLAGS="", that is what i
saw in git master yesterday...

1234