patch: make dpb and ports more uniform

classic Classic list List threaded Threaded
16 messages Options
Reply | Threaded
Open this post in threaded view
|

patch: make dpb and ports more uniform

Marc Espie-2
So dpb has been using MAKE_JOBS for a while, and landry said it would be
cool to actually have it work with regular builds as well.

This patch:
- admits that parallel make is going to be used;
- renames PARALLEL_BUILD to something that reflects its actual usage
(and consumers as well): PARALLEL_USES_MAKE
- adds a PARALLEL_MAKE_JOBS that allows the user to tweak the number
of MAKE_JOBS used if they want (set to hw.ncpuonline)


The names are somewhat straightforward, though not particularly inspired,
so I'm game for suggestions.

The logic itself is also possibly improvable...




Index: infrastructure/mk/bsd.port.mk
===================================================================
RCS file: /cvs/ports/infrastructure/mk/bsd.port.mk,v
retrieving revision 1.1505
diff -u -p -r1.1505 bsd.port.mk
--- infrastructure/mk/bsd.port.mk 26 Nov 2019 11:49:02 -0000 1.1505
+++ infrastructure/mk/bsd.port.mk 1 Dec 2019 10:26:58 -0000
@@ -454,19 +454,23 @@ ALL_FAKE_FLAGS= ${MAKE_FLAGS:N-j[0-9]*}
 _PKG_ADD += -L ${LOCALBASE}
 .endif
 
-# XXX this stuff is not production-ready, because there are too many bugs in
-# parallel make yet.  MAKE_JOBS>1 is known to work on a few ports and used
-# sparingly by dpb (DPB_PROPERTIES=parallel) for obvious gains.
+# XXX there are still many bugs in parallel make.
+# so MAKE_JOBS is used sparingly by dpb for obvious gains.
 #
-PARALLEL_BUILD ?= Yes
-PARALLEL_INSTALL ?= ${PARALLEL_BUILD}
+PARALLEL_USES_MAKE ?= Yes
+
+.if !defined(MAKE_JOBS) && ${DPB_PROPERTIES:Mparallel})
+.  if defined(PARALLEL_MAKE_JOBS)
+MAKE_JOBS = ${PARALLEL_MAKE_JOBS}
+.  else
+MAKE_JOBS !!= sysctl -n hw.ncpuonline
+.  endif
+.endif
 MAKE_JOBS ?= 1
 
 .if ${MAKE_JOBS} != 1
-.  if ${PARALLEL_BUILD:L} == "yes"
+.  if ${PARALLEL_USES_MAKE:L} == "yes"
 MAKE_FLAGS += -j${MAKE_JOBS}
-.  endif
-.  if ${PARALLEL_INSTALL:L} == "yes"
 ALL_FAKE_FLAGS += -j${MAKE_JOBS}
 .  endif
 .endif
Index: editors/libreoffice/Makefile
===================================================================
RCS file: /cvs/ports/editors/libreoffice/Makefile,v
retrieving revision 1.221
diff -u -p -r1.221 Makefile
--- editors/libreoffice/Makefile 6 Nov 2019 12:21:06 -0000 1.221
+++ editors/libreoffice/Makefile 1 Dec 2019 10:26:58 -0000
@@ -354,11 +354,11 @@ CONFIGURE_ARGS+=--without-java
 .endif
 
 # XXX libreoffice has its own way to build things in parallel
-PARALLEL_BUILD = No
+PARALLEL_USES_MAKE = No
 .if defined(MAKE_JOBS)
 NCPU= ${MAKE_JOBS}
 .elif ${FLAVOR:Mparallel}
-NCPU!= echo $$((`sysctl -n hw.ncpuonline` + 1))
+NCPU!!= echo $$((`sysctl -n hw.ncpuonline` + 1))
 .else
 NCPU= 1
 .endif
Index: devel/jdk/1.8/Makefile
===================================================================
RCS file: /cvs/ports/devel/jdk/1.8/Makefile,v
retrieving revision 1.48
diff -u -p -r1.48 Makefile
--- devel/jdk/1.8/Makefile 21 Nov 2019 17:39:08 -0000 1.48
+++ devel/jdk/1.8/Makefile 1 Dec 2019 10:26:58 -0000
@@ -84,7 +84,7 @@ USE_GMAKE= Yes
 GMAKE= ${LOCALBASE}/bin/gmake
 
 # Don't call make -jN, instead pass MAKE_JOBS to --with-jobs
-PARALLEL_BUILD= No
+PARALLEL_USES_MAKE= No
 
 CONFIGURE_STYLE=gnu
 CONFIGURE_ARGS= --disable-option-checking \
Index: devel/jdk/11/Makefile
===================================================================
RCS file: /cvs/ports/devel/jdk/11/Makefile,v
retrieving revision 1.11
diff -u -p -r1.11 Makefile
--- devel/jdk/11/Makefile 26 Oct 2019 17:04:15 -0000 1.11
+++ devel/jdk/11/Makefile 1 Dec 2019 10:26:58 -0000
@@ -84,7 +84,7 @@ USE_GMAKE= Yes
 GMAKE= ${LOCALBASE}/bin/gmake
 
 # Don't call make -jN, instead pass MAKE_JOBS to --with-jobs
-PARALLEL_BUILD= No
+PARALLEL_USES_MAKE= No
 
 CONFIGURE_STYLE=gnu
 CONFIGURE_ENV+= AUTOCONF_VERSION=${AUTOCONF_VERSION} \

Reply | Threaded
Open this post in threaded view
|

Re: patch: make dpb and ports more uniform

Landry Breuil-5
On Sun, Dec 01, 2019 at 11:31:40AM +0100, Marc Espie wrote:

> So dpb has been using MAKE_JOBS for a while, and landry said it would be
> cool to actually have it work with regular builds as well.
>
> This patch:
> - admits that parallel make is going to be used;
> - renames PARALLEL_BUILD to something that reflects its actual usage
> (and consumers as well): PARALLEL_USES_MAKE
> - adds a PARALLEL_MAKE_JOBS that allows the user to tweak the number
> of MAKE_JOBS used if they want (set to hw.ncpuonline)
>
>
> The names are somewhat straightforward, though not particularly inspired,
> so I'm game for suggestions.
>
> The logic itself is also possibly improvable...

I like this, and it does the right thing for mozilla ports. No opinion
on the naming, and then maybe some bsd.port.mk(5) bits ?

Reply | Threaded
Open this post in threaded view
|

Re: patch: make dpb and ports more uniform

Marc Espie-2
On Mon, Dec 02, 2019 at 10:17:44AM +0100, Landry Breuil wrote:

> On Sun, Dec 01, 2019 at 11:31:40AM +0100, Marc Espie wrote:
> > So dpb has been using MAKE_JOBS for a while, and landry said it would be
> > cool to actually have it work with regular builds as well.
> >
> > This patch:
> > - admits that parallel make is going to be used;
> > - renames PARALLEL_BUILD to something that reflects its actual usage
> > (and consumers as well): PARALLEL_USES_MAKE
> > - adds a PARALLEL_MAKE_JOBS that allows the user to tweak the number
> > of MAKE_JOBS used if they want (set to hw.ncpuonline)
> >
> >
> > The names are somewhat straightforward, though not particularly inspired,
> > so I'm game for suggestions.
> >
> > The logic itself is also possibly improvable...
>
> I like this, and it does the right thing for mozilla ports. No opinion
> on the naming, and then maybe some bsd.port.mk(5) bits ?
>
>
Sure there will be bsd.port.mk(5) bits.  As soon as I get opinions (or no
opinions) on the naming.

Reply | Threaded
Open this post in threaded view
|

Re: patch: make dpb and ports more uniform

Kurt Miller-3
In reply to this post by Marc Espie-2
On Sun, 2019-12-01 at 11:31 +0100, Marc Espie wrote:

> So dpb has been using MAKE_JOBS for a while, and landry said it would be
> cool to actually have it work with regular builds as well.
>
> This patch:
> - admits that parallel make is going to be used;
> - renames PARALLEL_BUILD to something that reflects its actual usage
> (and consumers as well): PARALLEL_USES_MAKE
> - adds a PARALLEL_MAKE_JOBS that allows the user to tweak the number
> of MAKE_JOBS used if they want (set to hw.ncpuonline)
>
>
> The names are somewhat straightforward, though not particularly inspired,
> so I'm game for suggestions.
>
> The logic itself is also possibly improvable...
>

I like where this ended up. Thank you.

okay kurt@

>
>
> Index: infrastructure/mk/bsd.port.mk
> ===================================================================
> RCS file: /cvs/ports/infrastructure/mk/bsd.port.mk,v
> retrieving revision 1.1505
> diff -u -p -r1.1505 bsd.port.mk
> --- infrastructure/mk/bsd.port.mk 26 Nov 2019 11:49:02 -0000 1.1505
> +++ infrastructure/mk/bsd.port.mk 1 Dec 2019 10:26:58 -0000
> @@ -454,19 +454,23 @@ ALL_FAKE_FLAGS= ${MAKE_FLAGS:N-j[0-9]*} 
>  _PKG_ADD += -L ${LOCALBASE}
>  .endif
>  
> -# XXX this stuff is not production-ready, because there are too many bugs in
> -# parallel make yet.  MAKE_JOBS>1 is known to work on a few ports and used
> -# sparingly by dpb (DPB_PROPERTIES=parallel) for obvious gains.
> +# XXX there are still many bugs in parallel make.
> +# so MAKE_JOBS is used sparingly by dpb for obvious gains.
>  #
> -PARALLEL_BUILD ?= Yes
> -PARALLEL_INSTALL ?= ${PARALLEL_BUILD}
> +PARALLEL_USES_MAKE ?= Yes
> +
> +.if !defined(MAKE_JOBS) && ${DPB_PROPERTIES:Mparallel})
> +.  if defined(PARALLEL_MAKE_JOBS)
> +MAKE_JOBS = ${PARALLEL_MAKE_JOBS}
> +.  else
> +MAKE_JOBS !!= sysctl -n hw.ncpuonline
> +.  endif
> +.endif
>  MAKE_JOBS ?= 1
>  
>  .if ${MAKE_JOBS} != 1
> -.  if ${PARALLEL_BUILD:L} == "yes"
> +.  if ${PARALLEL_USES_MAKE:L} == "yes"
>  MAKE_FLAGS += -j${MAKE_JOBS}
> -.  endif
> -.  if ${PARALLEL_INSTALL:L} == "yes"
>  ALL_FAKE_FLAGS += -j${MAKE_JOBS}
>  .  endif
>  .endif
> Index: editors/libreoffice/Makefile
> ===================================================================
> RCS file: /cvs/ports/editors/libreoffice/Makefile,v
> retrieving revision 1.221
> diff -u -p -r1.221 Makefile
> --- editors/libreoffice/Makefile 6 Nov 2019 12:21:06 -0000 1.221
> +++ editors/libreoffice/Makefile 1 Dec 2019 10:26:58 -0000
> @@ -354,11 +354,11 @@ CONFIGURE_ARGS+=--without-java
>  .endif
>  
>  # XXX libreoffice has its own way to build things in parallel
> -PARALLEL_BUILD = No
> +PARALLEL_USES_MAKE = No
>  .if defined(MAKE_JOBS)
>  NCPU= ${MAKE_JOBS}
>  .elif ${FLAVOR:Mparallel}
> -NCPU!= echo $$((`sysctl -n hw.ncpuonline` + 1))
> +NCPU!!= echo $$((`sysctl -n hw.ncpuonline` + 1))
>  .else
>  NCPU= 1
>  .endif
> Index: devel/jdk/1.8/Makefile
> ===================================================================
> RCS file: /cvs/ports/devel/jdk/1.8/Makefile,v
> retrieving revision 1.48
> diff -u -p -r1.48 Makefile
> --- devel/jdk/1.8/Makefile 21 Nov 2019 17:39:08 -0000 1.48
> +++ devel/jdk/1.8/Makefile 1 Dec 2019 10:26:58 -0000
> @@ -84,7 +84,7 @@ USE_GMAKE= Yes
>  GMAKE= ${LOCALBASE}/bin/gmake
>  
>  # Don't call make -jN, instead pass MAKE_JOBS to --with-jobs
> -PARALLEL_BUILD= No
> +PARALLEL_USES_MAKE= No
>  
>  CONFIGURE_STYLE=gnu
>  CONFIGURE_ARGS= --disable-option-checking \
> Index: devel/jdk/11/Makefile
> ===================================================================
> RCS file: /cvs/ports/devel/jdk/11/Makefile,v
> retrieving revision 1.11
> diff -u -p -r1.11 Makefile
> --- devel/jdk/11/Makefile 26 Oct 2019 17:04:15 -0000 1.11
> +++ devel/jdk/11/Makefile 1 Dec 2019 10:26:58 -0000
> @@ -84,7 +84,7 @@ USE_GMAKE= Yes
>  GMAKE= ${LOCALBASE}/bin/gmake
>  
>  # Don't call make -jN, instead pass MAKE_JOBS to --with-jobs
> -PARALLEL_BUILD= No
> +PARALLEL_USES_MAKE= No
>  
>  CONFIGURE_STYLE=gnu
>  CONFIGURE_ENV+= AUTOCONF_VERSION=${AUTOCONF_VERSION} \

Reply | Threaded
Open this post in threaded view
|

Re: patch: make dpb and ports more uniform

Christian Weisgerber
In reply to this post by Marc Espie-2
On 2019-12-01, Marc Espie <[hidden email]> wrote:

> This patch:
> - admits that parallel make is going to be used;
> - renames PARALLEL_BUILD to something that reflects its actual usage
> (and consumers as well): PARALLEL_USES_MAKE
> - adds a PARALLEL_MAKE_JOBS that allows the user to tweak the number
> of MAKE_JOBS used if they want (set to hw.ncpuonline)

What this actually does: If you set PARALLEL_MAKE_JOBS, then ports
that are marked DPB_PROPERTIES=parallel will be built with
MAKE_JOBS=${PARALLEL_MAKE_JOBS} for any regular "make build" outside
of dpb.

So basically this (ab)uses DPB_PROPERTIES=parallel as a marker for
ports that are parallel-safe.  Okay, whatever.

I think PARALLEL_BUILD is a perfectly fine name and it took me ten
minutes to understand "PARALLEL_USES_MAKE".  I'll probably be puzzled
again the next time I look at this.  Yes, there are a very few ports
that set PARALLEL_BUILD=No and still do some form of parallel
building.  That's why they have a corresponding comment.

> --- editors/libreoffice/Makefile 6 Nov 2019 12:21:06 -0000 1.221
> +++ editors/libreoffice/Makefile 1 Dec 2019 10:26:58 -0000
> @@ -354,11 +354,11 @@ CONFIGURE_ARGS+=--without-java
>  .endif
>  
>  # XXX libreoffice has its own way to build things in parallel
> -PARALLEL_BUILD = No
> +PARALLEL_USES_MAKE = No
>  .if defined(MAKE_JOBS)
>  NCPU= ${MAKE_JOBS}
>  .elif ${FLAVOR:Mparallel}
> -NCPU!= echo $$((`sysctl -n hw.ncpuonline` + 1))
> +NCPU!!= echo $$((`sysctl -n hw.ncpuonline` + 1))
>  .else
>  NCPU= 1
>  .endif

The point of the infrastructure change is to dispense with the need
for the "parallel" pseudoflavor here, so the pseudoflavor should
just be removed.

--
Christian "naddy" Weisgerber                          [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: patch: make dpb and ports more uniform

Marc Espie-2
On Mon, Dec 02, 2019 at 09:46:11PM -0000, Christian Weisgerber wrote:

> On 2019-12-01, Marc Espie <[hidden email]> wrote:
>
> > This patch:
> > - admits that parallel make is going to be used;
> > - renames PARALLEL_BUILD to something that reflects its actual usage
> > (and consumers as well): PARALLEL_USES_MAKE
> > - adds a PARALLEL_MAKE_JOBS that allows the user to tweak the number
> > of MAKE_JOBS used if they want (set to hw.ncpuonline)
>
> What this actually does: If you set PARALLEL_MAKE_JOBS, then ports
> that are marked DPB_PROPERTIES=parallel will be built with
> MAKE_JOBS=${PARALLEL_MAKE_JOBS} for any regular "make build" outside
> of dpb.
>
> So basically this (ab)uses DPB_PROPERTIES=parallel as a marker for
> ports that are parallel-safe.  Okay, whatever.
>
> I think PARALLEL_BUILD is a perfectly fine name and it took me ten
> minutes to understand "PARALLEL_USES_MAKE".  I'll probably be puzzled
> again the next time I look at this.  Yes, there are a very few ports
> that set PARALLEL_BUILD=No and still do some form of parallel
> building.  That's why they have a corresponding comment.

There are no ports that set PARALLEL_BUILD *for any other purpose than
wanting DPB_PROPERTIES=parallel and using something that's NOT make-j
to achieve parallelism*.

Hence my wanting to change the name to reflect reality.

Reply | Threaded
Open this post in threaded view
|

Re: patch: make dpb and ports more uniform

Antoine Jacoutot-7
In reply to this post by Christian Weisgerber
On Mon, Dec 02, 2019 at 09:46:11PM -0000, Christian Weisgerber wrote:

> On 2019-12-01, Marc Espie <[hidden email]> wrote:
>
> > This patch:
> > - admits that parallel make is going to be used;
> > - renames PARALLEL_BUILD to something that reflects its actual usage
> > (and consumers as well): PARALLEL_USES_MAKE
> > - adds a PARALLEL_MAKE_JOBS that allows the user to tweak the number
> > of MAKE_JOBS used if they want (set to hw.ncpuonline)
>
> What this actually does: If you set PARALLEL_MAKE_JOBS, then ports
> that are marked DPB_PROPERTIES=parallel will be built with
> MAKE_JOBS=${PARALLEL_MAKE_JOBS} for any regular "make build" outside
> of dpb.

Ah.
Thanks :-)

> So basically this (ab)uses DPB_PROPERTIES=parallel as a marker for
> ports that are parallel-safe.  Okay, whatever.
>
> I think PARALLEL_BUILD is a perfectly fine name and it took me ten
> minutes to understand "PARALLEL_USES_MAKE".  I'll probably be puzzled

Thank you for telling us, I stopped trying to understand after 2 minutes.

> again the next time I look at this.  Yes, there are a very few ports
> that set PARALLEL_BUILD=No and still do some form of parallel
> building.  That's why they have a corresponding comment.
>
> > --- editors/libreoffice/Makefile 6 Nov 2019 12:21:06 -0000 1.221
> > +++ editors/libreoffice/Makefile 1 Dec 2019 10:26:58 -0000
> > @@ -354,11 +354,11 @@ CONFIGURE_ARGS+=--without-java
> >  .endif
> >  
> >  # XXX libreoffice has its own way to build things in parallel
> > -PARALLEL_BUILD = No
> > +PARALLEL_USES_MAKE = No

I still don't understand that naming.
If I don't use make(1) I can't do parallel builds?


> >  .if defined(MAKE_JOBS)
> >  NCPU= ${MAKE_JOBS}
> >  .elif ${FLAVOR:Mparallel}
> > -NCPU!= echo $$((`sysctl -n hw.ncpuonline` + 1))
> > +NCPU!!= echo $$((`sysctl -n hw.ncpuonline` + 1))
> >  .else
> >  NCPU= 1
> >  .endif
>
> The point of the infrastructure change is to dispense with the need
> for the "parallel" pseudoflavor here, so the pseudoflavor should
> just be removed.
>
> --
> Christian "naddy" Weisgerber                          [hidden email]
>

--
Antoine

Reply | Threaded
Open this post in threaded view
|

Re: patch: make dpb and ports more uniform

Marc Espie-2
On Tue, Dec 03, 2019 at 12:37:25AM +0100, Antoine Jacoutot wrote:

> On Mon, Dec 02, 2019 at 09:46:11PM -0000, Christian Weisgerber wrote:
> > On 2019-12-01, Marc Espie <[hidden email]> wrote:
> >
> > > This patch:
> > > - admits that parallel make is going to be used;
> > > - renames PARALLEL_BUILD to something that reflects its actual usage
> > > (and consumers as well): PARALLEL_USES_MAKE
> > > - adds a PARALLEL_MAKE_JOBS that allows the user to tweak the number
> > > of MAKE_JOBS used if they want (set to hw.ncpuonline)
> >
> > What this actually does: If you set PARALLEL_MAKE_JOBS, then ports
> > that are marked DPB_PROPERTIES=parallel will be built with
> > MAKE_JOBS=${PARALLEL_MAKE_JOBS} for any regular "make build" outside
> > of dpb.
>
> Ah.
> Thanks :-)
>
> > So basically this (ab)uses DPB_PROPERTIES=parallel as a marker for
> > ports that are parallel-safe.  Okay, whatever.
> >
> > I think PARALLEL_BUILD is a perfectly fine name and it took me ten
> > minutes to understand "PARALLEL_USES_MAKE".  I'll probably be puzzled
>
> Thank you for telling us, I stopped trying to understand after 2 minutes.
>
> > again the next time I look at this.  Yes, there are a very few ports
> > that set PARALLEL_BUILD=No and still do some form of parallel
> > building.  That's why they have a corresponding comment.
> >
> > > --- editors/libreoffice/Makefile 6 Nov 2019 12:21:06 -0000 1.221
> > > +++ editors/libreoffice/Makefile 1 Dec 2019 10:26:58 -0000
> > > @@ -354,11 +354,11 @@ CONFIGURE_ARGS+=--without-java
> > >  .endif
> > >  
> > >  # XXX libreoffice has its own way to build things in parallel
> > > -PARALLEL_BUILD = No
> > > +PARALLEL_USES_MAKE = No
>
> I still don't understand that naming.
> If I don't use make(1) I can't do parallel builds?

No, the way it works is that *by default* MAKE_JOBS is passed to
make as make -j${MAKE_JOBS}

Look at both ports that override it, they explicitly set it to NO because
they pass "MAKE_JOBS" *elsewhere* (and don't want to use make -j)

I tried for a less confusing naming than PARALLEL_BUILD.  Tell me what
to call it then.!

Reply | Threaded
Open this post in threaded view
|

Re: patch: make dpb and ports more uniform

Stuart Henderson
On 2019/12/03 15:58, Marc Espie wrote:

> On Tue, Dec 03, 2019 at 12:37:25AM +0100, Antoine Jacoutot wrote:
> > On Mon, Dec 02, 2019 at 09:46:11PM -0000, Christian Weisgerber wrote:
> > > On 2019-12-01, Marc Espie <[hidden email]> wrote:
> > >
> > > > This patch:
> > > > - admits that parallel make is going to be used;
> > > > - renames PARALLEL_BUILD to something that reflects its actual usage
> > > > (and consumers as well): PARALLEL_USES_MAKE
> > > > - adds a PARALLEL_MAKE_JOBS that allows the user to tweak the number
> > > > of MAKE_JOBS used if they want (set to hw.ncpuonline)
> > >
> > > What this actually does: If you set PARALLEL_MAKE_JOBS, then ports
> > > that are marked DPB_PROPERTIES=parallel will be built with
> > > MAKE_JOBS=${PARALLEL_MAKE_JOBS} for any regular "make build" outside
> > > of dpb.
> >
> > Ah.
> > Thanks :-)
> >
> > > So basically this (ab)uses DPB_PROPERTIES=parallel as a marker for
> > > ports that are parallel-safe.  Okay, whatever.
> > >
> > > I think PARALLEL_BUILD is a perfectly fine name and it took me ten
> > > minutes to understand "PARALLEL_USES_MAKE".  I'll probably be puzzled
> >
> > Thank you for telling us, I stopped trying to understand after 2 minutes.
> >
> > > again the next time I look at this.  Yes, there are a very few ports
> > > that set PARALLEL_BUILD=No and still do some form of parallel
> > > building.  That's why they have a corresponding comment.
> > >
> > > > --- editors/libreoffice/Makefile 6 Nov 2019 12:21:06 -0000 1.221
> > > > +++ editors/libreoffice/Makefile 1 Dec 2019 10:26:58 -0000
> > > > @@ -354,11 +354,11 @@ CONFIGURE_ARGS+=--without-java
> > > >  .endif
> > > >  
> > > >  # XXX libreoffice has its own way to build things in parallel
> > > > -PARALLEL_BUILD = No
> > > > +PARALLEL_USES_MAKE = No
> >
> > I still don't understand that naming.
> > If I don't use make(1) I can't do parallel builds?
>
> No, the way it works is that *by default* MAKE_JOBS is passed to
> make as make -j${MAKE_JOBS}
>
> Look at both ports that override it, they explicitly set it to NO because
> they pass "MAKE_JOBS" *elsewhere* (and don't want to use make -j)
>
> I tried for a less confusing naming than PARALLEL_BUILD.  Tell me what
> to call it then.!
>

# XXX libreoffice has its own way to build things in parallel
USES_PARALLEL_MAKE = No

?

Reply | Threaded
Open this post in threaded view
|

Re: patch: make dpb and ports more uniform

Marc Espie-2
On Tue, Dec 03, 2019 at 03:41:03PM +0000, Stuart Henderson wrote:

> On 2019/12/03 15:58, Marc Espie wrote:
> > On Tue, Dec 03, 2019 at 12:37:25AM +0100, Antoine Jacoutot wrote:
> > > On Mon, Dec 02, 2019 at 09:46:11PM -0000, Christian Weisgerber wrote:
> > > > On 2019-12-01, Marc Espie <[hidden email]> wrote:
> > > >
> > > > > This patch:
> > > > > - admits that parallel make is going to be used;
> > > > > - renames PARALLEL_BUILD to something that reflects its actual usage
> > > > > (and consumers as well): PARALLEL_USES_MAKE
> > > > > - adds a PARALLEL_MAKE_JOBS that allows the user to tweak the number
> > > > > of MAKE_JOBS used if they want (set to hw.ncpuonline)
> > > >
> > > > What this actually does: If you set PARALLEL_MAKE_JOBS, then ports
> > > > that are marked DPB_PROPERTIES=parallel will be built with
> > > > MAKE_JOBS=${PARALLEL_MAKE_JOBS} for any regular "make build" outside
> > > > of dpb.
> > >
> > > Ah.
> > > Thanks :-)
> > >
> > > > So basically this (ab)uses DPB_PROPERTIES=parallel as a marker for
> > > > ports that are parallel-safe.  Okay, whatever.
> > > >
> > > > I think PARALLEL_BUILD is a perfectly fine name and it took me ten
> > > > minutes to understand "PARALLEL_USES_MAKE".  I'll probably be puzzled
> > >
> > > Thank you for telling us, I stopped trying to understand after 2 minutes.
> > >
> > > > again the next time I look at this.  Yes, there are a very few ports
> > > > that set PARALLEL_BUILD=No and still do some form of parallel
> > > > building.  That's why they have a corresponding comment.
> > > >
> > > > > --- editors/libreoffice/Makefile 6 Nov 2019 12:21:06 -0000 1.221
> > > > > +++ editors/libreoffice/Makefile 1 Dec 2019 10:26:58 -0000
> > > > > @@ -354,11 +354,11 @@ CONFIGURE_ARGS+=--without-java
> > > > >  .endif
> > > > >  
> > > > >  # XXX libreoffice has its own way to build things in parallel
> > > > > -PARALLEL_BUILD = No
> > > > > +PARALLEL_USES_MAKE = No
> > >
> > > I still don't understand that naming.
> > > If I don't use make(1) I can't do parallel builds?
> >
> > No, the way it works is that *by default* MAKE_JOBS is passed to
> > make as make -j${MAKE_JOBS}
> >
> > Look at both ports that override it, they explicitly set it to NO because
> > they pass "MAKE_JOBS" *elsewhere* (and don't want to use make -j)
> >
> > I tried for a less confusing naming than PARALLEL_BUILD.  Tell me what
> > to call it then.!
> >
>
> # XXX libreoffice has its own way to build things in parallel
> USES_PARALLEL_MAKE = No
>
> ?

Ah, looks nicer indeed.

Reply | Threaded
Open this post in threaded view
|

Re: patch: make dpb and ports more uniform

Antoine Jacoutot-7
In reply to this post by Stuart Henderson
On Tue, Dec 03, 2019 at 03:41:03PM +0000, Stuart Henderson wrote:

> On 2019/12/03 15:58, Marc Espie wrote:
> > On Tue, Dec 03, 2019 at 12:37:25AM +0100, Antoine Jacoutot wrote:
> > > On Mon, Dec 02, 2019 at 09:46:11PM -0000, Christian Weisgerber wrote:
> > > > On 2019-12-01, Marc Espie <[hidden email]> wrote:
> > > >
> > > > > This patch:
> > > > > - admits that parallel make is going to be used;
> > > > > - renames PARALLEL_BUILD to something that reflects its actual usage
> > > > > (and consumers as well): PARALLEL_USES_MAKE
> > > > > - adds a PARALLEL_MAKE_JOBS that allows the user to tweak the number
> > > > > of MAKE_JOBS used if they want (set to hw.ncpuonline)
> > > >
> > > > What this actually does: If you set PARALLEL_MAKE_JOBS, then ports
> > > > that are marked DPB_PROPERTIES=parallel will be built with
> > > > MAKE_JOBS=${PARALLEL_MAKE_JOBS} for any regular "make build" outside
> > > > of dpb.
> > >
> > > Ah.
> > > Thanks :-)
> > >
> > > > So basically this (ab)uses DPB_PROPERTIES=parallel as a marker for
> > > > ports that are parallel-safe.  Okay, whatever.
> > > >
> > > > I think PARALLEL_BUILD is a perfectly fine name and it took me ten
> > > > minutes to understand "PARALLEL_USES_MAKE".  I'll probably be puzzled
> > >
> > > Thank you for telling us, I stopped trying to understand after 2 minutes.
> > >
> > > > again the next time I look at this.  Yes, there are a very few ports
> > > > that set PARALLEL_BUILD=No and still do some form of parallel
> > > > building.  That's why they have a corresponding comment.
> > > >
> > > > > --- editors/libreoffice/Makefile 6 Nov 2019 12:21:06 -0000 1.221
> > > > > +++ editors/libreoffice/Makefile 1 Dec 2019 10:26:58 -0000
> > > > > @@ -354,11 +354,11 @@ CONFIGURE_ARGS+=--without-java
> > > > >  .endif
> > > > >  
> > > > >  # XXX libreoffice has its own way to build things in parallel
> > > > > -PARALLEL_BUILD = No
> > > > > +PARALLEL_USES_MAKE = No
> > >
> > > I still don't understand that naming.
> > > If I don't use make(1) I can't do parallel builds?
> >
> > No, the way it works is that *by default* MAKE_JOBS is passed to
> > make as make -j${MAKE_JOBS}
> >
> > Look at both ports that override it, they explicitly set it to NO because
> > they pass "MAKE_JOBS" *elsewhere* (and don't want to use make -j)
> >
> > I tried for a less confusing naming than PARALLEL_BUILD.  Tell me what
> > to call it then.!
> >
>
> # XXX libreoffice has its own way to build things in parallel
> USES_PARALLEL_MAKE = No
>
> ?

+1

--
Antoine

Reply | Threaded
Open this post in threaded view
|

Re: patch: make dpb and ports more uniform

Christian Weisgerber
On 2019-12-03, Antoine Jacoutot <[hidden email]> wrote:

>> # XXX libreoffice has its own way to build things in parallel
>> USES_PARALLEL_MAKE = No
>>
>> ?
>
> +1

Like USES_GMAKE, USES_GROFF, USES_LIBTOOL, ...?

--
Christian "naddy" Weisgerber                          [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: patch: make dpb and ports more uniform

Stuart Henderson
On 2019/12/04 12:43, Christian Weisgerber wrote:

> On 2019-12-03, Antoine Jacoutot <[hidden email]> wrote:
>
> >> # XXX libreoffice has its own way to build things in parallel
> >> USES_PARALLEL_MAKE = No
> >>
> >> ?
> >
> > +1
>
> Like USES_GMAKE, USES_GROFF, USES_LIBTOOL, ...?

hmm, fair point.

Perhaps we could do something like this instead, and set PARALLEL_MAKE_FLAGS
to nothing in libreoffice etc.

(I'm unsure about ALL_FAKE_FLAGS, I don't think anything currently
uses PARALLEL_INSTALL=No, but there have been cases before where
parallel-build is OK but parallel-install isn't).


Index: bsd.port.mk
===================================================================
RCS file: /cvs/ports/infrastructure/mk/bsd.port.mk,v
retrieving revision 1.1509
diff -u -p -r1.1509 bsd.port.mk
--- bsd.port.mk 2 Dec 2019 18:59:03 -0000 1.1509
+++ bsd.port.mk 4 Dec 2019 12:51:40 -0000
@@ -454,21 +454,23 @@ ALL_FAKE_FLAGS= ${MAKE_FLAGS:N-j[0-9]*}
 _PKG_ADD += -L ${LOCALBASE}
 .endif
 
-# XXX this stuff is not production-ready, because there are too many bugs in
-# parallel make yet.  MAKE_JOBS>1 is known to work on a few ports and used
-# sparingly by dpb (DPB_PROPERTIES=parallel) for obvious gains.
+# XXX there are still many bugs in parallel make.
+# so MAKE_JOBS is used sparingly by dpb for obvious gains.
 #
-PARALLEL_BUILD ?= Yes
-PARALLEL_INSTALL ?= ${PARALLEL_BUILD}
+PARALLEL_MAKE_FLAGS ?= -j${MAKE_JOBS}
+
+.if !defined(MAKE_JOBS) && ${DPB_PROPERTIES:Mparallel})
+.  if defined(PARALLEL_MAKE_JOBS)
+MAKE_JOBS = ${PARALLEL_MAKE_JOBS}
+.  else
+MAKE_JOBS !!= sysctl -n hw.ncpuonline
+.  endif
+.endif
 MAKE_JOBS ?= 1
 
 .if ${MAKE_JOBS} != 1
-.  if ${PARALLEL_BUILD:L} == "yes"
-MAKE_FLAGS += -j${MAKE_JOBS}
-.  endif
-.  if ${PARALLEL_INSTALL:L} == "yes"
-ALL_FAKE_FLAGS += -j${MAKE_JOBS}
-.  endif
+MAKE_FLAGS += ${PARALLEL_MAKE_FLAGS}
+ALL_FAKE_FLAGS += ${PARALLEL_MAKE_FLAGS}
 .endif
 
 # Here comes the part that sets BUILD_PACKAGES and various IGNORE* up.

Reply | Threaded
Open this post in threaded view
|

Re: patch: make dpb and ports more uniform

Marc Espie-2
On Wed, Dec 04, 2019 at 12:55:33PM +0000, Stuart Henderson wrote:

> On 2019/12/04 12:43, Christian Weisgerber wrote:
> > On 2019-12-03, Antoine Jacoutot <[hidden email]> wrote:
> >
> > >> # XXX libreoffice has its own way to build things in parallel
> > >> USES_PARALLEL_MAKE = No
> > >>
> > >> ?
> > >
> > > +1
> >
> > Like USES_GMAKE, USES_GROFF, USES_LIBTOOL, ...?
>
> hmm, fair point.
>
> Perhaps we could do something like this instead, and set PARALLEL_MAKE_FLAGS
> to nothing in libreoffice etc.
>
> (I'm unsure about ALL_FAKE_FLAGS, I don't think anything currently
> uses PARALLEL_INSTALL=No, but there have been cases before where
> parallel-build is OK but parallel-install isn't).

None of them uses DPB_PROPERTIES=parallel

dpb thinks there are n jobs per machine during fake, only package becomes
single job.

I'm okay with your version.

Let's get this in before it turns into the bike-shed from hell

Reply | Threaded
Open this post in threaded view
|

Re: patch: make dpb and ports more uniform

Stuart Henderson
On 2019/12/04 14:29, Marc Espie wrote:

> On Wed, Dec 04, 2019 at 12:55:33PM +0000, Stuart Henderson wrote:
> > On 2019/12/04 12:43, Christian Weisgerber wrote:
> > > On 2019-12-03, Antoine Jacoutot <[hidden email]> wrote:
> > >
> > > >> # XXX libreoffice has its own way to build things in parallel
> > > >> USES_PARALLEL_MAKE = No
> > > >>
> > > >> ?
> > > >
> > > > +1
> > >
> > > Like USES_GMAKE, USES_GROFF, USES_LIBTOOL, ...?
> >
> > hmm, fair point.
> >
> > Perhaps we could do something like this instead, and set PARALLEL_MAKE_FLAGS
> > to nothing in libreoffice etc.
> >
> > (I'm unsure about ALL_FAKE_FLAGS, I don't think anything currently
> > uses PARALLEL_INSTALL=No, but there have been cases before where
> > parallel-build is OK but parallel-install isn't).
>
> None of them uses DPB_PROPERTIES=parallel
>
> dpb thinks there are n jobs per machine during fake, only package becomes
> single job.
>
> I'm okay with your version.
>
> Let's get this in before it turns into the bike-shed from hell
>

So here's a diff with the various parts for ports, including removing
the pseudo flavour for libreoffice, I've quickly edited README.internals
but not manpages.

I'm not sure about the change in default behaviour though, it means
that unless you set PARALLEL_MAKE_JOBS explicitly, it will now build
with hw.ncpuonline by default for all DPB_PROPERTIES=parallel ports.
This won't get well on smaller-memory machines.



Index: devel/jdk/1.8/Makefile
===================================================================
RCS file: /cvs/ports/devel/jdk/1.8/Makefile,v
retrieving revision 1.48
diff -u -p -r1.48 Makefile
--- devel/jdk/1.8/Makefile 21 Nov 2019 17:39:08 -0000 1.48
+++ devel/jdk/1.8/Makefile 4 Dec 2019 16:15:38 -0000
@@ -84,7 +84,7 @@ USE_GMAKE= Yes
 GMAKE= ${LOCALBASE}/bin/gmake
 
 # Don't call make -jN, instead pass MAKE_JOBS to --with-jobs
-PARALLEL_BUILD= No
+PARALLEL_MAKE_FLAGS=
 
 CONFIGURE_STYLE=gnu
 CONFIGURE_ARGS= --disable-option-checking \
Index: devel/jdk/11/Makefile
===================================================================
RCS file: /cvs/ports/devel/jdk/11/Makefile,v
retrieving revision 1.11
diff -u -p -r1.11 Makefile
--- devel/jdk/11/Makefile 26 Oct 2019 17:04:15 -0000 1.11
+++ devel/jdk/11/Makefile 4 Dec 2019 16:15:38 -0000
@@ -84,7 +84,7 @@ USE_GMAKE= Yes
 GMAKE= ${LOCALBASE}/bin/gmake
 
 # Don't call make -jN, instead pass MAKE_JOBS to --with-jobs
-PARALLEL_BUILD= No
+PARALLEL_MAKE_FLAGS=
 
 CONFIGURE_STYLE=gnu
 CONFIGURE_ENV+= AUTOCONF_VERSION=${AUTOCONF_VERSION} \
Index: editors/libreoffice/Makefile
===================================================================
RCS file: /cvs/ports/editors/libreoffice/Makefile,v
retrieving revision 1.221
diff -u -p -r1.221 Makefile
--- editors/libreoffice/Makefile 6 Nov 2019 12:21:06 -0000 1.221
+++ editors/libreoffice/Makefile 4 Dec 2019 16:15:38 -0000
@@ -136,7 +136,7 @@ PERMIT_PACKAGE= Yes
 
 MULTI_PACKAGES= -main -java -kde
 
-PSEUDO_FLAVORS= parallel no_lang
+PSEUDO_FLAVORS= no_lang
 FLAVORS= debug no_java
 FLAVOR?=
 
@@ -354,15 +354,8 @@ CONFIGURE_ARGS+=--without-java
 .endif
 
 # XXX libreoffice has its own way to build things in parallel
-PARALLEL_BUILD = No
-.if defined(MAKE_JOBS)
-NCPU= ${MAKE_JOBS}
-.elif ${FLAVOR:Mparallel}
-NCPU!= echo $$((`sysctl -n hw.ncpuonline` + 1))
-.else
-NCPU= 1
-.endif
-CONFIGURE_ARGS+=--with-parallelism=${NCPU}
+PARALLEL_MAKE_FLAGS =
+CONFIGURE_ARGS+=--with-parallelism=${MAKE_JOBS}
 
 .if ${FLAVOR:Mdebug}
 CONFIGURE_ARGS+=--enable-debug
Index: infrastructure/mk/bsd.port.mk
===================================================================
RCS file: /cvs/ports/infrastructure/mk/bsd.port.mk,v
retrieving revision 1.1509
diff -u -p -r1.1509 bsd.port.mk
--- infrastructure/mk/bsd.port.mk 2 Dec 2019 18:59:03 -0000 1.1509
+++ infrastructure/mk/bsd.port.mk 4 Dec 2019 16:15:38 -0000
@@ -454,21 +454,23 @@ ALL_FAKE_FLAGS= ${MAKE_FLAGS:N-j[0-9]*}
 _PKG_ADD += -L ${LOCALBASE}
 .endif
 
-# XXX this stuff is not production-ready, because there are too many bugs in
-# parallel make yet.  MAKE_JOBS>1 is known to work on a few ports and used
-# sparingly by dpb (DPB_PROPERTIES=parallel) for obvious gains.
+# XXX there are still many bugs in parallel make.
+# so MAKE_JOBS is used sparingly by dpb for obvious gains.
 #
-PARALLEL_BUILD ?= Yes
-PARALLEL_INSTALL ?= ${PARALLEL_BUILD}
+PARALLEL_MAKE_FLAGS ?= -j${MAKE_JOBS}
+
+.if !defined(MAKE_JOBS) && ${DPB_PROPERTIES:Mparallel})
+.  if defined(PARALLEL_MAKE_JOBS)
+MAKE_JOBS = ${PARALLEL_MAKE_JOBS}
+.  else
+MAKE_JOBS !!= sysctl -n hw.ncpuonline
+.  endif
+.endif
 MAKE_JOBS ?= 1
 
 .if ${MAKE_JOBS} != 1
-.  if ${PARALLEL_BUILD:L} == "yes"
-MAKE_FLAGS += -j${MAKE_JOBS}
-.  endif
-.  if ${PARALLEL_INSTALL:L} == "yes"
-ALL_FAKE_FLAGS += -j${MAKE_JOBS}
-.  endif
+MAKE_FLAGS += ${PARALLEL_MAKE_FLAGS}
+ALL_FAKE_FLAGS += ${PARALLEL_MAKE_FLAGS}
 .endif
 
 # Here comes the part that sets BUILD_PACKAGES and various IGNORE* up.
Index: infrastructure/mk/README.internals
===================================================================
RCS file: /cvs/ports/infrastructure/mk/README.internals,v
retrieving revision 1.16
diff -u -p -r1.16 README.internals
--- infrastructure/mk/README.internals 1 Dec 2019 09:36:29 -0000 1.16
+++ infrastructure/mk/README.internals 4 Dec 2019 16:15:39 -0000
@@ -437,15 +437,16 @@ _PERL_FIX_SHAR make visible ?
 TODO fully document:
  MAKE_JOBS
  PARALLEL_BUILD
- PARALLEL_INSTALL
  XAUTHORITY for interactive tests
 
-MAKE_JOBS is set by dpb based on DPB_PROPERTIES=parallel
-PARALLEL_BUILD is currently used to control passing MAKE_JOBS as make -j.
-(some ports do MAKE_JOBS differently: libreoffice, jdk)
+MAKE_JOBS is set by dpb based on DPB_PROPERTIES=parallel or can be set
+by the user. When MAKE_JOBS is >1, PARALLEL_MAKE_FLAGS is added to
+MAKE_FLAGS, by default this contains -j${MAKE_JOBS} which works for
+most ports. Some ports do MAKE_JOBS differently (libreoffice, jdk),
+in those cases "make -j" is not wanted, so PARALLEL_MAKE_FLAGS
+should be explicitly cleared.
 
 Probably time to rename that and really document it.
-PARALLEL_INSTALL is unused.
 fishy: FLAVORS tests ?
 fishy: TEST_STATUS_IGNORE
 

Reply | Threaded
Open this post in threaded view
|

Re: patch: make dpb and ports more uniform

Marc Espie-2
On Wed, Dec 04, 2019 at 04:29:19PM +0000, Stuart Henderson wrote:

> On 2019/12/04 14:29, Marc Espie wrote:
> > On Wed, Dec 04, 2019 at 12:55:33PM +0000, Stuart Henderson wrote:
> > > On 2019/12/04 12:43, Christian Weisgerber wrote:
> > > > On 2019-12-03, Antoine Jacoutot <[hidden email]> wrote:
> > > >
> > > > >> # XXX libreoffice has its own way to build things in parallel
> > > > >> USES_PARALLEL_MAKE = No
> > > > >>
> > > > >> ?
> > > > >
> > > > > +1
> > > >
> > > > Like USES_GMAKE, USES_GROFF, USES_LIBTOOL, ...?
> > >
> > > hmm, fair point.
> > >
> > > Perhaps we could do something like this instead, and set PARALLEL_MAKE_FLAGS
> > > to nothing in libreoffice etc.
> > >
> > > (I'm unsure about ALL_FAKE_FLAGS, I don't think anything currently
> > > uses PARALLEL_INSTALL=No, but there have been cases before where
> > > parallel-build is OK but parallel-install isn't).
> >
> > None of them uses DPB_PROPERTIES=parallel
> >
> > dpb thinks there are n jobs per machine during fake, only package becomes
> > single job.
> >
> > I'm okay with your version.
> >
> > Let's get this in before it turns into the bike-shed from hell
> >
>
> So here's a diff with the various parts for ports, including removing
> the pseudo flavour for libreoffice, I've quickly edited README.internals
> but not manpages.
>
> I'm not sure about the change in default behaviour though, it means
> that unless you set PARALLEL_MAKE_JOBS explicitly, it will now build
> with hw.ncpuonline by default for all DPB_PROPERTIES=parallel ports.
> This won't get well on smaller-memory machines.

Blah. You could make it half that on, say 32 bit arches.
This can happen later if neede be.