dpb: Patch to suprress errors seen with dpb -I

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

dpb: Patch to suprress errors seen with dpb -I

Preben Guldberg
Below is a patch that suppresses errors I get when running dpb with -I:

    Use of uninitialized value $ts in sprintf at /usr/ports/infrastructure/lib/DPB/Util.pm line 58.
    DPB::Util::ts2string(DPB::Util, <undef>) called at /usr/ports/infrastructure/lib/DPB/Job/Port.pm line 1107
    DPB::Job::BasePort::timings(DPB::Job::Port::Install=HASH(0xf3c120630d0)(sysutils/iwatch)) called at /usr/ports/infrastructure/lib/DPB/Job/Port.pm line 964
    DPB::Job::BasePort::__ANON__(DPB::Core=HASH(0xf3c1203be38)(localhost:sysutils/iwatch)) called at /usr/ports/infrastructure/lib/DPB/Job.pm line 189
    DPB::Job::Normal::finalize(DPB::Job::Port::Install=HASH(0xf3c120630d0)(sysutils/iwatch), DPB::Core=HASH(0xf3c1203be38)(localhost:sysutils/iwatch)) called at /
    [... snip - longer example at the end ...]

If I run with -P instead of -I, the errors are not seen.

The diff at the end avoids a DPB::Util->ts2string($self->{watched}{max})
call when $self->{watched} is undefined.  My feeling is that the install
phase does not watch for stuck processes at all, so it benefits from
this.

Whether this is the right way or place to address the problem, I don't
know, but with the patch the errors are suppressed for me.

To reproduce, I am starting dpb with a command like this:

    % doas /usr/ports/infrastructure/bin/dpb -I /tmp/ports.lst -L /home/obsd/ports/logs

My list of ports and mk.conf is below.

    % cat /tmp/ports.lst
    editors/vim,no_x11,-main
    mail/mutt,sasl
    shells/zsh
    sysutils/iwatch
    sysutils/parallel
    sysutils/pv
    devel/got

    % cat /etc/mk.conf
    SUDO=                   doas
    PORTSDIR_PATH=          /home/preben/OpenBSD/ports:${PORTSDIR}
    BULK_COOKIES_DIR?=      /home/obsd/ports/bulk
    DISTDIR?=               /home/obsd/ports/distfiles
    PACKAGE_REPOSITORY?=    /home/obsd/ports/packages
    PLIST_REPOSITORY?=      /home/obsd/ports/plist
    UPDATE_COOKIES_DIR?=    /home/obsd/ports/update
    WRKOBJDIR?=             /home/obsd/ports/pobj
    WRKOBJDIR_MFS?=         /tmp/pobj
    PORTS_PRIVSEP?=         Yes
    %

Longer output while dpb was running:

    4 Apr 21:20:08 [85831] control-ox-85831 elapsed: 00:00:03
    sysutils/parallel(install) [23258]
    sysutils/pv(install) [24069]
    LISTING [146] at devel/gmake
    sysutils/pv(install) [78151]
    editors/vim,no_x11,-main(install) [49646]
    Hosts: localhost
    I=16 B=7 Q=0 T=0 F=0 !=1
    Use of uninitialized value $ts in sprintf at /usr/ports/infrastructure/lib/DPB/Util.pm line 58, <__ANONIO__> line 1.
    DPB::Util::ts2string(DPB::Util, <undef>) called at /usr/ports/infrastructure/lib/DPB/Job/Port.pm line 1107
    DPB::Job::BasePort::timings(DPB::Job::Port::Install=HASH(0xf3b30e0fa78)(devel/got)) called at /usr/ports/infrastructure/lib/DPB/Job/Port.pm line 964
    DPB::Job::BasePort::__ANON__(DPB::Core=HASH(0xf3c1205e130)(localhost:devel/got)) called at /usr/ports/infrastructure/lib/DPB/Job.pm line 189
    DPB::Job::Normal::finalize(DPB::Job::Port::Install=HASH(0xf3b30e0fa78)(devel/got), DPB::Core=HASH(0xf3c1205e130)(localhost:devel/got)) called at /usr/ports/in
    frastructure/lib/DPB/Job/Port.pm line 1089
    DPB::Job::BasePort::finalize(DPB::Job::Port::Install=HASH(0xf3b30e0fa78)(devel/got), DPB::Core=HASH(0xf3c1205e130)(localhost:devel/got)) called at /usr/ports/
    infrastructure/lib/DPB/Core.pm line 413
    DPB::Core::WithJobs::start_task(DPB::Core=HASH(0xf3c1205e130)(localhost:devel/got)) called at /usr/ports/infrastructure/lib/DPB/Core.pm line 399
    DPB::Core::WithJobs::continue(DPB::Core=HASH(0xf3c1205e130)(localhost:devel/got)) called at /usr/ports/infrastructure/lib/DPB/Core.pm line 212
    DPB::Core::Abstract::reap_kid(DPB::Core, 20938) called at /usr/ports/infrastructure/lib/DPB/Core.pm line 225
    DPB::Core::Abstract::reap(DPB::Core) called at /usr/ports/infrastructure/bin/dpb line 216
    main::handle_non_waiting_jobs(DPB::Core) called at /usr/ports/infrastructure/lib/DPB/Grabber.pm line 108
    DPB::Grabber::finish(DPB::Grabber=HASH(0xf3bb511d2e0), HASH(0xf3c120614d8)) called at /usr/ports/infrastructure/lib/DPB/Grabber.pm line 215
    DPB::Grabber::__ANON__(HASH(0xf3c120614d8)) called at /usr/ports/infrastructure/lib/DPB/Vars.pm line 130
    DPB::Vars::__ANON__(HASH(0xf3c120614d8)) called at /usr/ports/infrastructure/lib/DPB/Vars.pm line 178
    DPB::Vars::grab_list(DPB::Vars, DPB::Core=HASH(0xf3c1203bdd8)(localhost:LISTING), DPB::Grabber=HASH(0xf3bb511d2e0), HASH(0xf3c00a38880), <undef>, 0, GLOB(0xf3
    bdfffbc10), fetch, CODE(0xf3c00a383e8)) called at /usr/ports/infrastructure/lib/DPB/Grabber.pm line 216
    DPB::Grabber::complete_subdirs(DPB::Grabber=HASH(0xf3bb511d2e0), DPB::Core=HASH(0xf3c1203bdd8)(localhost:LISTING), <undef>) called at /usr/ports/infrastructur
    e/bin/dpb line 309
    Use of uninitialized value $ts in sprintf at /usr/ports/infrastructure/lib/DPB/Util.pm line 58.
    DPB::Util::ts2string(DPB::Util, <undef>) called at /usr/ports/infrastructure/lib/DPB/Job/Port.pm line 1107
    DPB::Job::BasePort::timings(DPB::Job::Port::Install=HASH(0xf3c120630d0)(sysutils/iwatch)) called at /usr/ports/infrastructure/lib/DPB/Job/Port.pm line 964
    DPB::Job::BasePort::__ANON__(DPB::Core=HASH(0xf3c1203be38)(localhost:sysutils/iwatch)) called at /usr/ports/infrastructure/lib/DPB/Job.pm line 189
    DPB::Job::Normal::finalize(DPB::Job::Port::Install=HASH(0xf3c120630d0)(sysutils/iwatch), DPB::Core=HASH(0xf3c1203be38)(localhost:sysutils/iwatch)) called at /
    usr/ports/infrastructure/lib/DPB/Job/Port.pm line 1089
    DPB::Job::BasePort::finalize(DPB::Job::Port::Install=HASH(0xf3c120630d0)(sysutils/iwatch), DPB::Core=HASH(0xf3c1203be38)(localhost:sysutils/iwatch)) called at

And finally, the diff:

diff refs/heads/master refs/heads/pg
blob - aceb2d60195811d2215d000bb21c4e20fcf02a18
blob + f4f2b443ad8c92571f8e5dc032516a3b55ac191b
--- infrastructure/lib/DPB/Job/Port.pm
+++ infrastructure/lib/DPB/Job/Port.pm
@@ -1104,7 +1104,12 @@ sub totaltime
 sub timings
 {
  my $self = shift;
- return join('/', "max_stuck=".DPB::Util->ts2string($self->{watched}{max}), map {sprintf("%s=%.2f", $_->{phase}, $_->elapsed)} @{$self->{done}});
+ my $stuck_status = "unwatched";
+
+ if (defined($self->{watched})) {
+ $stuck_status = DPB::Util->ts2string($self->{watched}{max});
+ }
+ return join('/', "max_stuck=".$stuck_status, map {sprintf("%s=%.2f", $_->{phase}, $_->elapsed)} @{$self->{done}});
 }
 
 sub equates

Reply | Threaded
Open this post in threaded view
|

Re: dpb: Patch to suprress errors seen with dpb -I

Marc Espie-2
On Sun, Apr 04, 2021 at 09:56:49PM +0200, Preben Guldberg wrote:

> Below is a patch that suppresses errors I get when running dpb with -I:
>
>     Use of uninitialized value $ts in sprintf at /usr/ports/infrastructure/lib/DPB/Util.pm line 58.
>     DPB::Util::ts2string(DPB::Util, <undef>) called at /usr/ports/infrastructure/lib/DPB/Job/Port.pm line 1107
>     DPB::Job::BasePort::timings(DPB::Job::Port::Install=HASH(0xf3c120630d0)(sysutils/iwatch)) called at /usr/ports/infrastructure/lib/DPB/Job/Port.pm line 964
>     DPB::Job::BasePort::__ANON__(DPB::Core=HASH(0xf3c1203be38)(localhost:sysutils/iwatch)) called at /usr/ports/infrastructure/lib/DPB/Job.pm line 189
>     DPB::Job::Normal::finalize(DPB::Job::Port::Install=HASH(0xf3c120630d0)(sysutils/iwatch), DPB::Core=HASH(0xf3c1203be38)(localhost:sysutils/iwatch)) called at /
>     [... snip - longer example at the end ...]
>
> If I run with -P instead of -I, the errors are not seen.
>
> The diff at the end avoids a DPB::Util->ts2string($self->{watched}{max})
> call when $self->{watched} is undefined.  My feeling is that the install
> phase does not watch for stuck processes at all, so it benefits from
> this.
>
> Whether this is the right way or place to address the problem, I don't
> know, but with the patch the errors are suppressed for me.

Naddy reported something similar.

The actual fix would be somewhat different. Watch is set in an odd location
in PortBuilder, whereas it should probably be done at the end of
Job::BasePort->new

(Install could very well get stuck, because install will run build as well
if the package doesn't already exist, so watched should always be set)

I have a patch for this but I haven't been able to test it yet because my
dpb boxes are busy with something else.

Reply | Threaded
Open this post in threaded view
|

Re: dpb: Patch to suprress errors seen with dpb -I

Preben Guldberg
Marc Espie wrote:
> On Sun, Apr 04, 2021 at 09:56:49PM +0200, Preben Guldberg wrote:
> > Below is a patch that suppresses errors I get when running dpb with -I:

> The actual fix would be somewhat different. Watch is set in an odd location
> in PortBuilder, whereas it should probably be done at the end of
> Job::BasePort->new

> (Install could very well get stuck, because install will run build as well
> if the package doesn't already exist, so watched should always be set)

I hadn't considered that (or gotten as deep). Thanks for the explanation.

> I have a patch for this but I haven't been able to test it yet because my
> dpb boxes are busy with something else.

If you would like me to test the patch on my system, let me know.

Thank you,
Preben

Reply | Threaded
Open this post in threaded view
|

Re: dpb: Patch to suprress errors seen with dpb -I

Marc Espie-2
In reply to this post by Marc Espie-2
On Tue, Apr 06, 2021 at 08:49:10PM +0200, Marc Espie wrote:

> On Sun, Apr 04, 2021 at 09:56:49PM +0200, Preben Guldberg wrote:
> > Below is a patch that suppresses errors I get when running dpb with -I:
> >
> >     Use of uninitialized value $ts in sprintf at /usr/ports/infrastructure/lib/DPB/Util.pm line 58.
> >     DPB::Util::ts2string(DPB::Util, <undef>) called at /usr/ports/infrastructure/lib/DPB/Job/Port.pm line 1107
> >     DPB::Job::BasePort::timings(DPB::Job::Port::Install=HASH(0xf3c120630d0)(sysutils/iwatch)) called at /usr/ports/infrastructure/lib/DPB/Job/Port.pm line 964
> >     DPB::Job::BasePort::__ANON__(DPB::Core=HASH(0xf3c1203be38)(localhost:sysutils/iwatch)) called at /usr/ports/infrastructure/lib/DPB/Job.pm line 189
> >     DPB::Job::Normal::finalize(DPB::Job::Port::Install=HASH(0xf3c120630d0)(sysutils/iwatch), DPB::Core=HASH(0xf3c1203be38)(localhost:sysutils/iwatch)) called at /
> >     [... snip - longer example at the end ...]
> >
> > If I run with -P instead of -I, the errors are not seen.
> >
> > The diff at the end avoids a DPB::Util->ts2string($self->{watched}{max})
> > call when $self->{watched} is undefined.  My feeling is that the install
> > phase does not watch for stuck processes at all, so it benefits from
> > this.
> >
> > Whether this is the right way or place to address the problem, I don't
> > know, but with the patch the errors are suppressed for me.
>
> Naddy reported something similar.
>
> The actual fix would be somewhat different. Watch is set in an odd location
> in PortBuilder, whereas it should probably be done at the end of
> Job::BasePort->new
>
> (Install could very well get stuck, because install will run build as well
> if the package doesn't already exist, so watched should always be set)
>
> I have a patch for this but I haven't been able to test it yet because my
> dpb boxes are busy with something else.
Previous patch was a brainfart, I had somehow refactored the code in
the wrong base class (e.g., not a base class at all)

Setting a watched uniformously in DPB::Job::BasePort looks like this:

Index: PortBuilder.pm
===================================================================
RCS file: /cvs/ports/infrastructure/lib/DPB/PortBuilder.pm,v
retrieving revision 1.88
diff -u -p -r1.88 PortBuilder.pm
--- PortBuilder.pm 21 Mar 2021 19:17:34 -0000 1.88
+++ PortBuilder.pm 14 Apr 2021 15:19:18 -0000
@@ -255,7 +255,6 @@ sub build
  $self->report($v, $job, $core);
  &$final_sub($job->{failed});
  });
- $job->set_watch($self->logger, $v);
  $core->start_job($job, $v);
  # lonesome takes precedence for swallowing everything
  if ($job->{lonesome}) {
@@ -283,7 +282,6 @@ sub wipe
  $self->report($v, $job, $core);
  &$final_sub($job->{failed});
  });
- $job->set_watch($self->logger, $v);
  $core->start_job($job, $v);
 }
 
Index: Job/Port.pm
===================================================================
RCS file: /cvs/ports/infrastructure/lib/DPB/Job/Port.pm,v
retrieving revision 1.204
diff -u -p -r1.204 Port.pm
--- Job/Port.pm 21 Mar 2021 19:17:34 -0000 1.204
+++ Job/Port.pm 14 Apr 2021 15:19:18 -0000
@@ -969,6 +969,7 @@ sub new
  $job->{current} = '';
  # for stuff that doesn't really lock
  $job->{lock} //= DPB::DummyLock->new;
+ $job->set_watch($job->{builder}->logger, $job->{v});
 
  return $job;
 }

Reply | Threaded
Open this post in threaded view
|

Re: dpb: Patch to suprress errors seen with dpb -I

Preben Guldberg
Marc Espie wrote:
> On Tue, Apr 06, 2021 at 08:49:10PM +0200, Marc Espie wrote:
> > On Sun, Apr 04, 2021 at 09:56:49PM +0200, Preben Guldberg wrote:
> > > Below is a patch that suppresses errors I get when running dpb with -I:

> > >     Use of uninitialized value $ts in sprintf at /usr/ports/infrastructure/lib/DPB/Util.pm line 58.
> > >     DPB::Util::ts2string(DPB::Util, <undef>) called at /usr/ports/infrastructure/lib/DPB/Job/Port.pm line 1107
> > >     DPB::Job::BasePort::timings(DPB::Job::Port::Install=HASH(0xf3c120630d0)(sysutils/iwatch)) called at /usr/ports/infrastructure/lib/DPB/Job/Port.pm line 964
> > >     DPB::Job::BasePort::__ANON__(DPB::Core=HASH(0xf3c1203be38)(localhost:sysutils/iwatch)) called at /usr/ports/infrastructure/lib/DPB/Job.pm line 189
> > >     DPB::Job::Normal::finalize(DPB::Job::Port::Install=HASH(0xf3c120630d0)(sysutils/iwatch), DPB::Core=HASH(0xf3c1203be38)(localhost:sysutils/iwatch)) called at /
> > >     [... snip - longer example at the end ...]

> Previous patch was a brainfart, I had somehow refactored the code in
> the wrong base class (e.g., not a base class at all)

> Setting a watched uniformously in DPB::Job::BasePort looks like this:

FWIW, I tested on a freshly updated ports tree with and without the
patch below. The patch nicely makes the error message go away.

Thank you!
Preben

> Index: PortBuilder.pm
> ===================================================================
> RCS file: /cvs/ports/infrastructure/lib/DPB/PortBuilder.pm,v
> retrieving revision 1.88
> diff -u -p -r1.88 PortBuilder.pm
> --- PortBuilder.pm 21 Mar 2021 19:17:34 -0000 1.88
> +++ PortBuilder.pm 14 Apr 2021 15:19:18 -0000
> @@ -255,7 +255,6 @@ sub build
>   $self->report($v, $job, $core);
>   &$final_sub($job->{failed});
>   });
> - $job->set_watch($self->logger, $v);
>   $core->start_job($job, $v);
>   # lonesome takes precedence for swallowing everything
>   if ($job->{lonesome}) {
> @@ -283,7 +282,6 @@ sub wipe
>   $self->report($v, $job, $core);
>   &$final_sub($job->{failed});
>   });
> - $job->set_watch($self->logger, $v);
>   $core->start_job($job, $v);
>  }

> Index: Job/Port.pm
> ===================================================================
> RCS file: /cvs/ports/infrastructure/lib/DPB/Job/Port.pm,v
> retrieving revision 1.204
> diff -u -p -r1.204 Port.pm
> --- Job/Port.pm 21 Mar 2021 19:17:34 -0000 1.204
> +++ Job/Port.pm 14 Apr 2021 15:19:18 -0000
> @@ -969,6 +969,7 @@ sub new
>   $job->{current} = '';
>   # for stuff that doesn't really lock
>   $job->{lock} //= DPB::DummyLock->new;
> + $job->set_watch($job->{builder}->logger, $job->{v});

>   return $job;
>  }