File::Copy bug

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

File::Copy bug

Christian Weisgerber
Not sure who to report this to...

Perl's File::Copy::move has a bug or limitation that can cause
problems with autoconf, which uses File::Copy.

File::Copy::move tries to rename() the file, but necessarily falls
back to copy and unlink if source and destination are on different
file systems.  It also tries to copy the access and modification
timestamps, but uses utime(), which only has full second resolution.
Modern POSIX filesystems have nanosecond resolution timestamps.
Effectively, the timestamp of the moved file is truncated down to
a second, which means it travels back in time.  This can mean that
dependencies are out of date and make wants to rebuild files...

I guess File::Copy should use utimensat() or futimens().


(This is the ultimate source of sporadic, quasi-random build failures
I see during package bulk builds.)

--
Christian "naddy" Weisgerber                          [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: File::Copy bug

Raul Miller
search.cpan.org/~shay/perl-5.26.1/lib/File/Copy.pm suggests
perlmonks.org though the search.cpan.org/~shay/ email address might
also be good,

Good luck,

--
Raul

On Thu, Nov 2, 2017 at 11:39 AM, Christian Weisgerber
<[hidden email]> wrote:

> Not sure who to report this to...
>
> Perl's File::Copy::move has a bug or limitation that can cause
> problems with autoconf, which uses File::Copy.
>
> File::Copy::move tries to rename() the file, but necessarily falls
> back to copy and unlink if source and destination are on different
> file systems.  It also tries to copy the access and modification
> timestamps, but uses utime(), which only has full second resolution.
> Modern POSIX filesystems have nanosecond resolution timestamps.
> Effectively, the timestamp of the moved file is truncated down to
> a second, which means it travels back in time.  This can mean that
> dependencies are out of date and make wants to rebuild files...
>
> I guess File::Copy should use utimensat() or futimens().
>
>
> (This is the ultimate source of sporadic, quasi-random build failures
> I see during package bulk builds.)
>
> --
> Christian "naddy" Weisgerber                          [hidden email]
>

Reply | Threaded
Open this post in threaded view
|

Re: File::Copy bug

Andrew Fresh-3
In reply to this post by Christian Weisgerber
On Thu, Nov 02, 2017 at 04:39:23PM +0100, Christian Weisgerber wrote:
> Not sure who to report this to...

I'll submit a report to the perl5 rt issue tracker.  File::Copy is a
core module where the upstream is perl5-porters and the issue tracker at
rt.perl.org.

> Perl's File::Copy::move has a bug or limitation that can cause
> problems with autoconf, which uses File::Copy.
>
> File::Copy::move tries to rename() the file, but necessarily falls
> back to copy and unlink if source and destination are on different
> file systems.  It also tries to copy the access and modification
> timestamps, but uses utime(), which only has full second resolution.
> Modern POSIX filesystems have nanosecond resolution timestamps.
> Effectively, the timestamp of the moved file is truncated down to
> a second, which means it travels back in time.  This can mean that
> dependencies are out of date and make wants to rebuild files...
>
> I guess File::Copy should use utimensat() or futimens().

It probably should.  This is the patch I'm going to submit upstream if
you'd like to try it out.

If it works, I can also commit it in OpenBSD so we don't have to wait for
perl 5.28 or even 5.30 before having it.


Index: gnu/usr.bin/perl/lib/File/Copy.pm
===================================================================
RCS file: /cvs/src/gnu/usr.bin/perl/lib/File/Copy.pm,v
retrieving revision 1.17
diff -u -p -r1.17 Copy.pm
--- gnu/usr.bin/perl/lib/File/Copy.pm 5 Feb 2017 00:32:15 -0000 1.17
+++ gnu/usr.bin/perl/lib/File/Copy.pm 5 Nov 2017 21:05:33 -0000
@@ -16,6 +16,8 @@ use Config;
 # And then we need these games to avoid loading overload, as that will
 # confuse miniperl during the bootstrap of perl.
 my $Scalar_Util_loaded = eval q{ require Scalar::Util; require overload; 1 };
+# We want HiRes stat and utime if available
+eval q{ require Time::HiRes; Time::HiRes->import(qw( stat utime )); 1 };
 our(@ISA, @EXPORT, @EXPORT_OK, $VERSION, $Too_Big, $Syscopy_is_copy);
 sub copy;
 sub syscopy;
Index: gnu/usr.bin/perl/lib/File/Copy.t
===================================================================
RCS file: /cvs/src/gnu/usr.bin/perl/lib/File/Copy.t,v
retrieving revision 1.2
diff -u -p -r1.2 Copy.t
--- gnu/usr.bin/perl/lib/File/Copy.t 5 Feb 2017 00:32:15 -0000 1.2
+++ gnu/usr.bin/perl/lib/File/Copy.t 5 Nov 2017 21:05:33 -0000
@@ -24,6 +24,9 @@ BEGIN { *CORE::GLOBAL::rename = sub { CO
 use File::Copy qw(copy move cp);
 use Config;
 
+# If we have Time::HiRes, File::Copy loaded it for us.
+my $have_hires_utime = eval {Time::HiRes::d_hires_utime()};
+note "Testing Time::HiRes::utime support" if $have_hires_utime;
 
 foreach my $code ("copy()", "copy('arg')", "copy('arg', 'arg', 'arg', 'arg')",
                   "move()", "move('arg')", "move('arg', 'arg', 'arg')"
@@ -101,6 +104,7 @@ for my $cross_partition_test (0..1) {
 
   # Doesn't really matter what time it is as long as its not now.
   my $time = 1000000000;
+  $time += 0.1234 if $have_hires_utime;
   utime( $time, $time, "copy-$$" );
 
   # Recheck the mtime rather than rely on utime in case we're on a

--
andrew - http://afresh1.com

Adding manpower to a late software project makes it later.

Reply | Threaded
Open this post in threaded view
|

Re: File::Copy bug

Andrew Fresh-3
On Sun, Nov 05, 2017 at 02:07:02PM -0700, Andrew Hewus Fresh wrote:
> It probably should.  This is the patch I'm going to submit upstream if
> you'd like to try it out.

As noticed by naddy, my test was faulty and I need this to happen in a
BEGIN block because

The test needs to use the HiRes utime or else it loses the hires time we
tried to hard to set.


Index: gnu/usr.bin/perl/lib/File/Copy.pm
===================================================================
RCS file: /cvs/src/gnu/usr.bin/perl/lib/File/Copy.pm,v
retrieving revision 1.17
diff -u -p -r1.17 Copy.pm
--- gnu/usr.bin/perl/lib/File/Copy.pm 5 Feb 2017 00:32:15 -0000 1.17
+++ gnu/usr.bin/perl/lib/File/Copy.pm 5 Nov 2017 22:14:38 -0000
@@ -16,6 +16,8 @@ use Config;
 # And then we need these games to avoid loading overload, as that will
 # confuse miniperl during the bootstrap of perl.
 my $Scalar_Util_loaded = eval q{ require Scalar::Util; require overload; 1 };
+# We want HiRes stat and utime if available
+BEGIN { eval q{ use Time::HiRes qw( stat utime ) } };
 our(@ISA, @EXPORT, @EXPORT_OK, $VERSION, $Too_Big, $Syscopy_is_copy);
 sub copy;
 sub syscopy;
Index: gnu/usr.bin/perl/lib/File/Copy.t
===================================================================
RCS file: /cvs/src/gnu/usr.bin/perl/lib/File/Copy.t,v
retrieving revision 1.2
diff -u -p -r1.2 Copy.t
--- gnu/usr.bin/perl/lib/File/Copy.t 5 Feb 2017 00:32:15 -0000 1.2
+++ gnu/usr.bin/perl/lib/File/Copy.t 5 Nov 2017 22:14:38 -0000
@@ -24,6 +24,11 @@ BEGIN { *CORE::GLOBAL::rename = sub { CO
 use File::Copy qw(copy move cp);
 use Config;
 
+# If we have Time::HiRes, File::Copy loaded it for us.
+BEGIN {
+  eval { Time::HiRes->import(qw( stat utime )) };
+  note "Testing Time::HiRes::utime support" unless $@;
+}
 
 foreach my $code ("copy()", "copy('arg')", "copy('arg', 'arg', 'arg', 'arg')",
                   "move()", "move('arg')", "move('arg', 'arg', 'arg')"
@@ -100,7 +105,7 @@ for my $cross_partition_test (0..1) {
   ok -e "copy-$$",                '  target still there';
 
   # Doesn't really matter what time it is as long as its not now.
-  my $time = 1000000000;
+  my $time = 1000000000.12345;
   utime( $time, $time, "copy-$$" );
 
   # Recheck the mtime rather than rely on utime in case we're on a

--
andrew - http://afresh1.com

Whatever happened to the days when hacking started at the cerebral cortex
and not the keyboard?
                      -- Sid from UserFriendly.org

Reply | Threaded
Open this post in threaded view
|

Re: File::Copy bug

Marc Espie-2
On Sun, Nov 05, 2017 at 03:20:16PM -0700, Andrew Hewus Fresh wrote:
> On Sun, Nov 05, 2017 at 02:07:02PM -0700, Andrew Hewus Fresh wrote:
> > It probably should.  This is the patch I'm going to submit upstream if
> > you'd like to try it out.
>
> As noticed by naddy, my test was faulty and I need this to happen in a
> BEGIN block because
>
> The test needs to use the HiRes utime or else it loses the hires time we
> tried to hard to set.

You might also want to put a no warnings; in the eval block so that it's
100% silent ?