FAQ's duplicating file systems, both methods fail to reproduce correctly

classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

FAQ's duplicating file systems, both methods fail to reproduce correctly

webmaster
Forgive problems with this email.
I saw how my emails showed up on marc.info
Scary. This is just temporary.

OK. I've tried to use both methods and just don't
get true duplication.

tar
It can't work with file and directory names
that are OK in filesystem, but too long for itself.
Quite a while back I lost a lot of unimportant files
and directories that had absolute paths too long.
Why is this happening with tar? Can this be fixed?
If not, I'd like to add a note about that to the FAQ.

dump
I had to move /usr/local to a bigger partition. growfs,
etc. I kept the /usr/local untouched and then dumped it
to the new partition, expecting a true duplication.
Nope.
It changed all of the program symlinks permissions.
Why is dump doing this? Can this be fixed?
Otherwise, a note about this should be added to the FAQ
also.

Question:
Can dd be used to do what I did with dump or tar?
Smaller partition copied to a bigger partition.

I'm willing to try and help out, but I'm going through
both laptop and server hell at the moment.

Thanks,
Chris Bennett

Reply | Threaded
Open this post in threaded view
|

Re: FAQ's duplicating file systems, both methods fail to reproduce correctly

webmaster
I'm not able to try it right now, but would gtar
accomplish what that our tar doesn't for this?
As in maybe pull something out of it into our tar?

Chris Bennett


Reply | Threaded
Open this post in threaded view
|

Re: FAQ's duplicating file systems, both methods fail to reproduce correctly

vincent delft-2
In reply to this post by webmaster
Hello,

Did you tried pax ?
some thing like: pax -rw -pe <source> <destination>

I don't know if this the best tool, but I'm using it to duplicate a 1TB
drive (having lot of hard links) onto an other one.
I've done it couple of time, and I've do not see issues.

rgds










On Sun, Dec 10, 2017 at 7:03 PM, <[hidden email]> wrote:

> Forgive problems with this email.
> I saw how my emails showed up on marc.info
> Scary. This is just temporary.
>
> OK. I've tried to use both methods and just don't
> get true duplication.
>
> tar
> It can't work with file and directory names
> that are OK in filesystem, but too long for itself.
> Quite a while back I lost a lot of unimportant files
> and directories that had absolute paths too long.
> Why is this happening with tar? Can this be fixed?
> If not, I'd like to add a note about that to the FAQ.
>
> dump
> I had to move /usr/local to a bigger partition. growfs,
> etc. I kept the /usr/local untouched and then dumped it
> to the new partition, expecting a true duplication.
> Nope.
> It changed all of the program symlinks permissions.
> Why is dump doing this? Can this be fixed?
> Otherwise, a note about this should be added to the FAQ
> also.
>
> Question:
> Can dd be used to do what I did with dump or tar?
> Smaller partition copied to a bigger partition.
>
> I'm willing to try and help out, but I'm going through
> both laptop and server hell at the moment.
>
> Thanks,
> Chris Bennett
>
>
Reply | Threaded
Open this post in threaded view
|

Re: FAQ's duplicating file systems, both methods fail to reproduce correctly

Philip Guenther-2
On Sun, 10 Dec 2017, vincent delft wrote:
> Did you tried pax ?
> some thing like: pax -rw -pe <source> <destination>
>
> I don't know if this the best tool, but I'm using it to duplicate a 1TB
> drive (having lot of hard links) onto an other one.
> I've done it couple of time, and I've do not see issues.

'pax' and 'tar' are actually the same binary so they have the same
limitation from the file formats that are supported, as well as any purely
internal limitations.  "pax -rw" actually has file format limitations by
design, so it doesn't automagically free you from those limitations.


> On Sun, Dec 10, 2017 at 7:03 PM, <[hidden email]> wrote:
...

> > OK. I've tried to use both methods and just don't
> > get true duplication.
> >
> > tar
> > It can't work with file and directory names
> > that are OK in filesystem, but too long for itself.
> > Quite a while back I lost a lot of unimportant files
> > and directories that had absolute paths too long.
> > Why is this happening with tar? Can this be fixed?
> > If not, I'd like to add a note about that to the FAQ.

tar/pax should have emitted warnings about such files when generating the
archive; if that didn't happen it's a bug and we should fix it.  
Depending on the exact failure you hit there may be ways to fix what you
hit.


> > dump
> > I had to move /usr/local to a bigger partition. growfs,
> > etc. I kept the /usr/local untouched and then dumped it
> > to the new partition, expecting a true duplication.
> > Nope.
> > It changed all of the program symlinks permissions.

You do know that the mode of a symlink has *no* effect on how the kernel
processes it, don't you?  As far as the kernel is concerned, you can do
the exact same operations on a mode 0 symlink as on a mode 777 symlink.


> > Why is dump doing this? Can this be fixed?

restore did that because (a) it didn't matter, and (b) there was no API to
modify the mode of a symlink (because it didn't matter).

An API that can chmod a symlink _was_ eventually added: fchmodat(2).  The
diff below makes restore preserve symlink mode.


Philip Guenther


Index: tape.c
===================================================================
RCS file: /data/src/openbsd/src/sbin/restore/tape.c,v
retrieving revision 1.49
diff -u -p -r1.49 tape.c
--- tape.c 21 Jan 2017 08:31:44 -0000 1.49
+++ tape.c 10 Dec 2017 21:33:10 -0000
@@ -564,6 +564,8 @@ extractfile(char *name)
  if (linkit(lnkbuf, name, SYMLINK) == FAIL)
  return (FAIL);
  (void)lchown(name, uid, gid);
+ (void)fchmodat(AT_FDCWD, name, mode,
+    AT_SYMLINK_NOFOLLOW);
  return (GOOD);
  }
 

Reply | Threaded
Open this post in threaded view
|

Re: FAQ's duplicating file systems, both methods fail to reproduce correctly

webmaster
In reply to this post by webmaster


>
> 'pax' and 'tar' are actually the same binary so they have the same
> limitation from the file formats that are supported, as well as any purely
> internal limitations.  "pax -rw" actually has file format limitations by
> design, so it doesn't automagically free you from those limitations.
>
>
> > On Sun, Dec 10, 2017 at 7:03 PM, <[hidden email]> wrote:
> ...
> > > OK. I've tried to use both methods and just don't
> > > get true duplication.
> > >
> > > tar
> > > It can't work with file and directory names
> > > that are OK in filesystem, but too long for itself.
> > > Quite a while back I lost a lot of unimportant files
> > > and directories that had absolute paths too long.
> > > Why is this happening with tar? Can this be fixed?
> > > If not, I'd like to add a note about that to the FAQ.
>
> tar/pax should have emitted warnings about such files when generating the
> archive; if that didn't happen it's a bug and we should fix it.  
> Depending on the exact failure you hit there may be ways to fix what you
> hit.

Yes, I got warnings, I was pulling all of the files off of five failing
hard drives. Luckily, the files were just some pr0n videos, but it could
have been really bad if the hard drive was on it's very last run.

>
> > > dump
> > > I had to move /usr/local to a bigger partition. growfs,
> > > etc. I kept the /usr/local untouched and then dumped it
> > > to the new partition, expecting a true duplication.
> > > Nope.
> > > It changed all of the program symlinks permissions.
>
> You do know that the mode of a symlink has *no* effect on how the kernel
> processes it, don't you?  As far as the kernel is concerned, you can do
> the exact same operations on a mode 0 symlink as on a mode 777 symlink.
>

No, I didn't know. I have had lots of problems when ownership changes
with
the symlinks, so I wrote I program to delete and restore them with the
proper owners.
Thanks for letting me know. I can delete the files I had left on the old
partition.

>
> > > Why is dump doing this? Can this be fixed?
>
> restore did that because (a) it didn't matter, and (b) there was no API to
> modify the mode of a symlink (because it didn't matter).
>
> An API that can chmod a symlink _was_ eventually added: fchmodat(2).  The
> diff below makes restore preserve symlink mode.
>

Thanks,
Chris Bennett


Reply | Threaded
Open this post in threaded view
|

Re: FAQ's duplicating file systems, both methods fail to reproduce correctly

Philip Guenther-2
On Sun, Dec 10, 2017 at 3:24 PM, <[hidden email]> wrote:
...

>
> > > > dump
> > > > I had to move /usr/local to a bigger partition. growfs,
> > > > etc. I kept the /usr/local untouched and then dumped it
> > > > to the new partition, expecting a true duplication.
> > > > Nope.
> > > > It changed all of the program symlinks permissions.
> >
> > You do know that the mode of a symlink has *no* effect on how the kernel
> > processes it, don't you?  As far as the kernel is concerned, you can do
> > the exact same operations on a mode 0 symlink as on a mode 777 symlink.
>
> No, I didn't know. I have had lots of problems when ownership changes
> with the symlinks, so I wrote I program to delete and restore them with the
> proper owners. Thanks for letting me know. I can delete the files I had
> left on the old
> partition.
>

Wait, you previously said your problem was with symlinks *permissions* but
now you're saying *ownership*!  I can confirm that restore(8) didn't
preserve the permissions (thus the patch I sent), but as long as you ran it
with sufficient privilege it should have always restored symlink
*ownership*.  Was that a slip of the tongue/fingers?

(Symlink ownership doesn't affect the ability to follow the symlink; it
only matters for directory-entry/inode level operations, such as deciding
who can remove a symlink from a mode 1777 directory like /tmp and who can
change the group or perms on the symlink.  AFAIK, the _group_ of a symlink
is never checked.)


Philip Guenther
Reply | Threaded
Open this post in threaded view
|

Re: FAQ's duplicating file systems, both methods fail to reproduce correctly

webmaster
In reply to this post by webmaster



>
> Wait, you previously said your problem was with symlinks *permissions* but
> now you're saying *ownership*!  I can confirm that restore(8) didn't
> preserve the permissions (thus the patch I sent), but as long as you ran it
> with sufficient privilege it should have always restored symlink
> *ownership*.  Was that a slip of the tongue/fingers?
>

Sorry, I was just blathering about a different unrelated problem I had
with
website symlinks. My bad.

Chris Bennett


Reply | Threaded
Open this post in threaded view
|

Re: FAQ's duplicating file systems, both methods fail to reproduce correctly

Philip Guenther-2
On Sun, Dec 10, 2017 at 3:46 PM, <[hidden email]> wrote:

>
> > Wait, you previously said your problem was with symlinks *permissions*
> but
> > now you're saying *ownership*!  I can confirm that restore(8) didn't
> > preserve the permissions (thus the patch I sent), but as long as you ran
> it
> > with sufficient privilege it should have always restored symlink
> > *ownership*.  Was that a slip of the tongue/fingers?
>
> Sorry, I was just blathering about a different unrelated problem I had with
> website symlinks. My bad.
>

Heh, no problem!
Reply | Threaded
Open this post in threaded view
|

Re: FAQ's duplicating file systems, both methods fail to reproduce correctly

Robert Paschedag
In reply to this post by vincent delft-2
Am 10. Dezember 2017 22:17:11 MEZ schrieb vincent delft <[hidden email]>:

>Hello,
>
>Did you tried pax ?
>some thing like: pax -rw -pe <source> <destination>
>
>I don't know if this the best tool, but I'm using it to duplicate a 1TB
>drive (having lot of hard links) onto an other one.
>I've done it couple of time, and I've do not see issues.
>
>rgds
>

Is "rsync" not an option?

>
>
>
>
>
>
>
>
>
>On Sun, Dec 10, 2017 at 7:03 PM, <[hidden email]>
>wrote:
>
>> Forgive problems with this email.
>> I saw how my emails showed up on marc.info
>> Scary. This is just temporary.
>>
>> OK. I've tried to use both methods and just don't
>> get true duplication.
>>
>> tar
>> It can't work with file and directory names
>> that are OK in filesystem, but too long for itself.
>> Quite a while back I lost a lot of unimportant files
>> and directories that had absolute paths too long.
>> Why is this happening with tar? Can this be fixed?
>> If not, I'd like to add a note about that to the FAQ.
>>
>> dump
>> I had to move /usr/local to a bigger partition. growfs,
>> etc. I kept the /usr/local untouched and then dumped it
>> to the new partition, expecting a true duplication.
>> Nope.
>> It changed all of the program symlinks permissions.
>> Why is dump doing this? Can this be fixed?
>> Otherwise, a note about this should be added to the FAQ
>> also.
>>
>> Question:
>> Can dd be used to do what I did with dump or tar?
>> Smaller partition copied to a bigger partition.
>>
>> I'm willing to try and help out, but I'm going through
>> both laptop and server hell at the moment.
>>
>> Thanks,
>> Chris Bennett
>>
>>

Reply | Threaded
Open this post in threaded view
|

Re: FAQ's duplicating file systems, both methods fail to reproduce correctly

vincent delft-2
In reply to this post by Philip Guenther-2
On Sun, Dec 10, 2017 at 10:39 PM, Philip Guenther <[hidden email]>
wrote:


> 'pax' and 'tar' are actually the same binary so they have the same
> limitation from the file formats that are supported, as well as any purely
> internal limitations.  "pax -rw" actually has file format limitations by
> design, so it doesn't automagically free you from those limitations.
>
>
By Looking a tar man page, I do not find comments regarding "length"
constraints :(.
On pax, it stated that with the default archive format "ustar", filenames
must be 100 char max and path names must be 256 char max.

But It's not clear which archive format the command Tar is using. Is it
using the "old tar" format or the new "ustar" format as described in the
pax man pages ?


Amazing for me to see that both are the same binary

t420:~$ ls -ali /bin/tar
32778 -r-xr-xr-x  3 root  bin  433488 Oct  4 05:13 /bin/tar
t420:~$ ls -ali /bin/pax
32778 -r-xr-xr-x  3 root  bin  433488 Oct  4 05:13 /bin/pax
Reply | Threaded
Open this post in threaded view
|

Re: FAQ's duplicating file systems, both methods fail to reproduce correctly

x9p-2
In reply to this post by Robert Paschedag

On Mon, December 11, 2017 4:28 am, Robert Paschedag wrote:
>>
>
> Is "rsync" not an option?
>

+1 for rsync. never had a problem.

cheers.

--

x9p | PGP : 0x03B50AF5EA4C8D80 / 5135 92C1 AD36 5293 2BDF  DDCC 0DFA 74AE
1524 E7EE

Reply | Threaded
Open this post in threaded view
|

Re: FAQ's duplicating file systems, both methods fail to reproduce correctly

Steve Williams-16
In reply to this post by webmaster
Hi,

cpio has always been my "go to" for file system duplication because it
will re-create device nodes.

Cheers,
Steve Williams


On 10/12/2017 11:03 AM, [hidden email] wrote:

> Forgive problems with this email.
> I saw how my emails showed up on marc.info
> Scary. This is just temporary.
>
> OK. I've tried to use both methods and just don't
> get true duplication.
>
> tar
> It can't work with file and directory names
> that are OK in filesystem, but too long for itself.
> Quite a while back I lost a lot of unimportant files
> and directories that had absolute paths too long.
> Why is this happening with tar? Can this be fixed?
> If not, I'd like to add a note about that to the FAQ.
>
> dump
> I had to move /usr/local to a bigger partition. growfs,
> etc. I kept the /usr/local untouched and then dumped it
> to the new partition, expecting a true duplication.
> Nope.
> It changed all of the program symlinks permissions.
> Why is dump doing this? Can this be fixed?
> Otherwise, a note about this should be added to the FAQ
> also.
>
> Question:
> Can dd be used to do what I did with dump or tar?
> Smaller partition copied to a bigger partition.
>
> I'm willing to try and help out, but I'm going through
> both laptop and server hell at the moment.
>
> Thanks,
> Chris Bennett

Reply | Threaded
Open this post in threaded view
|

Re: FAQ's duplicating file systems, both methods fail to reproduce correctly

Otto Moerbeek
On Mon, Dec 11, 2017 at 08:30:54AM -0700, Steve Williams wrote:

> Hi,
>
> cpio has always been my "go to" for file system duplication because it will
> re-create device nodes.

Both pax and tar do that as well.

        -Otto

>
> Cheers,
> Steve Williams
>
>
> On 10/12/2017 11:03 AM, [hidden email] wrote:
> > Forgive problems with this email.
> > I saw how my emails showed up on marc.info
> > Scary. This is just temporary.
> >
> > OK. I've tried to use both methods and just don't
> > get true duplication.
> >
> > tar
> > It can't work with file and directory names
> > that are OK in filesystem, but too long for itself.
> > Quite a while back I lost a lot of unimportant files
> > and directories that had absolute paths too long.
> > Why is this happening with tar? Can this be fixed?
> > If not, I'd like to add a note about that to the FAQ.
> >
> > dump
> > I had to move /usr/local to a bigger partition. growfs,
> > etc. I kept the /usr/local untouched and then dumped it
> > to the new partition, expecting a true duplication.
> > Nope.
> > It changed all of the program symlinks permissions.
> > Why is dump doing this? Can this be fixed?
> > Otherwise, a note about this should be added to the FAQ
> > also.
> >
> > Question:
> > Can dd be used to do what I did with dump or tar?
> > Smaller partition copied to a bigger partition.
> >
> > I'm willing to try and help out, but I'm going through
> > both laptop and server hell at the moment.
> >
> > Thanks,
> > Chris Bennett

Reply | Threaded
Open this post in threaded view
|

Re: FAQ's duplicating file systems, both methods fail to reproduce correctly

Philip Guenther-2
On Mon, Dec 11, 2017 at 9:16 AM, Otto Moerbeek <[hidden email]> wrote:

> On Mon, Dec 11, 2017 at 08:30:54AM -0700, Steve Williams wrote:
> > cpio has always been my "go to" for file system duplication because it
> will
> > re-create device nodes.
>
> Both pax and tar do that as well.
>

Come on, you still remember using tar back in the 90's when it didn't
support devices, paths were 100 bytes _total_, and they didn't include
user/group names (only UID/GID), right?  Good times!

Philip
Reply | Threaded
Open this post in threaded view
|

Re: FAQ's duplicating file systems, both methods fail to reproduce correctly

Steve Williams-16


On 11/12/2017 12:27 PM, Philip Guenther wrote:

> On Mon, Dec 11, 2017 at 9:16 AM, Otto Moerbeek <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     On Mon, Dec 11, 2017 at 08:30:54AM -0700, Steve Williams wrote:
>     > cpio has always been my "go to" for file system duplication
>     because it will
>     > re-create device nodes.
>
>     Both pax and tar do that as well.
>
> Come on, you still remember using tar back in the 90's when it didn't
> support devices, paths were 100 bytes _total_, and they didn't include
> user/group names (only UID/GID), right?  Good times!
>
> Philip
>
Yes, my habits were born of SCO Xenix, IBM's AIX for the RT PC, etc.  
Old habits die hard!!!  lol

Cheers,
Steve W.