dd conv=fsync

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

dd conv=fsync

Alexander Bluhm
Hi,

GNU dd has the conv=fsync feature which does an fsync(2) after final
write to output.  I find this useful for write performance measurement
through the file system without buffer cache optimization.  Others
may like the reliable storage guarantee of fsync(2).

Do we want dd conv=fsync in OpenBSD?

bluhm

Index: bin/dd/args.c
===================================================================
RCS file: /data/mirror/openbsd/cvs/src/bin/dd/args.c,v
retrieving revision 1.30
diff -u -p -r1.30 args.c
--- bin/dd/args.c 25 Jul 2018 15:09:48 -0000 1.30
+++ bin/dd/args.c 14 Feb 2019 15:05:18 -0000
@@ -274,6 +274,7 @@ static const struct conv {
  { "ascii", C_ASCII, C_EBCDIC, e2a_POSIX },
  { "block", C_BLOCK, C_UNBLOCK, NULL },
  { "ebcdic", C_EBCDIC, C_ASCII, a2e_POSIX },
+ { "fsync", C_FSYNC, 0, NULL },
  { "ibm", C_EBCDIC, C_ASCII, a2ibm_POSIX },
  { "lcase", C_LCASE, C_UCASE, NULL },
  { "osync", C_OSYNC, C_BS, NULL },
Index: bin/dd/dd.1
===================================================================
RCS file: /data/mirror/openbsd/cvs/src/bin/dd/dd.1,v
retrieving revision 1.33
diff -u -p -r1.33 dd.1
--- bin/dd/dd.1 17 Aug 2016 21:23:01 -0000 1.33
+++ bin/dd/dd.1 14 Feb 2019 18:12:03 -0000
@@ -203,6 +203,8 @@ is a slightly different mapping, which i
 .At V
 .Cm ibm
 value.
+.It Cm fsync
+Write output to permanent storage device before finishing.
 .It Cm lcase
 Transform uppercase characters into lowercase characters.
 .It Cm noerror
Index: bin/dd/dd.c
===================================================================
RCS file: /data/mirror/openbsd/cvs/src/bin/dd/dd.c,v
retrieving revision 1.25
diff -u -p -r1.25 dd.c
--- bin/dd/dd.c 23 Jul 2018 23:09:37 -0000 1.25
+++ bin/dd/dd.c 14 Feb 2019 18:13:55 -0000
@@ -347,6 +347,10 @@ dd_close(void)
  }
  if (out.dbcnt)
  dd_out(1);
+ if (ddflags & C_FSYNC) {
+ if (fsync(out.fd) == -1)
+ err(1, "fsync %s", out.name);
+ }
 }
 
 void
Index: bin/dd/dd.h
===================================================================
RCS file: /data/mirror/openbsd/cvs/src/bin/dd/dd.h,v
retrieving revision 1.8
diff -u -p -r1.8 dd.h
--- bin/dd/dd.h 13 Aug 2017 02:06:42 -0000 1.8
+++ bin/dd/dd.h 14 Feb 2019 15:05:18 -0000
@@ -96,3 +96,4 @@ typedef struct {
 #define C_STATUS 0x200000
 #define C_NOXFER 0x400000
 #define C_NOINFO 0x800000
+#define C_FSYNC 0x1000000

Reply | Threaded
Open this post in threaded view
|

Re: dd conv=fsync

Ted Unangst-6
Alexander Bluhm wrote:
> GNU dd has the conv=fsync feature which does an fsync(2) after final
> write to output.  I find this useful for write performance measurement
> through the file system without buffer cache optimization.  Others
> may like the reliable storage guarantee of fsync(2).
>
> Do we want dd conv=fsync in OpenBSD?

What does gnu dd do if the output is stdout? Is it an error?

With this patch, I get dd: fsync stdout: Invalid argument
after the transfer is complete. Should there be an fstat check to make sure
it's a regular file?

> +.It Cm fsync
> +Write output to permanent storage device before finishing.

I know this wording is similar to fsync(2), but I think I'd prefer just a
reference. The output may not be permanent storage. What about

Call
.Xr fsync 2
on the output file before exiting.


Reply | Threaded
Open this post in threaded view
|

Re: dd conv=fsync

Brent Cook
On Thu, Feb 14, 2019 at 4:21 PM Ted Unangst <[hidden email]> wrote:

> Alexander Bluhm wrote:
> > GNU dd has the conv=fsync feature which does an fsync(2) after final
> > write to output.  I find this useful for write performance measurement
> > through the file system without buffer cache optimization.  Others
> > may like the reliable storage guarantee of fsync(2).
> >
> > Do we want dd conv=fsync in OpenBSD?
>
> What does gnu dd do if the output is stdout? Is it an error?
>

> With this patch, I get dd: fsync stdout: Invalid argument
>

gnu dd prints something similar to stderr, and returns exit code 1.

dd: fsync failed for 'standard output': Invalid argument
3+1 records in
3+1 records out
1784 bytes (1.8 kB, 1.7 KiB) copied, 9.4827e-05 s, 18.8 MB/s



> after the transfer is complete. Should there be an fstat check to make sure
> it's a regular file?
>
> > +.It Cm fsync
> > +Write output to permanent storage device before finishing.
>
> I know this wording is similar to fsync(2), but I think I'd prefer just a
> reference. The output may not be permanent storage. What about
>
> Call
> .Xr fsync 2
> on the output file before exiting.
>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: dd conv=fsync

Alexander Bluhm
In reply to this post by Ted Unangst-6
On Thu, Feb 14, 2019 at 05:20:13PM -0500, Ted Unangst wrote:
> What does gnu dd do if the output is stdout? Is it an error?

On Linux it fails with an error.

linux$ yes | dd count=4 bs=1 conv=fsync
y
y
dd: fsync failed for 'standard output': Invalid argument
4+0 records in
4+0 records out
4 bytes copied, 0.0004733 s, 8.5 kB/s
linux$ echo $?
1

On OpenBSD calling fsync(2) with a terminal works fine, but it fails
with a pipe.

openbsd$ yes | dd count=4 bs=1 conv=fsync      
y
y
4+0 records in
4+0 records out
4 bytes transferred in 0.001 secs (2400 bytes/sec)

openbsd$ yes | dd count=4 bs=1 conv=fsync | cat
dd: fsync stdout: Invalid argument
4+0 records in
4+0 records out
4 bytes transferred in 0.000 secs (4796 bytes/sec)
y
y

> Should there be an fstat check to make sure
> it's a regular file?

We should not implement the magic which device allows fsync(2) into
dd(1).  Just do what the user says, if it is nonsense, give him an
error.

> I know this wording is similar to fsync(2), but I think I'd prefer just a
> reference. The output may not be permanent storage. What about
>
> Call
> .Xr fsync 2
> on the output file before exiting.

Sure, new diff.

ok?

bluhm

Index: bin/dd/args.c
===================================================================
RCS file: /data/mirror/openbsd/cvs/src/bin/dd/args.c,v
retrieving revision 1.30
diff -u -p -r1.30 args.c
--- bin/dd/args.c 25 Jul 2018 15:09:48 -0000 1.30
+++ bin/dd/args.c 14 Feb 2019 15:05:18 -0000
@@ -274,6 +274,7 @@ static const struct conv {
  { "ascii", C_ASCII, C_EBCDIC, e2a_POSIX },
  { "block", C_BLOCK, C_UNBLOCK, NULL },
  { "ebcdic", C_EBCDIC, C_ASCII, a2e_POSIX },
+ { "fsync", C_FSYNC, 0, NULL },
  { "ibm", C_EBCDIC, C_ASCII, a2ibm_POSIX },
  { "lcase", C_LCASE, C_UCASE, NULL },
  { "osync", C_OSYNC, C_BS, NULL },
Index: bin/dd/dd.1
===================================================================
RCS file: /data/mirror/openbsd/cvs/src/bin/dd/dd.1,v
retrieving revision 1.33
diff -u -p -r1.33 dd.1
--- bin/dd/dd.1 17 Aug 2016 21:23:01 -0000 1.33
+++ bin/dd/dd.1 15 Feb 2019 12:17:29 -0000
@@ -203,6 +203,10 @@ is a slightly different mapping, which i
 .At V
 .Cm ibm
 value.
+.It Cm fsync
+Call
+.Xr fsync 2
+on the output file before exiting.
 .It Cm lcase
 Transform uppercase characters into lowercase characters.
 .It Cm noerror
Index: bin/dd/dd.c
===================================================================
RCS file: /data/mirror/openbsd/cvs/src/bin/dd/dd.c,v
retrieving revision 1.25
diff -u -p -r1.25 dd.c
--- bin/dd/dd.c 23 Jul 2018 23:09:37 -0000 1.25
+++ bin/dd/dd.c 14 Feb 2019 18:13:55 -0000
@@ -347,6 +347,10 @@ dd_close(void)
  }
  if (out.dbcnt)
  dd_out(1);
+ if (ddflags & C_FSYNC) {
+ if (fsync(out.fd) == -1)
+ err(1, "fsync %s", out.name);
+ }
 }
 
 void
Index: bin/dd/dd.h
===================================================================
RCS file: /data/mirror/openbsd/cvs/src/bin/dd/dd.h,v
retrieving revision 1.8
diff -u -p -r1.8 dd.h
--- bin/dd/dd.h 13 Aug 2017 02:06:42 -0000 1.8
+++ bin/dd/dd.h 14 Feb 2019 15:05:18 -0000
@@ -96,3 +96,4 @@ typedef struct {
 #define C_STATUS 0x200000
 #define C_NOXFER 0x400000
 #define C_NOINFO 0x800000
+#define C_FSYNC 0x1000000

Reply | Threaded
Open this post in threaded view
|

Re: dd conv=fsync

Klemens Nanni-2
On Fri, Feb 15, 2019 at 01:28:15PM +0100, Alexander Bluhm wrote:
> We should not implement the magic which device allows fsync(2) into
> dd(1).  Just do what the user says, if it is nonsense, give him an
> error.
Seems reasonable to me.

> > I know this wording is similar to fsync(2), but I think I'd prefer just a
> > reference. The output may not be permanent storage.
I agree with tedu.

OK kn

Reply | Threaded
Open this post in threaded view
|

Re: dd conv=fsync

Ted Unangst-6
In reply to this post by Alexander Bluhm
Alexander Bluhm wrote:
> We should not implement the magic which device allows fsync(2) into
> dd(1).  Just do what the user says, if it is nonsense, give him an
> error.

sure. ok.