What are the disadvantages of soft updates?

classic Classic list List threaded Threaded
30 messages Options
12
Reply | Threaded
Open this post in threaded view
|

What are the disadvantages of soft updates?

currellberry
Hello,

The FAQ[1] states that soft updates result in "a large performance
increase in disk writing performance," and links to a resource[2] which
claims that soft updates, in addition to being a performance
enhancement, "can also maintain better disk consistency."  Resource 2
links to several academic papers[3][4], which while they are a bit above
my level, contain discussions of how soft updates can increase
performance and speed recovery on crash.

My question is: what are the downsides of soft updates?  Also, does
journaling provide a better data-safety guarantee?

Thank you,
Currell Berry

[1]http://www.openbsd.org/faq/faq14.html#SoftUpdates
[2]http://www.mckusick.com/softdep/
[3]https://www.usenix.org/legacy/publications/library/proceedings/usenix99/mckusick.html
[4]https://www.usenix.org/legacy/publications/library/proceedings/usenix2000/general/seltzer.html

Reply | Threaded
Open this post in threaded view
|

Re: What are the disadvantages of soft updates?

Alexandre Ratchov-2
On Mon, Jan 19, 2015 at 03:59:34AM +0000, [hidden email] wrote:

> Hello,
>
> The FAQ[1] states that soft updates result in "a large performance increase
> in disk writing performance," and links to a resource[2] which claims that
> soft updates, in addition to being a performance enhancement, "can also
> maintain better disk consistency."  Resource 2 links to several academic
> papers[3][4], which while they are a bit above my level, contain discussions
> of how soft updates can increase performance and speed recovery on crash.
>
> My question is: what are the downsides of soft updates?

- softdep consumes more cpu in kernel mode, which hurts interactive
  programms on very slow machines. It has the reputation of
  consuming more memory.

- the softdep code is more complex (likely to have more bugs).

> Also, does journaling provide a better data-safety guarantee?

They are not the same. On OpenBSD, softdep makes cerain operations
much faster while ensuring that upon power loss, all
inconsistencies can be automatically fixed by fsck on next boot.

Journaling would write data twice (first in the journal, then in
the filesystem) and would allow last operations to be replayed on
next boot, so no need to run fsck, which in turn makes system boot
fast after a power loss.

In theory, from data safety point of view they are equivalent.

Reply | Threaded
Open this post in threaded view
|

Re: What are the disadvantages of soft updates?

currellberry
In reply to this post by currellberry
I infer from your response that soft updates possess:

1. increased overhead over default FFS settings.
2. increased implementation complexity over default FFS settings.

Also, I infer that journaling and soft updates provide equivalent data
safety
guarantees "in theory." Do they provide equivalent guarantees in
practice?

Thank you,
Currell

------ Original Message ------
From: "Alexandre Ratchov" <[hidden email]>
To: [hidden email]
Cc: [hidden email]
Sent: 1/19/2015 4:44:59 AM
Subject: Re: What are the disadvantages of soft updates?

>On Mon, Jan 19, 2015 at 03:59:34AM +0000, [hidden email] wrote:
>>  Hello,
>>
>>  The FAQ[1] states that soft updates result in "a large performance
>>increase
>>  in disk writing performance," and links to a resource[2] which claims
>>that
>>  soft updates, in addition to being a performance enhancement, "can
>>also
>>  maintain better disk consistency." Resource 2 links to several
>>academic
>>  papers[3][4], which while they are a bit above my level, contain
>>discussions
>>  of how soft updates can increase performance and speed recovery on
>>crash.
>>
>>  My question is: what are the downsides of soft updates?
>
>- softdep consumes more cpu in kernel mode, which hurts interactive
>   programms on very slow machines. It has the reputation of
>   consuming more memory.
>
>- the softdep code is more complex (likely to have more bugs).
>
>>  Also, does journaling provide a better data-safety guarantee?
>
>They are not the same. On OpenBSD, softdep makes cerain operations
>much faster while ensuring that upon power loss, all
>inconsistencies can be automatically fixed by fsck on next boot.
>
>Journaling would write data twice (first in the journal, then in
>the filesystem) and would allow last operations to be replayed on
>next boot, so no need to run fsck, which in turn makes system boot
>fast after a power loss.
>
>In theory, from data safety point of view they are equivalent.

Reply | Threaded
Open this post in threaded view
|

Re: What are the disadvantages of soft updates?

Nick Holland
On 01/19/15 14:10, Currell Berry wrote:
> I infer from your response that soft updates possess:
>
> 1. increased overhead over default FFS settings.
> 2. increased implementation complexity over default FFS settings.

for a "he stated" definition of "you infer", sure.

> Also, I infer that journaling and soft updates provide equivalent data
> safety

um. I think we have a terms issue here with "data safety"...

> guarantees "in theory." Do they provide equivalent guarantees in
> practice?

Being there are many journaled file systems in Linux, if you wish to get
to real life, you will have to specify one.

But ...
Being that FFS+soft updates has been in development and production
longer than just about any currently used Linux file system ("of the
week" -- sorry, I just feel the urge to add those three words after
referencing Linux file systems), and almost all the BSD file system
works goes into FFS, rather than split up among lots of options, I'd put
my money (and data) on FFS+softupdates.  But that's me.  I tend to put
my money where my mouth is -- I have no UPSs in use, and if it would
take longer to login and halt a machine than to wait for an fsck, I just
wack the power button or yank the cord.

Keep in mind, what softupdates promises is /file system/ integrity.
Journaling does similar.  If the power goes out or the system crashes
mid-Big Data Write, the goal is to get the file system back into sane
shape so the system can come back up and resume its tasks, NOT that the
1.7TB of a 1.8TB write will be sitting on disk waiting for you, or that
your database is consistent.

It is entirely likely -- probable, in fact -- that you will find your
actively written file truncated to zero bytes.  Depending on your
application, this is probably a GOOD thing -- if you find a zero byte
file, that normally means something went wrong (or hasn't yet gone
right).  A 1.7TB file?  You have no idea if that's complete or not.

If you want true "data safety", you probably want some kind of
application transaction tracking BEYOND the file system.

Nick.

>
> Thank you,
> Currell
>
> ------ Original Message ------
> From: "Alexandre Ratchov" <[hidden email]>
> To: [hidden email]
> Cc: [hidden email]
> Sent: 1/19/2015 4:44:59 AM
> Subject: Re: What are the disadvantages of soft updates?
>
>>On Mon, Jan 19, 2015 at 03:59:34AM +0000, [hidden email] wrote:
>>>  Hello,
>>>
>>>  The FAQ[1] states that soft updates result in "a large performance
>>>increase
>>>  in disk writing performance," and links to a resource[2] which claims
>>>that
>>>  soft updates, in addition to being a performance enhancement, "can
>>>also
>>>  maintain better disk consistency." Resource 2 links to several
>>>academic
>>>  papers[3][4], which while they are a bit above my level, contain
>>>discussions
>>>  of how soft updates can increase performance and speed recovery on
>>>crash.
>>>
>>>  My question is: what are the downsides of soft updates?
>>
>>- softdep consumes more cpu in kernel mode, which hurts interactive
>>   programms on very slow machines. It has the reputation of
>>   consuming more memory.
>>
>>- the softdep code is more complex (likely to have more bugs).
>>
>>>  Also, does journaling provide a better data-safety guarantee?
>>
>>They are not the same. On OpenBSD, softdep makes cerain operations
>>much faster while ensuring that upon power loss, all
>>inconsistencies can be automatically fixed by fsck on next boot.
>>
>>Journaling would write data twice (first in the journal, then in
>>the filesystem) and would allow last operations to be replayed on
>>next boot, so no need to run fsck, which in turn makes system boot
>>fast after a power loss.
>>
>>In theory, from data safety point of view they are equivalent.

Reply | Threaded
Open this post in threaded view
|

Re: What are the disadvantages of soft updates?

currellberry
Thank you for your answer.  That clarifies things for me.

w.r.t a couple of points:
I did make an inference.  Alexander stated several points, and
I used deduction to summarize his statements.

1. (Increased CPU)&(Increased Memory)->(Increased Overhead).
2. (I will grant that here I restated what he said using synonyms).

To your second point, which argues that journaling/soft updates do not
affect "data safety," I would respond as follows.

-- Metadata is a form of data.
-- Filesystem corruption can also cause application data loss.

Therefore data-safety is an applicable term, but probably not the most
precise term.

I was mainly curious as to why soft updates were not enabled by default
if they have so many good qualities.  Your answers explained this well.

--Currell

------ Original Message ------
From: "Nick Holland" <[hidden email]>
To: [hidden email]
Sent: 1/19/2015 4:25:51 PM
Subject: Re: What are the disadvantages of soft updates?

>On 01/19/15 14:10, Currell Berry wrote:
>>  I infer from your response that soft updates possess:
>>
>>  1. increased overhead over default FFS settings.
>>  2. increased implementation complexity over default FFS settings.
>
>for a "he stated" definition of "you infer", sure.
>
>>  Also, I infer that journaling and soft updates provide equivalent
>>data
>>  safety
>
>um. I think we have a terms issue here with "data safety"...
>
>>  guarantees "in theory." Do they provide equivalent guarantees in
>>  practice?
>
>Being there are many journaled file systems in Linux, if you wish to
>get
>to real life, you will have to specify one.
>
>But ...
>Being that FFS+soft updates has been in development and production
>longer than just about any currently used Linux file system ("of the
>week" -- sorry, I just feel the urge to add those three words after
>referencing Linux file systems), and almost all the BSD file system
>works goes into FFS, rather than split up among lots of options, I'd
>put
>my money (and data) on FFS+softupdates. But that's me. I tend to put
>my money where my mouth is -- I have no UPSs in use, and if it would
>take longer to login and halt a machine than to wait for an fsck, I
>just
>wack the power button or yank the cord.
>
>Keep in mind, what softupdates promises is /file system/ integrity.
>Journaling does similar. If the power goes out or the system crashes
>mid-Big Data Write, the goal is to get the file system back into sane
>shape so the system can come back up and resume its tasks, NOT that the
>1.7TB of a 1.8TB write will be sitting on disk waiting for you, or that
>your database is consistent.
>
>It is entirely likely -- probable, in fact -- that you will find your
>actively written file truncated to zero bytes. Depending on your
>application, this is probably a GOOD thing -- if you find a zero byte
>file, that normally means something went wrong (or hasn't yet gone
>right). A 1.7TB file? You have no idea if that's complete or not.
>
>If you want true "data safety", you probably want some kind of
>application transaction tracking BEYOND the file system.
>
>Nick.
>
>>
>>  Thank you,
>>  Currell
>>
>>  ------ Original Message ------
>>  From: "Alexandre Ratchov" <[hidden email]>
>>  To: [hidden email]
>>  Cc: [hidden email]
>>  Sent: 1/19/2015 4:44:59 AM
>>  Subject: Re: What are the disadvantages of soft updates?
>>
>>>On Mon, Jan 19, 2015 at 03:59:34AM +0000, [hidden email]
>>>wrote:
>>>>   Hello,
>>>>
>>>>   The FAQ[1] states that soft updates result in "a large performance
>>>>increase
>>>>   in disk writing performance," and links to a resource[2] which
>>>>claims
>>>>that
>>>>   soft updates, in addition to being a performance enhancement, "can
>>>>also
>>>>   maintain better disk consistency." Resource 2 links to several
>>>>academic
>>>>   papers[3][4], which while they are a bit above my level, contain
>>>>discussions
>>>>   of how soft updates can increase performance and speed recovery on
>>>>crash.
>>>>
>>>>   My question is: what are the downsides of soft updates?
>>>
>>>- softdep consumes more cpu in kernel mode, which hurts interactive
>>>    programms on very slow machines. It has the reputation of
>>>    consuming more memory.
>>>
>>>- the softdep code is more complex (likely to have more bugs).
>>>
>>>>   Also, does journaling provide a better data-safety guarantee?
>>>
>>>They are not the same. On OpenBSD, softdep makes cerain operations
>>>much faster while ensuring that upon power loss, all
>>>inconsistencies can be automatically fixed by fsck on next boot.
>>>
>>>Journaling would write data twice (first in the journal, then in
>>>the filesystem) and would allow last operations to be replayed on
>>>next boot, so no need to run fsck, which in turn makes system boot
>>>fast after a power loss.
>>>
>>>In theory, from data safety point of view they are equivalent.

Reply | Threaded
Open this post in threaded view
|

Re: What are the disadvantages of soft updates?

Janne Johansson-3
2015-01-20 1:46 GMT+01:00 Currell Berry <[hidden email]>:

> I was mainly curious as to why soft updates were not enabled by default
> if they have so many good qualities.  Your answers explained this well.
>

At least sun4c but also other memory starved machines (mostly those who do
not have
a lot of kernel memory mapped) will crash a lot if it is enabled by default.


--
May the most significant bit of your life be positive.

Reply | Threaded
Open this post in threaded view
|

Re: What are the disadvantages of soft updates?

Alexandre Ratchov-2
In reply to this post by currellberry
On Mon, Jan 19, 2015 at 07:10:53PM +0000, Currell Berry wrote:
> I infer from your response that soft updates possess:
>
> 1. increased overhead over default FFS settings.
> 2. increased implementation complexity over default FFS settings.
>
> Also, I infer that journaling and soft updates provide equivalent data
> safety
> guarantees "in theory." Do they provide equivalent guarantees in practice?

in *my* practice, yes. I lost no single file last 10 years despite
the frequent system crashes during kernel development &
experimenting.

Reply | Threaded
Open this post in threaded view
|

Re: What are the disadvantages of soft updates?

frantisek holop
Alexandre Ratchov, 20 Jan 2015 10:17:
> in *my* practice, yes. I lost no single file last 10 years despite
> the frequent system crashes during kernel development &
> experimenting.

very nice, i dont doubt that.

but in my experience it is not that hard to get a
corrupted filesystem with softupdates and i had to stop
using it.  but i seem to attract panics and
page faults.

in the recent past i had corrupted filesystems even
without softupdates, up to a point that nowadays i
mount -o sync,noatime.

i have a toshiba ssd, so it actually feels like having
softupdates on :)

when your hardware (and its drivers) are solid,
i am all for it though.  i just dont have that hw.

-f
--
day-by-day a day goes by.

Reply | Threaded
Open this post in threaded view
|

Re: What are the disadvantages of soft updates?

Mihai Popescu-3
In reply to this post by currellberry
> but in my experience it is not that hard to get a
> corrupted filesystem with softupdates and i had to stop
> using it.  but i seem to attract panics and
> page faults.

> in the recent past i had corrupted filesystems even
> without softupdates, up to a point that nowadays i
> mount -o sync,noatime.

> i have a toshiba ssd, so it actually feels like having
> softupdates on :)

> when your hardware (and its drivers) are solid,
> i am all for it though.  i just dont have that hw.

I might be quick on judgement or even mess with thread's topic, but
did you reported that problem anywere close to openbsd project lists?
SSD is a very used hardware under OpenBSD, it is not so exotic. I
think that are developers who can take a look at your problem if you
report it.

After watching this thread, I enabled softdep on all FFS partitions
thinking that Firefox will speed up a bit. I will see the results in
time.

Reply | Threaded
Open this post in threaded view
|

Re: What are the disadvantages of soft updates?

Nick Holland
On 01/21/15 07:34, Mihai Popescu wrote:
...
> After watching this thread, I enabled softdep on all FFS partitions
> thinking that Firefox will speed up a bit. I will see the results in
> time.

Keep in mind what softdeps do -- they improve performance of disk
writes.  They do nothing for disk reads.  Write a few small files, you
won't see much.  Write lots of files, you will see a huge difference.

Firefox is just plain slow at everything it seems, but writes are not
the major bottleneck, so you may not see much gain.

If you want to see a huge difference in softdep performance, unpack a
tar file with a lot of small files, such as the ports or source files.
No stopwatch will be needed to see the difference.

Nick.

Reply | Threaded
Open this post in threaded view
|

Re: What are the disadvantages of soft updates?

frantisek holop
In reply to this post by Mihai Popescu-3
Mihai Popescu, 21 Jan 2015 14:34:

> > but in my experience it is not that hard to get a
> > corrupted filesystem with softupdates and i had to stop
> > using it.  but i seem to attract panics and
> > page faults.
>
> > in the recent past i had corrupted filesystems even
> > without softupdates, up to a point that nowadays i
> > mount -o sync,noatime.
>
> > i have a toshiba ssd, so it actually feels like having
> > softupdates on :)
>
> > when your hardware (and its drivers) are solid,
> > i am all for it though.  i just dont have that hw.
>
> I might be quick on judgement or even mess with thread's topic, but
> did you reported that problem anywere close to openbsd project lists?
> SSD is a very used hardware under OpenBSD, it is not so exotic. I
> think that are developers who can take a look at your problem if you
> report it.

i meant hardware in general, not the disk.
sometimes laptops (esp. the cheaper ones)
contain questionable devices...

i saw panics with ath mainly.

-f
--
words are not food, though sometimes we must eat them.

Reply | Threaded
Open this post in threaded view
|

Re: What are the disadvantages of soft updates?

Steven Shockley
In reply to this post by frantisek holop
On 1/21/2015 5:50 AM, frantisek holop wrote:
> but in my experience it is not that hard to get a
> corrupted filesystem with softupdates and i had to stop
> using it.  but i seem to attract panics and
> page faults.

I've personally had problems with OpenBSD panics with softupdates when
running under ESXi when the back-end storage becomes high-latency
(aggressive SAN backups, not enough spindles).  I haven't tried recently
(it was difficult to repro on demand) but I didn't really consider it an
OpenBSD issue.  Presumably softupdate has shorter timeouts.

Reply | Threaded
Open this post in threaded view
|

Re: What are the disadvantages of soft updates?

Reyk Floeter-2
On Thu, Jan 22, 2015 at 09:02:51AM -0500, Steve Shockley wrote:

> On 1/21/2015 5:50 AM, frantisek holop wrote:
> >but in my experience it is not that hard to get a
> >corrupted filesystem with softupdates and i had to stop
> >using it.  but i seem to attract panics and
> >page faults.
>
> I've personally had problems with OpenBSD panics with softupdates when
> running under ESXi when the back-end storage becomes high-latency
> (aggressive SAN backups, not enough spindles).  I haven't tried recently (it
> was difficult to repro on demand) but I didn't really consider it an OpenBSD
> issue.  Presumably softupdate has shorter timeouts.
>

What release and what virtualized SCSI controller where you using?

Reyk

Reply | Threaded
Open this post in threaded view
|

Re: What are the disadvantages of soft updates?

Predrag Punosevac-2
In reply to this post by currellberry
I was following this discussion with the great interest but without
intend to participate in it until today.

Namely one of my OpenBSD servers (5.6 sparc64) runs Mollify and last
night I received an e-mail from an angry user who could not upload files
(the upload will fail or upload the file with file size zero). After
running df I noticed that /tmp was 100% full (4GB used) but the size of
individual files was only 12Kb. I thought for a second and I remember
seeing this with HAMMER on DF. Long story short I checked /etc/fstab and
sure enough I had rw,softdep next to each partition including tmp. I
removed softdep rebooted the sytem and /tmp usage dropped to 0%. More
importantly users could upload files again.

My questions is which partitions should be mounted with softdep option?
Is there a way to prune metadata which will save me for problems like
the one I encountered last night.

Best,
Predrag

Reply | Threaded
Open this post in threaded view
|

Re: What are the disadvantages of soft updates?

Ingo Schwarze
Hi Predrag,

Predrag Punosevac wrote on Fri, Jan 23, 2015 at 03:24:00PM -0500:

> I was following this discussion with the great interest but without
> intend to participate in it until today.
>
> Namely one of my OpenBSD servers (5.6 sparc64) runs Mollify and last
> night I received an e-mail from an angry user who could not upload files
> (the upload will fail or upload the file with file size zero). After
> running df I noticed that /tmp was 100% full (4GB used) but the size of
> individual files was only 12Kb.

That is unlikely to be due to softdep.  The usual reason for a file
system to be actually full and seemingly almost empty at the same
time is somebody doing the following sequence of operations:

 - open(2) a file for writing
 - writing a lot of data until the file system is full
 - unlink(2) the file
 - keep the process running that open(2)'ed it
 - let that process keep the file open and *not* close(2) it

Specifically, in /tmp, anybody can do that.

> I thought for a second and I remember seeing this with HAMMER on DF.

The above works with almost any file system.

> Long story short I checked /etc/fstab and
> sure enough I had rw,softdep next to each partition including tmp. I
> removed softdep rebooted the sytem and /tmp usage dropped to 0%.

That's not likely to be related to softdep either.  Chances are
just rebooting would have solved the issue as well - simply because
rebooting terminates all running processes, and consequently closes
all open files.

What you should have done instead was run fstat(1), look for processes
having files open in /tmp, use ls(1) -iRa /tmp to find the inode
numbers of linked files in /tmp, and kill the processes having files
open that were *not* linked until you found the one having the big
file open - and then have a friendly talk with the responsible user,
if any, or figure out what went wrong in case some daemon process
caused the issue.

> My questions is which partitions should be mounted with softdep
> option?

I'm not an expert on that and hardly ever use softdep, but i'd say
on file systems where file create/delete performance is *critically*
important, performace has been clearly demonstrated to be insufficient
without softdep, and you consider data loss harmless.

> Is there a way to prune metadata which will save me for problems like
> the one I encountered last night.

I'm not convinced that's a good question to ask.

Yours,
  Ingo

Reply | Threaded
Open this post in threaded view
|

Re: What are the disadvantages of soft updates?

Amit Kulkarni-5
In reply to this post by Predrag Punosevac-2
On Fri, Jan 23, 2015 at 2:24 PM, Predrag Punosevac <[hidden email]>
wrote:

> I was following this discussion with the great interest but without
> intend to participate in it until today.
>
> Namely one of my OpenBSD servers (5.6 sparc64) runs Mollify and last
> night I received an e-mail from an angry user who could not upload files
> (the upload will fail or upload the file with file size zero). After
> running df I noticed that /tmp was 100% full (4GB used) but the size of
> individual files was only 12Kb. I thought for a second and I remember
> seeing this with HAMMER on DF. Long story short I checked /etc/fstab and
> sure enough I had rw,softdep next to each partition including tmp. I
> removed softdep rebooted the sytem and /tmp usage dropped to 0%. More
> importantly users could upload files again.
>

Unless your server is heavily loaded softdep will sync and write to disk
within 30 secs to few minutes. IMHO, your /tmp being 4GB is unrelated to
softdep. What is the underlying issue why /tmp fills so much?

Reply | Threaded
Open this post in threaded view
|

Re: What are the disadvantages of soft updates?

Jorge Gabriel Lopez Paramount
In reply to this post by Predrag Punosevac-2
Quoting Predrag Punosevac <[hidden email]>:

> I was following this discussion with the great interest but without
> intend to participate in it until today.
>
> Namely one of my OpenBSD servers (5.6 sparc64) runs Mollify and last
> night I received an e-mail from an angry user who could not upload files
> (the upload will fail or upload the file with file size zero). After
> running df I noticed that /tmp was 100% full (4GB used) but the size of
> individual files was only 12Kb. I thought for a second and I remember
> seeing this with HAMMER on DF. Long story short I checked /etc/fstab and
> sure enough I had rw,softdep next to each partition including tmp. I
> removed softdep rebooted the sytem and /tmp usage dropped to 0%. More
> importantly users could upload files again.

Two things: UNIX servers like OpenBSD usually clean /tmp every reboot:

$ ls -la /tmp
total 20
drwxrwxrwt   5 root  wheel  512 Jan 23 15:00 .
drwxr-xr-x  16 root  wheel  512 Jan 23 14:58 ..
drwxrwxrwt   2 root  wheel  512 Jan 23 15:00 .ICE-unix
drwxrwxrwt   2 root  wheel  512 Jan 23 15:00 .X11-unix
drwxr-xr-x   2 root  wheel  512 Jan 23 15:00 aucat
$ uptime
  3:00PM  up 1 min, 1 user, load averages: 1.11, 0.41, 0.16

And one thing is space available and other different but related is  
inodes available:

$ df -i /tmp
Filesystem  512-blocks      Used     Avail Capacity iused   ifree  
%iused  Mounted on
/dev/wd0a      1920764    126340   1698388     7%    2439  127479     2%   /

If you have lots of small files you might have plenty of space  
available, but will be unable to create more files if there are no  
inodes available.

--
Best regards,
Jorge Lopez.


----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.

Reply | Threaded
Open this post in threaded view
|

Re: What are the disadvantages of soft updates?

Amit Kulkarni-5
In reply to this post by Ingo Schwarze
On Fri, Jan 23, 2015 at 2:53 PM, Ingo Schwarze <[hidden email]> wrote:

> Hi Predrag,
>
> Predrag Punosevac wrote on Fri, Jan 23, 2015 at 03:24:00PM -0500:
>
> > I was following this discussion with the great interest but without
> > intend to participate in it until today.
> >
> > Namely one of my OpenBSD servers (5.6 sparc64) runs Mollify and last
> > night I received an e-mail from an angry user who could not upload files
> > (the upload will fail or upload the file with file size zero). After
> > running df I noticed that /tmp was 100% full (4GB used) but the size of
> > individual files was only 12Kb.
>
> That is unlikely to be due to softdep.  The usual reason for a file
> system to be actually full and seemingly almost empty at the same
> time is somebody doing the following sequence of operations:
>
>  - open(2) a file for writing
>  - writing a lot of data until the file system is full
>  - unlink(2) the file
>  - keep the process running that open(2)'ed it
>  - let that process keep the file open and *not* close(2) it
>
> Specifically, in /tmp, anybody can do that.
>
> > I thought for a second and I remember seeing this with HAMMER on DF.
>
> The above works with almost any file system.
>
> > Long story short I checked /etc/fstab and
> > sure enough I had rw,softdep next to each partition including tmp. I
> > removed softdep rebooted the sytem and /tmp usage dropped to 0%.
>
> That's not likely to be related to softdep either.  Chances are
> just rebooting would have solved the issue as well - simply because
> rebooting terminates all running processes, and consequently closes
> all open files.
>

One more point to add to Ingo's detailed and very helpful reply. Reboot
actually clears /tmp.


>
> What you should have done instead was run fstat(1), look for processes
> having files open in /tmp, use ls(1) -iRa /tmp to find the inode
> numbers of linked files in /tmp, and kill the processes having files
> open that were *not* linked until you found the one having the big
> file open - and then have a friendly talk with the responsible user,
> if any, or figure out what went wrong in case some daemon process
> caused the issue.
>
> > My questions is which partitions should be mounted with softdep
> > option?
>
> I'm not an expert on that and hardly ever use softdep, but i'd say
> on file systems where file create/delete performance is *critically*
> important, performace has been clearly demonstrated to be insufficient
> without softdep, and you consider data loss harmless.
>
> > Is there a way to prune metadata which will save me for problems like
> > the one I encountered last night.
>
> I'm not convinced that's a good question to ask.
>
> Yours,
>   Ingo

Reply | Threaded
Open this post in threaded view
|

Re: What are the disadvantages of soft updates?

Ingo Schwarze
Hi,

Amit Kulkarni wrote on Fri, Jan 23, 2015 at 03:05:03PM -0600:

> One more point to add to Ingo's detailed and very helpful reply.
> Reboot actually clears /tmp.

The reason i didn't mention that is that it definitely doesn't
have anything to do with Predrag's problem, which was that /tmp
was full even though it contained almost no filename entries.

Rebooting clears /tmp in the sense that it deletes some filenames.
That's irrelevant here because Predrag reports that there were
no filenames pointing to large files in the first place.

Yours,
  Ingo

Reply | Threaded
Open this post in threaded view
|

Re: What are the disadvantages of soft updates?

Predrag Punosevac-2
In reply to this post by Ingo Schwarze
Ingo Schwarze <[hidden email]> wrote:

> Hi Predrag,
>

Hi Ingo,

> Predrag Punosevac wrote on Fri, Jan 23, 2015 at 03:24:00PM -0500:
>
> > I was following this discussion with the great interest but without
> > intend to participate in it until today.
> >
> > Namely one of my OpenBSD servers (5.6 sparc64) runs Mollify and last
> > night I received an e-mail from an angry user who could not upload files
> > (the upload will fail or upload the file with file size zero). After
> > running df I noticed that /tmp was 100% full (4GB used) but the size of
> > individual files was only 12Kb.
>
> That is unlikely to be due to softdep.  The usual reason for a file
> system to be actually full and seemingly almost empty at the same
> time is somebody doing the following sequence of operations:
>

Now it seems obvious that I made a mistake and put the blame on metadata
instead of running fstat but at the moment I had a mindset of DF user.
Namely couple of months ago I had a rogue process which keeps altering
log files on one of DF machines. Long story short after couple of hours
my /var had only about 1GB of log files but HAMMER history almost
completely filled the rest of /var due to the frequent changes of log
files. I regained the space by clearing HAMMER history and learned how
to tune HAMMER retention parameters.

It seems from what you are saying and from what seems clear to me before
last night incident that I made a fundamental mistake of thinking of
soft updates as honest journaling file system while in reality they are
just a way of maintaining file system meta-data integrity. Hence softdep
are safe from HAMMER like history retention problems.

Most Kind Regards,
Predrag

P.S. Since you are developer I can't resist not to ask if anybody is
looking at Bitrig code which brings WAPBL essentially into OpenBSD?


>  - open(2) a file for writing
>  - writing a lot of data until the file system is full
>  - unlink(2) the file
>  - keep the process running that open(2)'ed it
>  - let that process keep the file open and *not* close(2) it
>
> Specifically, in /tmp, anybody can do that.
>
> > I thought for a second and I remember seeing this with HAMMER on DF.
>
> The above works with almost any file system.
>
> > Long story short I checked /etc/fstab and
> > sure enough I had rw,softdep next to each partition including tmp. I
> > removed softdep rebooted the sytem and /tmp usage dropped to 0%.
>
> That's not likely to be related to softdep either.  Chances are
> just rebooting would have solved the issue as well - simply because
> rebooting terminates all running processes, and consequently closes
> all open files.
>
> What you should have done instead was run fstat(1), look for processes
> having files open in /tmp, use ls(1) -iRa /tmp to find the inode
> numbers of linked files in /tmp, and kill the processes having files
> open that were *not* linked until you found the one having the big
> file open - and then have a friendly talk with the responsible user,
> if any, or figure out what went wrong in case some daemon process
> caused the issue.
>
> > My questions is which partitions should be mounted with softdep
> > option?
>
> I'm not an expert on that and hardly ever use softdep, but i'd say
> on file systems where file create/delete performance is *critically*
> important, performace has been clearly demonstrated to be insufficient
> without softdep, and you consider data loss harmless.
>
> > Is there a way to prune metadata which will save me for problems like
> > the one I encountered last night.
>
> I'm not convinced that's a good question to ask.
>
> Yours,
>   Ingo

12