Odd /tmp behavior

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Odd /tmp behavior

Raymond, David
On an AMD-64 workstation /tmp fills up to 105% according to df,
apparently as a result of UNIX pipes in a shell script passing a whole
lot of moderately big files. Examination of /tmp with du and ls -gal
on /tmp shows no big files and trying to delete everything that is
there has no effect.  Rebooting cleans out /tmp.

I had /tmp mounted with the standard options + softdep.  I eliminated
softdep and the problem appears to have gone away.

Any ideas on what is going on with softdep here?  Dmesg shows a long
series of "/tmp file system full" messages.

Dave Raymond

--
David J. Raymond
[hidden email]
http://physics.nmt.edu/~raymond

Reply | Threaded
Open this post in threaded view
|

Re: Odd /tmp behavior

sven falempin
On Tue, Jan 7, 2020 at 12:18 PM Raymond, David <[hidden email]>
wrote:

> On an AMD-64 workstation /tmp fills up to 105% according to df,
> apparently as a result of UNIX pipes in a shell script passing a whole
> lot of moderately big files. Examination of /tmp with du and ls -gal
> on /tmp shows no big files and trying to delete everything that is
> there has no effect.  Rebooting cleans out /tmp.
>
> I had /tmp mounted with the standard options + softdep.  I eliminated
> softdep and the problem appears to have gone away.
>
> Any ideas on what is going on with softdep here?  Dmesg shows a long
> series of "/tmp file system full" messages.
>
> Dave Raymond
>
> --
> David J. Raymond
> [hidden email]
> http://physics.nmt.edu/~raymond
>
> man fstat

--
--
---------------------------------------------------------------------------------------------------------------------
Knowing is not enough; we must apply. Willing is not enough; we must do
Reply | Threaded
Open this post in threaded view
|

Re: Odd /tmp behavior

Jordan Geoghegan-3
In reply to this post by Raymond, David


On 2020-01-07 09:16, Raymond, David wrote:

> On an AMD-64 workstation /tmp fills up to 105% according to df,
> apparently as a result of UNIX pipes in a shell script passing a whole
> lot of moderately big files. Examination of /tmp with du and ls -gal
> on /tmp shows no big files and trying to delete everything that is
> there has no effect.  Rebooting cleans out /tmp.
>
> I had /tmp mounted with the standard options + softdep.  I eliminated
> softdep and the problem appears to have gone away.
>
> Any ideas on what is going on with softdep here?  Dmesg shows a long
> series of "/tmp file system full" messages.
>
> Dave Raymond
>

Using softdep on /tmp is a silly idea. I see way too many people who
don't understand how softdep works.

Reply | Threaded
Open this post in threaded view
|

Re: Odd /tmp behavior

Karel Gardas


On 1/7/20 7:38 PM, Jordan Geoghegan wrote:
>  > Using softdep on /tmp is a silly idea. >
Why? To naive eyes it may look like a natural solution: e.g. before temp
file is even created (on drive), it may be deleted which means there is
no meta-data change hence speedup of operation on /tmp. In case of
classical ffs, you will need to create file (sync meta-data update),
save some data (async), delete file (sync meta-data update). But
honestly still need to read the code...

Reply | Threaded
Open this post in threaded view
|

Re: Odd /tmp behavior

Jordan Geoghegan-3


On 2020-01-07 11:06, Karel Gardas wrote:

>
>
> On 1/7/20 7:38 PM, Jordan Geoghegan wrote:
>>  > Using softdep on /tmp is a silly idea. >
> Why? To naive eyes it may look like a natural solution: e.g. before
> temp file is even created (on drive), it may be deleted which means
> there is no meta-data change hence speedup of operation on /tmp. In
> case of classical ffs, you will need to create file (sync meta-data
> update), save some data (async), delete file (sync meta-data update).
> But honestly still need to read the code...
>
softdep is very complex, and it is by no means perfect or bug free. A
good rule of thumb is to only use softdep on larger partitions that will
see a large number of writes (particularly smaller and/or random writes).

Softdep can chew up a fair amount of kernel memory as well, and in
certain cases changes can take over a minute to actually make their way
on to disk. If softdep was the magic bullet that some people think it
is, it would be enabled by default.

Reply | Threaded
Open this post in threaded view
|

Re: Odd /tmp behavior

Juan Francisco Cantero Hurtado
In reply to this post by Raymond, David
On Tue, Jan 07, 2020 at 10:16:22AM -0700, Raymond, David wrote:

> On an AMD-64 workstation /tmp fills up to 105% according to df,
> apparently as a result of UNIX pipes in a shell script passing a whole
> lot of moderately big files. Examination of /tmp with du and ls -gal
> on /tmp shows no big files and trying to delete everything that is
> there has no effect.  Rebooting cleans out /tmp.
>
> I had /tmp mounted with the standard options + softdep.  I eliminated
> softdep and the problem appears to have gone away.
>
> Any ideas on what is going on with softdep here?  Dmesg shows a long
> series of "/tmp file system full" messages.

If you're using current and that started to happens in the last week or
so, then maybe there is some bug somewhere in the softdep code. Some
devs are reworking some parts of softdep.

If the problem is not new or you're using -stable, then maybe the
problem is that you're using a too small /tmp partition. Softdep delays
the writing of metadata. Maybe you're writing and deleting too much data
without giving softdep a chance to update the metadata on the disk.

Giving more space to /tmp should fix the problem. Even if you're not
going to use so much space, softdep will need the extra space between
the metadata updates.

BTW, I prefer to use "async" for /tmp.


--
Juan Francisco Cantero Hurtado http://juanfra.info

Reply | Threaded
Open this post in threaded view
|

Re: Odd /tmp behavior

Nick Holland
In reply to this post by Karel Gardas
On 2020-01-07 14:06, Karel Gardas wrote:

>
>
> On 1/7/20 7:38 PM, Jordan Geoghegan wrote:
>>  > Using softdep on /tmp is a silly idea. >
> Why? To naive eyes it may look like a natural solution: e.g. before temp
> file is even created (on drive), it may be deleted which means there is
> no meta-data change hence speedup of operation on /tmp. In case of
> classical ffs, you will need to create file (sync meta-data update),
> save some data (async), delete file (sync meta-data update). But
> honestly still need to read the code...
>

I'm not going to go nearly as far as to say it's a silly idea (as I
do it myself) but ... be aware softdep is funky.  Weird stuff happens
when Softdeps are working as designed.

When you do things out of order, things happen...well, out of order.
So ...
  create file
  delete file
  create file
  delete file
  create file
  delete file
  create file
  delete file
  create file
  delete file
sounds perfectly safe, as long as "file" is smaller than available
disk space, right?  Softdeps...no so much.  This can actually result
in running out of disk space, as the deletes may not happen until
after the creates.  

Another place where softdeps will sometimes bite you is when you
unpack tar balls that overwrite existing files -- simple thought
process says, "as long as you have enough space to cover the growth,
fine".  Softdeps might surprise you.  You may get an "out of disk
space" error, and a minute later, see much more space than you
thought you could ever need to accomplish the task, once the deletions
have time to take effect.

So ... make sure you have lots of extra disk space...if things are
snug, it's a bad place to use softdeps.

Nick.

Reply | Threaded
Open this post in threaded view
|

Re: Odd /tmp behavior

Jan Stary
On Jan 08 08:31:26, [hidden email] wrote:
> Another place where softdeps will sometimes bite you is when you
> unpack tar balls that overwrite existing files -- simple thought
> process says, "as long as you have enough space to cover the growth,
> fine".  Softdeps might surprise you.  You may get an "out of disk
> space" error, and a minute later, see much more space than you
> thought you could ever need to accomplish the task, once the deletions
> have time to take effect.

Another case of the same (imho)
is pkg_add -u when /usr/local is tight.

Reply | Threaded
Open this post in threaded view
|

Re: Odd /tmp behavior

Christian Weisgerber
In reply to this post by Nick Holland
On 2020-01-08, Nick Holland <[hidden email]> wrote:

> Weird stuff happens when Softdeps are working as designed.

To put it simply: Meta-data writes are delayed.

--
Christian "naddy" Weisgerber                          [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Odd /tmp behavior

Raymond, David
In reply to this post by Jan Stary
Thanks for the input on softdep.  I read a lot on the pros and cons.
This certainly pushes me in the "con" direction.

I forgot to mention that I am using 6.6 stable, not current, so the
latest updates to softdep shouldn't be an issue.

Dave Raymond

On 1/8/20, Jan Stary <[hidden email]> wrote:

> On Jan 08 08:31:26, [hidden email] wrote:
>> Another place where softdeps will sometimes bite you is when you
>> unpack tar balls that overwrite existing files -- simple thought
>> process says, "as long as you have enough space to cover the growth,
>> fine".  Softdeps might surprise you.  You may get an "out of disk
>> space" error, and a minute later, see much more space than you
>> thought you could ever need to accomplish the task, once the deletions
>> have time to take effect.
>
> Another case of the same (imho)
> is pkg_add -u when /usr/local is tight.
>
>


--
David J. Raymond
[hidden email]
http://physics.nmt.edu/~raymond