Disk accesses freeze for a lot of seconds

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Disk accesses freeze for a lot of seconds

Federico Giannici
We have an OpenBSD 5.2 amd64 where every 5 minutes a few thousand of
".rrd" files from MRTG are written (actually, updated) to disk.

The problem is that for a few seconds (15-20) every other access to the
disk is totally blocked. So during those 15-20 seconds the access to the
graphs is freezed! And this is really annoying for a graphs server...

It's not a problem of CPU load (it's a quadruple core AMD Athlon II X4
630 Processor). Processes run smoothly, they freeze only when they try
to access the disk. Disk is a normal SATA, and the partition is FFS with
softdep.

Is there anything (some system tuning?) I can do to get rid of the
freezes, or at least to mitigate them?

Thanks.

Reply | Threaded
Open this post in threaded view
|

Re: Disk accesses freeze for a lot of seconds

Stefan Sperling-8
On Sun, Jan 06, 2013 at 12:22:44PM +0100, Federico Giannici wrote:

> We have an OpenBSD 5.2 amd64 where every 5 minutes a few thousand of
> ".rrd" files from MRTG are written (actually, updated) to disk.
>
> The problem is that for a few seconds (15-20) every other access to
> the disk is totally blocked. So during those 15-20 seconds the
> access to the graphs is freezed! And this is really annoying for a
> graphs server...
>
> It's not a problem of CPU load (it's a quadruple core AMD Athlon II
> X4 630 Processor). Processes run smoothly, they freeze only when
> they try to access the disk. Disk is a normal SATA, and the
> partition is FFS with softdep.

It's probably the bug in the buffer cache where the kernel would allow
userland to queue up so many writes that eventually the kernel is starved
out of buffers. Everything else (for example, read operations on your
graph files) then sleeps until enough writes have been spilled out to disk.

> Is there anything (some system tuning?) I can do to get rid of the
> freezes, or at least to mitigate them?

The best solution is an upgrade to -current where this has been fixed.
See http://marc.info/?l=openbsd-cvs&m=135231065926430&w=2 and other
related commits by Bob Beck.

If you'd rather stick to 5.2 you can try turning off softdep. softdep delays
some write operations so turning it off might help somewhat by allowing more
read operations to interleave with write operations. While the bug was
affecting -current I found that my systems where much more responsive with
softdep turned off.

Reply | Threaded
Open this post in threaded view
|

Re: Disk accesses freeze for a lot of seconds

Federico Giannici
You was right: turning off softdep made the freezes much shorter.

Thanks.


On 01/06/13 13:49, Stefan Sperling wrote:

> On Sun, Jan 06, 2013 at 12:22:44PM +0100, Federico Giannici wrote:
>> We have an OpenBSD 5.2 amd64 where every 5 minutes a few thousand of
>> ".rrd" files from MRTG are written (actually, updated) to disk.
>>
>> The problem is that for a few seconds (15-20) every other access to
>> the disk is totally blocked. So during those 15-20 seconds the
>> access to the graphs is freezed! And this is really annoying for a
>> graphs server...
>>
>> It's not a problem of CPU load (it's a quadruple core AMD Athlon II
>> X4 630 Processor). Processes run smoothly, they freeze only when
>> they try to access the disk. Disk is a normal SATA, and the
>> partition is FFS with softdep.
>
> It's probably the bug in the buffer cache where the kernel would allow
> userland to queue up so many writes that eventually the kernel is starved
> out of buffers. Everything else (for example, read operations on your
> graph files) then sleeps until enough writes have been spilled out to disk.
>
>> Is there anything (some system tuning?) I can do to get rid of the
>> freezes, or at least to mitigate them?
>
> The best solution is an upgrade to -current where this has been fixed.
> See http://marc.info/?l=openbsd-cvs&m=135231065926430&w=2 and other
> related commits by Bob Beck.
>
> If you'd rather stick to 5.2 you can try turning off softdep. softdep delays
> some write operations so turning it off might help somewhat by allowing more
> read operations to interleave with write operations. While the bug was
> affecting -current I found that my systems where much more responsive with
> softdep turned off.

Reply | Threaded
Open this post in threaded view
|

Re: Disk accesses freeze for a lot of seconds

Loïc Blot-2
I got same problem with squid when squid exit normally (/etc/rc.d/squid
stop), when mass squid disk cache is written, there is a one min freeze
on the server. (OpenBSD 5.2). The problem was also here under OpenBSD
5.1. CPU is also OK (10% of a big xeon quad). But for me softdeps aren't
activated.
The temporary solution i used, kill -9 squid process when stop/restart
is done.

--
Cordialement,
Loïc BLOT, UNIX systems, security and network expert
http://www.unix-experience.fr 

Le dimanche 06 janvier 2013 à 16:08 +0100, Federico Giannici a écrit :

> You was right: turning off softdep made the freezes much shorter.
>
> Thanks.
>
>
> On 01/06/13 13:49, Stefan Sperling wrote:
> > On Sun, Jan 06, 2013 at 12:22:44PM +0100, Federico Giannici wrote:
> >> We have an OpenBSD 5.2 amd64 where every 5 minutes a few thousand of
> >> ".rrd" files from MRTG are written (actually, updated) to disk.
> >>
> >> The problem is that for a few seconds (15-20) every other access to
> >> the disk is totally blocked. So during those 15-20 seconds the
> >> access to the graphs is freezed! And this is really annoying for a
> >> graphs server...
> >>
> >> It's not a problem of CPU load (it's a quadruple core AMD Athlon II
> >> X4 630 Processor). Processes run smoothly, they freeze only when
> >> they try to access the disk. Disk is a normal SATA, and the
> >> partition is FFS with softdep.
> >
> > It's probably the bug in the buffer cache where the kernel would allow
> > userland to queue up so many writes that eventually the kernel is starved
> > out of buffers. Everything else (for example, read operations on your
> > graph files) then sleeps until enough writes have been spilled out to disk.
> >
> >> Is there anything (some system tuning?) I can do to get rid of the
> >> freezes, or at least to mitigate them?
> >
> > The best solution is an upgrade to -current where this has been fixed.
> > See http://marc.info/?l=openbsd-cvs&m=135231065926430&w=2 and other
> > related commits by Bob Beck.
> >
> > If you'd rather stick to 5.2 you can try turning off softdep. softdep delays
> > some write operations so turning it off might help somewhat by allowing more
> > read operations to interleave with write operations. While the bug was
> > affecting -current I found that my systems where much more responsive with
> > softdep turned off.

Reply | Threaded
Open this post in threaded view
|

Re: Disk accesses freeze for a lot of seconds

David Diggles-2
Maybe the following will help.

See "Tuning for More"
http://wiki.squid-cache.org/BestOsForSquid

I use mount options: noatime and async.
I don't use softdep for squid cache either.

I found aufs worked best for storage scheme (in squid.conf).

I am curious. Anyone out there using diskd?

On Sun, Jan 06, 2013 at 07:49:27PM +0100, Lo?c BLOT wrote:

> I got same problem with squid when squid exit normally (/etc/rc.d/squid
> stop), when mass squid disk cache is written, there is a one min freeze
> on the server. (OpenBSD 5.2). The problem was also here under OpenBSD
> 5.1. CPU is also OK (10% of a big xeon quad). But for me softdeps aren't
> activated.
> The temporary solution i used, kill -9 squid process when stop/restart
> is done.
>
> --
> Cordialement,
> Lo??c BLOT, UNIX systems, security and network expert
> http://www.unix-experience.fr 
>
> Le dimanche 06 janvier 2013 ?? 16:08 +0100, Federico Giannici a ??crit :
>
> > You was right: turning off softdep made the freezes much shorter.
> >
> > Thanks.
> >
> >
> > On 01/06/13 13:49, Stefan Sperling wrote:
> > > On Sun, Jan 06, 2013 at 12:22:44PM +0100, Federico Giannici wrote:
> > >> We have an OpenBSD 5.2 amd64 where every 5 minutes a few thousand of
> > >> ".rrd" files from MRTG are written (actually, updated) to disk.
> > >>
> > >> The problem is that for a few seconds (15-20) every other access to
> > >> the disk is totally blocked. So during those 15-20 seconds the
> > >> access to the graphs is freezed! And this is really annoying for a
> > >> graphs server...
> > >>
> > >> It's not a problem of CPU load (it's a quadruple core AMD Athlon II
> > >> X4 630 Processor). Processes run smoothly, they freeze only when
> > >> they try to access the disk. Disk is a normal SATA, and the
> > >> partition is FFS with softdep.
> > >
> > > It's probably the bug in the buffer cache where the kernel would allow
> > > userland to queue up so many writes that eventually the kernel is starved
> > > out of buffers. Everything else (for example, read operations on your
> > > graph files) then sleeps until enough writes have been spilled out to disk.
> > >
> > >> Is there anything (some system tuning?) I can do to get rid of the
> > >> freezes, or at least to mitigate them?
> > >
> > > The best solution is an upgrade to -current where this has been fixed.
> > > See http://marc.info/?l=openbsd-cvs&m=135231065926430&w=2 and other
> > > related commits by Bob Beck.
> > >
> > > If you'd rather stick to 5.2 you can try turning off softdep. softdep delays
> > > some write operations so turning it off might help somewhat by allowing more
> > > read operations to interleave with write operations. While the bug was
> > > affecting -current I found that my systems where much more responsive with
> > > softdep turned off.

Reply | Threaded
Open this post in threaded view
|

Re: Disk accesses freeze for a lot of seconds

Amit Kulkarni-5
> Maybe the following will help.
>
> See "Tuning for More"
> http://wiki.squid-cache.org/BestOsForSquid
>
> I use mount options: noatime and async.
> I don't use softdep for squid cache either.

that is not good policy. you are asking for trouble.

Reply | Threaded
Open this post in threaded view
|

Re: Disk accesses freeze for a lot of seconds

David Diggles-2
> > Maybe the following will help.
> >
> > See "Tuning for More"
> > http://wiki.squid-cache.org/BestOsForSquid
> >
> > I use mount options: noatime and async.
> > I don't use softdep for squid cache either.
>
> that is not good policy. you are asking for trouble.

Thanks for the opinion.

Yeah I read the disclaimer about async in mount(8) and
don't mind taking the risk.

As for noatime.  Are you kidding me?

I forgot "tuning" => "idiot" to some on this list.

.d.d.

Reply | Threaded
Open this post in threaded view
|

Re: Disk accesses freeze for a lot of seconds

William Ahern-2
On Mon, Jan 07, 2013 at 12:53:01PM +1000, David Diggles wrote:

> > > Maybe the following will help.
> > >
> > > See "Tuning for More"
> > > http://wiki.squid-cache.org/BestOsForSquid
> > >
> > > I use mount options: noatime and async.
> > > I don't use softdep for squid cache either.
> >
> > that is not good policy. you are asking for trouble.
>
> Thanks for the opinion.
>
> Yeah I read the disclaimer about async in mount(8) and
> don't mind taking the risk.
>
> As for noatime.  Are you kidding me?
>

I briefly looked into adding relatime, but I stopped as soon as I realized
there was no more room for additional flags in the existing data structures.
I don't have the kernel experience to deal with such headaches
properly--that is, without it being excessively ugly or screwing people down
the road.

Having relatime would be awesome, though.