speedup shutdown

classic Classic list List threaded Threaded
17 messages Options
Reply | Threaded
Open this post in threaded view
|

speedup shutdown

quartz-2
I have a machine where tapping the front panel power button correctly
halts and powers off the machine.... however there's a solid 10 second
delay after I press the button before anything happens. Is there any way
to speed this process up?

Reply | Threaded
Open this post in threaded view
|

Re: speedup shutdown

Matthew Martin-2
On Sep 20, 2015 3:12 PM, "Quartz" <[hidden email]> wrote:
>
> I have a machine where tapping the front panel power button correctly
halts and powers off the machine.... however there's a solid 10 second
delay after I press the button before anything happens. Is there any way to
speed this process up?
>
Hold the button down.

Reply | Threaded
Open this post in threaded view
|

Re: speedup shutdown

Philip Guenther-2
On Sun, Sep 20, 2015 at 1:45 PM, Matthew Martin <[hidden email]> wrote:
> On Sep 20, 2015 3:12 PM, "Quartz" <[hidden email]> wrote:
>>
>> I have a machine where tapping the front panel power button correctly
> halts and powers off the machine.... however there's a solid 10 second
> delay after I press the button before anything happens. Is there any way to
> speed this process up?
>
> Hold the button down.

Heh, hard power-off, resulting in unclean filesystems probably isn't
what the original poster meant.

For power off via button, init runs "sh /etc/rc shutdown", then sends
all processes a SIGHUP, then waits 5 seconds.  If there are any
processes still alive it'll send SIGTERM and wait another 5 seconds.
If any are still alive at that point it'll send'em all SIGKILL and
wait another 5 seconds.  It'll then tell the kernel to halt the
system.

So, slow /etc/rc.d/* script delaying the /etc/rc shutdown step?  Or do
you have some daemon which isn't killed by its rc.d script, nor by
SIGHUP, thus requiring SIGTERM and at least 10 seconds?


Philip Guenther

Reply | Threaded
Open this post in threaded view
|

Re: speedup shutdown

Philip Guenther-2
Oops, I misdescribed init's waiting behavior:

On Sun, Sep 20, 2015 at 5:59 PM, Philip Guenther <[hidden email]> wrote:
...
> For power off via button, init runs "sh /etc/rc shutdown", then sends
> all processes a SIGHUP, then waits 5 seconds.  If there are any
> processes still alive it'll send SIGTERM and wait another 5 seconds.
> If any are still alive at that point it'll send'em all SIGKILL and
> wait another 5 seconds.  It'll then tell the kernel to halt the
> system.

It'll wait *up to* 5 seconds in each case: it'll continue immediately
to the kernel as soon as all processes are dead, so a 10 second wait
means slow /etc/rc.d script(s) or processes that won't die from SIGHUP
or SIGTERM, which is rude of them...

Philip Guenther

Reply | Threaded
Open this post in threaded view
|

Re: speedup shutdown

quartz-2
In reply to this post by Philip Guenther-2
> So, slow /etc/rc.d/* script delaying the /etc/rc shutdown step?  Or do
> you have some daemon which isn't killed by its rc.d script, nor by
> SIGHUP, thus requiring SIGTERM and at least 10 seconds?

This is a test system and it's pretty stock right now. Aside from the
standard services like pf and ntp the only installed pkg is I think
dnsmasq. It's possible there's something wrong there but I'm not sure
where I should start looking.

Reply | Threaded
Open this post in threaded view
|

Re: speedup shutdown

quartz-2
In reply to this post by Philip Guenther-2
> For power off via button, init runs "sh /etc/rc shutdown", then sends
> all processes a SIGHUP, then waits 5 seconds.  If there are any
> processes still alive it'll send SIGTERM and wait another 5 seconds.
> If any are still alive at that point it'll send'em all SIGKILL and
> wait another 5 seconds.  It'll then tell the kernel to halt the
> system.

Is there a way to watch this process as it's happening to see where the
holdup is? Watching it in general wouldn't be a bad idea. I guess a
large part of the issue is not so much that it takes 10 seconds, but
that there's no confirmation or indication that it's actually doing
anything. It just sits there like it ignored you and you can continue
typing at the command line. There's no output or anything until the
"syncing disks" line finally pops up.

Reply | Threaded
Open this post in threaded view
|

Re: speedup shutdown

Philip Guenther-2
In reply to this post by quartz-2
On Sun, Sep 20, 2015 at 6:12 PM, Quartz <[hidden email]> wrote:
>> So, slow /etc/rc.d/* script delaying the /etc/rc shutdown step?  Or do
>> you have some daemon which isn't killed by its rc.d script, nor by
>> SIGHUP, thus requiring SIGTERM and at least 10 seconds?
>
> This is a test system and it's pretty stock right now. Aside from the
> standard services like pf and ntp the only installed pkg is I think dnsmasq.
> It's possible there's something wrong there but I'm not sure where I should
> start looking.

Hmm?   How about replicate the process and observe the results?  "time
sh /etc/rc shutdown".  See what's still running.  kill -HUP everything
except init and your session and see what's still running 5 seconds
later.  Then again with kill -TERM.  Whatever still standing is
slowing you down; for each one figure out whether and when it should
have died.

Philip Guenther

Reply | Threaded
Open this post in threaded view
|

Re: speedup shutdown

quartz-2
> Hmm?   How about replicate the process and observe the results?

Well, I wasn't sure if that was the exact/entire process or just a summary.


"time
> sh /etc/rc shutdown".  See what's still running.  kill -HUP everything
> except init and your session and see what's still running 5 seconds
> later.

OK I'll try that, thanks.

Reply | Threaded
Open this post in threaded view
|

Re: speedup shutdown

quartz-2
>> Hmm? How about replicate the process and observe the results?
>
> Well, I wasn't sure if that was the exact/entire process or just a summary.
>
>
> "time
>> sh /etc/rc shutdown". See what's still running. kill -HUP everything
>> except init and your session and see what's still running 5 seconds
>> later.
>
> OK I'll try that, thanks.

I'm missing something.

Logged in as root, 'sh /etc/rc shutdown' returns instantly and according
'ps' everything's still running. Trying to then kill -HUP half the
processes doesn't work (they just restart).

Reply | Threaded
Open this post in threaded view
|

Re: speedup shutdown

Philip Guenther-2
On Sun, Sep 20, 2015 at 7:24 PM, Quartz <[hidden email]> wrote:
...
>> "time sh /etc/rc shutdown". See what's still running. kill -HUP everything
>>> except init and your session and see what's still running 5 seconds
>>> later.

Hmm, you truncated the suggested steps...


>> OK I'll try that, thanks.
>
>
> I'm missing something.
>
> Logged in as root, 'sh /etc/rc shutdown' returns instantly and according
> 'ps' everything's still running.

Okay, so it's not some pkg_script that's being slow.  That's good.


> Trying to then kill -HUP half the processes
> doesn't work (they just restart).

Saying "doesn't work" when they're behaving as documented on their
manpages is a bit odd.  Which of those survive after you do the step
that you truncated from the suggestion?  Checking one of my systems,
the answer is "NONE"...

(Why does init do HUP then TERM?  By default, HUP will terminate
logins and most user processes, getting most interactive processes out
of the way while logging and other base and network services are still
operating, so interactive processes can cleanly terminate.  TERM
should then kill the rest, leaving only processes having some sort of
problem (e.g., unresponsive NFS server) or that are badly behaved.)


Philip Guenther

Reply | Threaded
Open this post in threaded view
|

Re: speedup shutdown

quartz-2
>>> "time sh /etc/rc shutdown". See what's still running. kill -HUP everything
>>>> except init and your session and see what's still running 5 seconds
>>>> later.
>
> Hmm, you truncated the suggested steps...

You wrote:

"Hmm?   How about replicate the process and observe the results?  "time
sh /etc/rc shutdown".  See what's still running.  kill -HUP everything
except init and your session and see what's still running 5 seconds
later.  Then again with kill -TERM.  Whatever still standing is
slowing you down; for each one figure out whether and when it should
have died."

I took that to mean:

1) run (presumably as root) 'time sh /etc/rc shutdown'
2) check 'ps -aux' to see what's still running
3) 'kill -HUP [PID]' for each of the remaining processes
4) check 'ps -aux' again
5) 'kill -TERM [PID]' for each of the remaining processes
6) check 'ps -aux' again


I appear to be hung up near the beginning. 'sh /etc/rc shutdown' doesn't
appear to do anything, since it returns instantly and the ps output from
(2) is identical to ps output from before 'sh /etc/rc shutdown'. (3)
"doesn't work" in the sense that it doesn't appear to actually stop
[m]any services (presumably because I didn't do something correctly
before this point).

Like I said, I'm missing something. There were a couple assumptions in
there somewhere that I'm not picking up on. What exactly am I supposed
to do in what order?

Reply | Threaded
Open this post in threaded view
|

Re: speedup shutdown

Philip Guenther-2
On Sun, Sep 20, 2015 at 8:10 PM, Quartz <[hidden email]> wrote:

>>>> "time sh /etc/rc shutdown". See what's still running. kill -HUP
>>>> everything
>>>>>
>>>>> except init and your session and see what's still running 5 seconds
>>>>> later.
>>
>>
>> Hmm, you truncated the suggested steps...
>
>
> You wrote:
>
> "Hmm?   How about replicate the process and observe the results?  "time
> sh /etc/rc shutdown".  See what's still running.  kill -HUP everything
> except init and your session and see what's still running 5 seconds
> later.  Then again with kill -TERM.  Whatever still standing is
> slowing you down; for each one figure out whether and when it should
> have died."
>
> I took that to mean:
>
> 1) run (presumably as root) 'time sh /etc/rc shutdown'
> 2) check 'ps -aux' to see what's still running
> 3) 'kill -HUP [PID]' for each of the remaining processes
> 4) check 'ps -aux' again
> 5) 'kill -TERM [PID]' for each of the remaining processes
> 6) check 'ps -aux' again

Yes.  Perhaps it isn't clear that I would *expect* stuff to still be
running at step 4, and thus for shutdown like this to take at least 5
seconds.

You said shutdown was taking "a solid 10 seconds", so I was under the
belief that some process was hanging around *past* step 6, and
requiring a SIGKILL to be taken down, but I guess we don't have the
data to conclude that.


> I appear to be hung up near the beginning. 'sh /etc/rc shutdown' doesn't
> appear to do anything, since it returns instantly and the ps output from (2)
> is identical to ps output from before 'sh /etc/rc shutdown'. (3) "doesn't
> work" in the sense that it doesn't appear to actually stop [m]any services
> (presumably because I didn't do something correctly before this point).

It sounds like things are behaving as expected for steps 1-4: shut
down packaged services (none, or they're quick), then shut down
interactive sessions, giving them 5 seconds to do so.

If the next step, the one you didn't describe the results of, killing
daemons with SIGTERM, leaves services running then there may be a bug
in those programs worth fixing.  Or maybe everything dies after the
SIGTERM and it's the kernel's shut down work, flushing stuff to disk
and such that is rounding out the 10 seconds.


Philip Guenther

Reply | Threaded
Open this post in threaded view
|

Re: speedup shutdown

quartz-2
>> I took that to mean:
>>
>> 1) run (presumably as root) 'time sh /etc/rc shutdown'
>> 2) check 'ps -aux' to see what's still running
>> 3) 'kill -HUP [PID]' for each of the remaining processes
>> 4) check 'ps -aux' again
>> 5) 'kill -TERM [PID]' for each of the remaining processes
>> 6) check 'ps -aux' again
>
> Yes.  Perhaps it isn't clear that I would *expect* stuff to still be
> running at step 4, and thus for shutdown like this to take at least 5
> seconds.

> If the next step, the one you didn't describe the results of, killing
> daemons with SIGTERM,

OK, maybe this is where the communication gap is. Sending HUP to sshd
and syslogd and everything was effectively a no-op since they'd all just
immediately restart. I looped between (3) and (4) for a bit then gave
up. I assumed I was doing something wrong when by this point the state
of the system was identical to (0).

Just to be doubly clear, is it expected behavior that at (4) everything
will still be running?

(In the mean time, I'll try continuing on through (6) anyway and see
what happens).

Reply | Threaded
Open this post in threaded view
|

Re: speedup shutdown

Peter Hessler
On 2015 Sep 21 (Mon) at 09:37:11 -0400 (-0400), Quartz wrote:
:>>I took that to mean:
:>>
:>>1) run (presumably as root) 'time sh /etc/rc shutdown'
:>>2) check 'ps -aux' to see what's still running
:>>3) 'kill -HUP [PID]' for each of the remaining processes
:>>4) check 'ps -aux' again
:>>5) 'kill -TERM [PID]' for each of the remaining processes
:>>6) check 'ps -aux' again
:>
:>Yes.  Perhaps it isn't clear that I would *expect* stuff to still be
:>running at step 4, and thus for shutdown like this to take at least 5
:>seconds.
:
:>If the next step, the one you didn't describe the results of, killing
:>daemons with SIGTERM,
:
:OK, maybe this is where the communication gap is. Sending HUP to sshd and
:syslogd and everything was effectively a no-op since they'd all just
:immediately restart. I looped between (3) and (4) for a bit then gave up. I
:assumed I was doing something wrong when by this point the state of the
:system was identical to (0).
:

The two daemons you refer to, treat SIGHUP as a "please re-read your
configuration files and restart".  This is semi-common.  This happens to
also be the two daemons you are testing this with, causing some confusino.

Almost all signals can be caught and the default behaviour is changed.
Check the man page for signal(3) for some more information.

:Just to be doubly clear, is it expected behavior that at (4) everything will
:still be running?
:

Not everything, but some things will still be running.


:(In the mean time, I'll try continuing on through (6) anyway and see what
:happens).
:

After running commands #1, #3 and #5; almost everything should be
killed.  Command #1 should take care of the vast majority of daemons
started at boot; #3 and #5 are to catch the ones that aren't.  


--
If you keep anything long enough, you can throw it away.

Reply | Threaded
Open this post in threaded view
|

Re: speedup shutdown

quartz-2
> The two daemons you refer to, treat SIGHUP as a "please re-read your
> configuration files and restart".  This is semi-common.  This happens to
> also be the two daemons you are testing this with, causing some confusino.

> Not everything, but some things will still be running.

It wasn't just syslogd and sshd, -HUP also doesn't shut down any of the
pflogd/dhclient/cron stuff either. The only process it actually stops is
sndiod, all the others restart on their own.


> After running commands #1, #3 and #5; almost everything should be
> killed.  Command #1 should take care of the vast majority of daemons
> started at boot; #3 and #5 are to catch the ones that aren't.

Well, -TERM stops every PID I typed in (the four I didn't being init,
two ksh's and ps itself), so I'm not sure where that leave me. I guess
it's some kind of timing thing or race condition?

Reply | Threaded
Open this post in threaded view
|

Re: speedup shutdown

quartz-2
>> The two daemons you refer to, treat SIGHUP as a "please re-read your
>> configuration files and restart". This is semi-common. This happens to
>> also be the two daemons you are testing this with, causing some
>> confusino.
>
>> Not everything, but some things will still be running.
>
> It wasn't just syslogd and sshd, -HUP also doesn't shut down any of the
> pflogd/dhclient/cron stuff either. The only process it actually stops is
> sndiod, all the others restart on their own.
>
>
>> After running commands #1, #3 and #5; almost everything should be
>> killed. Command #1 should take care of the vast majority of daemons
>> started at boot; #3 and #5 are to catch the ones that aren't.
>
> Well, -TERM stops every PID I typed in (the four I didn't being init,
> two ksh's and ps itself), so I'm not sure where that leave me. I guess
> it's some kind of timing thing or race condition?

Also, FWIW, tapping the power button at this point yields a two second
delay before it does anything (down from the previous ten). Not sure if
that's useful information or not.

Reply | Threaded
Open this post in threaded view
|

Re: speedup shutdown

Joel Rees-2
In reply to this post by quartz-2
2015/09/22 3:21 "Quartz" <[hidden email]>:
>>
>> The two daemons you refer to, treat SIGHUP as a "please re-read your
>> configuration files and restart".  This is semi-common.  This happens to
>> also be the two daemons you are testing this with, causing some
confusino.
>
>> Not everything, but some things will still be running.
>
> It wasn't just syslogd and sshd, -HUP also doesn't shut down any of the
pflogd/dhclient/cron stuff either. The only process it actually stops is
sndiod, all the others restart on their own.
>
>
>> After running commands #1, #3 and #5; almost everything should be
>> killed.  Command #1 should take care of the vast majority of daemons
>> started at boot; #3 and #5 are to catch the ones that aren't.
>
> Well, -TERM stops every PID I typed in (the four I didn't being init, two
ksh's and ps itself), so I'm not sure where that leave me. I guess it's
some kind of timing thing or race condition?
>

I haven't tried this on openbsd, but I wrote a little tool for someone who
was fussing about debian taking too long to shut down:

http://joels-programming-fun.blogspot.jp/2014/08/this-is-demonstration-of-way-to.html

You'll want to tune some of it, probably, may not need to grep, may want to
change the timing. Just remember, writing to a file at shutdown will
interfere with the shutdown, especially if you use timing too fast to
finish one log entry before the next one starts. And you may want to
deliberately kill the process before the shutdown process does the final
sync.

And don't forget to remove things before you put the thing into production.

Joel Rees

Computer memory is just fancy paper,
CPUs just fancy pens.
All is a stream of text
flowing from the past into the future.