hang of 5.1

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

hang of 5.1

Han Boetes-2
Hi, I got this weird hang on my server. I could ping it but ssh
and httpd were unresponsive.

bt trace and dmesg attached from serial console. Yay for serial
consoles on OpenBSD.

[halt sent]
Stopped at      Debugger+0x5:   leave
ddb{0}> ps
   PID   PPID   PGRP    UID  S       FLAGS  WAIT          COMMAND
 18819  15214  18819      0  3    0x200000  inode         sshd
  9596  15214   9596      0  3    0x200000  inode         sshd
 26335  15214  26335      0  3    0x200000  inode         sshd
 11569  15214  11569      0  3    0x200000  inode         sshd
 30591  10750  10750      0  3    0x200000  inode         master
 13416  15214  13416      0  3    0x200000  inode         sshd
 10002  25204  26716   1000  3           0  inode         sh
 10951  25204  26716   1000  3           0  inode         sh
 12150  17962  17962   1000  3           0  inode         sh
  5231  17962  17962   1000  3           0  inode         sh
 12059   7433  12059      0  3           0  inode         cron
  7433   1522   1522      0  3        0x80  piperd        cron
  1216  29647   1216   1000  3        0x80  select        ssh-agent
 29647  19251  29647   1000  3        0x80  poll          tmux
 19251  22785  22785   1000  3           0  inode         sshd
 22785  15214  22785      0  3        0x80  poll          sshd
  6690  23658   6690   1000  3        0x80  select        ssh-agent
 23658      1  23658   1000  3      0x2080  ttyout        tmux
  8787  15214   8787      0  3           0  inode         sshd
 32356  32449  32449    507  3        0x80  nanosleep     perl
 32245  32449  32449    507  3        0x80  nanosleep     perl
  8031   9553   9553     67  3        0x80  kqread        nginx
 24243   9553   9553     67  3           0  inode         nginx
 19315   9553   9553     67  3           0  inode         nginx
 30796   9553   9553     67  3           0  inode         nginx
 14775   9553   9553     67  3           0  inode         nginx
 31193  19535  31193   1000  3        0x80  ttyin         zsh
 19535  18076  19535   1000  3        0x88  pause         sh
 19362   4815   4815     74  3        0x80  bpf           pflogd
  4815      1   4815      0  3        0x80  netio         pflogd
  5376  10750  10750    507  3        0x80  kqread        qmgr
  1619  32494  32494     67  3        0x80  poll          php-fpm-5.3
 23509  32494  32494     67  3        0x80  poll          php-fpm-5.3
 10761  32494  32494     67  3        0x80  poll          php-fpm-5.3
 32494      1  32494      0  3        0x80  poll          php-fpm-5.3
 23646  13534  13534   1000  3           0  inode         mutt
 13534  18076  13534   1000  3        0x88  pause         sh
  6713      1   6713      0  3        0x80  poll          emacs-24.2.50
  4585      1   4585    601  3        0x80  kqread        unbound
 18578  24276  24276   1000  3           0  inode         rtorrent
 24276  18076  24276   1000  3        0x88  pause         sh
  8042   6286   6286     67  3        0x80  netcon        httpd
 29190   6286   6286     67  3        0x80  netcon        httpd
  6727   6286   6286     67  3        0x80  netcon        httpd
  6638   6286   6286     67  3        0x80  netcon        httpd
  7280   6286   6286     67  3        0x80  netcon        httpd
  6286      1   6286      0  3        0x80  select        httpd
  9641  32013  32013   1000  4   0x8000000                emacsclient
 32013  30562  32013   1000  4   0x8001088                sh
 28064      1  28064   1000  3        0x80  poll          emacs-24.2.50
 24930  16156  16156    250  3           0  inode         bitlbee
 16156      1  16156    250  3        0x80  poll          bitlbee
 26754   3350   3350   1000  3        0x80  poll          irssi
  3350  18076   3350   1000  3        0x88  pause         sh
 18773      1  18773    539  3           0  inode         clamd
 12322  14129  12322   1000  3        0x80  ttyin         zsh
 30562  24979  30562   1000  3        0x80  ttyin         zsh
 19651  15004  19651   1000  3        0x80  ttyin         zsh
  6185   2796   6185   1000  3        0x80  ttyin         zsh
 14129  18076  14129   1000  3        0x88  pause         sh
 15004  18076  15004   1000  3        0x88  pause         sh
 24979  18076  24979   1000  3        0x88  pause         sh
  2796  18076   2796   1000  3        0x88  pause         sh
 18076      1  18076   1000  3           0  inode         tmux
 15038   9462  20303   1000  3        0x80  select        mpc
 10750      1  10750      0  3        0x80  kqread        master
  9553      1   9553      0  3        0x88  pause         nginx
 15147  31127  31127     70  3        0x80  select        named
 31127      1  31127      0  3        0x80  netio         named
 21619      1  21619      0  3           0  inode         getty
  9462      1  20303   1000  3        0x88  pause         sh
 14145      1  14145      0  3        0x80  select        rsync
 15214      1  15214      0  3        0x80  select        sshd
 17962      1  17962   1000  3        0x80  piperd        sh
 20302  30868  26716   1000  3        0x80  piperd        sh
 30403  30868  26716   1000  3        0x80  kqread        tail
 30868      1  26716   1000  3        0x88  pause         sh
 25204      1  26716   1000  3        0x80  piperd        sh
 22792   4269   2917     83  3        0x80  poll          ntpd
  4269   2917   2917     83  3        0x80  poll          ntpd
  2917      1   2917      0  3        0x80  poll          ntpd
 20765  26845  20765      0  3        0x80  poll          syslog-ng
 26845      1  17854      0  3        0x80  wait          syslog-ng
  1522      1   1522      0  3           0  inode         cron
 26641      1  26641      0  3        0x80  ttyin         getty
  9435      1   9435      0  3        0x80  ttyin         getty
 19690      1  19690      0  3        0x80  ttyin         getty
  9245      1   9245      0  3        0x80  ttyin         getty
 15782      1  15782      0  3        0x80  ttyin         getty
 32449      1  32449    507  3        0x80  wait          perl
 28704      1  28704    569  3        0x80  poll          dkim-filter
 32274      1  32274    560  3        0x80  poll          mpd
 12412  13936   8620    502  3        0x80  poll          mysqld
 13936      1   8620      0  3        0x88  pause         sh
  7106      1   7106   1001  3        0x80  poll          openvpn
  7503   8867   8867     62  3        0x80  piperd        spamd
 17788      1  17788     62  3        0x80  bpf           spamlogd
  8159   8867   8867     62  3        0x80  select        spamd
  8867      1   8867     62  3           0  inode         spamd
 31779      1  31779      0  3        0x80  netcon        vsftpd
  9864      1   9864      0  3        0x80  netcon        vsftpd
 18563  11213  11213      0  3        0x80  nfsd          nfsd
  2756  11213  11213      0  3        0x80  nfsd          nfsd
 20830  11213  11213      0  3        0x80  nfsd          nfsd
 20795  11213  11213      0  3        0x80  nfsd          nfsd
 11213      1  11213      0  3        0x80  netcon        nfsd
 23780      1  23780      0  3        0x80  select        mountd
 15095      1  15095     28  3        0x80  poll          portmap
    20      0      0      0  3    0x100200  aiodoned      aiodoned
    19      0      0      0  3    0x100200  syncer        update
    18      0      0      0  3    0x100200  cleaner       cleaner
    17      0      0      0  3    0x100200  reaper        reaper
    16      0      0      0  3    0x100200  pgdaemon      pagedaemon
    15      0      0      0  3    0x100200  bored         crypto
    14      0      0      0  3    0x100200  pftm          pfpurge
    13      0      0      0  3    0x100200  usbtsk        usbtask
    12      0      0      0  3    0x100200  usbatsk       usbatsk
    11      0      0      0  3    0x100200  acpi0         acpi0
    10      0      0      0  7  0x40100200                idle7
     9      0      0      0  7  0x40100200                idle6
     8      0      0      0  7  0x40100200                idle5
     7      0      0      0  7  0x40100200                idle4
     6      0      0      0  7  0x40100200                idle3
     5      0      0      0  7  0x40100200                idle2
     4      0      0      0  7  0x40100200                idle1
     3      0      0      0  3    0x100200  bored         syswq
*    2      0      0      0  7  0x40100200                idle0
     1      0      1      0  3        0x80  wait          init
     0     -1      0      0  3       0x200  scheduler     swapper
  3839   8787   8787   1000  5      0x2000                sshd
ddb{0}> trace
Debugger() at Debugger+0x5
comintr() at comintr+0x268
Xintr_ioapic_edge4() at Xintr_ioapic_edge4+0xe8
--- interrupt ---
Bad frame pointer: 0xffff800022151f10
end trace frame: 0xffff800022151f10, count: -3
cpu_idle_cycle+0x13:
ddb{0}> boot reboot
rebooting...
           >> OpenBSD/amd64 BOOT 3.18ies Ltd.
boot> ghts Reserved
booting hd0a:/bsd: 5669864+1601484+935608+0+617568 [89+499848+323884]=0xd351b8
entry point at 0x1001e0 [7205c766, 34000004, 24448b12, 3cb8a304]
[ using 824664 bytes of bsd ELF symbol table ]or = 4
Copyright (c) 1982, 1986, 1989, 1991, 199333GHz
Intel(R)The Regents of the University of California.  All rights reserved.
Copyright (c) 1995-2012 OpenBSD. All rights reserved.  http://www.OpenBSD.org

OpenBSD 5.1 (GENERIC.MP) #207: Sun Feb 12 09:42:14 MST 2012
    [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 4292935680 (4094MB)
avail mem = 4164493312 (3971MB)
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.31 @ 0xdc010 (57 entries)
bios0: vendor HP version "O08" date 06/03/2009
bios0: HP ProLiant DL140 G3
acpi0 at bios0: rev 0
acpi0: sleep states S0 S4 S5
acpi0: tables DSDT FACP SPMI APIC MCFG BOOT SPCR SSDT
acpi0: wakeup devices BPD0(S5) BMF3(S5) P0P4(S5) P0P6(S5) PEX0(S5) PEX1(S5) PEX2(S5) PEX3(S5) USB1(S5) USB2(S5) USB3(S5) EUSB(S5) PCIB(S5)
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel(R) Xeon(R) CPU E5345 @ 2.33GHz, 2327.81 MHz
cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,NXE,LONG,LAHF
cpu0: 4MB 64b/line 16-way L2 cache
cpu0: apic clock running at 332MHz
cpu1 at mainbus0: apid 4 (application processor)
cpu1: Intel(R) Xeon(R) CPU E5345 @ 2.33GHz, 2327.51 MHz
cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,NXE,LONG,LAHF
cpu1: 4MB 64b/line 16-way L2 cache
cpu2 at mainbus0: apid 1 (application processor)
cpu2: Intel(R) Xeon(R) CPU E5345 @ 2.33GHz, 2327.50 MHz
cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,NXE,LONG,LAHF
cpu2: 4MB 64b/line 16-way L2 cache
cpu3 at mainbus0: apid 5 (application processor)
cpu3: Intel(R) Xeon(R) CPU E5345 @ 2.33GHz, 2327.50 MHz
cpu3: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,NXE,LONG,LAHF
cpu3: 4MB 64b/line 16-way L2 cache
cpu4 at mainbus0: apid 2 (application processor)
cpu4: Intel(R) Xeon(R) CPU E5345 @ 2.33GHz, 2327.50 MHz
cpu4: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,NXE,LONG,LAHF
cpu4: 4MB 64b/line 16-way L2 cache
cpu5 at mainbus0: apid 6 (application processor)
cpu5: Intel(R) Xeon(R) CPU E5345 @ 2.33GHz, 2327.50 MHz
cpu5: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,NXE,LONG,LAHF
cpu5: 4MB 64b/line 16-way L2 cache
cpu6 at mainbus0: apid 3 (application processor)
cpu6: Intel(R) Xeon(R) CPU E5345 @ 2.33GHz, 2327.50 MHz
cpu6: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,NXE,LONG,LAHF
cpu6: 4MB 64b/line 16-way L2 cache
cpu7 at mainbus0: apid 7 (application processor)
cpu7: Intel(R) Xeon(R) CPU E5345 @ 2.33GHz, 2327.50 MHz
cpu7: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,NXE,LONG,LAHF
cpu7: 4MB 64b/line 16-way L2 cache
ioapic0 at mainbus0: apid 8 pa 0xfec00000, version 20, 24 pins
ioapic1 at mainbus0: apid 9 pa 0xfec80000, version 20, 24 pins
acpimcfg0 at acpi0 addr 0xe0000000, bus 0-31
acpiprt0 at acpi0: bus 1 (P0P2)
acpiprt1 at acpi0: bus 2 (BMD0)
acpiprt2 at acpi0: bus 3 (BPD0)
acpiprt3 at acpi0: bus -1 (BPD1)
acpiprt4 at acpi0: bus -1 (BPD2)
acpiprt5 at acpi0: bus 11 (BMF3)
acpiprt6 at acpi0: bus 16 (P0P4)
acpiprt7 at acpi0: bus 18 (P0P6)
acpiprt8 at acpi0: bus 0 (PCI0)
acpiprt9 at acpi0: bus 30 (PEX0)
acpiprt10 at acpi0: bus 31 (PEX1)
acpiprt11 at acpi0: bus -1 (PEX2)
acpiprt12 at acpi0: bus -1 (PEX3)
acpiprt13 at acpi0: bus 32 (PCIB)
acpicpu0 at acpi0
acpicpu1 at acpi0
acpicpu2 at acpi0
acpicpu3 at acpi0
acpicpu4 at acpi0
acpicpu5 at acpi0
acpicpu6 at acpi0
acpicpu7 at acpi0
acpibtn0 at acpi0: PWRB
ipmi at mainbus0 not configured
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel 5000X Host" rev 0x31
ppb0 at pci0 dev 2 function 0 "Intel 5000 PCIE x8" rev 0x31
pci1 at ppb0 bus 1
ppb1 at pci1 dev 0 function 0 "Intel 6321ESB PCIE" rev 0x01
pci2 at ppb1 bus 2
ppb2 at pci2 dev 0 function 0 "Intel 6321ESB PCIE" rev 0x01
pci3 at ppb2 bus 3
ppb3 at pci1 dev 0 function 3 "Intel 6321ESB PCIE-PCIX" rev 0x01
pci4 at ppb3 bus 11
mpi0 at pci4 dev 1 function 0 "Symbios Logic SAS1068" rev 0x01: msi
scsibus0 at mpi0: 112 targets
sd0 at scsibus0 targ 1 lun 0: <LSILOGIC, Logical Volume, 3000> SCSI2 0/direct fixed naa.600508e000000000f37fc1e28b72c109
sd0: 1907348MB, 512 bytes/sector, 3906248704 sectors
ppb4 at pci0 dev 3 function 0 "Intel 5000 PCIE" rev 0x31
pci5 at ppb4 bus 12
ppb5 at pci0 dev 4 function 0 "Intel 5000 PCIE x16" rev 0x31: msi
pci6 at ppb5 bus 16
ppb6 at pci0 dev 5 function 0 "Intel 5000 PCIE" rev 0x31: msi
pci7 at ppb6 bus 17
ppb7 at pci0 dev 6 function 0 "Intel 5000 PCIE" rev 0x31: msi
pci8 at ppb7 bus 18
ppb8 at pci0 dev 7 function 0 "Intel 5000 PCIE" rev 0x31: msi
pci9 at ppb8 bus 19
pchb1 at pci0 dev 16 function 0 "Intel 5000 Error Reporting" rev 0x31
pchb2 at pci0 dev 16 function 1 "Intel 5000 Error Reporting" rev 0x31
pchb3 at pci0 dev 16 function 2 "Intel 5000 Error Reporting" rev 0x31
pchb4 at pci0 dev 17 function 0 "Intel 5000 Reserved" rev 0x31
pchb5 at pci0 dev 19 function 0 "Intel 5000 Reserved" rev 0x31
pchb6 at pci0 dev 21 function 0 "Intel 5000 FBD" rev 0x31
pchb7 at pci0 dev 22 function 0 "Intel 5000 FBD" rev 0x31
ppb9 at pci0 dev 28 function 0 "Intel 6321ESB PCIE" rev 0x09: msi
pci10 at ppb9 bus 30
bge0 at pci10 dev 0 function 0 "Broadcom BCM5721" rev 0x11, BCM5750 B1 (0x4101): apic 8 int 16, address 00:1a:4b:d4:1c:3e
brgphy0 at bge0 phy 1: BCM5750 10/100/1000baseT PHY, rev. 0
ppb10 at pci0 dev 28 function 1 "Intel 6321ESB PCIE" rev 0x09: msi
pci11 at ppb10 bus 31
bge1 at pci11 dev 0 function 0 "Broadcom BCM5721" rev 0x11, BCM5750 B1 (0x4101): apic 8 int 17, address 00:1a:4b:d4:1c:3f
brgphy1 at bge1 phy 1: BCM5750 10/100/1000baseT PHY, rev. 0
uhci0 at pci0 dev 29 function 0 "Intel 6321ESB USB" rev 0x09: apic 8 int 23
uhci1 at pci0 dev 29 function 1 "Intel 6321ESB USB" rev 0x09: apic 8 int 23
uhci2 at pci0 dev 29 function 2 "Intel 6321ESB USB" rev 0x09: apic 8 int 23
ehci0 at pci0 dev 29 function 7 "Intel 6321ESB USB" rev 0x09: apic 8 int 23
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
ppb11 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0xd9
pci12 at ppb11 bus 32
vga1 at pci12 dev 2 function 0 "Matrox MGA G200e (ServerEngines)" rev 0x02
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
pcib0 at pci0 dev 31 function 0 "Intel 6321ESB LPC" rev 0x09
pciide0 at pci0 dev 31 function 1 "Intel 6321ESB IDE" rev 0x09: DMA, channel 0 configured to compatibility, channel 1 configured to compatibility
pciide0: channel 0 disabled (no drives)
pciide0: channel 1 ignored (disabled)
ichiic0 at pci0 dev 31 function 3 "Intel 6321ESB SMBus" rev 0x09: apic 8 int 19
iic0 at ichiic0
usb1 at uhci0: USB revision 1.0
uhub1 at usb1 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb2 at uhci1: USB revision 1.0
uhub2 at usb2 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb3 at uhci2: USB revision 1.0
uhub3 at usb3 "Intel UHCI root hub" rev 1.00/1.00 addr 1
isa0 at pcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
com0: console
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
mtrr: Pentium Pro MTRR support
uhidev0 at uhub3 port 1 configuration 1 interface 0 "ServerEngines SE USB Device" rev 1.10/0.01 addr 2
uhidev0: iclass 3/1
ukbd0 at uhidev0: 8 modifier keys, 6 key codes
wskbd1 at ukbd0 mux 1
wskbd1: connecting to wsdisplay0
uhidev1 at uhub3 port 1 configuration 1 interface 1 "ServerEngines SE USB Device" rev 1.10/0.01 addr 2
uhidev1: iclass 3/1
ums0 at uhidev1: 8 buttons, Z dir
wsmouse0 at ums0 mux 0
vscsi0 at root
scsibus1 at vscsi0: 256 targets
softraid0 at root
scsibus2 at softraid0: 256 targets
root on sd0a (42c44955279fd28c.a) swap on sd0b dump on sd0b
WARNING: / was not properly unmounted
Automatic boot in progress: starting file system checks.




# Han

Reply | Threaded
Open this post in threaded view
|

Re: hang of 5.1

Stuart Henderson
On 2012/10/25 20:17, Han Boetes wrote:

> Hi, I got this weird hang on my server. I could ping it but ssh
> and httpd were unresponsive.
>
> bt trace and dmesg attached from serial console. Yay for serial
> consoles on OpenBSD.
>
> [halt sent]
> Stopped at      Debugger+0x5:   leave
> ddb{0}> ps
>    PID   PPID   PGRP    UID  S       FLAGS  WAIT          COMMAND
>  18819  15214  18819      0  3    0x200000  inode         sshd

all processes wanting disk i/o stuck with waitchan inode - I've seen
that before as have a few others I think - I haven't hit it myself on
-current for months now though.



>   9596  15214   9596      0  3    0x200000  inode         sshd
>  26335  15214  26335      0  3    0x200000  inode         sshd
>  11569  15214  11569      0  3    0x200000  inode         sshd
>  30591  10750  10750      0  3    0x200000  inode         master
>  13416  15214  13416      0  3    0x200000  inode         sshd
>  10002  25204  26716   1000  3           0  inode         sh
>  10951  25204  26716   1000  3           0  inode         sh
>  12150  17962  17962   1000  3           0  inode         sh
>   5231  17962  17962   1000  3           0  inode         sh
>  12059   7433  12059      0  3           0  inode         cron
>   7433   1522   1522      0  3        0x80  piperd        cron
>   1216  29647   1216   1000  3        0x80  select        ssh-agent
>  29647  19251  29647   1000  3        0x80  poll          tmux
>  19251  22785  22785   1000  3           0  inode         sshd
>  22785  15214  22785      0  3        0x80  poll          sshd
>   6690  23658   6690   1000  3        0x80  select        ssh-agent
>  23658      1  23658   1000  3      0x2080  ttyout        tmux
>   8787  15214   8787      0  3           0  inode         sshd
>  32356  32449  32449    507  3        0x80  nanosleep     perl
>  32245  32449  32449    507  3        0x80  nanosleep     perl
>   8031   9553   9553     67  3        0x80  kqread        nginx
>  24243   9553   9553     67  3           0  inode         nginx
>  19315   9553   9553     67  3           0  inode         nginx
>  30796   9553   9553     67  3           0  inode         nginx
>  14775   9553   9553     67  3           0  inode         nginx
>  31193  19535  31193   1000  3        0x80  ttyin         zsh
>  19535  18076  19535   1000  3        0x88  pause         sh
>  19362   4815   4815     74  3        0x80  bpf           pflogd
>   4815      1   4815      0  3        0x80  netio         pflogd
>   5376  10750  10750    507  3        0x80  kqread        qmgr
>   1619  32494  32494     67  3        0x80  poll          php-fpm-5.3
>  23509  32494  32494     67  3        0x80  poll          php-fpm-5.3
>  10761  32494  32494     67  3        0x80  poll          php-fpm-5.3
>  32494      1  32494      0  3        0x80  poll          php-fpm-5.3
>  23646  13534  13534   1000  3           0  inode         mutt
>  13534  18076  13534   1000  3        0x88  pause         sh
>   6713      1   6713      0  3        0x80  poll          emacs-24.2.50
>   4585      1   4585    601  3        0x80  kqread        unbound
>  18578  24276  24276   1000  3           0  inode         rtorrent
>  24276  18076  24276   1000  3        0x88  pause         sh
>   8042   6286   6286     67  3        0x80  netcon        httpd
>  29190   6286   6286     67  3        0x80  netcon        httpd
>   6727   6286   6286     67  3        0x80  netcon        httpd
>   6638   6286   6286     67  3        0x80  netcon        httpd
>   7280   6286   6286     67  3        0x80  netcon        httpd
>   6286      1   6286      0  3        0x80  select        httpd
>   9641  32013  32013   1000  4   0x8000000                emacsclient
>  32013  30562  32013   1000  4   0x8001088                sh
>  28064      1  28064   1000  3        0x80  poll          emacs-24.2.50
>  24930  16156  16156    250  3           0  inode         bitlbee
>  16156      1  16156    250  3        0x80  poll          bitlbee
>  26754   3350   3350   1000  3        0x80  poll          irssi
>   3350  18076   3350   1000  3        0x88  pause         sh
>  18773      1  18773    539  3           0  inode         clamd
>  12322  14129  12322   1000  3        0x80  ttyin         zsh
>  30562  24979  30562   1000  3        0x80  ttyin         zsh
>  19651  15004  19651   1000  3        0x80  ttyin         zsh
>   6185   2796   6185   1000  3        0x80  ttyin         zsh
>  14129  18076  14129   1000  3        0x88  pause         sh
>  15004  18076  15004   1000  3        0x88  pause         sh
>  24979  18076  24979   1000  3        0x88  pause         sh
>   2796  18076   2796   1000  3        0x88  pause         sh
>  18076      1  18076   1000  3           0  inode         tmux
>  15038   9462  20303   1000  3        0x80  select        mpc
>  10750      1  10750      0  3        0x80  kqread        master
>   9553      1   9553      0  3        0x88  pause         nginx
>  15147  31127  31127     70  3        0x80  select        named
>  31127      1  31127      0  3        0x80  netio         named
>  21619      1  21619      0  3           0  inode         getty
>   9462      1  20303   1000  3        0x88  pause         sh
>  14145      1  14145      0  3        0x80  select        rsync
>  15214      1  15214      0  3        0x80  select        sshd
>  17962      1  17962   1000  3        0x80  piperd        sh
>  20302  30868  26716   1000  3        0x80  piperd        sh
>  30403  30868  26716   1000  3        0x80  kqread        tail
>  30868      1  26716   1000  3        0x88  pause         sh
>  25204      1  26716   1000  3        0x80  piperd        sh
>  22792   4269   2917     83  3        0x80  poll          ntpd
>   4269   2917   2917     83  3        0x80  poll          ntpd
>   2917      1   2917      0  3        0x80  poll          ntpd
>  20765  26845  20765      0  3        0x80  poll          syslog-ng
>  26845      1  17854      0  3        0x80  wait          syslog-ng
>   1522      1   1522      0  3           0  inode         cron
>  26641      1  26641      0  3        0x80  ttyin         getty
>   9435      1   9435      0  3        0x80  ttyin         getty
>  19690      1  19690      0  3        0x80  ttyin         getty
>   9245      1   9245      0  3        0x80  ttyin         getty
>  15782      1  15782      0  3        0x80  ttyin         getty
>  32449      1  32449    507  3        0x80  wait          perl
>  28704      1  28704    569  3        0x80  poll          dkim-filter
>  32274      1  32274    560  3        0x80  poll          mpd
>  12412  13936   8620    502  3        0x80  poll          mysqld
>  13936      1   8620      0  3        0x88  pause         sh
>   7106      1   7106   1001  3        0x80  poll          openvpn
>   7503   8867   8867     62  3        0x80  piperd        spamd
>  17788      1  17788     62  3        0x80  bpf           spamlogd
>   8159   8867   8867     62  3        0x80  select        spamd
>   8867      1   8867     62  3           0  inode         spamd
>  31779      1  31779      0  3        0x80  netcon        vsftpd
>   9864      1   9864      0  3        0x80  netcon        vsftpd
>  18563  11213  11213      0  3        0x80  nfsd          nfsd
>   2756  11213  11213      0  3        0x80  nfsd          nfsd
>  20830  11213  11213      0  3        0x80  nfsd          nfsd
>  20795  11213  11213      0  3        0x80  nfsd          nfsd
>  11213      1  11213      0  3        0x80  netcon        nfsd
>  23780      1  23780      0  3        0x80  select        mountd
>  15095      1  15095     28  3        0x80  poll          portmap
>     20      0      0      0  3    0x100200  aiodoned      aiodoned
>     19      0      0      0  3    0x100200  syncer        update
>     18      0      0      0  3    0x100200  cleaner       cleaner
>     17      0      0      0  3    0x100200  reaper        reaper
>     16      0      0      0  3    0x100200  pgdaemon      pagedaemon
>     15      0      0      0  3    0x100200  bored         crypto
>     14      0      0      0  3    0x100200  pftm          pfpurge
>     13      0      0      0  3    0x100200  usbtsk        usbtask
>     12      0      0      0  3    0x100200  usbatsk       usbatsk
>     11      0      0      0  3    0x100200  acpi0         acpi0
>     10      0      0      0  7  0x40100200                idle7
>      9      0      0      0  7  0x40100200                idle6
>      8      0      0      0  7  0x40100200                idle5
>      7      0      0      0  7  0x40100200                idle4
>      6      0      0      0  7  0x40100200                idle3
>      5      0      0      0  7  0x40100200                idle2
>      4      0      0      0  7  0x40100200                idle1
>      3      0      0      0  3    0x100200  bored         syswq
> *    2      0      0      0  7  0x40100200                idle0
>      1      0      1      0  3        0x80  wait          init
>      0     -1      0      0  3       0x200  scheduler     swapper
>   3839   8787   8787   1000  5      0x2000                sshd
> ddb{0}> trace
> Debugger() at Debugger+0x5
> comintr() at comintr+0x268
> Xintr_ioapic_edge4() at Xintr_ioapic_edge4+0xe8
> --- interrupt ---
> Bad frame pointer: 0xffff800022151f10
> end trace frame: 0xffff800022151f10, count: -3
> cpu_idle_cycle+0x13:
> ddb{0}> boot reboot
> rebooting...
>            >> OpenBSD/amd64 BOOT 3.18ies Ltd.
> boot> ghts Reserved
> booting hd0a:/bsd: 5669864+1601484+935608+0+617568 [89+499848+323884]=0xd351b8
> entry point at 0x1001e0 [7205c766, 34000004, 24448b12, 3cb8a304]
> [ using 824664 bytes of bsd ELF symbol table ]or = 4
> Copyright (c) 1982, 1986, 1989, 1991, 199333GHz
> Intel(R)The Regents of the University of California.  All rights reserved.
> Copyright (c) 1995-2012 OpenBSD. All rights reserved.  http://www.OpenBSD.org
>
> OpenBSD 5.1 (GENERIC.MP) #207: Sun Feb 12 09:42:14 MST 2012
>     [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
> real mem = 4292935680 (4094MB)
> avail mem = 4164493312 (3971MB)
> mainbus0 at root
> bios0 at mainbus0: SMBIOS rev. 2.31 @ 0xdc010 (57 entries)
> bios0: vendor HP version "O08" date 06/03/2009
> bios0: HP ProLiant DL140 G3
> acpi0 at bios0: rev 0
> acpi0: sleep states S0 S4 S5
> acpi0: tables DSDT FACP SPMI APIC MCFG BOOT SPCR SSDT
> acpi0: wakeup devices BPD0(S5) BMF3(S5) P0P4(S5) P0P6(S5) PEX0(S5) PEX1(S5) PEX2(S5) PEX3(S5) USB1(S5) USB2(S5) USB3(S5) EUSB(S5) PCIB(S5)
> acpitimer0 at acpi0: 3579545 Hz, 24 bits
> acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
> cpu0 at mainbus0: apid 0 (boot processor)
> cpu0: Intel(R) Xeon(R) CPU E5345 @ 2.33GHz, 2327.81 MHz
> cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,NXE,LONG,LAHF
> cpu0: 4MB 64b/line 16-way L2 cache
> cpu0: apic clock running at 332MHz
> cpu1 at mainbus0: apid 4 (application processor)
> cpu1: Intel(R) Xeon(R) CPU E5345 @ 2.33GHz, 2327.51 MHz
> cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,NXE,LONG,LAHF
> cpu1: 4MB 64b/line 16-way L2 cache
> cpu2 at mainbus0: apid 1 (application processor)
> cpu2: Intel(R) Xeon(R) CPU E5345 @ 2.33GHz, 2327.50 MHz
> cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,NXE,LONG,LAHF
> cpu2: 4MB 64b/line 16-way L2 cache
> cpu3 at mainbus0: apid 5 (application processor)
> cpu3: Intel(R) Xeon(R) CPU E5345 @ 2.33GHz, 2327.50 MHz
> cpu3: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,NXE,LONG,LAHF
> cpu3: 4MB 64b/line 16-way L2 cache
> cpu4 at mainbus0: apid 2 (application processor)
> cpu4: Intel(R) Xeon(R) CPU E5345 @ 2.33GHz, 2327.50 MHz
> cpu4: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,NXE,LONG,LAHF
> cpu4: 4MB 64b/line 16-way L2 cache
> cpu5 at mainbus0: apid 6 (application processor)
> cpu5: Intel(R) Xeon(R) CPU E5345 @ 2.33GHz, 2327.50 MHz
> cpu5: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,NXE,LONG,LAHF
> cpu5: 4MB 64b/line 16-way L2 cache
> cpu6 at mainbus0: apid 3 (application processor)
> cpu6: Intel(R) Xeon(R) CPU E5345 @ 2.33GHz, 2327.50 MHz
> cpu6: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,NXE,LONG,LAHF
> cpu6: 4MB 64b/line 16-way L2 cache
> cpu7 at mainbus0: apid 7 (application processor)
> cpu7: Intel(R) Xeon(R) CPU E5345 @ 2.33GHz, 2327.50 MHz
> cpu7: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,NXE,LONG,LAHF
> cpu7: 4MB 64b/line 16-way L2 cache
> ioapic0 at mainbus0: apid 8 pa 0xfec00000, version 20, 24 pins
> ioapic1 at mainbus0: apid 9 pa 0xfec80000, version 20, 24 pins
> acpimcfg0 at acpi0 addr 0xe0000000, bus 0-31
> acpiprt0 at acpi0: bus 1 (P0P2)
> acpiprt1 at acpi0: bus 2 (BMD0)
> acpiprt2 at acpi0: bus 3 (BPD0)
> acpiprt3 at acpi0: bus -1 (BPD1)
> acpiprt4 at acpi0: bus -1 (BPD2)
> acpiprt5 at acpi0: bus 11 (BMF3)
> acpiprt6 at acpi0: bus 16 (P0P4)
> acpiprt7 at acpi0: bus 18 (P0P6)
> acpiprt8 at acpi0: bus 0 (PCI0)
> acpiprt9 at acpi0: bus 30 (PEX0)
> acpiprt10 at acpi0: bus 31 (PEX1)
> acpiprt11 at acpi0: bus -1 (PEX2)
> acpiprt12 at acpi0: bus -1 (PEX3)
> acpiprt13 at acpi0: bus 32 (PCIB)
> acpicpu0 at acpi0
> acpicpu1 at acpi0
> acpicpu2 at acpi0
> acpicpu3 at acpi0
> acpicpu4 at acpi0
> acpicpu5 at acpi0
> acpicpu6 at acpi0
> acpicpu7 at acpi0
> acpibtn0 at acpi0: PWRB
> ipmi at mainbus0 not configured
> pci0 at mainbus0 bus 0
> pchb0 at pci0 dev 0 function 0 "Intel 5000X Host" rev 0x31
> ppb0 at pci0 dev 2 function 0 "Intel 5000 PCIE x8" rev 0x31
> pci1 at ppb0 bus 1
> ppb1 at pci1 dev 0 function 0 "Intel 6321ESB PCIE" rev 0x01
> pci2 at ppb1 bus 2
> ppb2 at pci2 dev 0 function 0 "Intel 6321ESB PCIE" rev 0x01
> pci3 at ppb2 bus 3
> ppb3 at pci1 dev 0 function 3 "Intel 6321ESB PCIE-PCIX" rev 0x01
> pci4 at ppb3 bus 11
> mpi0 at pci4 dev 1 function 0 "Symbios Logic SAS1068" rev 0x01: msi
> scsibus0 at mpi0: 112 targets
> sd0 at scsibus0 targ 1 lun 0: <LSILOGIC, Logical Volume, 3000> SCSI2 0/direct fixed naa.600508e000000000f37fc1e28b72c109
> sd0: 1907348MB, 512 bytes/sector, 3906248704 sectors
> ppb4 at pci0 dev 3 function 0 "Intel 5000 PCIE" rev 0x31
> pci5 at ppb4 bus 12
> ppb5 at pci0 dev 4 function 0 "Intel 5000 PCIE x16" rev 0x31: msi
> pci6 at ppb5 bus 16
> ppb6 at pci0 dev 5 function 0 "Intel 5000 PCIE" rev 0x31: msi
> pci7 at ppb6 bus 17
> ppb7 at pci0 dev 6 function 0 "Intel 5000 PCIE" rev 0x31: msi
> pci8 at ppb7 bus 18
> ppb8 at pci0 dev 7 function 0 "Intel 5000 PCIE" rev 0x31: msi
> pci9 at ppb8 bus 19
> pchb1 at pci0 dev 16 function 0 "Intel 5000 Error Reporting" rev 0x31
> pchb2 at pci0 dev 16 function 1 "Intel 5000 Error Reporting" rev 0x31
> pchb3 at pci0 dev 16 function 2 "Intel 5000 Error Reporting" rev 0x31
> pchb4 at pci0 dev 17 function 0 "Intel 5000 Reserved" rev 0x31
> pchb5 at pci0 dev 19 function 0 "Intel 5000 Reserved" rev 0x31
> pchb6 at pci0 dev 21 function 0 "Intel 5000 FBD" rev 0x31
> pchb7 at pci0 dev 22 function 0 "Intel 5000 FBD" rev 0x31
> ppb9 at pci0 dev 28 function 0 "Intel 6321ESB PCIE" rev 0x09: msi
> pci10 at ppb9 bus 30
> bge0 at pci10 dev 0 function 0 "Broadcom BCM5721" rev 0x11, BCM5750 B1 (0x4101): apic 8 int 16, address 00:1a:4b:d4:1c:3e
> brgphy0 at bge0 phy 1: BCM5750 10/100/1000baseT PHY, rev. 0
> ppb10 at pci0 dev 28 function 1 "Intel 6321ESB PCIE" rev 0x09: msi
> pci11 at ppb10 bus 31
> bge1 at pci11 dev 0 function 0 "Broadcom BCM5721" rev 0x11, BCM5750 B1 (0x4101): apic 8 int 17, address 00:1a:4b:d4:1c:3f
> brgphy1 at bge1 phy 1: BCM5750 10/100/1000baseT PHY, rev. 0
> uhci0 at pci0 dev 29 function 0 "Intel 6321ESB USB" rev 0x09: apic 8 int 23
> uhci1 at pci0 dev 29 function 1 "Intel 6321ESB USB" rev 0x09: apic 8 int 23
> uhci2 at pci0 dev 29 function 2 "Intel 6321ESB USB" rev 0x09: apic 8 int 23
> ehci0 at pci0 dev 29 function 7 "Intel 6321ESB USB" rev 0x09: apic 8 int 23
> usb0 at ehci0: USB revision 2.0
> uhub0 at usb0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
> ppb11 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0xd9
> pci12 at ppb11 bus 32
> vga1 at pci12 dev 2 function 0 "Matrox MGA G200e (ServerEngines)" rev 0x02
> wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
> wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
> pcib0 at pci0 dev 31 function 0 "Intel 6321ESB LPC" rev 0x09
> pciide0 at pci0 dev 31 function 1 "Intel 6321ESB IDE" rev 0x09: DMA, channel 0 configured to compatibility, channel 1 configured to compatibility
> pciide0: channel 0 disabled (no drives)
> pciide0: channel 1 ignored (disabled)
> ichiic0 at pci0 dev 31 function 3 "Intel 6321ESB SMBus" rev 0x09: apic 8 int 19
> iic0 at ichiic0
> usb1 at uhci0: USB revision 1.0
> uhub1 at usb1 "Intel UHCI root hub" rev 1.00/1.00 addr 1
> usb2 at uhci1: USB revision 1.0
> uhub2 at usb2 "Intel UHCI root hub" rev 1.00/1.00 addr 1
> usb3 at uhci2: USB revision 1.0
> uhub3 at usb3 "Intel UHCI root hub" rev 1.00/1.00 addr 1
> isa0 at pcib0
> isadma0 at isa0
> com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
> com0: console
> pckbc0 at isa0 port 0x60/5
> pckbd0 at pckbc0 (kbd slot)
> pckbc0: using irq 1 for kbd slot
> wskbd0 at pckbd0: console keyboard, using wsdisplay0
> pcppi0 at isa0 port 0x61
> spkr0 at pcppi0
> mtrr: Pentium Pro MTRR support
> uhidev0 at uhub3 port 1 configuration 1 interface 0 "ServerEngines SE USB Device" rev 1.10/0.01 addr 2
> uhidev0: iclass 3/1
> ukbd0 at uhidev0: 8 modifier keys, 6 key codes
> wskbd1 at ukbd0 mux 1
> wskbd1: connecting to wsdisplay0
> uhidev1 at uhub3 port 1 configuration 1 interface 1 "ServerEngines SE USB Device" rev 1.10/0.01 addr 2
> uhidev1: iclass 3/1
> ums0 at uhidev1: 8 buttons, Z dir
> wsmouse0 at ums0 mux 0
> vscsi0 at root
> scsibus1 at vscsi0: 256 targets
> softraid0 at root
> scsibus2 at softraid0: 256 targets
> root on sd0a (42c44955279fd28c.a) swap on sd0b dump on sd0b
> WARNING: / was not properly unmounted
> Automatic boot in progress: starting file system checks.
>
>
>
>
> # Han

Reply | Threaded
Open this post in threaded view
|

Re: hang of 5.1

Alexander Bluhm
On Thu, Oct 25, 2012 at 09:01:44PM +0100, Stuart Henderson wrote:
> On 2012/10/25 20:17, Han Boetes wrote:
> all processes wanting disk i/o stuck with waitchan inode - I've seen
> that before as have a few others I think - I haven't hit it myself on
> -current for months now though.

I know this bug, too.  When I analysed it on my 5.2 box, I found a
ksh that did try to exit, did hold the vnode log and was waiting
for ttyout.  The potential reader of the pseudo tty was a sshd
waiting for the vnode lock.  Could be something simmilar here.

This tmux tries to exit:

> >    PID   PPID   PGRP    UID  S       FLAGS  WAIT          COMMAND
> >  23658      1  23658   1000  3      0x2080  ttyout        tmux

In this state it holds a inode lock to its /dev/ttyp trying to write
some final output.  The reading end of the pseudo tty might be one
of those sshd waiting for the inode lock.  This would be the dead
lock:

> >  18819  15214  18819      0  3    0x200000  inode         sshd
> >   9596  15214   9596      0  3    0x200000  inode         sshd
> >  26335  15214  26335      0  3    0x200000  inode         sshd
> >  11569  15214  11569      0  3    0x200000  inode         sshd
> >  13416  15214  13416      0  3    0x200000  inode         sshd
> >  19251  22785  22785   1000  3           0  inode         sshd
> >   8787  15214   8787      0  3           0  inode         sshd

This is just a wild guess what might went wrong here, based on
something similar I have seen before.

bluhm