panic: softdep_deallocate_dependencies: unrecovered I/O error

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

panic: softdep_deallocate_dependencies: unrecovered I/O error

Klemens Nanni-2
Pinebook Pro running a -CURRENT kernel with patches on recent snapshots
paniced upon

        $ doas ifconfig bwfm0 down
        $ doas ifconfig bwfm0 up
        $ doas ifconfig bwfm0 down
        $ doas ifconfig bwfm0 up

Changes from GENERIC.MP include omission of unused drivers such as
radeondrm(4) (to build faster) and a few debug printfs.

The only possibly relevant diff is this one which I cherry-picked from
NetBSD to potentially fix hard hangs with bwfm(4) on the Pinebook Pro.
Above up/down dances were testing this diff (which seems promising):

https://github.com/NetBSD/src/commit/5f697873ce77ab855674a138ff1e660a0aa506bd
"clear all interrupts, not just those we expect from the hostintmask."

Index: dev/sdmmc/if_bwfm_sdio.c
===================================================================
RCS file: /cvs/src/sys/dev/sdmmc/if_bwfm_sdio.c,v
retrieving revision 1.39
diff -u -p -r1.39 if_bwfm_sdio.c
--- dev/sdmmc/if_bwfm_sdio.c 26 Feb 2021 00:07:41 -0000 1.39
+++ dev/sdmmc/if_bwfm_sdio.c 4 Apr 2021 19:47:57 -0000
@@ -704,7 +704,6 @@ bwfm_sdio_task(void *v)
  }
 
  intstat = bwfm_sdio_dev_read(sc, BWFM_SDPCMD_INTSTATUS);
- intstat &= (SDPCMD_INTSTATUS_HMB_SW_MASK|SDPCMD_INTSTATUS_CHIPACTIVE);
  /* XXX fc state */
  if (intstat)
  bwfm_sdio_dev_write(sc, BWFM_SDPCMD_INTSTATUS, intstat);


I'm still in ddb on serial; panic here, full boot log/dmesg at the end.

        panic: softdep_deallocate_dependencies: unrecovered I/O error
        Stopped at      panic+0x158:    mov     w0, w20
            TID    PID    UID     PRFLAGS     PFLAGS  CPU  COMMAND
         126512  49936   1001    0x100003       0x88    5  ksh
          66096  96033   1001    0x100003      0x480    1  top
             65  32064     48    0x100012      0x480    4  unwind
         386823  74211      0     0x14000      0x200    2  sensors
        *353750  80377      0     0x14000      0x200    3K sdmmc2
        db_enter() at panic+0x154
        panic() at softdep_deallocate_dependencies+0x38
        softdep_count_dependencies() at brelse+0x344
        brelse() at sd_buf_done+0x12c
        sd_buf_done() at scsi_done+0x28
        scsi_done() at sdmmc_complete_xs+0xa0
        sdmmc_complete_xs() at sdmmc_task_thread+0x104
        https://www.openbsd.org/ddb.html describes the minimum info required in bug
        reports.  Insufficient info makes it difficult to find and fix bugs.

I've also included `show all mounts' output to show mount flags, but
I'm surprised to see none of them having SOFTDEP listed -- pretty sure
I've mounted almost all filesytems with "softdep".

bwfm(4) or rather sdmmc(4) in pristine GENERIC.MP sporadically fail at
boot which can look like this (pretty sure it's not always the exact
same chain of errors):

        starting network
        bwfm0: HT avail timeout
        bwfm_sdio_buf_write: error 60
        bwfm0: could not load microcode
        bwfm0: could not init bus
        bwfm_sdio_buf_read: error 60
        bwfm_sdio_buf_write: error 60
        bwfm_sdio_buf_read: error 60
        bwfm_sdio_buf_read: error 60
        bwfm_sdio_buf_read: error 60
        bwfm_sdio_buf_read: error 60
        bwfm_sdio_buf_read: error 60
        bwfm0: HT avail timeout
        bwfm_sdio_buf_write: error 60
        bwfm0: could not load microcode
        bwfm0: could not init bus
        starting early daemons: syslogd ntpd.

If that happens, bwfm seems unrecoverable and `ifconfig bwfm0 down'
often makes the system hang with GENERIC.MP -- it has never paniced on
me before, though.

FWIW, few filesystems needed fsck(8) after such hangs and that's the
first panic I've had, so hopefully the filesystems shouldn't been too
wasted already.


>> OpenBSD/arm64 BOOTAA64 1.4
boot>
booting sd0a:/bsd.pbp: 3375304+761096+204288+767328 [211304+109+560280+269733]=0x7ebb38
type 0x2 pa 0x200000 va 0x200000 pages 0x4000 attr 0x8
type 0x7 pa 0x4200000 va 0x4200000 pages 0x3eee attr 0x8
type 0x9 pa 0x80ee000 va 0x80ee000 pages 0x24 attr 0x8
type 0x7 pa 0x8112000 va 0x8112000 pages 0xeb74a attr 0x8
type 0x2 pa 0xf385c000 va 0xf385c000 pages 0x583 attr 0x8
type 0x7 pa 0xf3ddf000 va 0xf3ddf000 pages 0x1 attr 0x8
type 0x2 pa 0xf3de0000 va 0xf3de0000 pages 0x100 attr 0x8
type 0x1 pa 0xf3ee0000 va 0xf3ee0000 pages 0x2a attr 0x8
type 0x0 pa 0xf3f0a000 va 0xf3f0a000 pages 0x5 attr 0x8
type 0x4 pa 0xf3f0f000 va 0xf3f0f000 pages 0x1 attr 0x8
type 0x6 pa 0xf3f10000 va 0x4d03a0a000 pages 0x4 attr 0x8000000000000008
type 0x4 pa 0xf3f14000 va 0xf3f14000 pages 0x1 attr 0x8
type 0x6 pa 0xf3f15000 va 0x4d03a0f000 pages 0x4 attr 0x8000000000000008
type 0x0 pa 0xf3f19000 va 0xf3f19000 pages 0x1 attr 0x8
type 0x4 pa 0xf3f1a000 va 0xf3f1a000 pages 0x1 attr 0x8
type 0x0 pa 0xf3f1b000 va 0xf3f1b000 pages 0x1 attr 0x8
type 0x4 pa 0xf3f1c000 va 0xf3f1c000 pages 0x2 attr 0x8
type 0x0 pa 0xf3f1e000 va 0xf3f1e000 pages 0x1 attr 0x8
type 0x4 pa 0xf3f1f000 va 0xf3f1f000 pages 0x1 attr 0x8
type 0x0 pa 0xf3f20000 va 0xf3f20000 pages 0x2 attr 0x8
type 0x4 pa 0xf3f22000 va 0xf3f22000 pages 0x1 attr 0x8
type 0x0 pa 0xf3f23000 va 0xf3f23000 pages 0x1 attr 0x8
type 0x2 pa 0xf3f24000 va 0xf3f24000 pages 0x300c attr 0x8
type 0x5 pa 0xf6f30000 va 0x4d06a2a000 pages 0x10 attr 0x8000000000000008
type 0x2 pa 0xf6f40000 va 0xf6f40000 pages 0x10c0 attr 0x8
[ using 1042392 bytes of bsd ELF symbol table ]
Copyright (c) 1982, 1986, 1989, 1991, 1993
        The Regents of the University of California.  All rights reserved.
Copyright (c) 1995-2021 OpenBSD. All rights reserved.  https://www.OpenBSD.org

OpenBSD 6.9-beta (PBP.MP) #4: Sun Apr  4 02:28:39 CEST 2021
    [hidden email]:/sys/arch/arm64/compile/PBP.MP
real mem  = 4083384320 (3894MB)
avail mem = 3890089984 (3709MB)
random: good seed from bootblocks
mainbus0 at root: Pine64 Pinebook Pro
psci0 at mainbus0: PSCI 1.1, SMCCC 1.2
cpu0 at mainbus0 mpidr 0: ARM Cortex-A53 r0p4
cpu0: 32KB 64b/line 2-way L1 VIPT I-cache, 32KB 64b/line 4-way L1 D-cache
cpu0: 512KB 64b/line 16-way L2 cache
cpu0: CRC32,SHA2,SHA1,AES+PMULL,ASID16
cpu1 at mainbus0 mpidr 1: ARM Cortex-A53 r0p4
pu1: 32KB 64b/line 2-way L1 VIPT I-cache, 32KB 64b/line 4-way L1 D-cache
cpu1: 512KB 64b/line 16-way L2 cache
cpu1: CRC32,SHA2,SHA1,AES+PMULL,ASID16
cpu2 at mainbus0 mpidr 2: ARM Cortex-A53 r0p4
cpu2: 32KB 64b/line 2-way L1 VIPT I-cache, 32KB 64b/line 4-way L1 D-cache
cpu2: 512KB 64b/line 16-way L2 cache
cpu2: CRC32,SHA2,SHA1,AES+PMULL,ASID16
cpu3 at mainbus0 mpidr 3: ARM Cortex-A53 r0p4
cpu3: 32KB 64b/line 2-way L1 VIPT I-cache, 32KB 64b/line 4-way L1 D-cache
cpu3: 512KB 64b/line 16-way L2 cache
cpu3: CRC32,SHA2,SHA1,AES+PMULL,ASID16
cpu4 at mainbus0 mpidr 100: ARM Cortex-A72 r0p2
cpu4: 48KB 64b/line 3-way L1 PIPT I-cache, 32KB 64b/line 2-way L1 D-cache
cpu4: 1024KB 64b/line 16-way L2 cache
cpu4: CRC32,SHA2,SHA1,AES+PMULL,ASID16
cpu5 at mainbus0 mpidr 101: ARM Cortex-A72 r0p2
cpu5: 48KB 64b/line 3-way L1 PIPT I-cache, 32KB 64b/line 2-way L1 D-cache
cpu5: 1024KB 64b/line 16-way L2 cache
cpu5: CRC32,SHA2,SHA1,AES+PMULL,ASID16
efi0 at mainbus0: UEFI 2.8
efi0: Das U-Boot rev 0x20210100
apm0 at mainbus0
agintc0 at mainbus0 sec shift 3:3 nirq 288 nredist 6 ipi: 0, 1: "interrupt-controller"
agintcmsi0 at agintc0
syscon0 at mainbus0: "qos"
syscon1 at mainbus0: "qos"
syscon2 at mainbus0: "qos"
syscon3 at mainbus0: "qos"
syscon4 at mainbus0: "qos"
syscon5 at mainbus0: "qos"
syscon6 at mainbus0: "qos"
syscon7 at mainbus0: "qos"
syscon8 at mainbus0: "qos"
syscon9 at mainbus0: "qos"
syscon10 at mainbus0: "qos"
syscon11 at mainbus0: "qos"
syscon12 at mainbus0: "qos"
syscon13 at mainbus0: "qos"
syscon14 at mainbus0: "qos"
syscon15 at mainbus0: "qos"
syscon16 at mainbus0: "qos"
syscon17 at mainbus0: "qos"
syscon18 at mainbus0: "qos"
syscon19 at mainbus0: "qos"
syscon20 at mainbus0: "qos"
syscon21 at mainbus0: "qos"
syscon22 at mainbus0: "qos"
syscon23 at mainbus0: "qos"
syscon24 at mainbus0: "qos"
syscon25 at mainbus0: "power-management"
"power-controller" at syscon25 not configured
syscon26 at mainbus0: "syscon"
"io-domains" at syscon26 not configured
rkclock0 at mainbus0
rkclock1 at mainbus0
syscon27 at mainbus0: "syscon"
"io-domains" at syscon27 not configured
"usb2-phy" at syscon27 not configured
"usb2-phy" at syscon27 not configured
rkemmcphy0 at syscon27
"pcie-phy" at syscon27 not configured
rkpinctrl0 at mainbus0: "pinctrl"
rkgpio0 at rkpinctrl0
rkgpio1 at rkpinctrl0
rkgpio2 at rkpinctrl0
rkgpio3 at rkpinctrl0
rkgpio4 at rkpinctrl0
pwmreg0 at mainbus0
rkdrm0 at mainbus0
drm0 at rkdrm0
"pmu_a53" at mainbus0 not configured
"pmu_a72" at mainbus0 not configured
agtimer0 at mainbus0: 24000 kHz
"xin24m" at mainbus0 not configured
simplebus0 at mainbus0: "bus"
"dma-controller" at simplebus0 not configured
"dma-controller" at simplebus0 not configured
rkpcie0 at mainbus0
rkpcie0: link training timeout
dwmmc0 at mainbus0: 50 MHz base clock
sdmmc0 at dwmmc0: 4-bit, sd high-speed, dma
dwmmc1 at mainbus0: 50 MHz base clock
sdmmc1 at dwmmc1: 4-bit, sd high-speed, mmc high-speed, dma
sdhc0 at mainbus0
sdhc0: SDHC 3.0, 200 MHz base clock
sdmmc2 at sdhc0: 8-bit, sd high-speed, mmc high-speed, dma
ehci0 at mainbus0
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 configuration 1 interface 0 "Generic EHCI root hub" rev 2.00/1.00 addr 1
ohci0 at mainbus0: version 1.0
ehci1 at mainbus0
usb1 at ehci1: USB revision 2.0
uhub1 at usb1 configuration 1 interface 0 "Generic EHCI root hub" rev 2.00/1.00 addr 1
ohci1 at mainbus0: version 1.0
rkdwusb0 at mainbus0: "usb"
xhci0 at rkdwusb0, xHCI 1.10
usb2 at xhci0: USB revision 3.0
uhub2 at usb2 configuration 1 interface 0 "Generic xHCI root hub" rev 3.00/1.00 addr 1
rkdwusb1 at mainbus0: "usb"
xhci1 at rkdwusb1, xHCI 1.10
usb3 at xhci1: USB revision 3.0
uhub3 at usb3 configuration 1 interface 0 "Generic xHCI root hub" rev 3.00/1.00 addr 1
"dp" at mainbus0 not configured
"saradc" at mainbus0 not configured
rkiic0 at mainbus0
iic0 at rkiic0
escodec0 at iic0 addr 0x11
rkiic1 at mainbus0
iic1 at rkiic1
com0 at mainbus0: ns16550, no working fifo
com1 at mainbus0: ns16550, no working fifo
com1: console
"spi" at mainbus0 not configured
rktemp0 at mainbus0
rkrng0 at mainbus0
rkiic2 at mainbus0
iic2 at rkiic2
rkpmic0 at iic2 addr 0x1b: RK808
fanpwr0 at iic2 addr 0x40: SYR827, 1.00 VDC
fanpwr1 at iic2 addr 0x41: SYR828, 1.00 VDC
rkiic3 at mainbus0
iic3 at rkiic3
fusbtc0 at iic3 addr 0x22
cwfg0 at iic3 addr 0x62
rkpwm0 at mainbus0
rkpwm1 at mainbus0
"video-codec" at mainbus0 not configured
"iommu" at mainbus0 not configured
"video-codec" at mainbus0 not configured
"iommu" at mainbus0 not configured
"rga" at mainbus0 not configured
"efuse" at mainbus0 not configured
"phy" at mainbus0 not configured
"phy" at mainbus0 not configured
"watchdog" at mainbus0 not configured
"rktimer" at mainbus0 not configured
rkiis0 at mainbus0
rkvop0 at mainbus0: RK3399 VOPL
"iommu" at mainbus0 not configured
rkvop1 at mainbus0: RK3399 VOPB
"iommu" at mainbus0 not configured
"iommu" at mainbus0 not configured
"iommu" at mainbus0 not configured
simpleaudio0 at mainbus0
rkanxdp0 at mainbus0: eDP TX
"gpu" at mainbus0 not configured
"opp-table0" at mainbus0 not configured
"opp-table1" at mainbus0 not configured
"opp-table2" at mainbus0 not configured
pwmbl0 at mainbus0
"battery" at mainbus0 not configured
simplepanel0 at mainbus0
"gpio-key-lid" at mainbus0 not configured
"gpio-key-power" at mainbus0 not configured
"leds" at mainbus0 not configured
"sdio-pwrseq" at mainbus0 not configured
simpleaudio1 at mainbus0
simpleamp0 at mainbus0
"vcc-sysin" at mainbus0 not configured
"vcc-12v" at mainbus0 not configured
"vcc3v3-sys" at mainbus0 not configured
"vcc5v0-usb-regulator" at mainbus0 not configured
"vcc-0v9" at mainbus0 not configured
"vcc1v8-s3" at mainbus0 not configured
"vcc3v0-sd" at mainbus0 not configured
"vcc3v3-panel" at mainbus0 not configured
"vcc3v3-ssd" at mainbus0 not configured
"vcc5v0-otg" at mainbus0 not configured
"vbus-5vout" at mainbus0 not configured
"vcca0v9-s3" at mainbus0 not configured
"dc-charger" at mainbus0 not configured
usb4 at ohci0: USB revision 1.0
uhub4 at usb4 configuration 1 interface 0 "Generic OHCI root hub" rev 1.00/1.00 addr 1
usb5 at ohci1: USB revision 1.0
uhub5 at usb5 configuration 1 interface 0 "Generic OHCI root hub" rev 1.00/1.00 addr 1
audio0 at simpleaudio1
dt: 443 probes
bwfm0 at sdmmc0 function 1
manufacturer 0x02d0, product 0xa9bf at sdmmc0 function 2 not configured
manufacturer 0x02d0, product 0xa9bf at sdmmc0 function 3 not configured
scsibus0 at sdmmc2: 2 targets, initiator 0
sd0 at scsibus0 targ 1 lun 0: <Sandisk, DA4064, 0000> removable
sd0: 59640MB, 512 bytes/sector, 122142720 sectors
uhub6 at uhub1 port 1 configuration 1 interface 0 "Genesys Logic USB2.0 Hub" rev 2.00/85.36 addr 2
uhidev0 at uhub4 port 1 configuration 1 interface 0 "HAILUCK CO.,LTD USB KEYBOARD" rev 1.10/1.00 addr 2
uhidev0: iclass 3/1
ukbd0 at uhidev0: 8 variable keys, 6 key codes
wskbd0 at ukbd0 mux 1
uhidev1 at uhub4 port 1 configuration 1 interface 1 "HAILUCK CO.,LTD USB KEYBOARD" rev 1.10/1.00 addr 2
uhidev1: iclass 3/0, 35 report ids
umt0 at uhidev1: clickpad, 4 contacts
wsmouse0 at umt0 mux 0
ums0 at uhidev1 reportid 1: 3 buttons
wsmouse1 at ums0 mux 0
uhid0 at uhidev1 reportid 2: input=1, output=0, feature=0
uhid1 at uhidev1 reportid 3: input=3, output=0, feature=0
uhid2 at uhidev1 reportid 5: input=0, output=0, feature=5
uhid3 at uhidev1 reportid 6: input=0, output=0, feature=255
uhid4 at uhidev1 reportid 9: input=1, output=0, feature=0
uhid5 at uhidev1 reportid 32: input=0, output=0, feature=255
uhid6 at uhidev1 reportid 35: input=0, output=0, feature=1
root on sd0a (31fb99ccf4db16db.a) swap on sd0b dump on sd0b
rkvop0: using CRTC 0 for RK3399 VOPL
rkvop1: using CRTC 1 for RK3399 VOPB
rkdrm0: 1920x1080, 32bpp
wsdisplay0 at rkdrm0 mux 1
wskbd0: connecting to wsdisplay0
wsdisplay0: screen 0-5 added (std, vt100 emulation)
bwfm0: HT avail timeout
bwfm_sdio_buf_write: error 60
bwfm0: could not load microcode
Automatic boot in progress: starting file system checks.
/dev/sd0a (31fb99ccf4db16db.a): file system is clean; not checking
/dev/sd0l (31fb99ccf4db16db.l): file system is clean; not checking
/dev/sd0d (31fb99ccf4db16db.d): file system is clean; not checking
/dev/sd0f (31fb99ccf4db16db.f): file system is clean; not checking
/dev/sd0g (31fb99ccf4db16db.g): file system is clean; not checking
/dev/sd0h (31fb99ccf4db16db.h): file system is clean; not checking
/dev/sd0k (31fb99ccf4db16db.k): file system is clean; not checking
/dev/sd0j (31fb99ccf4db16db.j): file system is clean; not checking
/dev/sd0e (31fb99ccf4db16db.e): file system is clean; not checking
display.brightness -> 9.41%
mouse.tp.tapping -> 1,3,2
pf enabled
vm.malloc_conf:  -> SU
kern.video.record: 0 -> 1
kern.audio.record: 0 -> 1
kern.allowkmem: 0 -> 1
kern.securelevel: 0 -> -1
starting network
bwfm0: HT avail timeout
bwfm_sdio_buf_write: error 60
bwfm0: could not load microcode
bwfm0: could not init bus
bwfm_sdio_buf_read: error 60
bwfm_sdio_buf_write: error 60
bwfm_sdio_buf_read: error 60
bwfm_sdio_buf_read: error 60
bwfm_sdio_buf_read: error 60
bwfm_sdio_buf_read: error 60
bwfm_sdio_buf_read: error 60
bwfm0: HT avail timeout
bwfm_sdio_buf_write: error 60
bwfm0: could not load microcode
bwfm0: could not init bus
starting early daemons: syslogd ntpd.
starting RPC daemons:.
savecore: no core dump
checking quotas: done.
clearing /tmp
turning on accounting
creating runtime link editor directory cache.
preserving editor files.
starting network daemons: sshd smtpd sndiod.
starting local daemons: apmd cron xenodm.
Sun Apr  4 21:21:20 CEST 2021

OpenBSD/arm64 (pine64) (console)

login: panic: softdep_deallocate_dependencies: unrecovered I/O error


ddb{3}> show all mounts
mountpoint 0xffffff800b0ff400
flags 205000<LOCAL,ROOTFS>
vnodecovered 0x0 syncer 0xffffff800aefe7f8 data 0xffffff800b0fd600
vfsconf: ops 0xffffff80006aff20 name "ffs" num 1 ref 9 flags 0x201000
statvfs cache: bsize 800 iosize 4000
blocks 504711 free 442856 avail 417621
  files 155518 ffiles 153775 favail 153775
  f_fsidx {0x400, 0x547e46f8} owner 0 ctime 0x606a11ab
  syncwrites 25 asyncwrites = 51
  syncreads 10964 asyncreads = 0
  fstype "ffs" mnton "/" mntfrom "/dev/sd0a" mntspec "31fb99ccf4db16db.a"
locked vnodes:
mountpoint 0xffffff800b0ffc00
flags 4201018<NOSUID,NODEV,LOCAL>
vnodecovered 0xffffff800aefe8d8 syncer 0xffffff800aefe9b8 data 0xffffff800b373e
00
vfsconf: ops 0xffffff80006aff20 name "ffs" num 1 ref 9 flags 0x201000
statvfs cache: bsize 800 iosize 4000
blocks 10045895 free 4382788 avail 3880494
  files 2617918 ffiles 2419802 favail 2419802
  f_fsidx {0x40b, 0xa4d659ca} owner 0 ctime 0x606a11a1
  syncwrites 2 asyncwrites = 14
  syncreads 288 asyncreads = 0
  fstype "ffs" mnton "/home" mntfrom "/dev/sd0l" mntspec "31fb99ccf4db16db.l"
locked vnodes:
mountpoint 0xffffff800b376000
flags 420901c<NOEXEC,NOSUID,NODEV,LOCAL,NOATIME>
vnodecovered 0xffffff800aefea98 syncer 0xffffff800aefed38 data 0xffffff800b373a
00
vfsconf: ops 0xffffff80006aff20 name "ffs" num 1 ref 9 flags 0x201000
statvfs cache: bsize 800 iosize 4000
blocks 1996183 free 1996177 avail 1896368
  files 518398 ffiles 518391 favail 518391
  f_fsidx {0x403, 0xb48075d4} owner 0 ctime 0x606a11a1
  syncwrites 2 asyncwrites = 46
  syncreads 510 asyncreads = 0
  fstype "ffs" mnton "/tmp" mntfrom "/dev/sd0d" mntspec "31fb99ccf4db16db.d"
locked vnodes:
mountpoint 0xffffff800b386800
flags 201011<RDONLY,NODEV,LOCAL>
vnodecovered 0xffffff800af258c8 syncer 0x0 data 0xffffff800b373800
vfsconf: ops 0xffffff80006aff20 name "ffs" num 1 ref 9 flags 0x201000
statvfs cache: bsize 800 iosize 4000
blocks 3044751 free 2575231 avail 2422994
  files 803518 ffiles 788286 favail 788286
  f_fsidx {0x405, 0x1508d3e5} owner 0 ctime 0x606a11a1
  syncwrites 0 asyncwrites = 0
  syncreads 7140 asyncreads = 0
  fstype "ffs" mnton "/usr" mntfrom "/dev/sd0f" mntspec "31fb99ccf4db16db.f"
locked vnodes:
mountpoint 0xffffff800b386c00
flags 201011<RDONLY,NODEV,LOCAL>
vnodecovered 0xffffff800aef7d40 syncer 0x0 data 0xffffff800b373c00
vfsconf: ops 0xffffff80006aff20 name "ffs" num 1 ref 9 flags 0x201000
statvfs cache: bsize 800 iosize 4000
blocks 504711 free 395847 avail 370612
  files 155518 ffiles 146417 favail 146417
  f_fsidx {0x406, 0x3cc6eb7b} owner 0 ctime 0x606a11a1
  syncwrites 0 asyncwrites = 0
  syncreads 3983 asyncreads = 0
  fstype "ffs" mnton "/usr/X11R6" mntfrom "/dev/sd0g" mntspec "31fb99ccf4db16db
.g"
locked vnodes:
mountpoint 0xffffff800b386400
flags 4209019<RDONLY,NOSUID,NODEV,LOCAL,NOATIME>
vnodecovered 0xffffff800aef7b80 syncer 0x0 data 0xffffff800b389200
vfsconf: ops 0xffffff80006aff20 name "ffs" num 1 ref 9 flags 0x201000
statvfs cache: bsize 800 iosize 4000
blocks 4137327 free 2992353 avail 2785487
  files 1088638 ffiles 1055178 favail 1055178
  f_fsidx {0x407, 0xde832573} owner 0 ctime 0x606a11a1
  syncwrites 0 asyncwrites = 0
  syncreads 406 asyncreads = 0
  fstype "ffs" mnton "/usr/local" mntfrom "/dev/sd0h" mntspec "31fb99ccf4db16db
.h"
locked vnodes:
mountpoint 0xffffff800b376400
flags 4209018<NOSUID,NODEV,LOCAL,NOATIME>
vnodecovered 0xffffff800aef7800 syncer 0xffffff800aef7720 data 0xffffff800b3896
00
vfsconf: ops 0xffffff80006aff20 name "ffs" num 1 ref 9 flags 0x201000
statvfs cache: bsize 800 iosize 4000
blocks 3044751 free 3002424 avail 2850187
  files 803518 ffiles 801841 favail 801841
  f_fsidx {0x40a, 0x34be98cb} owner 0 ctime 0x606a11a1
  syncwrites 2 asyncwrites = 0
  syncreads 1 asyncreads = 0
  fstype "ffs" mnton "/usr/obj" mntfrom "/dev/sd0k" mntspec "31fb99ccf4db16db.k
"
locked vnodes:
mountpoint 0xffffff800b38c800
flags 4209018<NOSUID,NODEV,LOCAL,NOATIME>
vnodecovered 0xffffff800aef7560 syncer 0xffffff800aef7480 data 0xffffff800b389a
00
vfsconf: ops 0xffffff80006aff20 name "ffs" num 1 ref 9 flags 0x201000
statvfs cache: bsize 800 iosize 4000
blocks 1012719 free 390134 avail 339499
  files 285118 ffiles 187377 favail 187377
  f_fsidx {0x409, 0x2b0bf7a1} owner 0 ctime 0x606a11a1
  syncwrites 2 asyncwrites = 0
  syncreads 75 asyncreads = 0
  fstype "ffs" mnton "/usr/src" mntfrom "/dev/sd0j" mntspec "31fb99ccf4db16db.j
"
locked vnodes:
mountpoint 0xffffff800b38cc00
flags 4209018<NOSUID,NODEV,LOCAL,NOATIME>
vnodecovered 0xffffff800af25d28 syncer 0xffffff800aef72c0 data 0xffffff800b389e
00
vfsconf: ops 0xffffff80006aff20 name "ffs" num 1 ref 9 flags 0x201000
statvfs cache: bsize 800 iosize 4000
blocks 3186583 free 3175597 avail 3016268
  files 829438 ffiles 828074 favail 828074
  f_fsidx {0x404, 0x2c941ea7} owner 0 ctime 0x606a11a1
  syncwrites 236 asyncwrites = 418
  syncreads 3904 asyncreads = 0
  fstype "ffs" mnton "/var" mntfrom "/dev/sd0e" mntspec "31fb99ccf4db16db.e"
locked vnodes:
  0xffffff800ad96ae8