lock order reversal - msync

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

lock order reversal - msync

David Hill-3
Hello -

I enabled witness(4) and vfs lock in hopes to discover a hardlock after
Oct 29.  I drop into the debugger on boot when nsd(8) starts:

OpenBSD 6.2-current (GENERIC.MP) #37: Thu Nov  9 09:52:26 EST 2017
    [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 17032458240 (16243MB)
avail mem = 16378048512 (15619MB)
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.7 @ 0xeb450 (75 entries)
bios0: vendor American Megatrends Inc. version "F15" date 10/23/2013
bios0: Gigabyte Technology Co., Ltd. B75M-D3H
acpi0 at bios0: rev 2
acpi0: sleep states S0 S3 S4 S5
acpi0: tables DSDT FACP APIC MCFG HPET SSDT SSDT SSDT DMAR
acpi0: wakeup devices PS2K(S3) P0P1(S4) USB1(S3) USB2(S3) USB3(S3) USB4(S3) USB5(S3) USB6(S3) USB7(S3) PXSX(S4) RP01(
S4) PXSX(S4) RP02(S4) PXSX(S4) RP03(S4) PXSX(S4) [...]
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz, 3392.78 MHz
cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT
,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLIN
E,AES,XSAVE,AVX,F16C,RDRAND,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,SENSOR,ARAT
cpu0: 256KB 64b/line 8-way L2 cache
acpitimer0: recalibrated TSC frequency 3392291239 Hz
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges
cpu0: apic clock running at 99MHz
cpu0: mwait min=64, max=64, C-substates=0.2.1.1, IBE
cpu1 at mainbus0: apid 2 (application processor)
cpu1: Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz, 3392.29 MHz
cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT
,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLIN
E,AES,XSAVE,AVX,F16C,RDRAND,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,SENSOR,ARAT
cpu1: 256KB 64b/line 8-way L2 cache
cpu1: smt 0, core 1, package 0
cpu2 at mainbus0: apid 4 (application processor)
cpu2: Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz, 3392.29 MHz
cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT
,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,SENSOR,ARAT
cpu2: 256KB 64b/line 8-way L2 cache
cpu2: smt 0, core 2, package 0
cpu3 at mainbus0: apid 6 (application processor)
cpu3: Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz, 3392.29 MHz
cpu3: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,SENSOR,ARAT
cpu3: 256KB 64b/line 8-way L2 cache
cpu3: smt 0, core 3, package 0
cpu4 at mainbus0: apid 1 (application processor)
cpu4: Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz, 3392.29 MHz
cpu4: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,SENSOR,ARAT
cpu4: 256KB 64b/line 8-way L2 cache
cpu4: smt 1, core 0, package 0
cpu5 at mainbus0: apid 3 (application processor)
cpu5: Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz, 3392.29 MHz
cpu5: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,SENSOR,ARAT
cpu5: 256KB 64b/line 8-way L2 cache
cpu5: smt 1, core 1, package 0
cpu6 at mainbus0: apid 5 (application processor)
cpu6: Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz, 3392.29 MHz
cpu6: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,SENSOR,ARAT
cpu6: 256KB 64b/line 8-way L2 cache
cpu6: smt 1, core 2, package 0
cpu7 at mainbus0: apid 7 (application processor)
cpu7: Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz, 3392.29 MHz
cpu7: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,SENSOR,ARAT
cpu7: 256KB 64b/line 8-way L2 cache
cpu7: smt 1, core 3, package 0
ioapic0 at mainbus0: apid 2 pa 0xfec00000, version 20, 24 pins
acpimcfg0 at acpi0 addr 0xf8000000, bus 0-63
acpihpet0 at acpi0: 14318179 Hz
acpihpet0: recalibrated TSC frequency 3392306800 Hz
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus 5 (P0P1)
acpiprt2 at acpi0: bus 1 (RP01)
acpiprt3 at acpi0: bus -1 (RP02)
acpiprt4 at acpi0: bus -1 (RP03)
acpiprt5 at acpi0: bus -1 (RP04)
acpiprt6 at acpi0: bus 4 (RP05)
acpiprt7 at acpi0: bus -1 (RP06)
acpiprt8 at acpi0: bus -1 (RP07)
acpiprt9 at acpi0: bus -1 (RP08)
acpiprt10 at acpi0: bus -1 (PEG0)
acpiprt11 at acpi0: bus -1 (PEG1)
acpiprt12 at acpi0: bus -1 (PEG2)
acpiprt13 at acpi0: bus -1 (PEG3)
acpiec0 at acpi0: not present
acpicpu0 at acpi0: C3(350@80 mwait.1@0x20), C1(1000@1 mwait.1), PSS
acpicpu1 at acpi0: C3(350@80 mwait.1@0x20), C1(1000@1 mwait.1), PSS
acpicpu2 at acpi0: C3(350@80 mwait.1@0x20), C1(1000@1 mwait.1), PSS
acpicpu3 at acpi0: C3(350@80 mwait.1@0x20), C1(1000@1 mwait.1), PSS
acpicpu4 at acpi0: C3(350@80 mwait.1@0x20), C1(1000@1 mwait.1), PSS
acpicpu5 at acpi0: C3(350@80 mwait.1@0x20), C1(1000@1 mwait.1), PSS
acpicpu6 at acpi0: C3(350@80 mwait.1@0x20), C1(1000@1 mwait.1), PSS
acpicpu7 at acpi0: C3(350@80 mwait.1@0x20), C1(1000@1 mwait.1), PSS
acpipwrres0 at acpi0: FN00, resource for FAN0
acpipwrres1 at acpi0: FN01, resource for FAN1
acpipwrres2 at acpi0: FN02, resource for FAN2
acpipwrres3 at acpi0: FN03, resource for FAN3
acpipwrres4 at acpi0: FN04, resource for FAN4
acpitz0 at acpi0: critical temperature is 106 degC
"INT3F0D" at acpi0 not configured
"PNP0400" at acpi0 not configured
acpibtn0 at acpi0: PWRB
"PNP0C0B" at acpi0 not configured
"PNP0C0B" at acpi0 not configured
"PNP0C0B" at acpi0 not configured
"PNP0C0B" at acpi0 not configured
"PNP0C0B" at acpi0 not configured
acpivideo0 at acpi0: GFX0
acpivout0 at acpivideo0: DD02
cpu0: Enhanced SpeedStep 3392 MHz: speeds: 3901, 3900, 3700, 3600, 3400, 3200, 3100, 2900, 2700, 2600, 2400, 2300, 2100, 1900, 1800, 1600 MHz
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel Core 3G Host" rev 0x09
inteldrm0 at pci0 dev 2 function 0 "Intel HD Graphics 4000" rev 0x09
drm0 at inteldrm0
inteldrm0: msi
inteldrm0: 1280x1024, 32bpp
wsdisplay0 at inteldrm0 mux 1: console (std, vt100 emulation)
wsdisplay0: screen 1-5 added (std, vt100 emulation)
xhci0 at pci0 dev 20 function 0 "Intel 7 Series xHCI" rev 0x04: msi
usb0 at xhci0: USB revision 3.0
uhub0 at usb0 configuration 1 interface 0 "Intel xHCI root hub" rev 3.00/1.00 addr 1
"Intel 7 Series MEI" rev 0x04 at pci0 dev 22 function 0 not configured
ehci0 at pci0 dev 26 function 0 "Intel 7 Series USB" rev 0x04: apic 2 int 16
usb1 at ehci0: USB revision 2.0
uhub1 at usb1 configuration 1 interface 0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
ppb0 at pci0 dev 28 function 0 "Intel 7 Series PCIE" rev 0xc4: msi
pci1 at ppb0 bus 1
ppb1 at pci1 dev 0 function 0 "Intel IOP333 PCIE-PCIX" rev 0x00
pci2 at ppb1 bus 2
mfi0 at pci2 dev 14 function 0 "Symbios Logic MegaRAID SAS 1064R" rev 0x00: apic 2 int 18
mfi0: "MegaRAID SAS 84016E", firmware 7.0.1-0084, 256MB cache
scsibus1 at mfi0: 64 targets
sd0 at scsibus1 targ 0 lun 0: <LSI, MegaRAID 84016E, 1.12> SCSI3 0/direct fixed naa.600605b001d5a4101c162d40411f21f2
sd0: 2858160MB, 512 bytes/sector, 5853511680 sectors
ppb2 at pci1 dev 0 function 2 "Intel IOP333 PCIE-PCIX" rev 0x00
pci3 at ppb2 bus 3
ppb3 at pci0 dev 28 function 4 "Intel 7 Series PCIE" rev 0xc4: msi
pci4 at ppb3 bus 4
re0 at pci4 dev 0 function 0 "Realtek 8168" rev 0x06: RTL8168E/8111E-VL (0x2c80), msi, address fc:aa:14:29:1d:1d
rgephy0 at re0 phy 7: RTL8169S/8110S/8211 PHY, rev. 5
ehci1 at pci0 dev 29 function 0 "Intel 7 Series USB" rev 0x04: apic 2 int 23
usb2 at ehci1: USB revision 2.0
uhub2 at usb2 configuration 1 interface 0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
ppb4 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0xa4
pci5 at ppb4 bus 5
fxp0 at pci5 dev 1 function 0 "Intel 8255x" rev 0x08, i82559: apic 2 int 19, address 00:0e:0c:35:70:48
inphy0 at fxp0 phy 1: i82555 10/100 PHY, rev. 4
pcib0 at pci0 dev 31 function 0 "Intel B75 LPC" rev 0x04
ahci0 at pci0 dev 31 function 2 "Intel 7 Series AHCI" rev 0x04: msi, AHCI 1.3
ahci0: port 0: 6.0Gb/s
scsibus2 at ahci0: 32 targets
sd1 at scsibus2 targ 0 lun 0: <ATA, Crucial_CT128MX1, MU01> SCSI3 0/direct fixed naa.500a07510d04dca2
sd1: 122104MB, 512 bytes/sector, 250069680 sectors, thin
ichiic0 at pci0 dev 31 function 3 "Intel 7 Series SMBus" rev 0x04: apic 2 int 18
iic0 at ichiic0
spdmem0 at iic0 addr 0x51: 8GB DDR3 SDRAM PC3-12800
spdmem1 at iic0 addr 0x53: 8GB DDR3 SDRAM PC3-12800
isa0 at pcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
pckbc0 at isa0 port 0x60/5 irq 1 irq 12
pckbd0 at pckbc0 (kbd slot)
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
lpt0 at isa0 port 0x378/4 irq 7
it0 at isa0 port 0x2e/2: IT8728F rev 1, EC port 0xa30
error: [drm:pid0:cpt_set_fifo_underrun_reporting] *ERROR* uncleared pch fifo underrun on pch transcoder A
error: [drm:pid0:intel_pch_fifo_underrun_irq_handler] *ERROR* PCH transcoder A FIFO underrun
uhidev0 at uhub0 port 2 configuration 1 interface 0 "American Power Conversion Back-UPS RS 700G FW:856.L5 -P.D USB FW:L5 -P" rev 2.00/0.90 addr 2
uhidev0: iclass 3/0, 146 report ids
upd0 at uhidev0
uhub3 at uhub1 port 1 configuration 1 interface 0 "Intel Rate Matching Hub" rev 2.00/0.00 addr 2
uhub4 at uhub2 port 1 configuration 1 interface 0 "Intel Rate Matching Hub" rev 2.00/0.00 addr 2
vscsi0 at root
scsibus3 at vscsi0: 256 targets
softraid0 at root
scsibus4 at softraid0: 256 targets
root on sd1a (7e3675a2dbcdd669.a) swap on sd1b dump on sd1b
lock order reversal:
 1st 0xffffff041c86cdd8 vmmaplk (&map->lock) @ /usr/src/sys/uvm/uvm_map.c:4288
 2nd 0xffffff03a140bd68 inode (&ip->i_lock) @ /usr/src/sys/ufs/ufs/ufs_vnops.c:1554
Stopped at      db_enter+0x5:   popq    %rbp
ddb{2}> db_enter(10,ffff800032f93e30,286,8,ffffffff81552a05,0) at db_enter+0x5
witness_checkorder(ffffff03a140bd58,ffffffff81876bd6,612,9,81,61232f93ed0) at witness_checkorder+0xaaa
_rw_enter(ffffff03a151a900,ffff8000ffff9410,ffffff039ea5abe0,ffffff03a151a900,ffffff03a140bd58,e578bfe1e390cb1d) at _rw_enter+0x55
_rrw_enter(81,ffffff03a140bd58,ffffffff8143e021,ffff800032f93f70,ffffff03a151a900,ffff8000ffff9410) at _rrw_enter+0x31
VOP_LOCK(e578bfe1e390cb1d,ffff8000ffff9410,ffffffff00000081,ffffff03a151a900,ffffffff8114a3b1,ffff800032f93fa0) at VOP_LOCK+0x31
vn_lock(3000,ffffff039ea5ac04,ffffff039ea5ac00,ffff8000000ce000,ffffffff814a6fd6,ffff800032f93fd0) at vn_lock+0x36
uvn_io(ffffff039ea5abe0,ffff800032f94134,ffff800032f94138,ffffff011929ec60,0,e578bfe1e390cb1d) at uvn_io+0x196
uvm_pager_put(0,ffffff039ea5abe0,3,3000,0,ffff0100000023) at uvm_pager_put+0xf6
uvn_flush(16,3,5b613545000,ffffff041c86cdc0,ffffff039ed6b2c8,e578bfe1e390cb1d) at uvn_flush+0x414
uvm_map_clean(ffff800032f94350,1000,ffff8000ffff9410,0,0,0) at uvm_map_clean+0x3b2
syscall() at syscall+0x26f
--- syscall (number 256) ---

syscall 256: msync

ddb{2}>    PID     TID   PPID    UID  S       FLAGS  WAIT
COMMAND
 58977  286365  74422     97  7    0x100010                nsd
 28780  356041  33889      0  7         0x3                unbound-ancho
r
 33889  483347  82627      0  3    0x10008b  pause         sh
 74422  127608  11923     97  7    0x100010                nsd
*11923  380801      1     97  7    0x100010                nsd
 15691  401174  60684      0  3        0x83  netcon2       nsd-control
 60684  325155      1      0  3    0x100089  pause         sh
 99327  483568  25389     74  7    0x100012                pflogd
 25389  397912      1      0  3        0x80  netio         pflogd
 82105   37970  79884      0  3    0x100083  nanosleep     sleep
 79884  274312      1      0  3    0x100089  pause         sh
 47206  259428  63258     73  3    0x100090  kqread        syslogd
 63258  318514      1      0  3    0x100082  netio         syslogd
  7284  432776  57552      0  3    0x100083  nanosleep     sleep
 57552  206272      1      0  3    0x100089  pause         sh
 83740  269622  91900    115  3    0x100092  kqread        slaacd
 94892   20474  91900    115  3    0x100092  kqread        slaacd
 91900  372270      1      0  3        0x80  kqread        slaacd
 82627   31822      1      0  3    0x10008b  pause         sh
 32205   11000      0      0  3     0x14200  pgzero        zerothread
 48666  278993      0      0  3     0x14200  aiodoned      aiodoned
 98945  425376      0      0  3     0x14200  syncer        update
 57380  211885      0      0  3     0x14200  cleaner       cleaner
 60523  485933      0      0  3     0x14200  reaper        reaper
 59005   88656      0      0  3     0x14200  pgdaemon      pagedaemon
 97138   22012      0      0  3     0x14200  bored         crynlk
 99275  229256      0      0  3     0x14200  bored         crypto
 22598  359088      0      0  3     0x14200  usbtsk        usbtask
 78577   73837      0      0  3     0x14200  usbatsk       usbatsk
 69923  464026      0      0  3     0x14200  bored         i915-hangchec
k
 83143  200995      0      0  3     0x14200  bored         i915-dp
 19762  395791      0      0  3     0x14200  bored         i915
 30743   95118      0      0  3  0x40014200  acpi0         acpi0
 21967   22061      0      0  7  0x40014200                idle7
 37037  122725      0      0  7  0x40014200                idle6
 54176   86948      0      0  3  0x40014200                idle5
 24075  170172      0      0  7  0x40014200                idle4
 22271  378890      0      0  3  0x40014200                idle3
 29153  398323      0      0  3  0x40014200                idle2
 21511  427093      0      0  3  0x40014200                idle1
 29105  218719      0      0  3     0x14200  bored         sensors
 36657  326264      0      0  3     0x14200  bored         softnet
 71913  427365      0      0  3     0x14200  bored         systqmp
 62318   22393      0      0  3     0x14200  bored         systq
 24917  294467      0      0  3  0x40014200  bored         softclock
 78233    7063      0      0  3  0x40014200                idle0
 84793    9847      0      0  3     0x14200  bored         sbar
     1  169260      0      0  3        0x82  wait          init
     0       0     -1      0  3     0x10200  scheduler     swapper


Reply | Threaded
Open this post in threaded view
|

Re: lock order reversal - msync

Mathieu --2
Hey,

Yeah I remember hitting one of those quite often a few weeks ago with
witness(4) enabled, a LOR between the inode lock and the vmmap lock.
My initial guess was that this was a false positive but the small amount
of time I dedicated to it wasn't enough to prove or disprove it...

Mathieu.

David Hill wrote:

> Hello -
>
> I enabled witness(4) and vfs lock in hopes to discover a hardlock after
> Oct 29.  I drop into the debugger on boot when nsd(8) starts:
>
> OpenBSD 6.2-current (GENERIC.MP) #37: Thu Nov  9 09:52:26 EST 2017
>     [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
> real mem = 17032458240 (16243MB)
> avail mem = 16378048512 (15619MB)
> mpath0 at root
> scsibus0 at mpath0: 256 targets
> mainbus0 at root
> bios0 at mainbus0: SMBIOS rev. 2.7 @ 0xeb450 (75 entries)
> bios0: vendor American Megatrends Inc. version "F15" date 10/23/2013
> bios0: Gigabyte Technology Co., Ltd. B75M-D3H
> acpi0 at bios0: rev 2
> acpi0: sleep states S0 S3 S4 S5
> acpi0: tables DSDT FACP APIC MCFG HPET SSDT SSDT SSDT DMAR
> acpi0: wakeup devices PS2K(S3) P0P1(S4) USB1(S3) USB2(S3) USB3(S3) USB4(S3) USB5(S3) USB6(S3) USB7(S3) PXSX(S4) RP01(
> S4) PXSX(S4) RP02(S4) PXSX(S4) RP03(S4) PXSX(S4) [...]
> acpitimer0 at acpi0: 3579545 Hz, 24 bits
> acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
> cpu0 at mainbus0: apid 0 (boot processor)
> cpu0: Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz, 3392.78 MHz
> cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT
> ,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLIN
> E,AES,XSAVE,AVX,F16C,RDRAND,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,SENSOR,ARAT
> cpu0: 256KB 64b/line 8-way L2 cache
> acpitimer0: recalibrated TSC frequency 3392291239 Hz
> cpu0: smt 0, core 0, package 0
> mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges
> cpu0: apic clock running at 99MHz
> cpu0: mwait min=64, max=64, C-substates=0.2.1.1, IBE
> cpu1 at mainbus0: apid 2 (application processor)
> cpu1: Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz, 3392.29 MHz
> cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT
> ,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLIN
> E,AES,XSAVE,AVX,F16C,RDRAND,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,SENSOR,ARAT
> cpu1: 256KB 64b/line 8-way L2 cache
> cpu1: smt 0, core 1, package 0
> cpu2 at mainbus0: apid 4 (application processor)
> cpu2: Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz, 3392.29 MHz
> cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT
> ,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,SENSOR,ARAT
> cpu2: 256KB 64b/line 8-way L2 cache
> cpu2: smt 0, core 2, package 0
> cpu3 at mainbus0: apid 6 (application processor)
> cpu3: Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz, 3392.29 MHz
> cpu3: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,SENSOR,ARAT
> cpu3: 256KB 64b/line 8-way L2 cache
> cpu3: smt 0, core 3, package 0
> cpu4 at mainbus0: apid 1 (application processor)
> cpu4: Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz, 3392.29 MHz
> cpu4: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,SENSOR,ARAT
> cpu4: 256KB 64b/line 8-way L2 cache
> cpu4: smt 1, core 0, package 0
> cpu5 at mainbus0: apid 3 (application processor)
> cpu5: Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz, 3392.29 MHz
> cpu5: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,SENSOR,ARAT
> cpu5: 256KB 64b/line 8-way L2 cache
> cpu5: smt 1, core 1, package 0
> cpu6 at mainbus0: apid 5 (application processor)
> cpu6: Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz, 3392.29 MHz
> cpu6: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,SENSOR,ARAT
> cpu6: 256KB 64b/line 8-way L2 cache
> cpu6: smt 1, core 2, package 0
> cpu7 at mainbus0: apid 7 (application processor)
> cpu7: Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz, 3392.29 MHz
> cpu7: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,SENSOR,ARAT
> cpu7: 256KB 64b/line 8-way L2 cache
> cpu7: smt 1, core 3, package 0
> ioapic0 at mainbus0: apid 2 pa 0xfec00000, version 20, 24 pins
> acpimcfg0 at acpi0 addr 0xf8000000, bus 0-63
> acpihpet0 at acpi0: 14318179 Hz
> acpihpet0: recalibrated TSC frequency 3392306800 Hz
> acpiprt0 at acpi0: bus 0 (PCI0)
> acpiprt1 at acpi0: bus 5 (P0P1)
> acpiprt2 at acpi0: bus 1 (RP01)
> acpiprt3 at acpi0: bus -1 (RP02)
> acpiprt4 at acpi0: bus -1 (RP03)
> acpiprt5 at acpi0: bus -1 (RP04)
> acpiprt6 at acpi0: bus 4 (RP05)
> acpiprt7 at acpi0: bus -1 (RP06)
> acpiprt8 at acpi0: bus -1 (RP07)
> acpiprt9 at acpi0: bus -1 (RP08)
> acpiprt10 at acpi0: bus -1 (PEG0)
> acpiprt11 at acpi0: bus -1 (PEG1)
> acpiprt12 at acpi0: bus -1 (PEG2)
> acpiprt13 at acpi0: bus -1 (PEG3)
> acpiec0 at acpi0: not present
> acpicpu0 at acpi0: C3(350@80 mwait.1@0x20), C1(1000@1 mwait.1), PSS
> acpicpu1 at acpi0: C3(350@80 mwait.1@0x20), C1(1000@1 mwait.1), PSS
> acpicpu2 at acpi0: C3(350@80 mwait.1@0x20), C1(1000@1 mwait.1), PSS
> acpicpu3 at acpi0: C3(350@80 mwait.1@0x20), C1(1000@1 mwait.1), PSS
> acpicpu4 at acpi0: C3(350@80 mwait.1@0x20), C1(1000@1 mwait.1), PSS
> acpicpu5 at acpi0: C3(350@80 mwait.1@0x20), C1(1000@1 mwait.1), PSS
> acpicpu6 at acpi0: C3(350@80 mwait.1@0x20), C1(1000@1 mwait.1), PSS
> acpicpu7 at acpi0: C3(350@80 mwait.1@0x20), C1(1000@1 mwait.1), PSS
> acpipwrres0 at acpi0: FN00, resource for FAN0
> acpipwrres1 at acpi0: FN01, resource for FAN1
> acpipwrres2 at acpi0: FN02, resource for FAN2
> acpipwrres3 at acpi0: FN03, resource for FAN3
> acpipwrres4 at acpi0: FN04, resource for FAN4
> acpitz0 at acpi0: critical temperature is 106 degC
> "INT3F0D" at acpi0 not configured
> "PNP0400" at acpi0 not configured
> acpibtn0 at acpi0: PWRB
> "PNP0C0B" at acpi0 not configured
> "PNP0C0B" at acpi0 not configured
> "PNP0C0B" at acpi0 not configured
> "PNP0C0B" at acpi0 not configured
> "PNP0C0B" at acpi0 not configured
> acpivideo0 at acpi0: GFX0
> acpivout0 at acpivideo0: DD02
> cpu0: Enhanced SpeedStep 3392 MHz: speeds: 3901, 3900, 3700, 3600, 3400, 3200, 3100, 2900, 2700, 2600, 2400, 2300, 2100, 1900, 1800, 1600 MHz
> pci0 at mainbus0 bus 0
> pchb0 at pci0 dev 0 function 0 "Intel Core 3G Host" rev 0x09
> inteldrm0 at pci0 dev 2 function 0 "Intel HD Graphics 4000" rev 0x09
> drm0 at inteldrm0
> inteldrm0: msi
> inteldrm0: 1280x1024, 32bpp
> wsdisplay0 at inteldrm0 mux 1: console (std, vt100 emulation)
> wsdisplay0: screen 1-5 added (std, vt100 emulation)
> xhci0 at pci0 dev 20 function 0 "Intel 7 Series xHCI" rev 0x04: msi
> usb0 at xhci0: USB revision 3.0
> uhub0 at usb0 configuration 1 interface 0 "Intel xHCI root hub" rev 3.00/1.00 addr 1
> "Intel 7 Series MEI" rev 0x04 at pci0 dev 22 function 0 not configured
> ehci0 at pci0 dev 26 function 0 "Intel 7 Series USB" rev 0x04: apic 2 int 16
> usb1 at ehci0: USB revision 2.0
> uhub1 at usb1 configuration 1 interface 0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
> ppb0 at pci0 dev 28 function 0 "Intel 7 Series PCIE" rev 0xc4: msi
> pci1 at ppb0 bus 1
> ppb1 at pci1 dev 0 function 0 "Intel IOP333 PCIE-PCIX" rev 0x00
> pci2 at ppb1 bus 2
> mfi0 at pci2 dev 14 function 0 "Symbios Logic MegaRAID SAS 1064R" rev 0x00: apic 2 int 18
> mfi0: "MegaRAID SAS 84016E", firmware 7.0.1-0084, 256MB cache
> scsibus1 at mfi0: 64 targets
> sd0 at scsibus1 targ 0 lun 0: <LSI, MegaRAID 84016E, 1.12> SCSI3 0/direct fixed naa.600605b001d5a4101c162d40411f21f2
> sd0: 2858160MB, 512 bytes/sector, 5853511680 sectors
> ppb2 at pci1 dev 0 function 2 "Intel IOP333 PCIE-PCIX" rev 0x00
> pci3 at ppb2 bus 3
> ppb3 at pci0 dev 28 function 4 "Intel 7 Series PCIE" rev 0xc4: msi
> pci4 at ppb3 bus 4
> re0 at pci4 dev 0 function 0 "Realtek 8168" rev 0x06: RTL8168E/8111E-VL (0x2c80), msi, address fc:aa:14:29:1d:1d
> rgephy0 at re0 phy 7: RTL8169S/8110S/8211 PHY, rev. 5
> ehci1 at pci0 dev 29 function 0 "Intel 7 Series USB" rev 0x04: apic 2 int 23
> usb2 at ehci1: USB revision 2.0
> uhub2 at usb2 configuration 1 interface 0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
> ppb4 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0xa4
> pci5 at ppb4 bus 5
> fxp0 at pci5 dev 1 function 0 "Intel 8255x" rev 0x08, i82559: apic 2 int 19, address 00:0e:0c:35:70:48
> inphy0 at fxp0 phy 1: i82555 10/100 PHY, rev. 4
> pcib0 at pci0 dev 31 function 0 "Intel B75 LPC" rev 0x04
> ahci0 at pci0 dev 31 function 2 "Intel 7 Series AHCI" rev 0x04: msi, AHCI 1.3
> ahci0: port 0: 6.0Gb/s
> scsibus2 at ahci0: 32 targets
> sd1 at scsibus2 targ 0 lun 0: <ATA, Crucial_CT128MX1, MU01> SCSI3 0/direct fixed naa.500a07510d04dca2
> sd1: 122104MB, 512 bytes/sector, 250069680 sectors, thin
> ichiic0 at pci0 dev 31 function 3 "Intel 7 Series SMBus" rev 0x04: apic 2 int 18
> iic0 at ichiic0
> spdmem0 at iic0 addr 0x51: 8GB DDR3 SDRAM PC3-12800
> spdmem1 at iic0 addr 0x53: 8GB DDR3 SDRAM PC3-12800
> isa0 at pcib0
> isadma0 at isa0
> com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
> pckbc0 at isa0 port 0x60/5 irq 1 irq 12
> pckbd0 at pckbc0 (kbd slot)
> wskbd0 at pckbd0: console keyboard, using wsdisplay0
> pcppi0 at isa0 port 0x61
> spkr0 at pcppi0
> lpt0 at isa0 port 0x378/4 irq 7
> it0 at isa0 port 0x2e/2: IT8728F rev 1, EC port 0xa30
> error: [drm:pid0:cpt_set_fifo_underrun_reporting] *ERROR* uncleared pch fifo underrun on pch transcoder A
> error: [drm:pid0:intel_pch_fifo_underrun_irq_handler] *ERROR* PCH transcoder A FIFO underrun
> uhidev0 at uhub0 port 2 configuration 1 interface 0 "American Power Conversion Back-UPS RS 700G FW:856.L5 -P.D USB FW:L5 -P" rev 2.00/0.90 addr 2
> uhidev0: iclass 3/0, 146 report ids
> upd0 at uhidev0
> uhub3 at uhub1 port 1 configuration 1 interface 0 "Intel Rate Matching Hub" rev 2.00/0.00 addr 2
> uhub4 at uhub2 port 1 configuration 1 interface 0 "Intel Rate Matching Hub" rev 2.00/0.00 addr 2
> vscsi0 at root
> scsibus3 at vscsi0: 256 targets
> softraid0 at root
> scsibus4 at softraid0: 256 targets
> root on sd1a (7e3675a2dbcdd669.a) swap on sd1b dump on sd1b
> lock order reversal:
>  1st 0xffffff041c86cdd8 vmmaplk (&map->lock) @ /usr/src/sys/uvm/uvm_map.c:4288
>  2nd 0xffffff03a140bd68 inode (&ip->i_lock) @ /usr/src/sys/ufs/ufs/ufs_vnops.c:1554
> Stopped at      db_enter+0x5:   popq    %rbp
> ddb{2}> db_enter(10,ffff800032f93e30,286,8,ffffffff81552a05,0) at db_enter+0x5
> witness_checkorder(ffffff03a140bd58,ffffffff81876bd6,612,9,81,61232f93ed0) at witness_checkorder+0xaaa
> _rw_enter(ffffff03a151a900,ffff8000ffff9410,ffffff039ea5abe0,ffffff03a151a900,ffffff03a140bd58,e578bfe1e390cb1d) at _rw_enter+0x55
> _rrw_enter(81,ffffff03a140bd58,ffffffff8143e021,ffff800032f93f70,ffffff03a151a900,ffff8000ffff9410) at _rrw_enter+0x31
> VOP_LOCK(e578bfe1e390cb1d,ffff8000ffff9410,ffffffff00000081,ffffff03a151a900,ffffffff8114a3b1,ffff800032f93fa0) at VOP_LOCK+0x31
> vn_lock(3000,ffffff039ea5ac04,ffffff039ea5ac00,ffff8000000ce000,ffffffff814a6fd6,ffff800032f93fd0) at vn_lock+0x36
> uvn_io(ffffff039ea5abe0,ffff800032f94134,ffff800032f94138,ffffff011929ec60,0,e578bfe1e390cb1d) at uvn_io+0x196
> uvm_pager_put(0,ffffff039ea5abe0,3,3000,0,ffff0100000023) at uvm_pager_put+0xf6
> uvn_flush(16,3,5b613545000,ffffff041c86cdc0,ffffff039ed6b2c8,e578bfe1e390cb1d) at uvn_flush+0x414
> uvm_map_clean(ffff800032f94350,1000,ffff8000ffff9410,0,0,0) at uvm_map_clean+0x3b2
> syscall() at syscall+0x26f
> --- syscall (number 256) ---
>
> syscall 256: msync
>
> ddb{2}>    PID     TID   PPID    UID  S       FLAGS  WAIT
> COMMAND
>  58977  286365  74422     97  7    0x100010                nsd
>  28780  356041  33889      0  7         0x3                unbound-ancho
> r
>  33889  483347  82627      0  3    0x10008b  pause         sh
>  74422  127608  11923     97  7    0x100010                nsd
> *11923  380801      1     97  7    0x100010                nsd
>  15691  401174  60684      0  3        0x83  netcon2       nsd-control
>  60684  325155      1      0  3    0x100089  pause         sh
>  99327  483568  25389     74  7    0x100012                pflogd
>  25389  397912      1      0  3        0x80  netio         pflogd
>  82105   37970  79884      0  3    0x100083  nanosleep     sleep
>  79884  274312      1      0  3    0x100089  pause         sh
>  47206  259428  63258     73  3    0x100090  kqread        syslogd
>  63258  318514      1      0  3    0x100082  netio         syslogd
>   7284  432776  57552      0  3    0x100083  nanosleep     sleep
>  57552  206272      1      0  3    0x100089  pause         sh
>  83740  269622  91900    115  3    0x100092  kqread        slaacd
>  94892   20474  91900    115  3    0x100092  kqread        slaacd
>  91900  372270      1      0  3        0x80  kqread        slaacd
>  82627   31822      1      0  3    0x10008b  pause         sh
>  32205   11000      0      0  3     0x14200  pgzero        zerothread
>  48666  278993      0      0  3     0x14200  aiodoned      aiodoned
>  98945  425376      0      0  3     0x14200  syncer        update
>  57380  211885      0      0  3     0x14200  cleaner       cleaner
>  60523  485933      0      0  3     0x14200  reaper        reaper
>  59005   88656      0      0  3     0x14200  pgdaemon      pagedaemon
>  97138   22012      0      0  3     0x14200  bored         crynlk
>  99275  229256      0      0  3     0x14200  bored         crypto
>  22598  359088      0      0  3     0x14200  usbtsk        usbtask
>  78577   73837      0      0  3     0x14200  usbatsk       usbatsk
>  69923  464026      0      0  3     0x14200  bored         i915-hangchec
> k
>  83143  200995      0      0  3     0x14200  bored         i915-dp
>  19762  395791      0      0  3     0x14200  bored         i915
>  30743   95118      0      0  3  0x40014200  acpi0         acpi0
>  21967   22061      0      0  7  0x40014200                idle7
>  37037  122725      0      0  7  0x40014200                idle6
>  54176   86948      0      0  3  0x40014200                idle5
>  24075  170172      0      0  7  0x40014200                idle4
>  22271  378890      0      0  3  0x40014200                idle3
>  29153  398323      0      0  3  0x40014200                idle2
>  21511  427093      0      0  3  0x40014200                idle1
>  29105  218719      0      0  3     0x14200  bored         sensors
>  36657  326264      0      0  3     0x14200  bored         softnet
>  71913  427365      0      0  3     0x14200  bored         systqmp
>  62318   22393      0      0  3     0x14200  bored         systq
>  24917  294467      0      0  3  0x40014200  bored         softclock
>  78233    7063      0      0  3  0x40014200                idle0
>  84793    9847      0      0  3     0x14200  bored         sbar
>      1  169260      0      0  3        0x82  wait          init
>      0       0     -1      0  3     0x10200  scheduler     swapper
>
>