lock order reversal when starting net/knot

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

lock order reversal when starting net/knot

Jeremie Courreges-Anglas-2

>Synopsis: lock order reversal when starting net/knot
>Category: kernel
>Environment:
        System      : OpenBSD 6.3
        Details     : OpenBSD 6.3-current (GENERIC.MP) #125: Wed Jul 11 10:18:39 MDT 2018
                         [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP

        Architecture: OpenBSD.amd64
        Machine     : amd64
>Description:
        When I run knot (an authoritative name server daemon that uses
        LMDB) the following entries appear in dmesg:

lock order reversal:
 1st 0xffffff01ff732468 vmmaplk (&map->lock) @ /usr/src/sys/uvm/uvm_map.c:4433
 2nd 0xffffff01fe312098 inode (&ip->i_lock) @ /usr/src/sys/ufs/ufs/ufs_vnops.c:1544
lock order "&ip->i_lock"(rrwlock) -> "&map->lock"(rwlock) first seen at:
#0  witness_checkorder+0x4c0
#1  _rw_enter+0x68
#2  vm_map_lock_ln+0xbc
#3  uvm_map+0x1a1
#4  km_alloc+0x16a
#5  pool_multi_alloc_ni+0xbb
#6  pool_p_alloc+0x56
#7  pool_do_get+0xe4
#8  pool_get+0xaf
#9  ufsdirhash_build+0x31e
#10 ufs_lookup+0x19d
#11 VOP_LOOKUP+0x4f
#12 vfs_lookup+0x27e
#13 namei+0x226
#14 start_init+0xb2
lock order "&map->lock"(rwlock) -> "&ip->i_lock"(rrwlock) first seen at:
#0  witness_checkorder+0x4c0
#1  _rw_enter+0x68
#2  _rrw_enter+0x3e
#3  VOP_LOCK+0x3d
#4  vn_lock+0x34
#5  uvn_io+0x1b8
#6  uvm_pager_put+0x109
#7  uvn_flush+0x424
#8  uvm_map_clean+0x3e7
#9  syscall+0x32a
#10 Xsyscall+0xef

fwiw this looks very similar to the lock order traces seen by Vijay
Sankar when testing an update to samba-4.8.3 (ports/net/samba):

  https://marc.info/?l=openbsd-ports&m=153118057112027&w=2

>How-To-Repeat:
        Start knot (at boot time or from a shell)
>Fix:
        Unknown


dmesg:
OpenBSD 6.3-current (GENERIC.MP) #125: Wed Jul 11 10:18:39 MDT 2018
    [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 8573018112 (8175MB)
avail mem = 8234561536 (7853MB)
pmap_enter_special: special diffing mapping at ffffffff819ff000
enter_shared_special_pages: entered kutext.page va 0xffffffff819ff000 pa 0x1c3c000
pmap_enter_special: special diffing mapping at ffffffff81a00000
enter_shared_special_pages: entered kutext.page va 0xffffffff81a00000 pa 0x1c3d000
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.8 @ 0xf6890 (11 entries)
bios0: vendor SeaBIOS version "1.10.2-1" date 04/01/2014
bios0: QEMU Standard PC (i440FX + PIIX, 1996)
acpi0 at bios0: rev 0
acpi0: sleep states S3 S4 S5
acpi0: tables DSDT FACP APIC
acpi0: wakeup devices
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
pmap_enter_special: special diffing mapping at ffffffff81cb1000
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel Core Processor (Broadwell), 3408.45 MHz
cpu0: FPU,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SSE3,PCLMUL,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,OSXSAVE,AVX,HV,NXE,RDTSCP,LONG,LAHF,3DNOWP,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,RDSEED,ADX,SMAP,XSAVEOPT,MELTDOWN
cpu0: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 512KB 64b/line 16-way L2 cache
cpu0: ITLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu0: DTLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
cpu0: apic clock running at 999MHz
cpu1 at mainbus0: apid 1 (application processor)
cpu1: Intel Core Processor (Broadwell), 3408.03 MHz
cpu1: FPU,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SSE3,PCLMUL,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,OSXSAVE,AVX,HV,NXE,RDTSCP,LONG,LAHF,3DNOWP,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,RDSEED,ADX,SMAP,XSAVEOPT,MELTDOWN
cpu1: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 512KB 64b/line 16-way L2 cache
cpu1: ITLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu1: DTLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu1: smt 0, core 0, package 1
ioapic0 at mainbus0: apid 0 pa 0xfec00000, version 11, 24 pins
acpiprt0 at acpi0: bus 0 (PCI0)
acpicpu0 at acpi0: C1(@1 halt!)
acpicpu1 at acpi0: C1(@1 halt!)
"ACPI0006" at acpi0 not configured
acpicmos0 at acpi0
"PNP0A06" at acpi0 not configured
"PNP0A06" at acpi0 not configured
"PNP0A06" at acpi0 not configured
"QEMU0002" at acpi0 not configured
pvbus0 at mainbus0: KVM
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel 82441FX" rev 0x02
pcib0 at pci0 dev 1 function 0 "Intel 82371SB ISA" rev 0x00
pciide0 at pci0 dev 1 function 1 "Intel 82371SB IDE" rev 0x00: DMA, channel 0 wired to compatibility, channel 1 wired to compatibility
atapiscsi0 at pciide0 channel 0 drive 0
scsibus1 at atapiscsi0: 2 targets
cd0 at scsibus1 targ 0 lun 0: <QEMU, QEMU DVD-ROM, 2.1.> ATAPI 5/cdrom removable
cd0(pciide0:0:0): using PIO mode 4, DMA mode 2
pciide0: channel 1 disabled (no drives)
piixpm0 at pci0 dev 1 function 3 "Intel 82371AB Power" rev 0x03: apic 0 int 9
iic0 at piixpm0
virtio0 at pci0 dev 2 function 0 "Qumranet Virtio Network" rev 0x00
vio0 at virtio0: address 52:54:00:12:01:90
virtio0: msix shared
virtio1 at pci0 dev 4 function 0 "Qumranet Virtio Storage" rev 0x00
vioblk0 at virtio1
scsibus2 at vioblk0: 2 targets
sd0 at scsibus2 targ 0 lun 0: <VirtIO, Block Device, > SCSI3 0/direct fixed
sd0: 256000MB, 512 bytes/sector, 524288000 sectors
virtio1: msix shared
virtio2 at pci0 dev 5 function 0 "Qumranet Virtio Memory" rev 0x00
viomb0 at virtio2
virtio2: apic 0 int 10
virtio3 at pci0 dev 6 function 0 "Qumranet Virtio RNG" rev 0x00
viornd0 at virtio3
virtio3: apic 0 int 10
isa0 at pcib0
isadma0 at isa0
fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
com0: console
pckbc0 at isa0 port 0x60/5 irq 1 irq 12
pckbd0 at pckbc0 (kbd slot)
wskbd0 at pckbd0 mux 1
pms0 at pckbc0 (aux slot)
wsmouse0 at pms0 mux 0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
vscsi0 at root
scsibus3 at vscsi0: 256 targets
softraid0 at root
scsibus4 at softraid0: 256 targets
root on sd0a (871b81ad3a5d5c9e.a) swap on sd0b dump on sd0b
fd0 at fdc0 drive 1: density unknown

usbdevs:
usbdevs: no USB controllers found

pcidump:
Domain /dev/pci0:
 0:0:0: Intel 82441FX
        0x0000: Vendor ID: 8086 Product ID: 1237
        0x0004: Command: 0103 Status: 0000
        0x0008: Class: 06 Subclass: 00 Interface: 00 Revision: 02
        0x000c: BIST: 00 Header Type: 00 Latency Timer: 00 Cache Line Size: 00
        0x0010: BAR empty (00000000)
        0x0014: BAR empty (00000000)
        0x0018: BAR empty (00000000)
        0x001c: BAR empty (00000000)
        0x0020: BAR empty (00000000)
        0x0024: BAR empty (00000000)
        0x0028: Cardbus CIS: 00000000
        0x002c: Subsystem Vendor ID: 1af4 Product ID: 1100
        0x0030: Expansion ROM Base Address: 00000000
        0x0038: 00000000
        0x003c: Interrupt Pin: 00 Line: 00 Min Gnt: 00 Max Lat: 00
        0x0000: 12378086 00000103 06000002 00000000
        0x0010: 00000000 00000000 00000000 00000000
        0x0020: 00000000 00000000 00000000 11001af4
        0x0030: 00000000 00000000 00000000 00000000
        0x0040: 00000000 00000000 00000000 00000000
        0x0050: 00000000 ff000000 11111000 31111111
        0x0060: 00000000 00000000 00000000 00000000
        0x0070: 00020000 00000000 00000000 00000000
        0x0080: 00000000 00000000 00000000 00000000
        0x0090: 00000000 00000000 00000000 00000000
        0x00a0: 00000000 00000000 00000000 00000000
        0x00b0: 00000000 00000000 00000000 00000000
        0x00c0: 00000000 00000000 00000000 00000000
        0x00d0: 00000000 00000000 00000000 00000000
        0x00e0: 00000000 00000000 00000000 00000000
        0x00f0: 00000000 00000000 00000000 00000000
 0:1:0: Intel 82371SB ISA
        0x0000: Vendor ID: 8086 Product ID: 7000
        0x0004: Command: 0103 Status: 0200
        0x0008: Class: 06 Subclass: 01 Interface: 00 Revision: 00
        0x000c: BIST: 00 Header Type: 80 Latency Timer: 00 Cache Line Size: 00
        0x0010: BAR empty (00000000)
        0x0014: BAR empty (00000000)
        0x0018: BAR empty (00000000)
        0x001c: BAR empty (00000000)
        0x0020: BAR empty (00000000)
        0x0024: BAR empty (00000000)
        0x0028: Cardbus CIS: 00000000
        0x002c: Subsystem Vendor ID: 1af4 Product ID: 1100
        0x0030: Expansion ROM Base Address: 00000000
        0x0038: 00000000
        0x003c: Interrupt Pin: 00 Line: 00 Min Gnt: 00 Max Lat: 00
        0x0000: 70008086 02000103 06010000 00800000
        0x0010: 00000000 00000000 00000000 00000000
        0x0020: 00000000 00000000 00000000 11001af4
        0x0030: 00000000 00000000 00000000 00000000
        0x0040: 00000000 00000000 00000000 0003004d
        0x0050: 00000000 00000000 00000000 00000000
        0x0060: 0b0b0a0a 00000000 00000200 00000000
        0x0070: 00000080 0c0c0000 00000002 00000000
        0x0080: 00000000 00000000 00000000 00000000
        0x0090: 00000000 00000000 00000000 00000000
        0x00a0: 00000008 00000000 0000000f 00000000
        0x00b0: 00000000 00000000 00000000 00000000
        0x00c0: 00000000 00000000 00000000 00000000
        0x00d0: 00000000 00000000 00000000 00000000
        0x00e0: 00000000 00000000 00000000 00000000
        0x00f0: 00000000 00000000 00000000 00000000
 0:1:1: Intel 82371SB IDE
        0x0000: Vendor ID: 8086 Product ID: 7010
        0x0004: Command: 0107 Status: 0280
        0x0008: Class: 01 Subclass: 01 Interface: 80 Revision: 00
        0x000c: BIST: 00 Header Type: 00 Latency Timer: 00 Cache Line Size: 00
        0x0010: BAR empty (00000000)
        0x0014: BAR empty (00000000)
        0x0018: BAR empty (00000000)
        0x001c: BAR empty (00000000)
        0x0020: BAR io addr: 0x0000c0a0/0x0010
        0x0024: BAR empty (00000000)
        0x0028: Cardbus CIS: 00000000
        0x002c: Subsystem Vendor ID: 1af4 Product ID: 1100
        0x0030: Expansion ROM Base Address: 00000000
        0x0038: 00000000
        0x003c: Interrupt Pin: 00 Line: 00 Min Gnt: 00 Max Lat: 00
        0x0000: 70108086 02800107 01018000 00000000
        0x0010: 00000000 00000000 00000000 00000000
        0x0020: 0000c0a1 00000000 00000000 11001af4
        0x0030: 00000000 00000000 00000000 00000000
        0x0040: 0000a307 00000000 00000000 00000000
        0x0050: 00000000 00000000 00000000 00000000
        0x0060: 00000000 00000000 00000000 00000000
        0x0070: 00000000 00000000 00000000 00000000
        0x0080: 00000000 00000000 00000000 00000000
        0x0090: 00000000 00000000 00000000 00000000
        0x00a0: 00000000 00000000 00000000 00000000
        0x00b0: 00000000 00000000 00000000 00000000
        0x00c0: 00000000 00000000 00000000 00000000
        0x00d0: 00000000 00000000 00000000 00000000
        0x00e0: 00000000 00000000 00000000 00000000
        0x00f0: 00000000 00000000 00000000 00000000
 0:1:3: Intel 82371AB Power
        0x0000: Vendor ID: 8086 Product ID: 7113
        0x0004: Command: 0103 Status: 0280
        0x0008: Class: 06 Subclass: 80 Interface: 00 Revision: 03
        0x000c: BIST: 00 Header Type: 00 Latency Timer: 00 Cache Line Size: 00
        0x0010: BAR empty (00000000)
        0x0014: BAR empty (00000000)
        0x0018: BAR empty (00000000)
        0x001c: BAR empty (00000000)
        0x0020: BAR empty (00000000)
        0x0024: BAR empty (00000000)
        0x0028: Cardbus CIS: 00000000
        0x002c: Subsystem Vendor ID: 1af4 Product ID: 1100
        0x0030: Expansion ROM Base Address: 00000000
        0x0038: 00000000
        0x003c: Interrupt Pin: 01 Line: 09 Min Gnt: 00 Max Lat: 00
        0x0000: 71138086 02800103 06800003 00000000
        0x0010: 00000000 00000000 00000000 00000000
        0x0020: 00000000 00000000 00000000 11001af4
        0x0030: 00000000 00000000 00000000 00000109
        0x0040: 00000601 00000000 00000000 00000000
        0x0050: 00000000 00000000 02000000 10000000
        0x0060: 60000000 08000000 00000000 00000000
        0x0070: 00000000 00000000 00000000 00000000
        0x0080: 00000001 00000000 00000000 00000000
        0x0090: 00000701 00000000 00000000 00000000
        0x00a0: 00000000 00000000 00000000 00000000
        0x00b0: 00000000 00000000 00000000 00000000
        0x00c0: 00000000 00000000 00000000 00000000
        0x00d0: 00090000 00000000 00000000 00000000
        0x00e0: 00000000 00000000 00000000 00000000
        0x00f0: 00000000 00000000 00000000 00000000
 0:2:0: Qumranet Virtio Network
        0x0000: Vendor ID: 1af4 Product ID: 1000
        0x0004: Command: 0107 Status: 0010
        0x0008: Class: 02 Subclass: 00 Interface: 00 Revision: 00
        0x000c: BIST: 00 Header Type: 00 Latency Timer: 00 Cache Line Size: 00
        0x0010: BAR io addr: 0x0000c040/0x0020
        0x0014: BAR mem 32bit addr: 0xfebc0000/0x00001000
        0x0018: BAR empty (00000000)
        0x001c: BAR empty (00000000)
        0x0020: BAR empty (00000000)
        0x0024: BAR empty (00000000)
        0x0028: Cardbus CIS: 00000000
        0x002c: Subsystem Vendor ID: 1af4 Product ID: 0001
        0x0030: Expansion ROM Base Address: feb80000
        0x0038: 00000000
        0x003c: Interrupt Pin: 01 Line: 0a Min Gnt: 00 Max Lat: 00
        0x0040: Capability 0x11: Extended Message Signalled Interrupts (MSI-X)
        0x0000: 10001af4 00100107 02000000 00000000
        0x0010: 0000c041 febc0000 00000000 00000000
        0x0020: 00000000 00000000 00000000 00011af4
        0x0030: feb80000 00000040 00000000 0000010a
        0x0040: 80020011 00000001 00000801 00000000
        0x0050: 00000000 00000000 00000000 00000000
        0x0060: 00000000 00000000 00000000 00000000
        0x0070: 00000000 00000000 00000000 00000000
        0x0080: 00000000 00000000 00000000 00000000
        0x0090: 00000000 00000000 00000000 00000000
        0x00a0: 00000000 00000000 00000000 00000000
        0x00b0: 00000000 00000000 00000000 00000000
        0x00c0: 00000000 00000000 00000000 00000000
        0x00d0: 00000000 00000000 00000000 00000000
        0x00e0: 00000000 00000000 00000000 00000000
        0x00f0: 00000000 00000000 00000000 00000000
 0:4:0: Qumranet Virtio Storage
        0x0000: Vendor ID: 1af4 Product ID: 1001
        0x0004: Command: 0107 Status: 0010
        0x0008: Class: 01 Subclass: 00 Interface: 00 Revision: 00
        0x000c: BIST: 00 Header Type: 00 Latency Timer: 00 Cache Line Size: 00
        0x0010: BAR io addr: 0x0000c000/0x0040
        0x0014: BAR mem 32bit addr: 0xfebc1000/0x00001000
        0x0018: BAR empty (00000000)
        0x001c: BAR empty (00000000)
        0x0020: BAR empty (00000000)
        0x0024: BAR empty (00000000)
        0x0028: Cardbus CIS: 00000000
        0x002c: Subsystem Vendor ID: 1af4 Product ID: 0002
        0x0030: Expansion ROM Base Address: 00000000
        0x0038: 00000000
        0x003c: Interrupt Pin: 01 Line: 0b Min Gnt: 00 Max Lat: 00
        0x0040: Capability 0x11: Extended Message Signalled Interrupts (MSI-X)
        0x0000: 10011af4 00100107 01000000 00000000
        0x0010: 0000c001 febc1000 00000000 00000000
        0x0020: 00000000 00000000 00000000 00021af4
        0x0030: 00000000 00000040 00000000 0000010b
        0x0040: 80010011 00000001 00000801 00000000
        0x0050: 00000000 00000000 00000000 00000000
        0x0060: 00000000 00000000 00000000 00000000
        0x0070: 00000000 00000000 00000000 00000000
        0x0080: 00000000 00000000 00000000 00000000
        0x0090: 00000000 00000000 00000000 00000000
        0x00a0: 00000000 00000000 00000000 00000000
        0x00b0: 00000000 00000000 00000000 00000000
        0x00c0: 00000000 00000000 00000000 00000000
        0x00d0: 00000000 00000000 00000000 00000000
        0x00e0: 00000000 00000000 00000000 00000000
        0x00f0: 00000000 00000000 00000000 00000000
 0:5:0: Qumranet Virtio Memory
        0x0000: Vendor ID: 1af4 Product ID: 1002
        0x0004: Command: 0107 Status: 0000
        0x0008: Class: 00 Subclass: ff Interface: 00 Revision: 00
        0x000c: BIST: 00 Header Type: 00 Latency Timer: 00 Cache Line Size: 00
        0x0010: BAR io addr: 0x0000c060/0x0020
        0x0014: BAR empty (00000000)
        0x0018: BAR empty (00000000)
        0x001c: BAR empty (00000000)
        0x0020: BAR empty (00000000)
        0x0024: BAR empty (00000000)
        0x0028: Cardbus CIS: 00000000
        0x002c: Subsystem Vendor ID: 1af4 Product ID: 0005
        0x0030: Expansion ROM Base Address: 00000000
        0x0038: 00000000
        0x003c: Interrupt Pin: 01 Line: 0a Min Gnt: 00 Max Lat: 00
        0x0000: 10021af4 00000107 00ff0000 00000000
        0x0010: 0000c061 00000000 00000000 00000000
        0x0020: 00000000 00000000 00000000 00051af4
        0x0030: 00000000 00000000 00000000 0000010a
        0x0040: 00000000 00000000 00000000 00000000
        0x0050: 00000000 00000000 00000000 00000000
        0x0060: 00000000 00000000 00000000 00000000
        0x0070: 00000000 00000000 00000000 00000000
        0x0080: 00000000 00000000 00000000 00000000
        0x0090: 00000000 00000000 00000000 00000000
        0x00a0: 00000000 00000000 00000000 00000000
        0x00b0: 00000000 00000000 00000000 00000000
        0x00c0: 00000000 00000000 00000000 00000000
        0x00d0: 00000000 00000000 00000000 00000000
        0x00e0: 00000000 00000000 00000000 00000000
        0x00f0: 00000000 00000000 00000000 00000000
 0:6:0: Qumranet Virtio RNG
        0x0000: Vendor ID: 1af4 Product ID: 1005
        0x0004: Command: 0107 Status: 0000
        0x0008: Class: 00 Subclass: ff Interface: 00 Revision: 00
        0x000c: BIST: 00 Header Type: 00 Latency Timer: 00 Cache Line Size: 00
        0x0010: BAR io addr: 0x0000c080/0x0020
        0x0014: BAR empty (00000000)
        0x0018: BAR empty (00000000)
        0x001c: BAR empty (00000000)
        0x0020: BAR empty (00000000)
        0x0024: BAR empty (00000000)
        0x0028: Cardbus CIS: 00000000
        0x002c: Subsystem Vendor ID: 1af4 Product ID: 0004
        0x0030: Expansion ROM Base Address: 00000000
        0x0038: 00000000
        0x003c: Interrupt Pin: 01 Line: 0a Min Gnt: 00 Max Lat: 00
        0x0000: 10051af4 00000107 00ff0000 00000000
        0x0010: 0000c081 00000000 00000000 00000000
        0x0020: 00000000 00000000 00000000 00041af4
        0x0030: 00000000 00000000 00000000 0000010a
        0x0040: 00000000 00000000 00000000 00000000
        0x0050: 00000000 00000000 00000000 00000000
        0x0060: 00000000 00000000 00000000 00000000
        0x0070: 00000000 00000000 00000000 00000000
        0x0080: 00000000 00000000 00000000 00000000
        0x0090: 00000000 00000000 00000000 00000000
        0x00a0: 00000000 00000000 00000000 00000000
        0x00b0: 00000000 00000000 00000000 00000000
        0x00c0: 00000000 00000000 00000000 00000000
        0x00d0: 00000000 00000000 00000000 00000000
        0x00e0: 00000000 00000000 00000000 00000000
        0x00f0: 00000000 00000000 00000000 00000000

acpidump:
begin-base64 644 APIC.3
QVBJQ4AAAAAB2kJPQ0hTIEJYUENBUElDAQAAAEJYUEMBAAAAAADg/gEAAAAACAAAAQAAAAAIAQEB
AAAAAQwAAAAAwP4AAAAAAgoAAAIAAAAAAAIKAAUFAAAADQACCgAJCQAAAA0AAgoACgoAAAANAAIK
AAsLAAAADQAEBv8AAAE=
====
begin-base64 644 DSDT.2
RFNEVJoVAAAB9kJPQ0hTIEJYUENEU0RUAQAAAEJYUEMBAAAAEEkEXABbgERCR18BCwIEAVuBC0RC
R18BREJHQggULERCVUcBmGhglmBgdIdgAWFwAGKiEJViYXCDiGBiAERCR0J1YnAKCkRCR0IQIl9T
Ql9bghtQQ0kwCF9ISUQMQdAKAwhfQURSAAhfVUlEARBNCF9TQl9bgkUISFBFVAhfSElEDEHQAQMI
X1VJRABbgEhQVE0ADAAA0P4LAARbgRBIUFRNE1ZFTkQgUFJEXyAUNl9TVEEAcFZFTkRgcFBSRF9h
emAKEGCgDJGTYACTYAv//6QAoA6Rk2EAlGEMAOH1BaQApAoPCF9DUlMREQoOhgkAAAAA0P4ABAAA
eQAQJS5fU0JfUENJMFuCGVBYMTMIX0FEUgwDAAEAW4BQMTNDAgAK/xBABS5fU0JfUENJMFuCQwRJ
U0FfCF9BRFIMAAABAFuAUDQwQwIKYAoEW4EmXi5QWDEzUDEzQwAASC8AB0xQRU4BADgAA0NBRU4B
AANDQkVOARBAHC8DX1NCX1BDSTBJU0FfW4ItUlRDXwhfSElEDEHQCwAIX0NSUxEYChVHAXAAcAAQ
AiIAAUcBcgByAAIGeQBbgjdLQkRfCF9ISUQMQdADAxQJX1NUQQCkCg8IX0NSUxEYChVHAWAAYAAB
AUcBZABkAAEBIgIAeQBbgidNT1VfCF9ISUQMQdAPExQJX1NUQQCkCg8IX0NSUxEICgUiABB5AFuC
TgRGREMwCF9ISUQMQdAHAAhfQ1JTERsKGEcB8gPyAwAERwH3A/cDAAEiQAAqBAB5AAhfRkRFERcK
FAAAAAAAAAAAAAAAAAAAAAACAAAAW4I+TFBUXwhfSElEDEHQBAAUGF9TVEEAcExQRU5goAaTYACk
AKEEpAoPCF9DUlMREAoNRwF4A3gDCAgigAB5AFuCRQRDT00xCF9ISUQMQdAFAQhfVUlEARQYX1NU
QQBwQ0FFTmCgBpNgAKQAoQSkCg8IX0NSUxEQCg1HAfgD+AMACCIQAHkAW4JGBENPTTIIX0hJRAxB
0AUBCF9VSUQKAhQYX1NUQQBwQ0JFTmCgBpNgAKQAoQSkCg8IX0NSUxEQCg1HAfgC+AIACCIIAHkA
EEgILl9TQl9QQ0kwW4BQQ1NUAQsArgoIW4EQUENTVENQQ0lVIFBDSUQgW4BTRUpfAQsIrgoEW4EL
U0VKX0NCMEVKIFuAQk5NUgELEK4KBFuBC0JOTVJDQk5VTSBbAUJMQ0sAFCVQQ0VKAlsjQkxDS///
cGhCTlVNcHkBaQBCMEVKWydCTENLpAAQTjZfU0JfEEsKUENJMBRECl9QUlQAcBICgGBwAGGiQgmV
YQqAcHphCgIAYnB7cmFiAAoDAGOgEJNjAHASCQQAAExOS0QAZKAkk2MBoBGTYQoEcBIJBAAATE5L
UwBkoQ1wEgkEAABMTktBAGSgEZNjCgJwEgkEAABMTktCAGSgEZNjCgNwEgkEAABMTktDAGRwfXli
ChAAC///AIhkAABwe2EKAwCIZAEAcGSIYGEAdWGkYFuBJC8DUENJMElTQV9QNDBDAVBSUTAIUFJR
MQhQUlEyCFBSUTMIFBNJUVNUAaAJewqAaACkCgmkCgsUNklRQ1IJCFBSUjARDgoLiQYACQEAAAAA
eQCKUFJSMAoFUFJSSaALlWgKgHBoUFJSSaRQUlIwW4JMB0xOS0EIX0hJRAxB0AwPCF9VSUQACF9Q
UlMRFgoTiQ4ACQMFAAAACgAAAAsAAAB5ABQPX1NUQQCkSVFTVFBSUTAUEV9ESVMAfVBSUTAKgFBS
UTAUD19DUlMApElRQ1JQUlEwFBdfU1JTAYpoCgVQUlJJcFBSUklQUlEwW4JMB0xOS0IIX0hJRAxB
0AwPCF9VSUQBCF9QUlMRFgoTiQ4ACQMFAAAACgAAAAsAAAB5ABQPX1NUQQCkSVFTVFBSUTEUEV9E
SVMAfVBSUTEKgFBSUTEUD19DUlMApElRQ1JQUlExFBdfU1JTAYpoCgVQUlJJcFBSUklQUlExW4JN
B0xOS0MIX0hJRAxB0AwPCF9VSUQKAghfUFJTERYKE4kOAAkDBQAAAAoAAAALAAAAeQAUD19TVEEA
pElRU1RQUlEyFBFfRElTAH1QUlEyCoBQUlEyFA9fQ1JTAKRJUUNSUFJRMhQXX1NSUwGKaAoFUFJS
SXBQUlJJUFJRMluCTQdMTktECF9ISUQMQdAMDwhfVUlECgMIX1BSUxEWChOJDgAJAwUAAAAKAAAA
CwAAAHkAFA9fU1RBAKRJUVNUUFJRMxQRX0RJUwB9UFJRMwqAUFJRMxQPX0NSUwCkSVFDUlBSUTMU
F19TUlMBimgKBVBSUklwUFJSSVBSUTNbgk8ETE5LUwhfSElEDEHQDA8IX1VJRAoECF9QUlMRDgoL
iQYACQEJAAAAeQAUCV9TVEEApAoLFAZfRElTABQLX0NSUwCkX1BSUxQGX1NSUwEQRxxfU0JfFDVD
UE1BAnCDiENQT05oAGBwEQsKCAAIAAAAAAAAYXBpiGEKAgBwaIhhCgMAcGCIYQoEAKRhFBpDUFNU
AXCDiENQT05oAGCgBWCkCg+hA6QAFApDUEVKAlsiCsgUSgZQUlNDAHBQUlNfZXAAYnAAYKJGBZVg
h0NQT05wg4hDUE9OYABhoAp7YAoHAHpiAWKhDHCDiGV6YAoDAABicHtiAQBjoCKSk2FjcGOIQ1BP
TmAAoAqTYwFOVEZZYAGhCE5URllgCgN1YFuCSwQuUENJMFBSRVMIX0hJRAxB0AoGCF9VSUQNQ1BV
IEhvdHBsdWcgcmVzb3VyY2VzAAhfU1RBCgsIX0NSUxENCgpHAQCvAK8BIHkAW4BQUlNUAQsArwog
W4EMUFJTVAFQUlNfQBBbgzRDUDAwAAAAAAAAFA1fTUFUAKRDUE1BAAAUDF9TVEEApENQU1QAFA1f
RUowAaRDUEVKAGhbgzRDUDAxAQAAAAAAFA1fTUFUAKRDUE1BAQEUDF9TVEEApENQU1QBFA1fRUow
AaRDUEVKAWgUHE5URlkCoAqTaACGQ1AwMGmgCpNoAYZDUDAxaQhDUE9OEgQCAQEUFlwuX0dQRV9F
MDIAXC5fU0JfUFJTQxBOLS5fU0JfUENJMFuCQS1NSFBECF9ISUQNUE5QMEEwNgAIX1VJRA1NZW1v
cnkgaG90cGx1ZyByZXNvdXJjZXMAFBNfU1RBAKAJk01ETlIApACkCgtbAU1MQ0sAFEEGTVNDTgCg
CZNNRE5SAKQAcABgWyNNTENL//+iPJVgTUROUnBgTVNFTKATk01JTlMBTVRGWWABcAFNSU5ToRag
FJNNUk1WAU1URllgCgNwAU1STVZyYAFgWydNTENLpAEULU1SU1QBcABgWyNNTENL//9wmWgATVNF
TKALk01FU18BcAoPYFsnTUxDS6RgFEEYTUNSUwlbI01MQ0v//3CZaABNU0VMCE1SNjQRMwowiisA
AAwDAAAAAAAAAAAAAAAAAAAAAP7/////////AAAAAAAAAAD//////////3kAik1SNjQKDk1JTkyK
TVI2NAoSTUlOSIpNUjY0CiZMRU5Mik1SNjQKKkxFTkiKTVI2NAoWTUFYTIpNUjY0ChpNQVhIcE1S
QkhNSU5IcE1SQkxNSU5McE1STEhMRU5IcE1STExMRU5Mck1JTkxMRU5MTUFYTHJNSU5ITEVOSE1B
WEigFJVNQVhMTUlOTHJNQVhIAU1BWEigEZVNQVhMAXRNQVhIAU1BWEh0TUFYTAFNQVhMoEQHk01B
WEgACE1SMzIRHwochxcAAAwDAAAAAAAAAAD+////AAAAAP////95AIpNUjMyCgpNSU5fik1SMzIK
Dk1BWF+KTVIzMgoWTEVOX3BNSU5MTUlOX3BNQVhMTUFYX3BMRU5MTEVOX1snTUxDS6RNUjMyWydN
TENLpE1SNjQUJE1QWE0BWyNNTENL//9wmWgATVNFTHBNUFhfYFsnTUxDS6RgFChNT1NUBFsjTUxD
S///cJloAE1TRUxwaU1PRVZwak1PU0NbJ01MQ0sUIk1FSjACWyNNTENL//9wmWgATVNFTHABTUVK
X1snTUxDSxBJBl9HUEUIX0hJRA1BQ1BJMDAwNgAUOV9FMDEAWyNcLwNfU0JfUENJMEJMQ0v//1wv
A19TQl9QQ0kwUENOVFsnXC8DX1NCX1BDSTBCTENLFBlfRTAzAFwvBF9TQl9QQ0kwTUhQRE1TQ04Q
RRFcLl9TQl9QQ0kwCF9DUlMRQgcKbogNAAIMAAAAAAD/AAAAAAFHAfgM+AwBCIgNAAEMAwAAAAD3
DAAA+AyIDQABDAMAAAAN//8AAADzhxcAAAwDAAAAAAAACgD//wsAAAAAAAAAAgCHFwAADAEAAAAA
AAAAwP//v/4AAAAAAADAPnkAW4JDBEdQRTAIX0hJRA1QTlAwQTA2AAhfVUlEDUdQRTAgcmVzb3Vy
Y2VzAAhfU1RBCgsIX0NSUxENCgpHAeCv4K8BBHkAW4JKBFBIUFIIX0hJRA1QTlAwQTA2AAhfVUlE
DVBDSSBIb3RwbHVnIHJlc291cmNlcwAIX1NUQQoLCF9DUlMRDQoKRwEArgCuARR5ABApXAAIX1Mz
XxIGBAEBAAAIX1M0XxIIBAoCCgIAAAhfUzVfEgYEAAAAABA7XC5fU0JfUENJMFuCLkZXQ0YIX0hJ
RA1RRU1VMDAwMgAIX1NUQQoLCF9DUlMRDQoKRwEQBRAFAQJ5ABBGelxfU0JfEE0IXC8DX1NCX1BD
STBNSFBECE1ETlIACF9DUlMRDQoKRwEACgAKABh5AFuASFBNUgELAAoKGFuBH0hQTVIDTVJCTCBN
UkJIIE1STEwgTVJMSCBNUFhfIFuBHUhQTVJBAEAKTUVTXwFNSU5TAU1STVYBTUVKXwFbgRVIUE1S
A01TRUwgTU9FViBNT1NDIBQGTVRGWQIQSXBQQ0kwCEJTRUwAW4ILUzAwXwhfQURSAFuCKVMxMF8I
X0FEUgwAAAIACF9TVU4KAhQSX0VKMAFQQ0VKQlNFTF9TVU5bgilTMThfCF9TVU4KAwhfQURSDAAA
AwAUEl9FSjABUENFSkJTRUxfU1VOW4IpUzIwXwhfQURSDAAABAAIX1NVTgoEFBJfRUowAVBDRUpC
U0VMX1NVTluCKVMyOF8IX0FEUgwAAAUACF9TVU4KBRQSX0VKMAFQQ0VKQlNFTF9TVU5bgilTMzBf
CF9BRFIMAAAGAAhfU1VOCgYUEl9FSjABUENFSkJTRUxfU1VOW4IpUzM4XwhfU1VOCgcIX0FEUgwA
AAcAFBJfRUowAVBDRUpCU0VMX1NVTluCKVM0MF8IX1NVTgoICF9BRFIMAAAIABQSX0VKMAFQQ0VK
QlNFTF9TVU5bgilTNDhfCF9TVU4KCQhfQURSDAAACQAUEl9FSjABUENFSkJTRUxfU1VOW4IpUzUw
XwhfU1VOCgoIX0FEUgwAAAoAFBJfRUowAVBDRUpCU0VMX1NVTluCKVM1OF8IX1NVTgoLCF9BRFIM
AAALABQSX0VKMAFQQ0VKQlNFTF9TVU5bgilTNjBfCF9TVU4KDAhfQURSDAAADAAUEl9FSjABUENF
SkJTRUxfU1VOW4IpUzY4XwhfU1VOCg0IX0FEUgwAAA0AFBJfRUowAVBDRUpCU0VMX1NVTluCKVM3
MF8IX1NVTgoOCF9BRFIMAAAOABQSX0VKMAFQQ0VKQlNFTF9TVU5bgilTNzhfCF9TVU4KDwhfQURS
DAAADwAUEl9FSjABUENFSkJTRUxfU1VOW4IpUzgwXwhfU1VOChAIX0FEUgwAABAAFBJfRUowAVBD
RUpCU0VMX1NVTluCKVM4OF8IX1NVTgoRCF9BRFIMAAARABQSX0VKMAFQQ0VKQlNFTF9TVU5bgilT
OTBfCF9TVU4KEghfQURSDAAAEgAUEl9FSjABUENFSkJTRUxfU1VOW4IpUzk4XwhfU1VOChMIX0FE
UgwAABMAFBJfRUowAVBDRUpCU0VMX1NVTluCKVNBMF8IX1NVTgoUCF9BRFIMAAAUABQSX0VKMAFQ
Q0VKQlNFTF9TVU5bgilTQThfCF9TVU4KFQhfQURSDAAAFQAUEl9FSjABUENFSkJTRUxfU1VOW4Ip
U0IwXwhfU1VOChYIX0FEUgwAABYAFBJfRUowAVBDRUpCU0VMX1NVTluCKVNCOF8IX1NVTgoXCF9B
RFIMAAAXABQSX0VKMAFQQ0VKQlNFTF9TVU5bgilTQzBfCF9TVU4KGAhfQURSDAAAGAAUEl9FSjAB
UENFSkJTRUxfU1VOW4IpU0M4XwhfU1VOChkIX0FEUgwAABkAFBJfRUowAVBDRUpCU0VMX1NVTluC
KVNEMF8IX1NVTgoaCF9BRFIMAAAaABQSX0VKMAFQQ0VKQlNFTF9TVU5bgilTRDhfCF9TVU4KGwhf
QURSDAAAGwAUEl9FSjABUENFSkJTRUxfU1VOW4IpU0UwXwhfU1VOChwIX0FEUgwAABwAFBJfRUow
AVBDRUpCU0VMX1NVTluCKVNFOF8IX1NVTgodCF9BRFIMAAAdABQSX0VKMAFQQ0VKQlNFTF9TVU5b
gilTRjBfCF9TVU4KHghfQURSDAAAHgAUEl9FSjABUENFSkJTRUxfU1VOW4IpU0Y4XwhfU1VOCh8I
X0FEUgwAAB8AFBJfRUowAVBDRUpCU0VMX1NVThRFHERWTlQCoAx7aAoEAIZTMTBfaaAMe2gKCACG
UzE4X2mgDHtoChAAhlMyMF9poAx7aAogAIZTMjhfaaAMe2gKQACGUzMwX2mgDHtoCoAAhlMzOF9p
oA17aAsAAQCGUzQwX2mgDXtoCwACAIZTNDhfaaANe2gLAAQAhlM1MF9poA17aAsACACGUzU4X2mg
DXtoCwAQAIZTNjBfaaANe2gLACAAhlM2OF9poA17aAsAQACGUzcwX2mgDXtoCwCAAIZTNzhfaaAP
e2gMAAABAACGUzgwX2mgD3toDAAAAgAAhlM4OF9poA97aAwAAAQAAIZTOTBfaaAPe2gMAAAIAACG
Uzk4X2mgD3toDAAAEAAAhlNBMF9poA97aAwAACAAAIZTQThfaaAPe2gMAABAAACGU0IwX2mgD3to
DAAAgAAAhlNCOF9poA97aAwAAAABAIZTQzBfaaAPe2gMAAAAAgCGU0M4X2mgD3toDAAAAAQAhlNE
MF9poA97aAwAAAAIAIZTRDhfaaAPe2gMAAAAEACGU0UwX2mgD3toDAAAACAAhlNFOF9poA97aAwA
AABAAIZTRjBfaaAPe2gMAAAAgACGU0Y4X2kUH1BDTlQAcABCTlVNRFZOVFBDSVUBRFZOVFBDSUQK
Aw==
====
begin-base64 644 FACP.1
RkFDUHQAAAAB50JPQ0hTIEJYUENGQUNQAQAAAEJYUEMBAAAAAAD+v0AA/r8BAAkAsgAAAPHwAAAA
BgAAAAAAAAQGAAAAAAAAAAAAAAgGAADgrwAAAAAAAAQCAAQEAAAA/w//DwAAAAAAAAAAMgAAAKWA
AAA=
====
begin-base64 644 RSDT.0
UlNEVCwAAAABoUJPQ0hTIEJYUENSU0RUAQAAAEJYUEMBAAAA2hX+v04W/r8=
====
begin-base64 644 headers
ClJTRCBQVFI6IENoZWNrc3VtPTE3NywgT0VNSUQ9Qk9DSFMsIFJldmlzaW9uPTAsIFJzZHRBZGRy
ZXNzPTB4YmZmZTE2Y2UKCgpSU0RUOiBMZW5ndGg9NDQsIFJldmlzaW9uPTEsIENoZWNrc3VtPTE2
MSwKCU9FTUlEPUJPQ0hTLCBPRU0gVGFibGUgSUQ9QlhQQ1JTRFQsIE9FTSBSZXZpc2lvbj0weDEs
CglDcmVhdG9yIElEPUJYUEMsIENyZWF0b3IgUmV2aXNpb249MHgxCgoKCUVudHJpZXM9eyAweGJm
ZmUxNWRhLCAweGJmZmUxNjRlIH0KCgoJRFNEVD0weGJmZmUwMDQwCglJTlRfTU9ERUw9QVBJQwoJ
U0NJX0lOVD05CglTTUlfQ01EPTB4YjIsIEFDUElfRU5BQkxFPTB4ZjEsIEFDUElfRElTQUJMRT0w
eGYwLCBTNEJJT1NfUkVRPTB4MAoJUE0xYV9FVlRfQkxLPTB4NjAwLTB4NjAzCglQTTFhX0NOVF9C
TEs9MHg2MDQtMHg2MDUKCVBNMl9UTVJfQkxLPTB4NjA4LTB4NjBiCglQTTJfR1BFMF9CTEs9MHhh
ZmUwLTB4YWZlMwoJUF9MVkwyX0xBVD00MDk1bXMsIFBfTFZMM19MQVQ9NDA5NW1zCglGTFVTSF9T
SVpFPTAsIEZMVVNIX1NUUklERT0wCglEVVRZX09GRlNFVD0wLCBEVVRZX1dJRFRIPTAKCURBWV9B
TFJNPTAsIE1PTl9BTFJNPTAsIENFTlRVUlk9NTAKCUZsYWdzPXtXQklOVkQsUFJPQ19DMSxTTFBf
QlVUVE9OLFJUQ19TNH0KCgpEU0RUOiBMZW5ndGg9NTUzMCwgUmV2aXNpb249MSwgQ2hlY2tzdW09
MjQ2LAoJT0VNSUQ9Qk9DSFMsIE9FTSBUYWJsZSBJRD1CWFBDRFNEVCwgT0VNIFJldmlzaW9uPTB4
MSwKCUNyZWF0b3IgSUQ9QlhQQywgQ3JlYXRvciBSZXZpc2lvbj0weDEKCgpBUElDOiBMZW5ndGg9
MTI4LCBSZXZpc2lvbj0xLCBDaGVja3N1bT0yMTgsCglPRU1JRD1CT0NIUywgT0VNIFRhYmxlIElE
PUJYUENBUElDLCBPRU0gUmV2aXNpb249MHgxLAoJQ3JlYXRvciBJRD1CWFBDLCBDcmVhdG9yIFJl
dmlzaW9uPTB4MQoK
====


--
jca | PGP : 0x1524E7EE / 5135 92C1 AD36 5293 2BDF  DDCC 0DFA 74AE 1524 E7EE

Reply | Threaded
Open this post in threaded view
|

Re: lock order reversal when starting net/knot

Stuart Henderson
On 2018/07/11 21:35, Jeremie Courreges-Anglas wrote:

>
> >Synopsis: lock order reversal when starting net/knot
> >Category: kernel
> >Environment:
> System      : OpenBSD 6.3
> Details     : OpenBSD 6.3-current (GENERIC.MP) #125: Wed Jul 11 10:18:39 MDT 2018
> [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
>
> Architecture: OpenBSD.amd64
> Machine     : amd64
> >Description:
> When I run knot (an authoritative name server daemon that uses
> LMDB) the following entries appear in dmesg:
>
> lock order reversal:
>  1st 0xffffff01ff732468 vmmaplk (&map->lock) @ /usr/src/sys/uvm/uvm_map.c:4433
>  2nd 0xffffff01fe312098 inode (&ip->i_lock) @ /usr/src/sys/ufs/ufs/ufs_vnops.c:1544
> lock order "&ip->i_lock"(rrwlock) -> "&map->lock"(rwlock) first seen at:
> #0  witness_checkorder+0x4c0
> #1  _rw_enter+0x68
> #2  vm_map_lock_ln+0xbc
> #3  uvm_map+0x1a1
> #4  km_alloc+0x16a
> #5  pool_multi_alloc_ni+0xbb
> #6  pool_p_alloc+0x56
> #7  pool_do_get+0xe4
> #8  pool_get+0xaf
> #9  ufsdirhash_build+0x31e
> #10 ufs_lookup+0x19d
> #11 VOP_LOOKUP+0x4f
> #12 vfs_lookup+0x27e
> #13 namei+0x226
> #14 start_init+0xb2
> lock order "&map->lock"(rwlock) -> "&ip->i_lock"(rrwlock) first seen at:
> #0  witness_checkorder+0x4c0
> #1  _rw_enter+0x68
> #2  _rrw_enter+0x3e
> #3  VOP_LOCK+0x3d
> #4  vn_lock+0x34
> #5  uvn_io+0x1b8
> #6  uvm_pager_put+0x109
> #7  uvn_flush+0x424
> #8  uvm_map_clean+0x3e7
> #9  syscall+0x32a
> #10 Xsyscall+0xef
>
> fwiw this looks very similar to the lock order traces seen by Vijay
> Sankar when testing an update to samba-4.8.3 (ports/net/samba):

I see similar at system boot (and have done since at least mid June as it was
included in a list mail I sent then). Haven't stepped through daemon startups
to figure out which one it is on my side yet (would need to do it soon after
startup otherwise witness gets tired and goes home: "witness_lock_list_get:
witness exhausted".

Different trace though:

starting package daemons: redis messagebus avahi_daemon php70_fpm nginx rrdcached mysqld ido2db(failed) icinga nsca_ng bacula_fd slo
wcgi memcached rspamd tracd clamd_bglock order reversal:                                                                        
 1st 0xffffff039b2dca00 vmmaplk (&map->lock) @ /src/cvs-openbsd/sys/uvm/uvm_map.c:4433          
 2nd 0xffffff0392bbdc48 inode (&ip->i_lock) @ /src/cvs-openbsd/sys/ufs/ufs/ufs_vnops.c:1544                              
lock order "&ip->i_lock"(rrwlock) -> "&map->lock"(rwlock) first seen at:
#0  witness_checkorder+0x4c0                                      
#1  _rw_enter_read+0x49                                                                            
#2  uvmfault_lookup+0x8d                                          
#3  uvm_fault+0x72                                                
#4  pageflttrap+0x14c    
#5  kerntrap+0x8f                      
#6  alltraps_kern+0x7b      
#7  copyout+0x53                            
#8  ffs_read+0x1f0                  
#9  VOP_READ+0x49              
#10 vn_read+0xca                        
#11 dofilereadv+0x21c      
#12 sys_read+0x82                    
#13 syscall+0x32a        
#14 Xsyscall_untramp+0xe4
lock order "&map->lock"(rwlock) -> "&ip->i_lock"(rrwlock) first seen at:
#0  witness_checkorder+0x4c0
#1  _rw_enter+0x68                                                                
#2  _rrw_enter+0x3e        
#3  VOP_LOCK+0x3d  
#4  vn_lock+0x34          
#5  uvn_io+0x1b8            
#6  uvm_pager_put+0x109  
#7  uvn_flush+0x424                    
#8  uvm_map_clean+0x3e7      
#9  syscall+0x32a                            
#10 Xsyscall_untramp+0xe4          
 freeradius cupsd rsyncd smbd nmbd lldpd postfix dovecot unifi icinga2 solr postgresql supervisord.
.    


>   https://marc.info/?l=openbsd-ports&m=153118057112027&w=2
>
> >How-To-Repeat:
> Start knot (at boot time or from a shell)
> >Fix:
> Unknown
>
>
> dmesg:
> OpenBSD 6.3-current (GENERIC.MP) #125: Wed Jul 11 10:18:39 MDT 2018
>     [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
> real mem = 8573018112 (8175MB)
> avail mem = 8234561536 (7853MB)
> pmap_enter_special: special diffing mapping at ffffffff819ff000
> enter_shared_special_pages: entered kutext.page va 0xffffffff819ff000 pa 0x1c3c000
> pmap_enter_special: special diffing mapping at ffffffff81a00000
> enter_shared_special_pages: entered kutext.page va 0xffffffff81a00000 pa 0x1c3d000
> mpath0 at root
> scsibus0 at mpath0: 256 targets
> mainbus0 at root
> bios0 at mainbus0: SMBIOS rev. 2.8 @ 0xf6890 (11 entries)
> bios0: vendor SeaBIOS version "1.10.2-1" date 04/01/2014
> bios0: QEMU Standard PC (i440FX + PIIX, 1996)
> acpi0 at bios0: rev 0
> acpi0: sleep states S3 S4 S5
> acpi0: tables DSDT FACP APIC
> acpi0: wakeup devices
> acpitimer0 at acpi0: 3579545 Hz, 24 bits
> acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
> pmap_enter_special: special diffing mapping at ffffffff81cb1000
> cpu0 at mainbus0: apid 0 (boot processor)
> cpu0: Intel Core Processor (Broadwell), 3408.45 MHz
> cpu0: FPU,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SSE3,PCLMUL,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,OSXSAVE,AVX,HV,NXE,RDTSCP,LONG,LAHF,3DNOWP,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,RDSEED,ADX,SMAP,XSAVEOPT,MELTDOWN
> cpu0: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 512KB 64b/line 16-way L2 cache
> cpu0: ITLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
> cpu0: DTLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
> cpu0: smt 0, core 0, package 0
> mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
> cpu0: apic clock running at 999MHz
> cpu1 at mainbus0: apid 1 (application processor)
> cpu1: Intel Core Processor (Broadwell), 3408.03 MHz
> cpu1: FPU,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SSE3,PCLMUL,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,OSXSAVE,AVX,HV,NXE,RDTSCP,LONG,LAHF,3DNOWP,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,RDSEED,ADX,SMAP,XSAVEOPT,MELTDOWN
> cpu1: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 512KB 64b/line 16-way L2 cache
> cpu1: ITLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
> cpu1: DTLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
> cpu1: smt 0, core 0, package 1
> ioapic0 at mainbus0: apid 0 pa 0xfec00000, version 11, 24 pins
> acpiprt0 at acpi0: bus 0 (PCI0)
> acpicpu0 at acpi0: C1(@1 halt!)
> acpicpu1 at acpi0: C1(@1 halt!)
> "ACPI0006" at acpi0 not configured
> acpicmos0 at acpi0
> "PNP0A06" at acpi0 not configured
> "PNP0A06" at acpi0 not configured
> "PNP0A06" at acpi0 not configured
> "QEMU0002" at acpi0 not configured
> pvbus0 at mainbus0: KVM
> pci0 at mainbus0 bus 0
> pchb0 at pci0 dev 0 function 0 "Intel 82441FX" rev 0x02
> pcib0 at pci0 dev 1 function 0 "Intel 82371SB ISA" rev 0x00
> pciide0 at pci0 dev 1 function 1 "Intel 82371SB IDE" rev 0x00: DMA, channel 0 wired to compatibility, channel 1 wired to compatibility
> atapiscsi0 at pciide0 channel 0 drive 0
> scsibus1 at atapiscsi0: 2 targets
> cd0 at scsibus1 targ 0 lun 0: <QEMU, QEMU DVD-ROM, 2.1.> ATAPI 5/cdrom removable
> cd0(pciide0:0:0): using PIO mode 4, DMA mode 2
> pciide0: channel 1 disabled (no drives)
> piixpm0 at pci0 dev 1 function 3 "Intel 82371AB Power" rev 0x03: apic 0 int 9
> iic0 at piixpm0
> virtio0 at pci0 dev 2 function 0 "Qumranet Virtio Network" rev 0x00
> vio0 at virtio0: address 52:54:00:12:01:90
> virtio0: msix shared
> virtio1 at pci0 dev 4 function 0 "Qumranet Virtio Storage" rev 0x00
> vioblk0 at virtio1
> scsibus2 at vioblk0: 2 targets
> sd0 at scsibus2 targ 0 lun 0: <VirtIO, Block Device, > SCSI3 0/direct fixed
> sd0: 256000MB, 512 bytes/sector, 524288000 sectors
> virtio1: msix shared
> virtio2 at pci0 dev 5 function 0 "Qumranet Virtio Memory" rev 0x00
> viomb0 at virtio2
> virtio2: apic 0 int 10
> virtio3 at pci0 dev 6 function 0 "Qumranet Virtio RNG" rev 0x00
> viornd0 at virtio3
> virtio3: apic 0 int 10
> isa0 at pcib0
> isadma0 at isa0
> fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
> com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
> com0: console
> pckbc0 at isa0 port 0x60/5 irq 1 irq 12
> pckbd0 at pckbc0 (kbd slot)
> wskbd0 at pckbd0 mux 1
> pms0 at pckbc0 (aux slot)
> wsmouse0 at pms0 mux 0
> pcppi0 at isa0 port 0x61
> spkr0 at pcppi0
> vscsi0 at root
> scsibus3 at vscsi0: 256 targets
> softraid0 at root
> scsibus4 at softraid0: 256 targets
> root on sd0a (871b81ad3a5d5c9e.a) swap on sd0b dump on sd0b
> fd0 at fdc0 drive 1: density unknown
>
> usbdevs:
> usbdevs: no USB controllers found
>
> pcidump:
> Domain /dev/pci0:
>  0:0:0: Intel 82441FX
> 0x0000: Vendor ID: 8086 Product ID: 1237
> 0x0004: Command: 0103 Status: 0000
> 0x0008: Class: 06 Subclass: 00 Interface: 00 Revision: 02
> 0x000c: BIST: 00 Header Type: 00 Latency Timer: 00 Cache Line Size: 00
> 0x0010: BAR empty (00000000)
> 0x0014: BAR empty (00000000)
> 0x0018: BAR empty (00000000)
> 0x001c: BAR empty (00000000)
> 0x0020: BAR empty (00000000)
> 0x0024: BAR empty (00000000)
> 0x0028: Cardbus CIS: 00000000
> 0x002c: Subsystem Vendor ID: 1af4 Product ID: 1100
> 0x0030: Expansion ROM Base Address: 00000000
> 0x0038: 00000000
> 0x003c: Interrupt Pin: 00 Line: 00 Min Gnt: 00 Max Lat: 00
> 0x0000: 12378086 00000103 06000002 00000000
> 0x0010: 00000000 00000000 00000000 00000000
> 0x0020: 00000000 00000000 00000000 11001af4
> 0x0030: 00000000 00000000 00000000 00000000
> 0x0040: 00000000 00000000 00000000 00000000
> 0x0050: 00000000 ff000000 11111000 31111111
> 0x0060: 00000000 00000000 00000000 00000000
> 0x0070: 00020000 00000000 00000000 00000000
> 0x0080: 00000000 00000000 00000000 00000000
> 0x0090: 00000000 00000000 00000000 00000000
> 0x00a0: 00000000 00000000 00000000 00000000
> 0x00b0: 00000000 00000000 00000000 00000000
> 0x00c0: 00000000 00000000 00000000 00000000
> 0x00d0: 00000000 00000000 00000000 00000000
> 0x00e0: 00000000 00000000 00000000 00000000
> 0x00f0: 00000000 00000000 00000000 00000000
>  0:1:0: Intel 82371SB ISA
> 0x0000: Vendor ID: 8086 Product ID: 7000
> 0x0004: Command: 0103 Status: 0200
> 0x0008: Class: 06 Subclass: 01 Interface: 00 Revision: 00
> 0x000c: BIST: 00 Header Type: 80 Latency Timer: 00 Cache Line Size: 00
> 0x0010: BAR empty (00000000)
> 0x0014: BAR empty (00000000)
> 0x0018: BAR empty (00000000)
> 0x001c: BAR empty (00000000)
> 0x0020: BAR empty (00000000)
> 0x0024: BAR empty (00000000)
> 0x0028: Cardbus CIS: 00000000
> 0x002c: Subsystem Vendor ID: 1af4 Product ID: 1100
> 0x0030: Expansion ROM Base Address: 00000000
> 0x0038: 00000000
> 0x003c: Interrupt Pin: 00 Line: 00 Min Gnt: 00 Max Lat: 00
> 0x0000: 70008086 02000103 06010000 00800000
> 0x0010: 00000000 00000000 00000000 00000000
> 0x0020: 00000000 00000000 00000000 11001af4
> 0x0030: 00000000 00000000 00000000 00000000
> 0x0040: 00000000 00000000 00000000 0003004d
> 0x0050: 00000000 00000000 00000000 00000000
> 0x0060: 0b0b0a0a 00000000 00000200 00000000
> 0x0070: 00000080 0c0c0000 00000002 00000000
> 0x0080: 00000000 00000000 00000000 00000000
> 0x0090: 00000000 00000000 00000000 00000000
> 0x00a0: 00000008 00000000 0000000f 00000000
> 0x00b0: 00000000 00000000 00000000 00000000
> 0x00c0: 00000000 00000000 00000000 00000000
> 0x00d0: 00000000 00000000 00000000 00000000
> 0x00e0: 00000000 00000000 00000000 00000000
> 0x00f0: 00000000 00000000 00000000 00000000
>  0:1:1: Intel 82371SB IDE
> 0x0000: Vendor ID: 8086 Product ID: 7010
> 0x0004: Command: 0107 Status: 0280
> 0x0008: Class: 01 Subclass: 01 Interface: 80 Revision: 00
> 0x000c: BIST: 00 Header Type: 00 Latency Timer: 00 Cache Line Size: 00
> 0x0010: BAR empty (00000000)
> 0x0014: BAR empty (00000000)
> 0x0018: BAR empty (00000000)
> 0x001c: BAR empty (00000000)
> 0x0020: BAR io addr: 0x0000c0a0/0x0010
> 0x0024: BAR empty (00000000)
> 0x0028: Cardbus CIS: 00000000
> 0x002c: Subsystem Vendor ID: 1af4 Product ID: 1100
> 0x0030: Expansion ROM Base Address: 00000000
> 0x0038: 00000000
> 0x003c: Interrupt Pin: 00 Line: 00 Min Gnt: 00 Max Lat: 00
> 0x0000: 70108086 02800107 01018000 00000000
> 0x0010: 00000000 00000000 00000000 00000000
> 0x0020: 0000c0a1 00000000 00000000 11001af4
> 0x0030: 00000000 00000000 00000000 00000000
> 0x0040: 0000a307 00000000 00000000 00000000
> 0x0050: 00000000 00000000 00000000 00000000
> 0x0060: 00000000 00000000 00000000 00000000
> 0x0070: 00000000 00000000 00000000 00000000
> 0x0080: 00000000 00000000 00000000 00000000
> 0x0090: 00000000 00000000 00000000 00000000
> 0x00a0: 00000000 00000000 00000000 00000000
> 0x00b0: 00000000 00000000 00000000 00000000
> 0x00c0: 00000000 00000000 00000000 00000000
> 0x00d0: 00000000 00000000 00000000 00000000
> 0x00e0: 00000000 00000000 00000000 00000000
> 0x00f0: 00000000 00000000 00000000 00000000
>  0:1:3: Intel 82371AB Power
> 0x0000: Vendor ID: 8086 Product ID: 7113
> 0x0004: Command: 0103 Status: 0280
> 0x0008: Class: 06 Subclass: 80 Interface: 00 Revision: 03
> 0x000c: BIST: 00 Header Type: 00 Latency Timer: 00 Cache Line Size: 00
> 0x0010: BAR empty (00000000)
> 0x0014: BAR empty (00000000)
> 0x0018: BAR empty (00000000)
> 0x001c: BAR empty (00000000)
> 0x0020: BAR empty (00000000)
> 0x0024: BAR empty (00000000)
> 0x0028: Cardbus CIS: 00000000
> 0x002c: Subsystem Vendor ID: 1af4 Product ID: 1100
> 0x0030: Expansion ROM Base Address: 00000000
> 0x0038: 00000000
> 0x003c: Interrupt Pin: 01 Line: 09 Min Gnt: 00 Max Lat: 00
> 0x0000: 71138086 02800103 06800003 00000000
> 0x0010: 00000000 00000000 00000000 00000000
> 0x0020: 00000000 00000000 00000000 11001af4
> 0x0030: 00000000 00000000 00000000 00000109
> 0x0040: 00000601 00000000 00000000 00000000
> 0x0050: 00000000 00000000 02000000 10000000
> 0x0060: 60000000 08000000 00000000 00000000
> 0x0070: 00000000 00000000 00000000 00000000
> 0x0080: 00000001 00000000 00000000 00000000
> 0x0090: 00000701 00000000 00000000 00000000
> 0x00a0: 00000000 00000000 00000000 00000000
> 0x00b0: 00000000 00000000 00000000 00000000
> 0x00c0: 00000000 00000000 00000000 00000000
> 0x00d0: 00090000 00000000 00000000 00000000
> 0x00e0: 00000000 00000000 00000000 00000000
> 0x00f0: 00000000 00000000 00000000 00000000
>  0:2:0: Qumranet Virtio Network
> 0x0000: Vendor ID: 1af4 Product ID: 1000
> 0x0004: Command: 0107 Status: 0010
> 0x0008: Class: 02 Subclass: 00 Interface: 00 Revision: 00
> 0x000c: BIST: 00 Header Type: 00 Latency Timer: 00 Cache Line Size: 00
> 0x0010: BAR io addr: 0x0000c040/0x0020
> 0x0014: BAR mem 32bit addr: 0xfebc0000/0x00001000
> 0x0018: BAR empty (00000000)
> 0x001c: BAR empty (00000000)
> 0x0020: BAR empty (00000000)
> 0x0024: BAR empty (00000000)
> 0x0028: Cardbus CIS: 00000000
> 0x002c: Subsystem Vendor ID: 1af4 Product ID: 0001
> 0x0030: Expansion ROM Base Address: feb80000
> 0x0038: 00000000
> 0x003c: Interrupt Pin: 01 Line: 0a Min Gnt: 00 Max Lat: 00
> 0x0040: Capability 0x11: Extended Message Signalled Interrupts (MSI-X)
> 0x0000: 10001af4 00100107 02000000 00000000
> 0x0010: 0000c041 febc0000 00000000 00000000
> 0x0020: 00000000 00000000 00000000 00011af4
> 0x0030: feb80000 00000040 00000000 0000010a
> 0x0040: 80020011 00000001 00000801 00000000
> 0x0050: 00000000 00000000 00000000 00000000
> 0x0060: 00000000 00000000 00000000 00000000
> 0x0070: 00000000 00000000 00000000 00000000
> 0x0080: 00000000 00000000 00000000 00000000
> 0x0090: 00000000 00000000 00000000 00000000
> 0x00a0: 00000000 00000000 00000000 00000000
> 0x00b0: 00000000 00000000 00000000 00000000
> 0x00c0: 00000000 00000000 00000000 00000000
> 0x00d0: 00000000 00000000 00000000 00000000
> 0x00e0: 00000000 00000000 00000000 00000000
> 0x00f0: 00000000 00000000 00000000 00000000
>  0:4:0: Qumranet Virtio Storage
> 0x0000: Vendor ID: 1af4 Product ID: 1001
> 0x0004: Command: 0107 Status: 0010
> 0x0008: Class: 01 Subclass: 00 Interface: 00 Revision: 00
> 0x000c: BIST: 00 Header Type: 00 Latency Timer: 00 Cache Line Size: 00
> 0x0010: BAR io addr: 0x0000c000/0x0040
> 0x0014: BAR mem 32bit addr: 0xfebc1000/0x00001000
> 0x0018: BAR empty (00000000)
> 0x001c: BAR empty (00000000)
> 0x0020: BAR empty (00000000)
> 0x0024: BAR empty (00000000)
> 0x0028: Cardbus CIS: 00000000
> 0x002c: Subsystem Vendor ID: 1af4 Product ID: 0002
> 0x0030: Expansion ROM Base Address: 00000000
> 0x0038: 00000000
> 0x003c: Interrupt Pin: 01 Line: 0b Min Gnt: 00 Max Lat: 00
> 0x0040: Capability 0x11: Extended Message Signalled Interrupts (MSI-X)
> 0x0000: 10011af4 00100107 01000000 00000000
> 0x0010: 0000c001 febc1000 00000000 00000000
> 0x0020: 00000000 00000000 00000000 00021af4
> 0x0030: 00000000 00000040 00000000 0000010b
> 0x0040: 80010011 00000001 00000801 00000000
> 0x0050: 00000000 00000000 00000000 00000000
> 0x0060: 00000000 00000000 00000000 00000000
> 0x0070: 00000000 00000000 00000000 00000000
> 0x0080: 00000000 00000000 00000000 00000000
> 0x0090: 00000000 00000000 00000000 00000000
> 0x00a0: 00000000 00000000 00000000 00000000
> 0x00b0: 00000000 00000000 00000000 00000000
> 0x00c0: 00000000 00000000 00000000 00000000
> 0x00d0: 00000000 00000000 00000000 00000000
> 0x00e0: 00000000 00000000 00000000 00000000
> 0x00f0: 00000000 00000000 00000000 00000000
>  0:5:0: Qumranet Virtio Memory
> 0x0000: Vendor ID: 1af4 Product ID: 1002
> 0x0004: Command: 0107 Status: 0000
> 0x0008: Class: 00 Subclass: ff Interface: 00 Revision: 00
> 0x000c: BIST: 00 Header Type: 00 Latency Timer: 00 Cache Line Size: 00
> 0x0010: BAR io addr: 0x0000c060/0x0020
> 0x0014: BAR empty (00000000)
> 0x0018: BAR empty (00000000)
> 0x001c: BAR empty (00000000)
> 0x0020: BAR empty (00000000)
> 0x0024: BAR empty (00000000)
> 0x0028: Cardbus CIS: 00000000
> 0x002c: Subsystem Vendor ID: 1af4 Product ID: 0005
> 0x0030: Expansion ROM Base Address: 00000000
> 0x0038: 00000000
> 0x003c: Interrupt Pin: 01 Line: 0a Min Gnt: 00 Max Lat: 00
> 0x0000: 10021af4 00000107 00ff0000 00000000
> 0x0010: 0000c061 00000000 00000000 00000000
> 0x0020: 00000000 00000000 00000000 00051af4
> 0x0030: 00000000 00000000 00000000 0000010a
> 0x0040: 00000000 00000000 00000000 00000000
> 0x0050: 00000000 00000000 00000000 00000000
> 0x0060: 00000000 00000000 00000000 00000000
> 0x0070: 00000000 00000000 00000000 00000000
> 0x0080: 00000000 00000000 00000000 00000000
> 0x0090: 00000000 00000000 00000000 00000000
> 0x00a0: 00000000 00000000 00000000 00000000
> 0x00b0: 00000000 00000000 00000000 00000000
> 0x00c0: 00000000 00000000 00000000 00000000
> 0x00d0: 00000000 00000000 00000000 00000000
> 0x00e0: 00000000 00000000 00000000 00000000
> 0x00f0: 00000000 00000000 00000000 00000000
>  0:6:0: Qumranet Virtio RNG
> 0x0000: Vendor ID: 1af4 Product ID: 1005
> 0x0004: Command: 0107 Status: 0000
> 0x0008: Class: 00 Subclass: ff Interface: 00 Revision: 00
> 0x000c: BIST: 00 Header Type: 00 Latency Timer: 00 Cache Line Size: 00
> 0x0010: BAR io addr: 0x0000c080/0x0020
> 0x0014: BAR empty (00000000)
> 0x0018: BAR empty (00000000)
> 0x001c: BAR empty (00000000)
> 0x0020: BAR empty (00000000)
> 0x0024: BAR empty (00000000)
> 0x0028: Cardbus CIS: 00000000
> 0x002c: Subsystem Vendor ID: 1af4 Product ID: 0004
> 0x0030: Expansion ROM Base Address: 00000000
> 0x0038: 00000000
> 0x003c: Interrupt Pin: 01 Line: 0a Min Gnt: 00 Max Lat: 00
> 0x0000: 10051af4 00000107 00ff0000 00000000
> 0x0010: 0000c081 00000000 00000000 00000000
> 0x0020: 00000000 00000000 00000000 00041af4
> 0x0030: 00000000 00000000 00000000 0000010a
> 0x0040: 00000000 00000000 00000000 00000000
> 0x0050: 00000000 00000000 00000000 00000000
> 0x0060: 00000000 00000000 00000000 00000000
> 0x0070: 00000000 00000000 00000000 00000000
> 0x0080: 00000000 00000000 00000000 00000000
> 0x0090: 00000000 00000000 00000000 00000000
> 0x00a0: 00000000 00000000 00000000 00000000
> 0x00b0: 00000000 00000000 00000000 00000000
> 0x00c0: 00000000 00000000 00000000 00000000
> 0x00d0: 00000000 00000000 00000000 00000000
> 0x00e0: 00000000 00000000 00000000 00000000
> 0x00f0: 00000000 00000000 00000000 00000000
>
> acpidump:
> begin-base64 644 APIC.3
> QVBJQ4AAAAAB2kJPQ0hTIEJYUENBUElDAQAAAEJYUEMBAAAAAADg/gEAAAAACAAAAQAAAAAIAQEB
> AAAAAQwAAAAAwP4AAAAAAgoAAAIAAAAAAAIKAAUFAAAADQACCgAJCQAAAA0AAgoACgoAAAANAAIK
> AAsLAAAADQAEBv8AAAE=
> ====
> begin-base64 644 DSDT.2
> RFNEVJoVAAAB9kJPQ0hTIEJYUENEU0RUAQAAAEJYUEMBAAAAEEkEXABbgERCR18BCwIEAVuBC0RC
> R18BREJHQggULERCVUcBmGhglmBgdIdgAWFwAGKiEJViYXCDiGBiAERCR0J1YnAKCkRCR0IQIl9T
> Ql9bghtQQ0kwCF9ISUQMQdAKAwhfQURSAAhfVUlEARBNCF9TQl9bgkUISFBFVAhfSElEDEHQAQMI
> X1VJRABbgEhQVE0ADAAA0P4LAARbgRBIUFRNE1ZFTkQgUFJEXyAUNl9TVEEAcFZFTkRgcFBSRF9h
> emAKEGCgDJGTYACTYAv//6QAoA6Rk2EAlGEMAOH1BaQApAoPCF9DUlMREQoOhgkAAAAA0P4ABAAA
> eQAQJS5fU0JfUENJMFuCGVBYMTMIX0FEUgwDAAEAW4BQMTNDAgAK/xBABS5fU0JfUENJMFuCQwRJ
> U0FfCF9BRFIMAAABAFuAUDQwQwIKYAoEW4EmXi5QWDEzUDEzQwAASC8AB0xQRU4BADgAA0NBRU4B
> AANDQkVOARBAHC8DX1NCX1BDSTBJU0FfW4ItUlRDXwhfSElEDEHQCwAIX0NSUxEYChVHAXAAcAAQ
> AiIAAUcBcgByAAIGeQBbgjdLQkRfCF9ISUQMQdADAxQJX1NUQQCkCg8IX0NSUxEYChVHAWAAYAAB
> AUcBZABkAAEBIgIAeQBbgidNT1VfCF9ISUQMQdAPExQJX1NUQQCkCg8IX0NSUxEICgUiABB5AFuC
> TgRGREMwCF9ISUQMQdAHAAhfQ1JTERsKGEcB8gPyAwAERwH3A/cDAAEiQAAqBAB5AAhfRkRFERcK
> FAAAAAAAAAAAAAAAAAAAAAACAAAAW4I+TFBUXwhfSElEDEHQBAAUGF9TVEEAcExQRU5goAaTYACk
> AKEEpAoPCF9DUlMREAoNRwF4A3gDCAgigAB5AFuCRQRDT00xCF9ISUQMQdAFAQhfVUlEARQYX1NU
> QQBwQ0FFTmCgBpNgAKQAoQSkCg8IX0NSUxEQCg1HAfgD+AMACCIQAHkAW4JGBENPTTIIX0hJRAxB
> 0AUBCF9VSUQKAhQYX1NUQQBwQ0JFTmCgBpNgAKQAoQSkCg8IX0NSUxEQCg1HAfgC+AIACCIIAHkA
> EEgILl9TQl9QQ0kwW4BQQ1NUAQsArgoIW4EQUENTVENQQ0lVIFBDSUQgW4BTRUpfAQsIrgoEW4EL
> U0VKX0NCMEVKIFuAQk5NUgELEK4KBFuBC0JOTVJDQk5VTSBbAUJMQ0sAFCVQQ0VKAlsjQkxDS///
> cGhCTlVNcHkBaQBCMEVKWydCTENLpAAQTjZfU0JfEEsKUENJMBRECl9QUlQAcBICgGBwAGGiQgmV
> YQqAcHphCgIAYnB7cmFiAAoDAGOgEJNjAHASCQQAAExOS0QAZKAkk2MBoBGTYQoEcBIJBAAATE5L
> UwBkoQ1wEgkEAABMTktBAGSgEZNjCgJwEgkEAABMTktCAGSgEZNjCgNwEgkEAABMTktDAGRwfXli
> ChAAC///AIhkAABwe2EKAwCIZAEAcGSIYGEAdWGkYFuBJC8DUENJMElTQV9QNDBDAVBSUTAIUFJR
> MQhQUlEyCFBSUTMIFBNJUVNUAaAJewqAaACkCgmkCgsUNklRQ1IJCFBSUjARDgoLiQYACQEAAAAA
> eQCKUFJSMAoFUFJSSaALlWgKgHBoUFJSSaRQUlIwW4JMB0xOS0EIX0hJRAxB0AwPCF9VSUQACF9Q
> UlMRFgoTiQ4ACQMFAAAACgAAAAsAAAB5ABQPX1NUQQCkSVFTVFBSUTAUEV9ESVMAfVBSUTAKgFBS
> UTAUD19DUlMApElRQ1JQUlEwFBdfU1JTAYpoCgVQUlJJcFBSUklQUlEwW4JMB0xOS0IIX0hJRAxB
> 0AwPCF9VSUQBCF9QUlMRFgoTiQ4ACQMFAAAACgAAAAsAAAB5ABQPX1NUQQCkSVFTVFBSUTEUEV9E
> SVMAfVBSUTEKgFBSUTEUD19DUlMApElRQ1JQUlExFBdfU1JTAYpoCgVQUlJJcFBSUklQUlExW4JN
> B0xOS0MIX0hJRAxB0AwPCF9VSUQKAghfUFJTERYKE4kOAAkDBQAAAAoAAAALAAAAeQAUD19TVEEA
> pElRU1RQUlEyFBFfRElTAH1QUlEyCoBQUlEyFA9fQ1JTAKRJUUNSUFJRMhQXX1NSUwGKaAoFUFJS
> SXBQUlJJUFJRMluCTQdMTktECF9ISUQMQdAMDwhfVUlECgMIX1BSUxEWChOJDgAJAwUAAAAKAAAA
> CwAAAHkAFA9fU1RBAKRJUVNUUFJRMxQRX0RJUwB9UFJRMwqAUFJRMxQPX0NSUwCkSVFDUlBSUTMU
> F19TUlMBimgKBVBSUklwUFJSSVBSUTNbgk8ETE5LUwhfSElEDEHQDA8IX1VJRAoECF9QUlMRDgoL
> iQYACQEJAAAAeQAUCV9TVEEApAoLFAZfRElTABQLX0NSUwCkX1BSUxQGX1NSUwEQRxxfU0JfFDVD
> UE1BAnCDiENQT05oAGBwEQsKCAAIAAAAAAAAYXBpiGEKAgBwaIhhCgMAcGCIYQoEAKRhFBpDUFNU
> AXCDiENQT05oAGCgBWCkCg+hA6QAFApDUEVKAlsiCsgUSgZQUlNDAHBQUlNfZXAAYnAAYKJGBZVg
> h0NQT05wg4hDUE9OYABhoAp7YAoHAHpiAWKhDHCDiGV6YAoDAABicHtiAQBjoCKSk2FjcGOIQ1BP
> TmAAoAqTYwFOVEZZYAGhCE5URllgCgN1YFuCSwQuUENJMFBSRVMIX0hJRAxB0AoGCF9VSUQNQ1BV
> IEhvdHBsdWcgcmVzb3VyY2VzAAhfU1RBCgsIX0NSUxENCgpHAQCvAK8BIHkAW4BQUlNUAQsArwog
> W4EMUFJTVAFQUlNfQBBbgzRDUDAwAAAAAAAAFA1fTUFUAKRDUE1BAAAUDF9TVEEApENQU1QAFA1f
> RUowAaRDUEVKAGhbgzRDUDAxAQAAAAAAFA1fTUFUAKRDUE1BAQEUDF9TVEEApENQU1QBFA1fRUow
> AaRDUEVKAWgUHE5URlkCoAqTaACGQ1AwMGmgCpNoAYZDUDAxaQhDUE9OEgQCAQEUFlwuX0dQRV9F
> MDIAXC5fU0JfUFJTQxBOLS5fU0JfUENJMFuCQS1NSFBECF9ISUQNUE5QMEEwNgAIX1VJRA1NZW1v
> cnkgaG90cGx1ZyByZXNvdXJjZXMAFBNfU1RBAKAJk01ETlIApACkCgtbAU1MQ0sAFEEGTVNDTgCg
> CZNNRE5SAKQAcABgWyNNTENL//+iPJVgTUROUnBgTVNFTKATk01JTlMBTVRGWWABcAFNSU5ToRag
> FJNNUk1WAU1URllgCgNwAU1STVZyYAFgWydNTENLpAEULU1SU1QBcABgWyNNTENL//9wmWgATVNF
> TKALk01FU18BcAoPYFsnTUxDS6RgFEEYTUNSUwlbI01MQ0v//3CZaABNU0VMCE1SNjQRMwowiisA
> AAwDAAAAAAAAAAAAAAAAAAAAAP7/////////AAAAAAAAAAD//////////3kAik1SNjQKDk1JTkyK
> TVI2NAoSTUlOSIpNUjY0CiZMRU5Mik1SNjQKKkxFTkiKTVI2NAoWTUFYTIpNUjY0ChpNQVhIcE1S
> QkhNSU5IcE1SQkxNSU5McE1STEhMRU5IcE1STExMRU5Mck1JTkxMRU5MTUFYTHJNSU5ITEVOSE1B
> WEigFJVNQVhMTUlOTHJNQVhIAU1BWEigEZVNQVhMAXRNQVhIAU1BWEh0TUFYTAFNQVhMoEQHk01B
> WEgACE1SMzIRHwochxcAAAwDAAAAAAAAAAD+////AAAAAP////95AIpNUjMyCgpNSU5fik1SMzIK
> Dk1BWF+KTVIzMgoWTEVOX3BNSU5MTUlOX3BNQVhMTUFYX3BMRU5MTEVOX1snTUxDS6RNUjMyWydN
> TENLpE1SNjQUJE1QWE0BWyNNTENL//9wmWgATVNFTHBNUFhfYFsnTUxDS6RgFChNT1NUBFsjTUxD
> S///cJloAE1TRUxwaU1PRVZwak1PU0NbJ01MQ0sUIk1FSjACWyNNTENL//9wmWgATVNFTHABTUVK
> X1snTUxDSxBJBl9HUEUIX0hJRA1BQ1BJMDAwNgAUOV9FMDEAWyNcLwNfU0JfUENJMEJMQ0v//1wv
> A19TQl9QQ0kwUENOVFsnXC8DX1NCX1BDSTBCTENLFBlfRTAzAFwvBF9TQl9QQ0kwTUhQRE1TQ04Q
> RRFcLl9TQl9QQ0kwCF9DUlMRQgcKbogNAAIMAAAAAAD/AAAAAAFHAfgM+AwBCIgNAAEMAwAAAAD3
> DAAA+AyIDQABDAMAAAAN//8AAADzhxcAAAwDAAAAAAAACgD//wsAAAAAAAAAAgCHFwAADAEAAAAA
> AAAAwP//v/4AAAAAAADAPnkAW4JDBEdQRTAIX0hJRA1QTlAwQTA2AAhfVUlEDUdQRTAgcmVzb3Vy
> Y2VzAAhfU1RBCgsIX0NSUxENCgpHAeCv4K8BBHkAW4JKBFBIUFIIX0hJRA1QTlAwQTA2AAhfVUlE
> DVBDSSBIb3RwbHVnIHJlc291cmNlcwAIX1NUQQoLCF9DUlMRDQoKRwEArgCuARR5ABApXAAIX1Mz
> XxIGBAEBAAAIX1M0XxIIBAoCCgIAAAhfUzVfEgYEAAAAABA7XC5fU0JfUENJMFuCLkZXQ0YIX0hJ
> RA1RRU1VMDAwMgAIX1NUQQoLCF9DUlMRDQoKRwEQBRAFAQJ5ABBGelxfU0JfEE0IXC8DX1NCX1BD
> STBNSFBECE1ETlIACF9DUlMRDQoKRwEACgAKABh5AFuASFBNUgELAAoKGFuBH0hQTVIDTVJCTCBN
> UkJIIE1STEwgTVJMSCBNUFhfIFuBHUhQTVJBAEAKTUVTXwFNSU5TAU1STVYBTUVKXwFbgRVIUE1S
> A01TRUwgTU9FViBNT1NDIBQGTVRGWQIQSXBQQ0kwCEJTRUwAW4ILUzAwXwhfQURSAFuCKVMxMF8I
> X0FEUgwAAAIACF9TVU4KAhQSX0VKMAFQQ0VKQlNFTF9TVU5bgilTMThfCF9TVU4KAwhfQURSDAAA
> AwAUEl9FSjABUENFSkJTRUxfU1VOW4IpUzIwXwhfQURSDAAABAAIX1NVTgoEFBJfRUowAVBDRUpC
> U0VMX1NVTluCKVMyOF8IX0FEUgwAAAUACF9TVU4KBRQSX0VKMAFQQ0VKQlNFTF9TVU5bgilTMzBf
> CF9BRFIMAAAGAAhfU1VOCgYUEl9FSjABUENFSkJTRUxfU1VOW4IpUzM4XwhfU1VOCgcIX0FEUgwA
> AAcAFBJfRUowAVBDRUpCU0VMX1NVTluCKVM0MF8IX1NVTgoICF9BRFIMAAAIABQSX0VKMAFQQ0VK
> QlNFTF9TVU5bgilTNDhfCF9TVU4KCQhfQURSDAAACQAUEl9FSjABUENFSkJTRUxfU1VOW4IpUzUw
> XwhfU1VOCgoIX0FEUgwAAAoAFBJfRUowAVBDRUpCU0VMX1NVTluCKVM1OF8IX1NVTgoLCF9BRFIM
> AAALABQSX0VKMAFQQ0VKQlNFTF9TVU5bgilTNjBfCF9TVU4KDAhfQURSDAAADAAUEl9FSjABUENF
> SkJTRUxfU1VOW4IpUzY4XwhfU1VOCg0IX0FEUgwAAA0AFBJfRUowAVBDRUpCU0VMX1NVTluCKVM3
> MF8IX1NVTgoOCF9BRFIMAAAOABQSX0VKMAFQQ0VKQlNFTF9TVU5bgilTNzhfCF9TVU4KDwhfQURS
> DAAADwAUEl9FSjABUENFSkJTRUxfU1VOW4IpUzgwXwhfU1VOChAIX0FEUgwAABAAFBJfRUowAVBD
> RUpCU0VMX1NVTluCKVM4OF8IX1NVTgoRCF9BRFIMAAARABQSX0VKMAFQQ0VKQlNFTF9TVU5bgilT
> OTBfCF9TVU4KEghfQURSDAAAEgAUEl9FSjABUENFSkJTRUxfU1VOW4IpUzk4XwhfU1VOChMIX0FE
> UgwAABMAFBJfRUowAVBDRUpCU0VMX1NVTluCKVNBMF8IX1NVTgoUCF9BRFIMAAAUABQSX0VKMAFQ
> Q0VKQlNFTF9TVU5bgilTQThfCF9TVU4KFQhfQURSDAAAFQAUEl9FSjABUENFSkJTRUxfU1VOW4Ip
> U0IwXwhfU1VOChYIX0FEUgwAABYAFBJfRUowAVBDRUpCU0VMX1NVTluCKVNCOF8IX1NVTgoXCF9B
> RFIMAAAXABQSX0VKMAFQQ0VKQlNFTF9TVU5bgilTQzBfCF9TVU4KGAhfQURSDAAAGAAUEl9FSjAB
> UENFSkJTRUxfU1VOW4IpU0M4XwhfU1VOChkIX0FEUgwAABkAFBJfRUowAVBDRUpCU0VMX1NVTluC
> KVNEMF8IX1NVTgoaCF9BRFIMAAAaABQSX0VKMAFQQ0VKQlNFTF9TVU5bgilTRDhfCF9TVU4KGwhf
> QURSDAAAGwAUEl9FSjABUENFSkJTRUxfU1VOW4IpU0UwXwhfU1VOChwIX0FEUgwAABwAFBJfRUow
> AVBDRUpCU0VMX1NVTluCKVNFOF8IX1NVTgodCF9BRFIMAAAdABQSX0VKMAFQQ0VKQlNFTF9TVU5b
> gilTRjBfCF9TVU4KHghfQURSDAAAHgAUEl9FSjABUENFSkJTRUxfU1VOW4IpU0Y4XwhfU1VOCh8I
> X0FEUgwAAB8AFBJfRUowAVBDRUpCU0VMX1NVThRFHERWTlQCoAx7aAoEAIZTMTBfaaAMe2gKCACG
> UzE4X2mgDHtoChAAhlMyMF9poAx7aAogAIZTMjhfaaAMe2gKQACGUzMwX2mgDHtoCoAAhlMzOF9p
> oA17aAsAAQCGUzQwX2mgDXtoCwACAIZTNDhfaaANe2gLAAQAhlM1MF9poA17aAsACACGUzU4X2mg
> DXtoCwAQAIZTNjBfaaANe2gLACAAhlM2OF9poA17aAsAQACGUzcwX2mgDXtoCwCAAIZTNzhfaaAP
> e2gMAAABAACGUzgwX2mgD3toDAAAAgAAhlM4OF9poA97aAwAAAQAAIZTOTBfaaAPe2gMAAAIAACG
> Uzk4X2mgD3toDAAAEAAAhlNBMF9poA97aAwAACAAAIZTQThfaaAPe2gMAABAAACGU0IwX2mgD3to
> DAAAgAAAhlNCOF9poA97aAwAAAABAIZTQzBfaaAPe2gMAAAAAgCGU0M4X2mgD3toDAAAAAQAhlNE
> MF9poA97aAwAAAAIAIZTRDhfaaAPe2gMAAAAEACGU0UwX2mgD3toDAAAACAAhlNFOF9poA97aAwA
> AABAAIZTRjBfaaAPe2gMAAAAgACGU0Y4X2kUH1BDTlQAcABCTlVNRFZOVFBDSVUBRFZOVFBDSUQK
> Aw==
> ====
> begin-base64 644 FACP.1
> RkFDUHQAAAAB50JPQ0hTIEJYUENGQUNQAQAAAEJYUEMBAAAAAAD+v0AA/r8BAAkAsgAAAPHwAAAA
> BgAAAAAAAAQGAAAAAAAAAAAAAAgGAADgrwAAAAAAAAQCAAQEAAAA/w//DwAAAAAAAAAAMgAAAKWA
> AAA=
> ====
> begin-base64 644 RSDT.0
> UlNEVCwAAAABoUJPQ0hTIEJYUENSU0RUAQAAAEJYUEMBAAAA2hX+v04W/r8=
> ====
> begin-base64 644 headers
> ClJTRCBQVFI6IENoZWNrc3VtPTE3NywgT0VNSUQ9Qk9DSFMsIFJldmlzaW9uPTAsIFJzZHRBZGRy
> ZXNzPTB4YmZmZTE2Y2UKCgpSU0RUOiBMZW5ndGg9NDQsIFJldmlzaW9uPTEsIENoZWNrc3VtPTE2
> MSwKCU9FTUlEPUJPQ0hTLCBPRU0gVGFibGUgSUQ9QlhQQ1JTRFQsIE9FTSBSZXZpc2lvbj0weDEs
> CglDcmVhdG9yIElEPUJYUEMsIENyZWF0b3IgUmV2aXNpb249MHgxCgoKCUVudHJpZXM9eyAweGJm
> ZmUxNWRhLCAweGJmZmUxNjRlIH0KCgoJRFNEVD0weGJmZmUwMDQwCglJTlRfTU9ERUw9QVBJQwoJ
> U0NJX0lOVD05CglTTUlfQ01EPTB4YjIsIEFDUElfRU5BQkxFPTB4ZjEsIEFDUElfRElTQUJMRT0w
> eGYwLCBTNEJJT1NfUkVRPTB4MAoJUE0xYV9FVlRfQkxLPTB4NjAwLTB4NjAzCglQTTFhX0NOVF9C
> TEs9MHg2MDQtMHg2MDUKCVBNMl9UTVJfQkxLPTB4NjA4LTB4NjBiCglQTTJfR1BFMF9CTEs9MHhh
> ZmUwLTB4YWZlMwoJUF9MVkwyX0xBVD00MDk1bXMsIFBfTFZMM19MQVQ9NDA5NW1zCglGTFVTSF9T
> SVpFPTAsIEZMVVNIX1NUUklERT0wCglEVVRZX09GRlNFVD0wLCBEVVRZX1dJRFRIPTAKCURBWV9B
> TFJNPTAsIE1PTl9BTFJNPTAsIENFTlRVUlk9NTAKCUZsYWdzPXtXQklOVkQsUFJPQ19DMSxTTFBf
> QlVUVE9OLFJUQ19TNH0KCgpEU0RUOiBMZW5ndGg9NTUzMCwgUmV2aXNpb249MSwgQ2hlY2tzdW09
> MjQ2LAoJT0VNSUQ9Qk9DSFMsIE9FTSBUYWJsZSBJRD1CWFBDRFNEVCwgT0VNIFJldmlzaW9uPTB4
> MSwKCUNyZWF0b3IgSUQ9QlhQQywgQ3JlYXRvciBSZXZpc2lvbj0weDEKCgpBUElDOiBMZW5ndGg9
> MTI4LCBSZXZpc2lvbj0xLCBDaGVja3N1bT0yMTgsCglPRU1JRD1CT0NIUywgT0VNIFRhYmxlIElE
> PUJYUENBUElDLCBPRU0gUmV2aXNpb249MHgxLAoJQ3JlYXRvciBJRD1CWFBDLCBDcmVhdG9yIFJl
> dmlzaW9uPTB4MQoK
> ====
>
>
> --
> jca | PGP : 0x1524E7EE / 5135 92C1 AD36 5293 2BDF  DDCC 0DFA 74AE 1524 E7EE
>