IPsec traffic causes system to halt with 'kernel: double fault trap, code=0' at 'pf_setup_pdesc+0x3f'

[Peter Müller]

>Synopsis: IPsec traffic causes system to halt with 'kernel: double fault trap, code=0' at 'pf_setup_pdesc+0x3f'
>Category: kernel
>Environment:
        System      : OpenBSD 6.6
        Details     : OpenBSD 6.6 (GENERIC.MP) #4: Wed Jan 15 10:55:43 MST 2020
                         [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP

        Architecture: OpenBSD.amd64
        Machine     : amd64
>Description:
        An OpenBSD machine acting as an upstream proxy for an remote IPsec client stalls
        as soon as a considerable amount of IPsec traffic (~ 10 MBit/Sec. over ~ 30 minutes)
        is generated by accessing websites via it, especially when it comes to video streaming.
       
        This behaviour seems to be somewhat reproducible under load after some time, as it
        does not appear if the IPsec is used for keep-alive ("INFORMATIONAL") and ping messages only.
       
        Used HTTP proxy is Squid 4.9 as installed from ports, and OpenIKED for the IPsec connection.
       
        After stalling, the system cannot be reached via network anymore and prints these lines
        on the physical console:
       
        kernel: double fault trap, code=0
        Stopped at      pf_setup_pdesc+0x3f:    callq    memset+0x4
        ddb{0}> trace
        pf_setup_pdesc(ffff8000210e4098,Z,Z,ffff80000016c400,fffffd803e71ca00,ffff8000210e41ae) at pf_setup_pdesc+0x3f
        pf_test(2,2,ffff80000013f000,ffff8000210e4280) at pf_test+0xfe
        ip_output(fffffd803e71ca00,0,fffffd806547d2a8,800,0,fffffd806547d238) at ip_output+0x7cf
        tcp_output(ffff800000584ee0) at tcp_output+0x15c1
        tcp_output(ffff800000584ee0) at tcp_output+0x1941
        tcp_output(ffff800000584ee0) at tcp_output+0x1941
        tcp_output(ffff800000584ee0) at tcp_output+0x1941
        tcp_output(ffff800000584ee0) at tcp_output+0x1941
        tcp_output(ffff800000584ee0) at tcp_output+0x1941
        tcp_output(ffff800000584ee0) at tcp_output+0x1941
        tcp_output(ffff800000584ee0) at tcp_output+0x1941
        tcp_output(ffff800000584ee0) at tcp_output+0x1941
        tcp_output(ffff800000584ee0) at tcp_output+0x1941
        tcp_output(ffff800000584ee0) at tcp_output+0x1941
        tcp_output(ffff800000584ee0) at tcp_output+0x1941
        tcp_output(ffff800000584ee0) at tcp_output+0x1941
        tcp_output(ffff800000584ee0) at tcp_output+0x1941
        tcp_output(ffff800000584ee0) at tcp_output+0x1941
        tcp_output(ffff800000584ee0) at tcp_output+0x1941
        tcp_output(ffff800000584ee0) at tcp_output+0x1941
        tcp_output(ffff800000584ee0) at tcp_output+0x1941
        tcp_output(ffff800000584ee0) at tcp_output+0x1941
        tcp_output(ffff800000584ee0) at tcp_output+0x1941
        tcp_output(ffff800000584ee0) at tcp_output+0x1941
        tcp_output(ffff800000584ee0) at tcp_output+0x1941
        tcp_output(ffff800000584ee0) at tcp_output+0x1941
        tcp_output(ffff800000584ee0) at tcp_output+0x1941
        tcp_output(ffff800000584ee0) at tcp_output+0x1941
        tcp_output(ffff800000584ee0) at tcp_output+0x1941
        tcp_output(ffff800000584ee0) at tcp_output+0x1941
        tcp_output(ffff800000584ee0) at tcp_output+0x1941
        tcp_output(ffff800000584ee0) at tcp_output+0x1941
        tcp_output(ffff800000584ee0) at tcp_output+0x1941
        tcp_output(ffff800000584ee0) at tcp_output+0x1941
        tcp_output(ffff800000584ee0) at tcp_output+0x1941
        tcp_output(ffff800000584ee0) at tcp_output+0x1941
        tcp_output(ffff800000584ee0) at tcp_output+0x1941
        tcp_output(ffff800000584ee0) at tcp_output+0x1941
        tcp_output(ffff800000584ee0) at tcp_output+0x1941
        tcp_output(ffff800000584ee0) at tcp_output+0x1941
        tcp_output(ffff800000584ee0) at tcp_output+0x1941
        tcp_timer_rexmt(ffff800000584ee0) at tcp_timer_rexmt+0x3f5
        softclock_thread(ffff8000210d2ed0) at softclock_thread+0xfb
        end trace frame: 0x0, count: -40
        ddb{0}>
       
        Content of /etc/iked.conf is:
       
        set fragmentation
        ikev2 "[REDACTED]" active esp \
                from 10.xxx.xxx.2/32 to 10.xxx.xxx.0/24 \
                local [REDACTED] peer [REDACTED] \
                ikesa auth hmac-sha2-512 enc aes-256 prf hmac-sha2-512 group curve25519 \
                childsa enc aes-256-gcm group curve25519 \
                srcid [REDACTED] dstid [REDACTED] \
                ikelifetime 3h \
                lifetime 1h

        Not sure whether it is related or not, but due to MTU issues, /etc/pf.conf
        contains the following line:

        match on enc0 scrub (max-mss 1394)

>How-To-Repeat:
        To reproduce this behaviour, seems to be enough to set up an OpenBSD machine
        running the OpenIKED configuration above and a local Squid instance accessed
        from the other end of the IPsec connection.
   
        After approximately 30 minutes of random website-related traffic (video streaming
        works best), system halts showing the mentioned output on its console.

>Fix:
        N/A

dmesg:
OpenBSD 6.6 (GENERIC.MP) #4: Wed Jan 15 10:55:43 MST 2020
    [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 2130694144 (2031MB)
avail mem = 2053459968 (1958MB)
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.4 @ 0x7ffffe70 (11 entries)
bios0: vendor Bochs version "Bochs" date 01/01/2007
bios0: Joyent SmartDC HVM
acpi0 at bios0: ACPI 1.0
acpi0: sleep states S3 S4 S5
acpi0: tables DSDT FACP SSDT APIC HPET
acpi0: wakeup devices
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz, 2937.47 MHz, 06-3e-04
cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,SSE3,SSSE3,CX16,SSE4.1,SSE4.2,x2APIC,POPCNT,AES,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,XSAVEOPT,XSAVEC,XSAVES,MELTDOWN
cpu0: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 512KB 64b/line 16-way L2 cache
cpu0: ITLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu0: DTLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
cpu0: apic clock running at 1009MHz
cpu1 at mainbus0: apid 1 (application processor)
cpu1: Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz, 2828.08 MHz, 06-3e-04
cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,SSE3,SSSE3,CX16,SSE4.1,SSE4.2,x2APIC,POPCNT,AES,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,XSAVEOPT,XSAVEC,XSAVES,MELTDOWN
cpu1: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 512KB 64b/line 16-way L2 cache
cpu1: ITLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu1: DTLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu1: smt 0, core 0, package 0
ioapic0 at mainbus0: apid 2 pa 0xfec00000, version 11, 24 pins, remapped
acpihpet0 at acpi0: 100000000 Hz
acpiprt0 at acpi0: bus 0 (PCI0)
acpicpu0 at acpi0: C1(@1 halt!)
acpicpu1 at acpi0: C1(@1 halt!)
"ACPI0006" at acpi0 not configured
acpipci0 at acpi0 PCI0: _OSC failed
acpicmos0 at acpi0
"ACPI0007" at acpi0 not configured
"ACPI0007" at acpi0 not configured
cpu0: using IvyBridge MDS workaround
pvbus0 at mainbus0: KVM
pvclock0 at pvbus0
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel 82441FX" rev 0x02
pcib0 at pci0 dev 1 function 0 "Intel 82371SB ISA" rev 0x00
pciide0 at pci0 dev 1 function 1 "Intel 82371SB IDE" rev 0x00: DMA, channel 0 wired to compatibility, channel 1 wired to compatibility
pciide0: channel 0 disabled (no drives)
atapiscsi0 at pciide0 channel 1 drive 0
scsibus1 at atapiscsi0: 2 targets
cd0 at scsibus1 targ 0 lun 0: <QEMU, QEMU DVD-ROM, 0.14> removable
cd0(pciide0:1:0): using PIO mode 4, DMA mode 2
uhci0 at pci0 dev 1 function 2 "Intel 82371SB USB" rev 0x01: apic 2 int 11
piixpm0 at pci0 dev 1 function 3 "Intel 82371AB Power" rev 0x03: apic 2 int 10
iic0 at piixpm0
iic0: addr 0x18 00=b4 01=a3 02=a3 03=a3 04=a3 05=a3 06=a3 07=a3 08=a3 09=fa 0a=d7 3e=b4 48=b4 4a=b4 4e=b4 fc=b4 fe=b4 words 00=ffff 01=ffff 02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x19 3e=b4 48=b4 4a=b4 4e=b4 fc=b4 fe=b4 words 00=ffff 01=ffff 02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x1a 00=b4 01=a3 02=a3 03=a3 04=a3 05=a3 06=a3 07=a3 08=a3 09=fa 0a=d7 3e=b4 48=b4 4a=b4 4e=b4 fc=b4 fe=b4 words 00=ffff 01=ffff 02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x1b 3e=b4 48=b4 4a=b4 4e=b4 fc=b4 fe=b4 words 00=ffff 01=ffff 02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x1c 0f=06 3e=b4 48=b4 4a=b4 4e=b4 fc=b4 fe=b4 words 00=ffff 01=ffff 02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x1d 0f=06 3e=b4 48=b4 4a=b4 4e=b4 fc=b4 fe=b4 words 00=ffff 01=ffff 02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x1e 3e=b4 48=b4 4a=b4 4e=b4 fc=b4 fe=b4 words 00=ffff 01=ffff 02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x1f 3e=b4 48=b4 4a=b4 4e=b4 fc=b4 fe=b4 words 00=ffff 01=ffff 02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x20 3e=b4 48=b4 4a=b4 4e=b4 fc=b4 fe=b4 words 00=ffff 01=ffff 02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x21 3e=b4 48=b4 4a=b4 4e=b4 fc=b4 fe=b4 words 00=ffff 01=ffff 02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x22 3e=b4 48=b4 4a=b4 4e=b4 fc=b4 fe=b4 words 00=ffff 01=ffff 02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x23 3e=b4 48=b4 4a=b4 4e=b4 fc=b4 fe=b4 words 00=ffff 01=ffff 02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x24 3e=b4 48=b4 4a=b4 4e=b4 fc=b4 fe=b4 words 00=ffff 01=ffff 02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x25 3e=b4 48=b4 4a=b4 4e=b4 fc=b4 fe=b4 words 00=ffff 01=ffff 02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x26 3e=b4 48=b4 4a=b4 4e=b4 fc=b4 fe=b4 words 00=ffff 01=ffff 02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x27 3e=b4 48=b4 4a=b4 4e=b4 fc=b4 fe=b4 words 00=ffff 01=ffff 02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x28 3e=b4 48=b4 4a=b4 4e=b4 4f=b4 fc=b4 fe=b4 words 00=ffff 01=ffff 02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x29 00=b4 01=a3 02=a3 03=a3 04=a3 05=a3 06=a3 07=a3 08=a3 09=fa 0a=d7 3e=b4 48=b4 4a=b4 4e=b4 4f=b4 fc=b4 fe=b4 words 00=ffff 01=ffff 02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x2a 3e=b4 48=b4 4a=b4 4e=b4 4f=b4 fc=b4 fe=b4 words 00=ffff 01=ffff 02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x2b 00=b4 01=a3 02=a3 03=a3 04=a3 05=a3 06=a3 07=a3 08=a3 09=fa 0a=d7 3e=b4 48=b4 4a=b4 4e=b4 4f=b4 fc=b4 fe=b4 words 00=ffff 01=ffff 02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x2c 00=b4 0e=b4 16=b4 3e=b4 48=b4 4a=b4 4e=b4 4f=b4 fc=b4 fe=b4 words 00=ffff 01=ffff 02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x2d 00=b4 04=b4 0e=b4 16=b4 3e=b4 48=b4 4a=b4 4e=b4 4f=b4 fc=b4 fe=b4 words 00=ffff 01=ffff 02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x2e 00=b4 04=b4 0e=b4 16=b4 3e=b4 48=b4 4a=b4 4e=b4 4f=b4 fc=b4 fe=b4 words 00=ffff 01=ffff 02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x2f 00=b4 0e=b4 3e=b4 48=b4 4a=b4 4e=b4 4f=b4 fc=b4 fe=b4 words 00=ffff 01=ffff 02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x48 3e=b4 48=b4 4a=b4 4e=b4 fc=b4 fe=b4 words 00=ffff 01=ffff 02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x49 3e=b4 48=b4 4a=b4 4e=b4 fc=b4 fe=b4 words 00=ffff 01=ffff 02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x4a 3e=b4 48=b4 4a=b4 4e=b4 fc=b4 fe=b4 words 00=ffff 01=ffff 02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x4b 3e=b4 48=b4 4a=b4 4e=b4 fc=b4 fe=b4 words 00=ffff 01=ffff 02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x4c 00=00 01=00 02=00 03=00 04=00 05=00 06=00 07=00 08=00 3e=b4 48=b4 4a=b4 4e=b4 fc=b4 fe=b4 words 00=ffff 01=ffff 02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x4d 3e=b4 48=b4 4a=b4 4e=b4 fc=b4 fe=b4 words 00=ffff 01=ffff 02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x4e 00=00 01=00 02=00 03=00 04=00 05=00 06=00 07=00 08=00 3e=b4 48=b4 4a=b4 4e=b4 fc=b4 fe=b4 words 00=ffff 01=ffff 02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
vga1 at pci0 dev 2 function 0 "Bochs VGA" rev 0x00
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
virtio0 at pci0 dev 3 function 0 "Qumranet Virtio Network" rev 0x00
vio0 at virtio0: address 62:1f:27:83:34:12
virtio0: msix shared
virtio1 at pci0 dev 4 function 0 "Qumranet Virtio Storage" rev 0x00
vioblk0 at virtio1
scsibus2 at vioblk0: 2 targets
sd0 at scsibus2 targ 0 lun 0: <VirtIO, Block Device, >
sd0: 10240MB, 512 bytes/sector, 20971520 sectors
virtio1: msix shared
virtio2 at pci0 dev 5 function 0 "Qumranet Virtio Storage" rev 0x00
vioblk1 at virtio2
scsibus3 at vioblk1: 2 targets
sd1 at scsibus3 targ 0 lun 0: <VirtIO, Block Device, >
sd1: 30720MB, 512 bytes/sector, 62914560 sectors
virtio2: msix shared
isa0 at pcib0
isadma0 at isa0
fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
pckbc0 at isa0 port 0x60/5 irq 1 irq 12
pckbd0 at pckbc0 (kbd slot)
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pms0 at pckbc0 (aux slot)
wsmouse0 at pms0 mux 0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
usb0 at uhci0: USB revision 1.0
uhub0 at usb0 configuration 1 interface 0 "Intel UHCI root hub" rev 1.00/1.00 addr 1
uhidev0 at uhub0 port 1 configuration 1 interface 0 "QEMU 0.14.1 QEMU USB Tablet" rev 1.00/0.00 addr 2
uhidev0: iclass 3/1
ums0 at uhidev0: 3 buttons, Z dir
wsmouse1 at ums0 mux 0
vscsi0 at root
scsibus4 at vscsi0: 256 targets
softraid0 at root
scsibus5 at softraid0: 256 targets
root on sd0a (2f0ee6c14449efb2.a) swap on sd0b dump on sd0b
fd0 at fdc0 drive 0: density unknown
fd1 at fdc0 drive 1: density unknown

usbdevs:
Controller /dev/usb0:
addr 01: 8086:0000 Intel, UHCI root hub
         full speed, self powered, config 1, rev 1.00
         driver: uhub0
addr 02: 0627:0001 QEMU 0.14.1, QEMU USB Tablet
         full speed, power 100 mA, config 1, rev 0.00, iSerial 42
         driver: uhidev0

pcidump:
Domain /dev/pci0:
 0:0:0: Intel 82441FX
        0x0000: Vendor ID: 8086, Product ID: 1237
        0x0004: Command: 0003, Status: 0000
        0x0008: Class: 06 Bridge, Subclass: 00 Host,
                Interface: 00, Revision: 02
        0x000c: BIST: 00, Header Type: 00, Latency Timer: 00,
                Cache Line Size: 00
        0x0010: BAR empty (00000000)
        0x0014: BAR empty (00000000)
        0x0018: BAR empty (00000000)
        0x001c: BAR empty (00000000)
        0x0020: BAR empty (00000000)
        0x0024: BAR empty (00000000)
        0x0028: Cardbus CIS: 00000000
        0x002c: Subsystem Vendor ID: 1af4 Product ID: 1100
        0x0030: Expansion ROM Base Address: 00000000
        0x0038: 00000000
        0x003c: Interrupt Pin: 00 Line: 00 Min Gnt: 00 Max Lat: 00
        0x0000: 12378086 00000003 06000002 00000000
        0x0010: 00000000 00000000 00000000 00000000
        0x0020: 00000000 00000000 00000000 11001af4
        0x0030: 00000000 00000000 00000000 00000000
        0x0040: 00000000 00000000 00000000 00000000
        0x0050: 00000000 ff000000 11111000 33333333
        0x0060: 00000000 00000000 00000000 00000000
        0x0070: 00020000 00000000 00000000 00000000
        0x0080: 00000000 00000000 00000000 00000000
        0x0090: 00000000 00000000 00000000 00000000
        0x00a0: 00000000 00000000 00000000 00000000
        0x00b0: 00000000 00000000 00000000 00000000
        0x00c0: 00000000 00000000 00000000 00000000
        0x00d0: 00000000 00000000 00000000 00000000
        0x00e0: 00000000 00000000 00000000 00000000
        0x00f0: 00000000 00000000 00000000 00000000
 0:1:0: Intel 82371SB ISA
        0x0000: Vendor ID: 8086, Product ID: 7000
        0x0004: Command: 0003, Status: 0200
        0x0008: Class: 06 Bridge, Subclass: 01 ISA,
                Interface: 00, Revision: 00
        0x000c: BIST: 00, Header Type: 80, Latency Timer: 00,
                Cache Line Size: 00
        0x0010: BAR empty (00000000)
        0x0014: BAR empty (00000000)
        0x0018: BAR empty (00000000)
        0x001c: BAR empty (00000000)
        0x0020: BAR empty (00000000)
        0x0024: BAR empty (00000000)
        0x0028: Cardbus CIS: 00000000
        0x002c: Subsystem Vendor ID: 1af4 Product ID: 1100
        0x0030: Expansion ROM Base Address: 00000000
        0x0038: 00000000
        0x003c: Interrupt Pin: 00 Line: 00 Min Gnt: 00 Max Lat: 00
        0x0000: 70008086 02000003 06010000 00800000
        0x0010: 00000000 00000000 00000000 00000000
        0x0020: 00000000 00000000 00000000 11001af4
        0x0030: 00000000 00000000 00000000 00000000
        0x0040: 00000000 00000000 00000000 0003004d
        0x0050: 00000000 00000000 00000000 00000000
        0x0060: 0b0b0a0a 00000000 00000200 00000000
        0x0070: 00000080 0c0c0000 00000002 00000000
        0x0080: 00000000 00000000 00000000 00000000
        0x0090: 00000000 00000000 00000000 00000000
        0x00a0: 00000008 00000000 0000000f 00000000
        0x00b0: 00000000 00000000 00000000 00000000
        0x00c0: 00000000 00000000 00000000 00000000
        0x00d0: 00000000 00000000 00000000 00000000
        0x00e0: 00000000 00000000 00000000 00000000
        0x00f0: 00000000 00000000 00000000 00000000
 0:1:1: Intel 82371SB IDE
        0x0000: Vendor ID: 8086, Product ID: 7010
        0x0004: Command: 0007, Status: 0280
        0x0008: Class: 01 Mass Storage, Subclass: 01 IDE,
                Interface: 80, Revision: 00
        0x000c: BIST: 00, Header Type: 00, Latency Timer: 00,
                Cache Line Size: 00
        0x0010: BAR empty (00000000)
        0x0014: BAR empty (00000000)
        0x0018: BAR empty (00000000)
        0x001c: BAR empty (00000000)
        0x0020: BAR io addr: 0x0000c000/0x0010
        0x0024: BAR empty (00000000)
        0x0028: Cardbus CIS: 00000000
        0x002c: Subsystem Vendor ID: 1af4 Product ID: 1100
        0x0030: Expansion ROM Base Address: 00000000
        0x0038: 00000000
        0x003c: Interrupt Pin: 00 Line: 00 Min Gnt: 00 Max Lat: 00
        0x0000: 70108086 02800007 01018000 00000000
        0x0010: 00000000 00000000 00000000 00000000
        0x0020: 0000c001 00000000 00000000 11001af4
        0x0030: 00000000 00000000 00000000 00000000
        0x0040: a3070000 00000000 00000000 00000000
        0x0050: 00000000 00000000 00000000 00000000
        0x0060: 00000000 00000000 00000000 00000000
        0x0070: 00000000 00000000 00000000 00000000
        0x0080: 00000000 00000000 00000000 00000000
        0x0090: 00000000 00000000 00000000 00000000
        0x00a0: 00000000 00000000 00000000 00000000
        0x00b0: 00000000 00000000 00000000 00000000
        0x00c0: 00000000 00000000 00000000 00000000
        0x00d0: 00000000 00000000 00000000 00000000
        0x00e0: 00000000 00000000 00000000 00000000
        0x00f0: 00000000 00000000 00000000 00000000
 0:1:2: Intel 82371SB USB
        0x0000: Vendor ID: 8086, Product ID: 7020
        0x0004: Command: 0007, Status: 0000
        0x0008: Class: 0c Serial Bus, Subclass: 03 USB,
                Interface: 00, Revision: 01
        0x000c: BIST: 00, Header Type: 00, Latency Timer: 00,
                Cache Line Size: 00
        0x0010: BAR empty (00000000)
        0x0014: BAR empty (00000000)
        0x0018: BAR empty (00000000)
        0x001c: BAR empty (00000000)
        0x0020: BAR io addr: 0x0000c020/0x0020
        0x0024: BAR empty (00000000)
        0x0028: Cardbus CIS: 00000000
        0x002c: Subsystem Vendor ID: 1af4 Product ID: 1100
        0x0030: Expansion ROM Base Address: 00000000
        0x0038: 00000000
        0x003c: Interrupt Pin: 04 Line: 0b Min Gnt: 00 Max Lat: 00
        0x0000: 70208086 00000007 0c030001 00000000
        0x0010: 00000000 00000000 00000000 00000000
        0x0020: 0000c021 00000000 00000000 11001af4
        0x0030: 00000000 00000000 00000000 0000040b
        0x0040: 00000000 00000000 00000000 00000000
        0x0050: 00000000 00000000 00000000 00000000
        0x0060: 00000010 00000000 00010000 00000000
        0x0070: 00000000 00000000 00000000 00000000
        0x0080: 00000000 00000000 00000000 00000000
        0x0090: 00000000 00000000 00000000 00000000
        0x00a0: 00000000 00000000 00000000 00000000
        0x00b0: 00000000 00000000 00000000 00000000
        0x00c0: 00002000 00000000 00000000 00000000
        0x00d0: 00000000 00000000 00000000 00000000
        0x00e0: 00000000 00000000 00000000 00000000
        0x00f0: 00000000 00000000 00000000 00000000
 0:1:3: Intel 82371AB Power
        0x0000: Vendor ID: 8086, Product ID: 7113
        0x0004: Command: 0003, Status: 0280
        0x0008: Class: 06 Bridge, Subclass: 80 Miscellaneous,
                Interface: 00, Revision: 03
        0x000c: BIST: 00, Header Type: 00, Latency Timer: 00,
                Cache Line Size: 00
        0x0010: BAR empty (00000000)
        0x0014: BAR empty (00000000)
        0x0018: BAR empty (00000000)
        0x001c: BAR empty (00000000)
        0x0020: BAR empty (00000000)
        0x0024: BAR empty (00000000)
        0x0028: Cardbus CIS: 00000000
        0x002c: Subsystem Vendor ID: 1af4 Product ID: 1100
        0x0030: Expansion ROM Base Address: 00000000
        0x0038: 00000000
        0x003c: Interrupt Pin: 01 Line: 09 Min Gnt: 00 Max Lat: 00
        0x0000: 71138086 02800003 06800003 00000000
        0x0010: 00000000 00000000 00000000 00000000
        0x0020: 00000000 00000000 00000000 11001af4
        0x0030: 00000000 00000000 00000000 00000109
        0x0040: 0000b001 00000000 00000000 00000000
        0x0050: 00000000 00000000 02000000 10000000
        0x0060: 60000000 98000000 00000000 00000000
        0x0070: 00000000 00000000 00000000 00000000
        0x0080: 00000001 00000000 00000000 00000000
        0x0090: 0000b101 00000000 00000000 00000000
        0x00a0: 00000000 00000000 00000000 00000000
        0x00b0: 00000000 00000000 00000000 00000000
        0x00c0: 00000000 00000000 00000000 00000000
        0x00d0: 00090000 00000000 00000000 00000000
        0x00e0: 00000000 00000000 00000000 00000000
        0x00f0: 00000000 00000000 00000000 00000000
 0:2:0: Bochs VGA
        0x0000: Vendor ID: 1234, Product ID: 1111
        0x0004: Command: 0007, Status: 0000
        0x0008: Class: 03 Display, Subclass: 00 VGA,
                Interface: 00, Revision: 00
        0x000c: BIST: 00, Header Type: 00, Latency Timer: 00,
                Cache Line Size: 00
        0x0010: BAR mem prefetchable 32bit addr: 0xf0000000/0x01000000
        0x0014: BAR empty (00000000)
        0x0018: BAR empty (00000000)
        0x001c: BAR empty (00000000)
        0x0020: BAR empty (00000000)
        0x0024: BAR empty (00000000)
        0x0028: Cardbus CIS: 00000000
        0x002c: Subsystem Vendor ID: 1af4 Product ID: 1100
        0x0030: Expansion ROM Base Address: f1000000
        0x0038: 00000000
        0x003c: Interrupt Pin: 00 Line: 00 Min Gnt: 00 Max Lat: 00
        0x0000: 11111234 00000007 03000000 00000000
        0x0010: f0000008 00000000 00000000 00000000
        0x0020: 00000000 00000000 00000000 11001af4
        0x0030: f1000000 00000000 00000000 00000000
        0x0040: 00000000 00000000 00000000 00000000
        0x0050: 00000000 00000000 00000000 00000000
        0x0060: 00000000 00000000 00000000 00000000
        0x0070: 00000000 00000000 00000000 00000000
        0x0080: 00000000 00000000 00000000 00000000
        0x0090: 00000000 00000000 00000000 00000000
        0x00a0: 00000000 00000000 00000000 00000000
        0x00b0: 00000000 00000000 00000000 00000000
        0x00c0: 00000000 00000000 00000000 00000000
        0x00d0: 00000000 00000000 00000000 00000000
        0x00e0: 00000000 00000000 00000000 00000000
        0x00f0: 00000000 00000000 00000000 00000000
 0:3:0: Qumranet Virtio Network
        0x0000: Vendor ID: 1af4, Product ID: 1000
        0x0004: Command: 0007, Status: 0010
        0x0008: Class: 02 Network, Subclass: 00 Ethernet,
                Interface: 00, Revision: 00
        0x000c: BIST: 00, Header Type: 00, Latency Timer: 00,
                Cache Line Size: 00
        0x0010: BAR io addr: 0x0000c040/0x0020
        0x0014: BAR mem 32bit addr: 0xf1010000/0x00001000
        0x0018: BAR empty (00000000)
        0x001c: BAR empty (00000000)
        0x0020: BAR empty (00000000)
        0x0024: BAR empty (00000000)
        0x0028: Cardbus CIS: 00000000
        0x002c: Subsystem Vendor ID: 1af4 Product ID: 0001
        0x0030: Expansion ROM Base Address: f1020000
        0x0038: 00000000
        0x003c: Interrupt Pin: 01 Line: 0b Min Gnt: 00 Max Lat: 00
        0x0040: Capability 0x11: Extended Message Signalled Interrupts (MSI-X)
                Enabled: yes; table size 3 (BAR 1:0)
        0x0000: 10001af4 00100007 02000000 00000000
        0x0010: 0000c041 f1010000 00000000 00000000
        0x0020: 00000000 00000000 00000000 00011af4
        0x0030: f1020000 00000040 00000000 0000010b
        0x0040: 80020011 00000001 00000801 00000000
        0x0050: 00000000 00000000 00000000 00000000
        0x0060: 00000000 00000000 00000000 00000000
        0x0070: 00000000 00000000 00000000 00000000
        0x0080: 00000000 00000000 00000000 00000000
        0x0090: 00000000 00000000 00000000 00000000
        0x00a0: 00000000 00000000 00000000 00000000
        0x00b0: 00000000 00000000 00000000 00000000
        0x00c0: 00000000 00000000 00000000 00000000
        0x00d0: 00000000 00000000 00000000 00000000
        0x00e0: 00000000 00000000 00000000 00000000
        0x00f0: 00000000 00000000 00000000 00000000
 0:4:0: Qumranet Virtio Storage
        0x0000: Vendor ID: 1af4, Product ID: 1001
        0x0004: Command: 0007, Status: 0010
        0x0008: Class: 01 Mass Storage, Subclass: 00 SCSI,
                Interface: 00, Revision: 00
        0x000c: BIST: 00, Header Type: 00, Latency Timer: 00,
                Cache Line Size: 00
        0x0010: BAR io addr: 0x0000c080/0x0040
        0x0014: BAR mem 32bit addr: 0xf1040000/0x00001000
        0x0018: BAR empty (00000000)
        0x001c: BAR empty (00000000)
        0x0020: BAR empty (00000000)
        0x0024: BAR empty (00000000)
        0x0028: Cardbus CIS: 00000000
        0x002c: Subsystem Vendor ID: 1af4 Product ID: 0002
        0x0030: Expansion ROM Base Address: 00000000
        0x0038: 00000000
        0x003c: Interrupt Pin: 01 Line: 0b Min Gnt: 00 Max Lat: 00
        0x0040: Capability 0x11: Extended Message Signalled Interrupts (MSI-X)
                Enabled: yes; table size 2 (BAR 1:0)
        0x0000: 10011af4 00100007 01000000 00000000
        0x0010: 0000c081 f1040000 00000000 00000000
        0x0020: 00000000 00000000 00000000 00021af4
        0x0030: 00000000 00000040 00000000 0000010b
        0x0040: 80010011 00000001 00000801 00000000
        0x0050: 00000000 00000000 00000000 00000000
        0x0060: 00000000 00000000 00000000 00000000
        0x0070: 00000000 00000000 00000000 00000000
        0x0080: 00000000 00000000 00000000 00000000
        0x0090: 00000000 00000000 00000000 00000000
        0x00a0: 00000000 00000000 00000000 00000000
        0x00b0: 00000000 00000000 00000000 00000000
        0x00c0: 00000000 00000000 00000000 00000000
        0x00d0: 00000000 00000000 00000000 00000000
        0x00e0: 00000000 00000000 00000000 00000000
        0x00f0: 00000000 00000000 00000000 00000000
 0:5:0: Qumranet Virtio Storage
        0x0000: Vendor ID: 1af4, Product ID: 1001
        0x0004: Command: 0007, Status: 0010
        0x0008: Class: 01 Mass Storage, Subclass: 00 SCSI,
                Interface: 00, Revision: 00
        0x000c: BIST: 00, Header Type: 00, Latency Timer: 00,
                Cache Line Size: 00
        0x0010: BAR io addr: 0x0000c0c0/0x0040
        0x0014: BAR mem 32bit addr: 0xf1041000/0x00001000
        0x0018: BAR empty (00000000)
        0x001c: BAR empty (00000000)
        0x0020: BAR empty (00000000)
        0x0024: BAR empty (00000000)
        0x0028: Cardbus CIS: 00000000
        0x002c: Subsystem Vendor ID: 1af4 Product ID: 0002
        0x0030: Expansion ROM Base Address: 00000000
        0x0038: 00000000
        0x003c: Interrupt Pin: 01 Line: 0a Min Gnt: 00 Max Lat: 00
        0x0040: Capability 0x11: Extended Message Signalled Interrupts (MSI-X)
                Enabled: yes; table size 2 (BAR 1:0)
        0x0000: 10011af4 00100007 01000000 00000000
        0x0010: 0000c0c1 f1041000 00000000 00000000
        0x0020: 00000000 00000000 00000000 00021af4
        0x0030: 00000000 00000040 00000000 0000010a
        0x0040: 80010011 00000001 00000801 00000000
        0x0050: 00000000 00000000 00000000 00000000
        0x0060: 00000000 00000000 00000000 00000000
        0x0070: 00000000 00000000 00000000 00000000
        0x0080: 00000000 00000000 00000000 00000000
        0x0090: 00000000 00000000 00000000 00000000
        0x00a0: 00000000 00000000 00000000 00000000
        0x00b0: 00000000 00000000 00000000 00000000
        0x00c0: 00000000 00000000 00000000 00000000
        0x00d0: 00000000 00000000 00000000 00000000
        0x00e0: 00000000 00000000 00000000 00000000
        0x00f0: 00000000 00000000 00000000 00000000

acpidump:
begin-base64 644 APIC.4
QVBJQ3oAAAAB6EJPQ0hTIEJYUENBUElDAQAAAEJYUEMBAAAAAADg/gEAAAAACAAAAQAAAAAIAQEB
AAAAAQwCAAAAwP4AAAAAAgoAAAIAAAAAAAIKAAUFAAAADQACCgAJCQAAAA0AAgoACgoAAAANAAIK
AAsLAAAADQA=
====
begin-base64 644 DSDT.2
RFNEVLcfAAABGUJYUEMAAEJYRFNEVAAAAQAAAElOVEwjAQkgEBxcAFuAREJHXwELRLAKBFuBC0RC
R18DREJHTCAQStZfU0JfW4JL0VBDSTAIX0hJRAxB0AoDCF9BRFIACF9VSUQBCF9QUlQSS3OAEgsE
C///AExOS0QAEgsEC///AUxOS0EAEgwEC///CgJMTktCABIMBAv//woDTE5LQwASDQQM//8BAABM
TktTABINBAz//wEAAUxOS0IAEg4EDP//AQAKAkxOS0MAEg4EDP//AQAKA0xOS0QAEg0EDP//AgAA
TE5LQgASDQQM//8CAAFMTktDABIOBAz//wIACgJMTktEABIOBAz//wIACgNMTktBABINBAz//wMA
AExOS0MAEg0EDP//AwABTE5LRAASDgQM//8DAAoCTE5LQQASDgQM//8DAAoDTE5LQgASDQQM//8E
AABMTktEABINBAz//wQAAUxOS0EAEg4EDP//BAAKAkxOS0IAEg4EDP//BAAKA0xOS0MAEg0EDP//
BQAATE5LQQASDQQM//8FAAFMTktCABIOBAz//wUACgJMTktDABIOBAz//wUACgNMTktEABINBAz/
/wYAAExOS0IAEg0EDP//BgABTE5LQwASDgQM//8GAAoCTE5LRAASDgQM//8GAAoDTE5LQQASDQQM
//8HAABMTktDABINBAz//wcAAUxOS0QAEg4EDP//BwAKAkxOS0EAEg4EDP//BwAKA0xOS0IAEg0E
DP//CAAATE5LRAASDQQM//8IAAFMTktBABIOBAz//wgACgJMTktCABIOBAz//wgACgNMTktDABIN
BAz//wkAAExOS0EAEg0EDP//CQABTE5LQgASDgQM//8JAAoCTE5LQwASDgQM//8JAAoDTE5LRAAS
DQQM//8KAABMTktCABINBAz//woAAUxOS0MAEg4EDP//CgAKAkxOS0QAEg4EDP//CgAKA0xOS0EA
Eg0EDP//CwAATE5LQwASDQQM//8LAAFMTktEABIOBAz//wsACgJMTktBABIOBAz//wsACgNMTktC
ABINBAz//wwAAExOS0QAEg0EDP//DAABTE5LQQASDgQM//8MAAoCTE5LQgASDgQM//8MAAoDTE5L
QwASDQQM//8NAABMTktBABINBAz//w0AAUxOS0IAEg4EDP//DQAKAkxOS0MAEg4EDP//DQAKA0xO
S0QAEg0EDP//DgAATE5LQgASDQQM//8OAAFMTktDABIOBAz//w4ACgJMTktEABIOBAz//w4ACgNM
TktBABINBAz//w8AAExOS0MAEg0EDP//DwABTE5LRAASDgQM//8PAAoCTE5LQQASDgQM//8PAAoD
TE5LQgASDQQM//8QAABMTktEABINBAz//xAAAUxOS0EAEg4EDP//EAAKAkxOS0IAEg4EDP//EAAK
A0xOS0MAEg0EDP//EQAATE5LQQASDQQM//8RAAFMTktCABIOBAz//xEACgJMTktDABIOBAz//xEA
CgNMTktEABINBAz//xIAAExOS0IAEg0EDP//EgABTE5LQwASDgQM//8SAAoCTE5LRAASDgQM//8S
AAoDTE5LQQASDQQM//8TAABMTktDABINBAz//xMAAUxOS0QAEg4EDP//EwAKAkxOS0EAEg4EDP//
EwAKA0xOS0IAEg0EDP//FAAATE5LRAASDQQM//8UAAFMTktBABIOBAz//xQACgJMTktCABIOBAz/
/xQACgNMTktDABINBAz//xUAAExOS0EAEg0EDP//FQABTE5LQgASDgQM//8VAAoCTE5LQwASDgQM
//8VAAoDTE5LRAASDQQM//8WAABMTktCABINBAz//xYAAUxOS0MAEg4EDP//FgAKAkxOS0QAEg4E
DP//FgAKA0xOS0EAEg0EDP//FwAATE5LQwASDQQM//8XAAFMTktEABIOBAz//xcACgJMTktBABIO
BAz//xcACgNMTktCABINBAz//xgAAExOS0QAEg0EDP//GAABTE5LQQASDgQM//8YAAoCTE5LQgAS
DgQM//8YAAoDTE5LQwASDQQM//8ZAABMTktBABINBAz//xkAAUxOS0IAEg4EDP//GQAKAkxOS0MA
Eg4EDP//GQAKA0xOS0QAEg0EDP//GgAATE5LQgASDQQM//8aAAFMTktDABIOBAz//xoACgJMTktE
ABIOBAz//xoACgNMTktBABINBAz//xsAAExOS0MAEg0EDP//GwABTE5LRAASDgQM//8bAAoCTE5L
QQASDgQM//8bAAoDTE5LQgASDQQM//8cAABMTktEABINBAz//xwAAUxOS0EAEg4EDP//HAAKAkxO
S0IAEg4EDP//HAAKA0xOS0MAEg0EDP//HQAATE5LQQASDQQM//8dAAFMTktCABIOBAz//x0ACgJM
TktDABIOBAz//x0ACgNMTktEABINBAz//x4AAExOS0IAEg0EDP//HgABTE5LQwASDgQM//8eAAoC
TE5LRAASDgQM//8eAAoDTE5LQQASDQQM//8fAABMTktDABINBAz//x8AAUxOS0QAEg4EDP//HwAK
AkxOS0EAEg4EDP//HwAKA0xOS0IAW4BQQ1NUAQsArgoIW4EQUENTVENQQ0lVIFBDSUQgW4BTRUpf
AQsIrgoEW4ELU0VKX0NCMEVKIFuCJVMxX18IX0FEUgwAAAEAFA9fRUowAXAKAkIwRUqkAAhfU1VO
AVuCJlMyX18IX0FEUgwAAAIAFA9fRUowAXAKBEIwRUqkAAhfU1VOCgJbgiZTM19fCF9BRFIMAAAD
ABQPX0VKMAFwCghCMEVKpAAIX1NVTgoDW4ImUzRfXwhfQURSDAAABAAUD19FSjABcAoQQjBFSqQA
CF9TVU4KBFuCJlM1X18IX0FEUgwAAAUAFA9fRUowAXAKIEIwRUqkAAhfU1VOCgVbgiZTNl9fCF9B
RFIMAAAGABQPX0VKMAFwCkBCMEVKpAAIX1NVTgoGW4ImUzdfXwhfQURSDAAABwAUD19FSjABcAqA
QjBFSqQACF9TVU4KB1uCJ1M4X18IX0FEUgwAAAgAFBBfRUowAXALAAFCMEVKpAAIX1NVTgoIW4In
UzlfXwhfQURSDAAACQAUEF9FSjABcAsAAkIwRUqkAAhfU1VOCglbgidTMTBfCF9BRFIMAAAKABQQ
X0VKMAFwCwAEQjBFSqQACF9TVU4KCluCJ1MxMV8IX0FEUgwAAAsAFBBfRUowAXALAAhCMEVKpAAI
X1NVTgoLW4InUzEyXwhfQURSDAAADAAUEF9FSjABcAsAEEIwRUqkAAhfU1VOCgxbgidTMTNfCF9B
RFIMAAANABQQX0VKMAFwCwAgQjBFSqQACF9TVU4KDVuCJ1MxNF8IX0FEUgwAAA4AFBBfRUowAXAL
AEBCMEVKpAAIX1NVTgoOW4InUzE1XwhfQURSDAAADwAUEF9FSjABcAsAgEIwRUqkAAhfU1VOCg9b
gilTMTZfCF9BRFIMAAAQABQSX0VKMAFwDAAAAQBCMEVKpAAIX1NVTgoQW4IpUzE3XwhfQURSDAAA
EQAUEl9FSjABcAwAAAIAQjBFSqQACF9TVU4KEVuCKVMxOF8IX0FEUgwAABIAFBJfRUowAXAMAAAE
AEIwRUqkAAhfU1VOChJbgilTMTlfCF9BRFIMAAATABQSX0VKMAFwDAAACABCMEVKpAAIX1NVTgoT
W4IpUzIwXwhfQURSDAAAFAAUEl9FSjABcAwAABAAQjBFSqQACF9TVU4KFFuCKVMyMV8IX0FEUgwA
ABUAFBJfRUowAXAMAAAgAEIwRUqkAAhfU1VOChVbgilTMjJfCF9BRFIMAAAWABQSX0VKMAFwDAAA
QABCMEVKpAAIX1NVTgoWW4IpUzIzXwhfQURSDAAAFwAUEl9FSjABcAwAAIAAQjBFSqQACF9TVU4K
F1uCKVMyNF8IX0FEUgwAABgAFBJfRUowAXAMAAAAAUIwRUqkAAhfU1VOChhbgilTMjVfCF9BRFIM
AAAZABQSX0VKMAFwDAAAAAJCMEVKpAAIX1NVTgoZW4IpUzI2XwhfQURSDAAAGgAUEl9FSjABcAwA
AAAEQjBFSqQACF9TVU4KGluCKVMyN18IX0FEUgwAABsAFBJfRUowAXAMAAAACEIwRUqkAAhfU1VO
ChtbgilTMjhfCF9BRFIMAAAcABQSX0VKMAFwDAAAABBCMEVKpAAIX1NVTgocW4IpUzI5XwhfQURS
DAAAHQAUEl9FSjABcAwAAAAgQjBFSqQACF9TVU4KHVuCKVMzMF8IX0FEUgwAAB4AFBJfRUowAXAM
AAAAQEIwRUqkAAhfU1VOCh5bgilTMzFfCF9BRFIMAAAfABQSX0VKMAFwDAAAAIBCMEVKpAAIX1NV
TgofCF9DUlMRQgcKbogNAAIMAAAAAAD/AAAAAAFHAfgM+AwBCIgNAAEMAwAAAAD3DAAA+AyIDQAB
DAMAAAAN//8AAADzhxcAAAwDAAAAAAAACgD//wsAAAAAAAAAAgCHFwAADAEAAAAAAAAA4P//v/4A
AAAAAADAHnkAW4JFBEhQRVQIX0hJRAxB0AEDCF9VSUQAFAlfU1RBAKQKDwhfQ1JTER8KHIcXAAAN
AQAAAAAAAND+/wPQ/gAAAAAABAAAeQAQTSsuX1NCX1BDSTBbgipWR0FfCF9BRFIMAAACABQIX1Mx
RACkABQIX1MyRACkABQIX1MzRACkAFuCQiNJU0FfCF9BRFIMAAABAFuAUDQwQwIKYAoEW4ItUlRD
XwhfSElEDEHQCwAIX0NSUxEYChVHAXAAcAAQAiIAAUcBcgByAAIGeQBbgkQES0JEXwhfSElEDEHQ
AwMUCV9TVEEApAoPFClfQ1JTAAhUTVBfERgKFUcBYABgAAEBRwFkAGQAAQEiAgB5AKRUTVBfW4Iz
TU9VXwhfSElEDEHQDxMUCV9TVEEApAoPFBlfQ1JTAAhUTVBfEQgKBSIAEHkApFRNUF9bgkcERkRD
MAhfSElEDEHQBwAUCV9TVEEApAoPFCxfQ1JTAAhCVUYwERsKGEcB8gPyAwAERwH3A/cDAAEiQAAq
BAB5AKRCVUYwW4JLBUxQVF8IX0hJRAxB0AQAFChfU1RBAHBeXl4uUFgxM0RSU0Fge2AMAAAAgGCg
BpNgAKQAoQSkCg8UIV9DUlMACEJVRjAREAoNRwF4A3gDCAgigAB5AKRCVUYwW4JBBkNPTTEIX0hJ
RAxB0AUBCF9VSUQBFChfU1RBAHBeXl4uUFgxM0RSU0Nge2AMAAAACGCgBpNgAKQAoQSkCg8UIV9D
UlMACEJVRjAREAoNRwH4A/gDAAgiEAB5AKRCVUYwW4JCBkNPTTIIX0hJRAxB0AUBCF9VSUQKAhQo
X1NUQQBwXl5eLlBYMTNEUlNDYHtgDAAAAIBgoAaTYACkAKEEpAoPFCFfQ1JTAAhCVUYwERAKDUcB
+AL4AgAIIggAeQCkQlVGMFuCQAVQWDEzCF9BRFIMAwABAFuAUDEzQwIKXAokW4EzUDEzQwNEUlNB
IERSU0IgRFJTQyBEUlNFIERSU0YgRFJTRyBEUlNIIERSU0kgRFJTSiAQSzxfU0JfW4EkLwNQQ0kw
SVNBX1A0MEMBUFJRMAhQUlExCFBSUTIIUFJRMwhbgk0LTE5LQQhfSElEDEHQDA8IX1VJRAEIX1BS
UxEWChOJDgAJAwUAAAAKAAAACwAAAHkAFBpfU1RBAHAKC2CgDXsKgFBSUTBhcAoJYKRgFBFfRElT
AH1QUlEwCoBQUlEwFEUEX0NSUwAIUFJSMBEOCguJBgAJAQEAAAB5AIpQUlIwCgVUTVBfcFBSUTBg
oAuVYAqAcGBUTVBfoQdwAFRNUF+kUFJSMBQXX1NSUwGKaAoFVE1QX3BUTVBfUFJRMFuCTgtMTktC
CF9ISUQMQdAMDwhfVUlECgIIX1BSUxEWChOJDgAJAwUAAAAKAAAACwAAAHkAFBpfU1RBAHAKC2Cg
DXsKgFBSUTFhcAoJYKRgFBFfRElTAH1QUlExCoBQUlExFEUEX0NSUwAIUFJSMBEOCguJBgAJAQEA
AAB5AIpQUlIwCgVUTVBfcFBSUTFgoAuVYAqAcGBUTVBfoQdwAFRNUF+kUFJSMBQXX1NSUwGKaAoF
VE1QX3BUTVBfUFJRMVuCTgtMTktDCF9ISUQMQdAMDwhfVUlECgMIX1BSUxEWChOJDgAJAwUAAAAK
AAAACwAAAHkAFBpfU1RBAHAKC2CgDXsKgFBSUTJhcAoJYKRgFBFfRElTAH1QUlEyCoBQUlEyFEUE
X0NSUwAIUFJSMBEOCguJBgAJAQEAAAB5AIpQUlIwCgVUTVBfcFBSUTJgoAuVYAqAcGBUTVBfoQdw
AFRNUF+kUFJSMBQXX1NSUwGKaAoFVE1QX3BUTVBfUFJRMluCTgtMTktECF9ISUQMQdAMDwhfVUlE
CgQIX1BSUxEWChOJDgAJAwUAAAAKAAAACwAAAHkAFBpfU1RBAHAKC2CgDXsKgFBSUTNhcAoJYKRg
FBFfRElTAH1QUlEzCoBQUlEzFEUEX0NSUwAIUFJSMBEOCguJBgAJAQEAAAB5AIpQUlIwCgVUTVBf
cFBSUTNgoAuVYAqAcGBUTVBfoQdwAFRNUF+kUFJSMBQXX1NSUwGKaAoFVE1QX3BUTVBfUFJRM1uC
TglMTktTCF9ISUQMQdAMDwhfVUlECgUIX1BSUxEOCguJBgAJAQkAAAB5ABQaX1NUQQBwCgtgoA17
CoBQUlEwYXAKCWCkYBQRX0RJUwB9UFJRMAqAUFJRMBRFBF9DUlMACFBSUjARDgoLiQYACQEJAAAA
eQCKUFJSMAoFVE1QX3BQUlEwYKALlWAKgHBgVE1QX6EHcABUTVBfpFBSUjAIX1MzXxIGBAEBAAAI
X1M0XxIGBAAAAAAIX1M1XxIGBAAAAAAQSQ5fU0JfFDVDUE1BAXCDiENQT05oAGBwEQsKCAAIAAAA
AAAAYXBoiGEKAgBwaIhhCgMAcGCIYQoEAKRhFBpDUFNUAXCDiENQT05oAGCgBWCkCg+hA6QAFApD
UEVKAlsiCshbgFBSU1QBCwCvCiBbgQxQUlNUAVBSU19AEBRMBlBSU0MAcFBSU19lcABicABgokYF
lWCHQ1BPTnCDiENQT05gAGGgCntgCgcAemIBYqEMcIOIZXpgCgMAAGJwe2IBAGOgIpKTYWNwY4hD
UE9OYACgCpNjAU5URllgAaEITlRGWWAKA3VgpAEQQqdfR1BFCF9ISUQNQUNQSTAwMDYAFAhfTDAw
AKQBFEycX0wwMQCgJXtcLwNfU0JfUENJMFBDSVUKAgCGXC8DX1NCX1BDSTBTMV9fAaAme1wvA19T
Ql9QQ0kwUENJRAoCAIZcLwNfU0JfUENJMFMxX18KA6Ale1wvA19TQl9QQ0kwUENJVQoEAIZcLwNf
U0JfUENJMFMyX18BoCZ7XC8DX1NCX1BDSTBQQ0lECgQAhlwvA19TQl9QQ0kwUzJfXwoDoCV7XC8D
X1NCX1BDSTBQQ0lVCggAhlwvA19TQl9QQ0kwUzNfXwGgJntcLwNfU0JfUENJMFBDSUQKCACGXC8D
X1NCX1BDSTBTM19fCgOgJXtcLwNfU0JfUENJMFBDSVUKEACGXC8DX1NCX1BDSTBTNF9fAaAme1wv
A19TQl9QQ0kwUENJRAoQAIZcLwNfU0JfUENJMFM0X18KA6Ale1wvA19TQl9QQ0kwUENJVQogAIZc
LwNfU0JfUENJMFM1X18BoCZ7XC8DX1NCX1BDSTBQQ0lECiAAhlwvA19TQl9QQ0kwUzVfXwoDoCV7
XC8DX1NCX1BDSTBQQ0lVCkAAhlwvA19TQl9QQ0kwUzZfXwGgJntcLwNfU0JfUENJMFBDSUQKQACG
XC8DX1NCX1BDSTBTNl9fCgOgJXtcLwNfU0JfUENJMFBDSVUKgACGXC8DX1NCX1BDSTBTN19fAaAm
e1wvA19TQl9QQ0kwUENJRAqAAIZcLwNfU0JfUENJMFM3X18KA6Ame1wvA19TQl9QQ0kwUENJVQsA
AQCGXC8DX1NCX1BDSTBTOF9fAaAne1wvA19TQl9QQ0kwUENJRAsAAQCGXC8DX1NCX1BDSTBTOF9f
CgOgJntcLwNfU0JfUENJMFBDSVULAAIAhlwvA19TQl9QQ0kwUzlfXwGgJ3tcLwNfU0JfUENJMFBD
SUQLAAIAhlwvA19TQl9QQ0kwUzlfXwoDoCZ7XC8DX1NCX1BDSTBQQ0lVCwAEAIZcLwNfU0JfUENJ
MFMxMF8BoCd7XC8DX1NCX1BDSTBQQ0lECwAEAIZcLwNfU0JfUENJMFMxMF8KA6Ame1wvA19TQl9Q
Q0kwUENJVQsACACGXC8DX1NCX1BDSTBTMTFfAaAne1wvA19TQl9QQ0kwUENJRAsACACGXC8DX1NC
X1BDSTBTMTFfCgOgJntcLwNfU0JfUENJMFBDSVULABAAhlwvA19TQl9QQ0kwUzEyXwGgJ3tcLwNf
U0JfUENJMFBDSUQLABAAhlwvA19TQl9QQ0kwUzEyXwoDoCZ7XC8DX1NCX1BDSTBQQ0lVCwAgAIZc
LwNfU0JfUENJMFMxM18BoCd7XC8DX1NCX1BDSTBQQ0lECwAgAIZcLwNfU0JfUENJMFMxM18KA6Am
e1wvA19TQl9QQ0kwUENJVQsAQACGXC8DX1NCX1BDSTBTMTRfAaAne1wvA19TQl9QQ0kwUENJRAsA
QACGXC8DX1NCX1BDSTBTMTRfCgOgJntcLwNfU0JfUENJMFBDSVULAIAAhlwvA19TQl9QQ0kwUzE1
XwGgJ3tcLwNfU0JfUENJMFBDSUQLAIAAhlwvA19TQl9QQ0kwUzE1XwoDoCh7XC8DX1NCX1BDSTBQ
Q0lVDAAAAQAAhlwvA19TQl9QQ0kwUzE2XwGgKXtcLwNfU0JfUENJMFBDSUQMAAABAACGXC8DX1NC
X1BDSTBTMTZfCgOgKHtcLwNfU0JfUENJMFBDSVUMAAACAACGXC8DX1NCX1BDSTBTMTdfAaApe1wv
A19TQl9QQ0kwUENJRAwAAAIAAIZcLwNfU0JfUENJMFMxN18KA6Aoe1wvA19TQl9QQ0kwUENJVQwA
AAQAAIZcLwNfU0JfUENJMFMxOF8BoCl7XC8DX1NCX1BDSTBQQ0lEDAAABAAAhlwvA19TQl9QQ0kw
UzE4XwoDoCh7XC8DX1NCX1BDSTBQQ0lVDAAACAAAhlwvA19TQl9QQ0kwUzE5XwGgKXtcLwNfU0Jf
UENJMFBDSUQMAAAIAACGXC8DX1NCX1BDSTBTMTlfCgOgKHtcLwNfU0JfUENJMFBDSVUMAAAQAACG
XC8DX1NCX1BDSTBTMjBfAaApe1wvA19TQl9QQ0kwUENJRAwAABAAAIZcLwNfU0JfUENJMFMyMF8K
A6Aoe1wvA19TQl9QQ0kwUENJVQwAACAAAIZcLwNfU0JfUENJMFMyMV8BoCl7XC8DX1NCX1BDSTBQ
Q0lEDAAAIAAAhlwvA19TQl9QQ0kwUzIxXwoDoCh7XC8DX1NCX1BDSTBQQ0lVDAAAQAAAhlwvA19T
Ql9QQ0kwUzIyXwGgKXtcLwNfU0JfUENJMFBDSUQMAABAAACGXC8DX1NCX1BDSTBTMjJfCgOgKHtc
LwNfU0JfUENJMFBDSVUMAACAAACGXC8DX1NCX1BDSTBTMjNfAaApe1wvA19TQl9QQ0kwUENJRAwA
AIAAAIZcLwNfU0JfUENJMFMyM18KA6Aoe1wvA19TQl9QQ0kwUENJVQwAAAABAIZcLwNfU0JfUENJ
MFMyNF8BoCl7XC8DX1NCX1BDSTBQQ0lEDAAAAAEAhlwvA19TQl9QQ0kwUzI0XwoDoCh7XC8DX1NC
X1BDSTBQQ0lVDAAAAAIAhlwvA19TQl9QQ0kwUzI1XwGgKXtcLwNfU0JfUENJMFBDSUQMAAAAAgCG
XC8DX1NCX1BDSTBTMjVfCgOgKHtcLwNfU0JfUENJMFBDSVUMAAAABACGXC8DX1NCX1BDSTBTMjZf
AaApe1wvA19TQl9QQ0kwUENJRAwAAAAEAIZcLwNfU0JfUENJMFMyNl8KA6Aoe1wvA19TQl9QQ0kw
UENJVQwAAAAIAIZcLwNfU0JfUENJMFMyN18BoCl7XC8DX1NCX1BDSTBQQ0lEDAAAAAgAhlwvA19T
Ql9QQ0kwUzI3XwoDoCh7XC8DX1NCX1BDSTBQQ0lVDAAAABAAhlwvA19TQl9QQ0kwUzI4XwGgKXtc
LwNfU0JfUENJMFBDSUQMAAAAEACGXC8DX1NCX1BDSTBTMjhfCgOgKHtcLwNfU0JfUENJMFBDSVUM
AAAAIACGXC8DX1NCX1BDSTBTMjlfAaApe1wvA19TQl9QQ0kwUENJRAwAAAAgAIZcLwNfU0JfUENJ
MFMyOV8KA6Aoe1wvA19TQl9QQ0kwUENJVQwAAABAAIZcLwNfU0JfUENJMFMzMF8BoCl7XC8DX1NC
X1BDSTBQQ0lEDAAAAEAAhlwvA19TQl9QQ0kwUzMwXwoDoCh7XC8DX1NCX1BDSTBQQ0lVDAAAAIAA
hlwvA19TQl9QQ0kwUzMxXwGgKXtcLwNfU0JfUENJMFBDSUQMAAAAgACGXC8DX1NCX1BDSTBTMzFf
CgOkARQRX0wwMgCkXC5fU0JfUFJTQxQIX0wwMwCkARQIX0wwNACkARQIX0wwNQCkARQIX0wwNgCk
ARQIX0wwNwCkARQIX0wwOACkARQIX0wwOQCkARQIX0wwQQCkARQIX0wwQgCkARQIX0wwQwCkARQI
X0wwRACkARQIX0wwRQCkARQIX0wwRgCkAQ==
====
begin-base64 644 FACP.1
RkFDUHQAAAABf0JPQ0hTIEJYUENGQUNQAQAAAEJYUEMBAAAAgP3/f8Dd/38BAAkAsgAAAPHwAAAA
sAAAAAAAAASwAAAAAAAAAAAAAAiwAADgrwAAAAAAAAQCAAQEAAAA/w//DwAAAAAAAAAAAAAAAGUA
AAA=
====
begin-base64 644 HPET.5
SFBFVDgAAAABA0JPQ0hTIEJYUENIUEVUAQAAAEJYUEMBAAAAAaKGgAAAAAAAAND+AAAAAAAAAAA=
====
begin-base64 644 RSDT.0
UlNEVDQAAAABz0JPQ0hTIEJYUENSU0RUAQAAAEJYUEMBAAAA8P3/f8Dc/3/Q2/9/kNv/fw==
====
begin-base64 644 SSDT.3
U1NEVP8AAAABMEJPQ0hTIEJYUENTU0RUAQAAAEJYUEMBAAAAEIoNAF9TQl9bg0IFQ1AwMAAQsAAA
BghJRF9fCgAIX0hJRA1BQ1BJMDAwNwAUD19NQVQApENQTUFJRF9fFA9fU1RBAKRDUFNUSURfXxQP
X0VKMAFDUEVKSURfX2hbg0IFQ1AwMQEQsAAABghJRF9fCgEIX0hJRA1BQ1BJMDAwNwAUD19NQVQA
pENQTUFJRF9fFA9fU1RBAKRDUFNUSURfXxQPX0VKMAFDUEVKSURfX2gUTwFOVEZZAqALk2gKAIZD
UDAwaaALk2gKAYZDUDAxaQhDUE9OEkUAAgEB
====
begin-base64 644 headers
ClJTRCBQVFI6IENoZWNrc3VtPTE2OSwgT0VNSUQ9Qk9DSFMsIFJldmlzaW9uPTAsIFJzZHRBZGRy
ZXNzPTB4N2ZmZmRiNTAKCgpSU0RUOiBMZW5ndGg9NTIsIFJldmlzaW9uPTEsIENoZWNrc3VtPTIw
NywKCU9FTUlEPUJPQ0hTLCBPRU0gVGFibGUgSUQ9QlhQQ1JTRFQsIE9FTSBSZXZpc2lvbj0weDEs
CglDcmVhdG9yIElEPUJYUEMsIENyZWF0b3IgUmV2aXNpb249MHgxCgoKCUVudHJpZXM9eyAweDdm
ZmZmZGYwLCAweDdmZmZkY2MwLCAweDdmZmZkYmQwLCAweDdmZmZkYjkwIH0KCgoJRFNEVD0weDdm
ZmZkZGMwCglJTlRfTU9ERUw9QVBJQwoJU0NJX0lOVD05CglTTUlfQ01EPTB4YjIsIEFDUElfRU5B
QkxFPTB4ZjEsIEFDUElfRElTQUJMRT0weGYwLCBTNEJJT1NfUkVRPTB4MAoJUE0xYV9FVlRfQkxL
PTB4YjAwMC0weGIwMDMKCVBNMWFfQ05UX0JMSz0weGIwMDQtMHhiMDA1CglQTTJfVE1SX0JMSz0w
eGIwMDgtMHhiMDBiCglQTTJfR1BFMF9CTEs9MHhhZmUwLTB4YWZlMwoJUF9MVkwyX0xBVD00MDk1
bXMsIFBfTFZMM19MQVQ9NDA5NW1zCglGTFVTSF9TSVpFPTAsIEZMVVNIX1NUUklERT0wCglEVVRZ
X09GRlNFVD0wLCBEVVRZX1dJRFRIPTAKCURBWV9BTFJNPTAsIE1PTl9BTFJNPTAsIENFTlRVUlk9
MAoJRmxhZ3M9e1dCSU5WRCxQUk9DX0MxLFNMUF9CVVRUT04sRklYX1JUQ30KCgpEU0RUOiBMZW5n
dGg9ODExOSwgUmV2aXNpb249MSwgQ2hlY2tzdW09MjUsCglPRU1JRD1CWFBDLCBPRU0gVGFibGUg
SUQ9QlhEU0RULCBPRU0gUmV2aXNpb249MHgxLAoJQ3JlYXRvciBJRD1JTlRMLCBDcmVhdG9yIFJl
dmlzaW9uPTB4MjAwOTAxMjMKCgpTU0RUOiBMZW5ndGg9MjU1LCBSZXZpc2lvbj0xLCBDaGVja3N1
bT00OCwKCU9FTUlEPUJPQ0hTLCBPRU0gVGFibGUgSUQ9QlhQQ1NTRFQsIE9FTSBSZXZpc2lvbj0w
eDEsCglDcmVhdG9yIElEPUJYUEMsIENyZWF0b3IgUmV2aXNpb249MHgxCgoKQVBJQzogTGVuZ3Ro
PTEyMiwgUmV2aXNpb249MSwgQ2hlY2tzdW09MjMyLAoJT0VNSUQ9Qk9DSFMsIE9FTSBUYWJsZSBJ
RD1CWFBDQVBJQywgT0VNIFJldmlzaW9uPTB4MSwKCUNyZWF0b3IgSUQ9QlhQQywgQ3JlYXRvciBS
ZXZpc2lvbj0weDEKCgpIUEVUOiBMZW5ndGg9NTYsIFJldmlzaW9uPTEsIENoZWNrc3VtPTMsCglP
RU1JRD1CT0NIUywgT0VNIFRhYmxlIElEPUJYUENIUEVULCBPRU0gUmV2aXNpb249MHgxLAoJQ3Jl
YXRvciBJRD1CWFBDLCBDcmVhdG9yIFJldmlzaW9uPTB4MQoK
====

[Alexander Bluhm]

On Fri, Jan 31, 2020 at 03:21:00PM +0000, Peter M??ller wrote:
> tcp_output(ffff800000584ee0) at tcp_output+0x1941
> tcp_output(ffff800000584ee0) at tcp_output+0x1941
> tcp_output(ffff800000584ee0) at tcp_output+0x1941

Looks like stack exhaustion.  tcp_output() calls tcp_mtudisc() calls
tcp_output().

/usr/src/sys/netinet/tcp_output.c:1084

                if (error == EMSGSIZE) {
                        /*
                         * ip_output() will have already fixed the route
                         * for us.  tcp_mtudisc() will, as its last action,
                         * initiate retransmission, so it is important to
                         * not do so here.
                         */
                        tcp_mtudisc(tp->t_inpcb, -1);
                        return (0);
                }

bluhm

[Peter Müller]

Hello Alexander,

thank you for your reply. Is there anything I can do about this
like modifying configurations or provide further information?

Thanks, and best regards,
Peter Müller

> On Fri, Jan 31, 2020 at 03:21:00PM +0000, Peter M??ller wrote:
>> tcp_output(ffff800000584ee0) at tcp_output+0x1941
>> tcp_output(ffff800000584ee0) at tcp_output+0x1941
>> tcp_output(ffff800000584ee0) at tcp_output+0x1941
>
> Looks like stack exhaustion.  tcp_output() calls tcp_mtudisc() calls
> tcp_output().
>
> /usr/src/sys/netinet/tcp_output.c:1084
>
>                 if (error == EMSGSIZE) {
>                         /*
>                          * ip_output() will have already fixed the route
>                          * for us.  tcp_mtudisc() will, as its last action,
>                          * initiate retransmission, so it is important to
>                          * not do so here.
>                          */
>                         tcp_mtudisc(tp->t_inpcb, -1);
>                         return (0);
>                 }
>
> bluhm
>

[Peter Müller]

Hello *,

after experimenting with different MTU sizes and pf normalisation rules,
I am getting the feeling of a root cause lying somewhere near path MTU
discovery - perhaps in combination with IPsec.

These are the console log messages of another crash observed meanwhile:

kernel: double fault trap, code=0
Stopped at      rtable_l2+0xf:  pushq   %rdi
ddb{0}> trace
rtable_l2(0) at rtable_l2+0xf
pf_setup_pdesc(ffff8000210e40a8,2,2,ffff80000016c400,fffffd806ee32e00,fffff80000210e41be) at pf_setup_pdesc+0x7d
pf_test(2,2,ffff80000013f000,ffff8000210e4290) at pf_test+0xfe
ip_output(fffffd806ee32e00,0,fffffd807d95a5f8,800,0,fffffd807d95a588) at ip_output+0x7cf
tcp_output(ffff800000551980) at tcp_output+0x15c1
tcp_output(ffff800000551980) at tcp_output+0x1914
tcp_output(ffff800000551980) at tcp_output+0x1914
tcp_output(ffff800000551980) at tcp_output+0x1914
tcp_output(ffff800000551980) at tcp_output+0x1914
tcp_output(ffff800000551980) at tcp_output+0x1914
tcp_output(ffff800000551980) at tcp_output+0x1914
tcp_output(ffff800000551980) at tcp_output+0x1914
tcp_output(ffff800000551980) at tcp_output+0x1914
tcp_output(ffff800000551980) at tcp_output+0x1914
tcp_output(ffff800000551980) at tcp_output+0x1914
tcp_output(ffff800000551980) at tcp_output+0x1914
tcp_output(ffff800000551980) at tcp_output+0x1914
tcp_output(ffff800000551980) at tcp_output+0x1914
tcp_output(ffff800000551980) at tcp_output+0x1914
tcp_output(ffff800000551980) at tcp_output+0x1914
[... some identical lines omitted...]
tcp_timer_rexmt(ffff800000551980) at tcp_timer_rexmt+0x3f5
softclock_thread(ffff8000210d2c58) at softclock_thread+0xfb
end trace frame: 0x0, count: -50

While the first lines differ, the tcp_output(...) and tcp_timer_rexmt(...)
and softclock_thread(...) stay always the same.

At the moment, by reducing the MTU of my vio0 interface to 1488 bytes and
attempting to clear DF flags on packages related to IPsec payload traffic
(/etc/pf.conf snippet: "match on enc0 scrub (max-mss 1360 random-id no-df)"),
I managed to delay crashes from ~ 30 minutes up to some hours in productive
use scenarios. Again, there is no problem if the machine is running idle.

Since these stalls keep happening and I am out of ideas by now, I wonder
if anybody is successfully running a Squid upstream proxy in combination
with an IPsec site-to-site connection on the same machine.

Thanks, and best regards,
Peter Müller

> Hello Alexander,
>
> thank you for your reply. Is there anything I can do about this
> like modifying configurations or provide further information?
>
> Thanks, and best regards,
> Peter Müller
>
>
>> On Fri, Jan 31, 2020 at 03:21:00PM +0000, Peter M??ller wrote:
>>> tcp_output(ffff800000584ee0) at tcp_output+0x1941
>>> tcp_output(ffff800000584ee0) at tcp_output+0x1941
>>> tcp_output(ffff800000584ee0) at tcp_output+0x1941
>>
>> Looks like stack exhaustion.  tcp_output() calls tcp_mtudisc() calls
>> tcp_output().
>>
>> /usr/src/sys/netinet/tcp_output.c:1084
>>
>>                 if (error == EMSGSIZE) {
>>                         /*
>>                          * ip_output() will have already fixed the route
>>                          * for us.  tcp_mtudisc() will, as its last action,
>>                          * initiate retransmission, so it is important to
>>                          * not do so here.
>>                          */
>>                         tcp_mtudisc(tp->t_inpcb, -1);
>>                         return (0);
>>                 }
>>
>> bluhm
>>
>

[Claudio Jeker-3]

On Fri, Jan 31, 2020 at 06:20:33PM +0100, Alexander Bluhm wrote:

> On Fri, Jan 31, 2020 at 03:21:00PM +0000, Peter M??ller wrote:
> > tcp_output(ffff800000584ee0) at tcp_output+0x1941
> > tcp_output(ffff800000584ee0) at tcp_output+0x1941
> > tcp_output(ffff800000584ee0) at tcp_output+0x1941
>
> Looks like stack exhaustion.  tcp_output() calls tcp_mtudisc() calls
> tcp_output().
>
> /usr/src/sys/netinet/tcp_output.c:1084
>
>                 if (error == EMSGSIZE) {
>                         /*
>                          * ip_output() will have already fixed the route
>                          * for us.  tcp_mtudisc() will, as its last action,
>                          * initiate retransmission, so it is important to
>                          * not do so here.
>                          */
>                         tcp_mtudisc(tp->t_inpcb, -1);
>                         return (0);
>                 }
>

Looks like tcp_mss() is not reducing the mss enough and is probably
confused by IPsec stealing the packets via a flow and so the route used
for MSS is incorrect and in the end the MTU. My assumption is that the MSS
remains the same and the stack produces EMSGSIZE over and over again.
Not entierly sure how to fix this right now.
Alexander, do you have any ideas?

--
:wq Claudio

[Alexander Bluhm]

On Wed, Feb 05, 2020 at 05:52:00PM +0000, Peter M??ller wrote:
> after experimenting with different MTU sizes and pf normalisation rules,
> I am getting the feeling of a root cause lying somewhere near path MTU
> discovery - perhaps in combination with IPsec.

A coworker ran into the same trace with routing domains and increased
interface MTU for jumbo frames.

> kernel: double fault trap, code=0
> Stopped at      rtable_l2+0xf:  pushq   %rdi
> ddb{0}> trace
> rtable_l2(0) at rtable_l2+0xf
> pf_setup_pdesc(ffff8000210e40a8,2,2,ffff80000016c400,fffffd806ee32e00,fffff80000210e41be) at pf_setup_pdesc+0x7d
> pf_test(2,2,ffff80000013f000,ffff8000210e4290) at pf_test+0xfe
> ip_output(fffffd806ee32e00,0,fffffd807d95a5f8,800,0,fffffd807d95a588) at ip_output+0x7cf
> tcp_output(ffff800000551980) at tcp_output+0x15c1
> tcp_output(ffff800000551980) at tcp_output+0x1914
> tcp_output(ffff800000551980) at tcp_output+0x1914
> tcp_output(ffff800000551980) at tcp_output+0x1914
> tcp_output(ffff800000551980) at tcp_output+0x1914
> tcp_output(ffff800000551980) at tcp_output+0x1914
> tcp_output(ffff800000551980) at tcp_output+0x1914
> tcp_output(ffff800000551980) at tcp_output+0x1914
> tcp_output(ffff800000551980) at tcp_output+0x1914
> tcp_output(ffff800000551980) at tcp_output+0x1914
> tcp_output(ffff800000551980) at tcp_output+0x1914
> tcp_output(ffff800000551980) at tcp_output+0x1914
> tcp_output(ffff800000551980) at tcp_output+0x1914
> tcp_output(ffff800000551980) at tcp_output+0x1914
> tcp_output(ffff800000551980) at tcp_output+0x1914
> tcp_output(ffff800000551980) at tcp_output+0x1914
> [... some identical lines omitted...]
> tcp_timer_rexmt(ffff800000551980) at tcp_timer_rexmt+0x3f5
> softclock_thread(ffff8000210d2c58) at softclock_thread+0xfb
> end trace frame: 0x0, count: -50

I have created an automated test from his setup.

pair1 interface wit MTU 8000 <-> pair2 interface <-> loopback3 interface

All interfaces are in different routing domains.  Between loopback3
and pair2, pf switches the routing table.  If TCP sends from loopback3
to pair1, path MTU discovery fails and the kernel crashes at the
next rexmt timeout.

The problem is that ip_output() tries to add the PMTU route in
routing table 2, but tcp_output() expects it in routing table 3.
Then it goes back and forth until the stack is exhausted.

This diff creates the PMTU route in the original routing table.
Then PMTU discovery works.

Do you also use routing domains?

Sorry that the fix took a year.  I needed a precise and simple
description how to reproduce.  Do you still have this setup?  Does
my diff also fix your problem?

bluhm

Index: netinet/ip_output.c
===================================================================
RCS file: /data/mirror/openbsd/cvs/src/sys/netinet/ip_output.c,v
retrieving revision 1.363
diff -u -p -r1.363 ip_output.c
--- netinet/ip_output.c 2 Feb 2021 17:47:42 -0000 1.363
+++ netinet/ip_output.c 5 Feb 2021 00:46:02 -0000
@@ -107,7 +107,10 @@ ip_output(struct mbuf *m0, struct mbuf *
  struct sockaddr_in *dst;
  struct tdb *tdb = NULL;
  u_long mtu;
-#if defined(MROUTING)
+#if NPF > 0
+ u_int orig_rtableid;
+#endif
+#ifdef MROUTING
  int rv;
 #endif
 
@@ -150,6 +153,7 @@ ip_output(struct mbuf *m0, struct mbuf *
  }
 
 #if NPF > 0
+ orig_rtableid = m->m_pkthdr.ph_rtableid;
 reroute:
 #endif
 
@@ -480,6 +484,15 @@ sendit:
  ipsec_adjust_mtu(m, ifp->if_mtu);
 #endif
  error = EMSGSIZE;
+#if NPF > 0
+ /* pf changed routing table, use orig rtable for path MTU */
+ if (ro->ro_tableid != orig_rtableid) {
+ rtfree(ro->ro_rt);
+ ro->ro_tableid = orig_rtableid;
+ ro->ro_rt = icmp_mtudisc_clone(
+    satosin(&ro->ro_dst)->sin_addr, ro->ro_tableid, 0);
+ }
+#endif
  /*
  * This case can happen if the user changed the MTU
  * of an interface after enabling IP on it.  Because

[Alexander Bluhm]

On Fri, Feb 05, 2021 at 02:01:31AM +0100, Alexander Bluhm wrote:
> I have created an automated test from his setup.

I have commited a regress test that automatically builds the affected
setup.  In case someone wants to play with it, run

make -C /usr/src/regress/sys/net/pair clean run-tcpbench-3-1

You see that tcpbench does not transfer much as PMTU discovery does
not work.  It leaves a TCP socket in FIN_WAIT_1.  After a few seconds
the rexmit timeout triggers.  Then the kernel crashes.

When built with -O0 you see the stack trace jumping between
tcp_output() and tcp_mtudisc().

login: kernel: double fault trap, code=0
Stopped at      m_copydata+0x11:        pushq   %rdx
ddb> trace
m_copydata(fffffd807c7c7400,14,0,0) at m_copydata+0x11
pf_pull_hdr(fffffd807c7c7400,14,ffff8000213421a4,14,0,ffff8000213421fe) at pf_p
ull_hdr+0xa9
pf_setup_pdesc(ffff800021342100,2,2,ffff800000578900,fffffd807c7c7400,ffff80002
13421fe) at pf_setup_pdesc+0x20d
pf_test(2,2,ffff8000005b0000,ffff800021342330) at pf_test+0x16e
ip_output(fffffd807c7c7400,0,fffffd807ed0b078,800,0,fffffd807ed0b008) at ip_out
put+0xb96
tcp_output(ffff800000593420) at tcp_output+0x221b
tcp_mtudisc(fffffd807ed0b008,ffffffff) at tcp_mtudisc+0x11b
tcp_output(ffff800000593420) at tcp_output+0x234b
tcp_mtudisc(fffffd807ed0b008,ffffffff) at tcp_mtudisc+0x11b
tcp_output(ffff800000593420) at tcp_output+0x234b
tcp_mtudisc(fffffd807ed0b008,ffffffff) at tcp_mtudisc+0x11b
tcp_output(ffff800000593420) at tcp_output+0x234b
tcp_mtudisc(fffffd807ed0b008,ffffffff) at tcp_mtudisc+0x11b
tcp_output(ffff800000593420) at tcp_output+0x234b
tcp_mtudisc(fffffd807ed0b008,ffffffff) at tcp_mtudisc+0x11b
tcp_output(ffff800000593420) at tcp_output+0x234b
tcp_mtudisc(fffffd807ed0b008,ffffffff) at tcp_mtudisc+0x11b
tcp_output(ffff800000593420) at tcp_output+0x234b
tcp_mtudisc(fffffd807ed0b008,ffffffff) at tcp_mtudisc+0x11b
tcp_output(ffff800000593420) at tcp_output+0x234b
tcp_mtudisc(fffffd807ed0b008,ffffffff) at tcp_mtudisc+0x11b
tcp_output(ffff800000593420) at tcp_output+0x234b
tcp_mtudisc(fffffd807ed0b008,ffffffff) at tcp_mtudisc+0x11b
tcp_output(ffff800000593420) at tcp_output+0x234b
tcp_mtudisc(fffffd807ed0b008,ffffffff) at tcp_mtudisc+0x11b
tcp_output(ffff800000593420) at tcp_output+0x234b
tcp_mtudisc(fffffd807ed0b008,ffffffff) at tcp_mtudisc+0x11b
tcp_output(ffff800000593420) at tcp_output+0x234b
tcp_mtudisc(fffffd807ed0b008,ffffffff) at tcp_mtudisc+0x11b
tcp_output(ffff800000593420) at tcp_output+0x234b
tcp_mtudisc(fffffd807ed0b008,ffffffff) at tcp_mtudisc+0x11b
tcp_output(ffff800000593420) at tcp_output+0x234b
tcp_mtudisc(fffffd807ed0b008,ffffffff) at tcp_mtudisc+0x11b
tcp_output(ffff800000593420) at tcp_output+0x234b
tcp_mtudisc(fffffd807ed0b008,ffffffff) at tcp_mtudisc+0x11b
tcp_output(ffff800000593420) at tcp_output+0x234b
tcp_mtudisc(fffffd807ed0b008,ffffffff) at tcp_mtudisc+0x11b
tcp_output(ffff800000593420) at tcp_output+0x234b
tcp_mtudisc(fffffd807ed0b008,ffffffff) at tcp_mtudisc+0x11b
tcp_output(ffff800000593420) at tcp_output+0x234b
tcp_mtudisc(fffffd807ed0b008,ffffffff) at tcp_mtudisc+0x11b
tcp_output(ffff800000593420) at tcp_output+0x234b
tcp_mtudisc(fffffd807ed0b008,ffffffff) at tcp_mtudisc+0x11b
tcp_output(ffff800000593420) at tcp_output+0x234b
tcp_mtudisc(fffffd807ed0b008,ffffffff) at tcp_mtudisc+0x11b
tcp_output(ffff800000593420) at tcp_output+0x234b
tcp_mtudisc(fffffd807ed0b008,ffffffff) at tcp_mtudisc+0x11b
tcp_output(ffff800000593420) at tcp_output+0x234b
tcp_mtudisc(fffffd807ed0b008,ffffffff) at tcp_mtudisc+0x11b
tcp_output(ffff800000593420) at tcp_output+0x234b
tcp_timer_rexmt(ffff800000593420) at tcp_timer_rexmt+0x74f
softclock_thread(ffff800021336fc0) at softclock_thread+0x13f
end trace frame: 0x0, count: -52

Attached again diff that fixes it.  I guess IPv6 has the same
problem.  I will look into that after we know that my IPv4 fix is
correct.

bluhm

Index: netinet/ip_output.c
===================================================================
RCS file: /data/mirror/openbsd/cvs/src/sys/netinet/ip_output.c,v
retrieving revision 1.363
diff -u -p -r1.363 ip_output.c
--- netinet/ip_output.c 2 Feb 2021 17:47:42 -0000 1.363
+++ netinet/ip_output.c 5 Feb 2021 14:29:46 -0000
@@ -107,7 +107,10 @@ ip_output(struct mbuf *m0, struct mbuf *
  struct sockaddr_in *dst;
  struct tdb *tdb = NULL;
  u_long mtu;
-#if defined(MROUTING)
+#if NPF > 0
+ u_int orig_rtableid;
+#endif
+#ifdef MROUTING
  int rv;
 #endif
 
@@ -150,6 +153,7 @@ ip_output(struct mbuf *m0, struct mbuf *
  }
 
 #if NPF > 0
+ orig_rtableid = m->m_pkthdr.ph_rtableid;
 reroute:
 #endif
 
@@ -480,6 +484,15 @@ sendit:
  ipsec_adjust_mtu(m, ifp->if_mtu);
 #endif
  error = EMSGSIZE;
+#if NPF > 0
+ /* pf changed routing table, use orig rtable for path MTU */
+ if (ro->ro_tableid != orig_rtableid) {
+ rtfree(ro->ro_rt);
+ ro->ro_tableid = orig_rtableid;
+ ro->ro_rt = icmp_mtudisc_clone(
+    satosin(&ro->ro_dst)->sin_addr, ro->ro_tableid, 0);
+ }
+#endif
  /*
  * This case can happen if the user changed the MTU
  * of an interface after enabling IP on it.  Because