aesni/crypto related kernel panic on 6.3

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

aesni/crypto related kernel panic on 6.3

mabi
Hi,

I finally replaced my old OpenBSD 5.0 firewall with 6.3 which also serves as a site-to-site VPN using now iked instead of isakmpd. The problem is that when I start a big transfer over the VPN to the remote site, also an OpenBSD 6.3 firewall, the kernel panics. Crazy enough I tried to reproduce the problem to find out what it is related to and managed to even make both firewalls kernel panic at the same time. When this happens the hardware is frozen and won't take any input and won't even reboot automatically. I need to power it off and on again.

As I was logged into the serial console when I reproduced this problem I managed to get the following messages from the console:

fatal protection fault in supervisor mode
trap type 4 code 0 rip ffffffff8104d58a cs 8 rflags 10202 cr2  1dd9ec5f7c70 cpl a rsp ffff80002231ce28
panic: trap type 4, code=0, pc=ffffffff8104d58a
Starting stack trace...
panic() at panic+0x11c
trap() at trap+0x688
--- trap (number 4) ---
memcpy(ffff8000001be460,ffffff03a0f34188,0,0,16,a21ae232a3847235) at memcpy+0xa
aesni_process(ffffff03a0f34188) at aesni_process+0x124
crypto_invoke(ffffffff8116ebc0) at crypto_invoke+0xd0
taskq_thread(0) at taskq_thread+0x67
end trace frame: 0x0, count: 251
End of stack trace.
syncing disks...

​When this happened I just started to transfer over SSH a ZFS snapshot to the remote site using the IPSec VPN. The iked daemon was rekeying its SAs and then the kernel paniced...

Below I pasted the dmesg of the firewall corresponding to the kernel panic message above. I can't send now the dmesg of the remote firewall as I need to go on-site first. Please let me know if I should send any log files or other details.

Regards,
Mabi


OpenBSD 6.3 (GENERIC.MP) #107: Sat Mar 24 14:21:59 MDT 2018
    [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 17104490496 (16312MB)
avail mem = 16579031040 (15810MB)
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.8 @ 0xec1e0 (85 entries)
bios0: vendor American Megatrends Inc. version "4.6.5" date 02/05/2015
bios0: INTEL Corporation DENLOW_REFRESH_WS
acpi0 at bios0: rev 2
acpi0: sleep states S0 S5
acpi0: tables DSDT FACP APIC FPDT FIDT SSDT SSDT MCFG HPET SSDT SSDT ASF! SPCR DMAR EINJ ERST HEST BERT
acpi0: wakeup devices PEGP(S0) PEG0(S0) PEGP(S0) PEG1(S0) PEGP(S0) PEG2(S0) PXSX(S0) RP01(S0) PXSX(S0) RP02(S0) PXSX(S0) RP03(S0) PXSX(S0) RP04(S0) PXSX(S0) RP05(S0) [...]
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel(R) Xeon(R) CPU E3-1275 v3 @ 3.50GHz, 3691.99 MHz
cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,PERF,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,IBRS,IBPB,STIBP,SENSOR,ARAT,MELTDOWN
cpu0: 256KB 64b/line 8-way L2 cache
acpitimer0: recalibrated TSC frequency 3491911605 Hz
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges
cpu0: apic clock running at 99MHz
cpu0: mwait min=64, max=64, C-substates=0.2.1.2.4, IBE
cpu1 at mainbus0: apid 2 (application processor)
cpu1: Intel(R) Xeon(R) CPU E3-1275 v3 @ 3.50GHz, 3691.45 MHz
cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,PERF,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,IBRS,IBPB,STIBP,SENSOR,ARAT,MELTDOWN
cpu1: 256KB 64b/line 8-way L2 cache
cpu1: smt 0, core 1, package 0
cpu2 at mainbus0: apid 4 (application processor)
cpu2: Intel(R) Xeon(R) CPU E3-1275 v3 @ 3.50GHz, 3691.45 MHz
cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,PERF,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,IBRS,IBPB,STIBP,SENSOR,ARAT,MELTDOWN
cpu2: 256KB 64b/line 8-way L2 cache
cpu2: smt 0, core 2, package 0
cpu3 at mainbus0: apid 6 (application processor)
cpu3: Intel(R) Xeon(R) CPU E3-1275 v3 @ 3.50GHz, 3691.45 MHz
cpu3: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,PERF,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,IBRS,IBPB,STIBP,SENSOR,ARAT,MELTDOWN
cpu3: 256KB 64b/line 8-way L2 cache
cpu3: smt 0, core 3, package 0
cpu4 at mainbus0: apid 1 (application processor)
cpu4: Intel(R) Xeon(R) CPU E3-1275 v3 @ 3.50GHz, 3691.45 MHz
cpu4: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,PERF,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,IBRS,IBPB,STIBP,SENSOR,ARAT,MELTDOWN
cpu4: 256KB 64b/line 8-way L2 cache
cpu4: smt 1, core 0, package 0
cpu5 at mainbus0: apid 3 (application processor)
cpu5: Intel(R) Xeon(R) CPU E3-1275 v3 @ 3.50GHz, 3691.45 MHz
cpu5: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,PERF,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,IBRS,IBPB,STIBP,SENSOR,ARAT,MELTDOWN
cpu5: 256KB 64b/line 8-way L2 cache
cpu5: smt 1, core 1, package 0
cpu6 at mainbus0: apid 5 (application processor)
cpu6: Intel(R) Xeon(R) CPU E3-1275 v3 @ 3.50GHz, 3691.45 MHz
cpu6: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,PERF,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,IBRS,IBPB,STIBP,SENSOR,ARAT,MELTDOWN
cpu6: 256KB 64b/line 8-way L2 cache
cpu6: smt 1, core 2, package 0
cpu7 at mainbus0: apid 7 (application processor)
cpu7: Intel(R) Xeon(R) CPU E3-1275 v3 @ 3.50GHz, 3691.45 MHz
cpu7: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,PERF,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,IBRS,IBPB,STIBP,SENSOR,ARAT,MELTDOWN
cpu7: 256KB 64b/line 8-way L2 cache
cpu7: smt 1, core 3, package 0
ioapic0 at mainbus0: apid 8 pa 0xfec00000, version 20, 24 pins
acpimcfg0 at acpi0 addr 0xf8000000, bus 0-63
acpihpet0 at acpi0: 14318179 Hz
acpihpet0: recalibrated TSC frequency 3491928190 Hz
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus 1 (PEG0)
acpiprt2 at acpi0: bus 2 (PEG1)
acpiprt3 at acpi0: bus 3 (PEG2)
acpiprt4 at acpi0: bus -1 (RP01)
acpiprt5 at acpi0: bus -1 (RP02)
acpiprt6 at acpi0: bus -1 (RP03)
acpiprt7 at acpi0: bus 4 (RP04)
acpiprt8 at acpi0: bus -1 (RP05)
acpiprt9 at acpi0: bus -1 (RP06)
acpiprt10 at acpi0: bus -1 (RP07)
acpiprt11 at acpi0: bus -1 (RP08)
acpiec0 at acpi0: not present
acpicpu0 at acpi0: C1(@1 halt!)
acpicpu1 at acpi0: C1(@1 halt!)
acpicpu2 at acpi0: C1(@1 halt!)
acpicpu3 at acpi0: C1(@1 halt!)
acpicpu4 at acpi0: C1(@1 halt!)
acpicpu5 at acpi0: C1(@1 halt!)
acpicpu6 at acpi0: C1(@1 halt!)
acpicpu7 at acpi0: C1(@1 halt!)
acpipwrres0 at acpi0: PG00, resource for PEG0
acpipwrres1 at acpi0: PG01, resource for PEG1
acpipwrres2 at acpi0: PG02, resource for PEG2
acpipwrres3 at acpi0: FN00, resource for FAN0
acpipwrres4 at acpi0: FN01, resource for FAN1
acpipwrres5 at acpi0: FN02, resource for FAN2
acpipwrres6 at acpi0: FN03, resource for FAN3
acpipwrres7 at acpi0: FN04, resource for FAN4
acpitz0 at acpi0: critical temperature is 105 degC
acpitz1 at acpi0: critical temperature is 105 degC
"INT3F0D" at acpi0 not configured
acpibtn0 at acpi0: PWRB
acpibtn1 at acpi0: SLPB
"PNP0C0B" at acpi0 not configured
"PNP0C0B" at acpi0 not configured
"PNP0C0B" at acpi0 not configured
"PNP0C0B" at acpi0 not configured
"PNP0C0B" at acpi0 not configured
acpivideo0 at acpi0: GFX0
acpivout0 at acpivideo0: DD1F
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel Xeon E3-1200 v3 Host" rev 0x06
ppb0 at pci0 dev 1 function 0 "Intel Core 4G PCIE" rev 0x06: msi
pci1 at ppb0 bus 1
ix0 at pci1 dev 0 function 0 "Intel 82599" rev 0x01: msi, address <EDITED_CONTENT>
ix1 at pci1 dev 0 function 1 "Intel 82599" rev 0x01: msi, address <EDITED_CONTENT>
ppb1 at pci0 dev 1 function 1 "Intel Core 4G PCIE" rev 0x06: msi
pci2 at ppb1 bus 2
em0 at pci2 dev 0 function 0 "Intel I350" rev 0x01: msi, address <EDITED_CONTENT>
em1 at pci2 dev 0 function 1 "Intel I350" rev 0x01: msi, address <EDITED_CONTENT>
em2 at pci2 dev 0 function 2 "Intel I350" rev 0x01: msi, address <EDITED_CONTENT>
em3 at pci2 dev 0 function 3 "Intel I350" rev 0x01: msi, address <EDITED_CONTENT>
ppb2 at pci0 dev 1 function 2 "Intel Core 4G PCIE" rev 0x06: msi
pci3 at ppb2 bus 3
em4 at pci3 dev 0 function 0 "Intel I350" rev 0x01: msi, address <EDITED_CONTENT>
em5 at pci3 dev 0 function 1 "Intel I350" rev 0x01: msi, address <EDITED_CONTENT>
em6 at pci3 dev 0 function 2 "Intel I350" rev 0x01: msi, address <EDITED_CONTENT>
em7 at pci3 dev 0 function 3 "Intel I350" rev 0x01: msi, address <EDITED_CONTENT>
xhci0 at pci0 dev 20 function 0 "Intel 8 Series xHCI" rev 0x05: msi
usb0 at xhci0: USB revision 3.0
uhub0 at usb0 configuration 1 interface 0 "Intel xHCI root hub" rev 3.00/1.00 addr 1
ehci0 at pci0 dev 26 function 0 "Intel 8 Series USB" rev 0x05: apic 8 int 16
usb1 at ehci0: USB revision 2.0
uhub1 at usb1 configuration 1 interface 0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
ppb3 at pci0 dev 28 function 0 "Intel 8 Series PCIE" rev 0xd5: msi
pci4 at ppb3 bus 4
ppb4 at pci4 dev 0 function 0 "ASPEED Technology AST1150 PCI" rev 0x02
pci5 at ppb4 bus 5
vga1 at pci5 dev 0 function 0 "ASPEED Technology AST2000" rev 0x21
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
ehci1 at pci0 dev 29 function 0 "Intel 8 Series USB" rev 0x05: apic 8 int 23
usb2 at ehci1: USB revision 2.0
uhub2 at usb2 configuration 1 interface 0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
pcib0 at pci0 dev 31 function 0 "Intel C226 LPC" rev 0x05
ahci0 at pci0 dev 31 function 2 "Intel 8 Series AHCI" rev 0x05: msi, AHCI 1.3
ahci0: port 0: 6.0Gb/s
ahci0: port 2: 6.0Gb/s
scsibus1 at ahci0: 32 targets
sd0 at scsibus1 targ 0 lun 0: <ATA, KINGSTON SV300S3, 608A> SCSI3 0/direct fixed naa.50026b7265075225
sd0: 57241MB, 512 bytes/sector, 117231408 sectors, thin
sd1 at scsibus1 targ 2 lun 0: <ATA, KINGSTON SV300S3, 608A> SCSI3 0/direct fixed naa.50026b7265074c9b
sd1: 57241MB, 512 bytes/sector, 117231408 sectors, thin
ichiic0 at pci0 dev 31 function 3 "Intel 8 Series SMBus" rev 0x05: apic 8 int 18
iic0 at ichiic0
sdtemp0 at iic0 addr 0x18: cat34ts02
sdtemp1 at iic0 addr 0x1a: cat34ts02
lm1 at iic0 addr 0x2d: W83627DHG
spdmem0 at iic0 addr 0x50: 8GB DDR3 SDRAM ECC PC3-12800 with thermal sensor
spdmem1 at iic0 addr 0x52: 8GB DDR3 SDRAM ECC PC3-12800 with thermal sensor
isa0 at pcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
com0: console
com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
pckbc0 at isa0 port 0x60/5 irq 1 irq 12
pckbd0 at pckbc0 (kbd slot)
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
lpt0 at isa0 port 0x378/4 irq 7
wbsio0 at isa0 port 0x2e/2: NCT6776F rev 0x33
lm2 at wbsio0 port 0xa30/8: NCT6776F
lm1: disabling sensors due to alias with lm2
vmm0 at mainbus0: VMX/EPT
uhub3 at uhub1 port 1 configuration 1 interface 0 "Intel Rate Matching Hub" rev 2.00/0.05 addr 2
uhub4 at uhub2 port 1 configuration 1 interface 0 "Intel Rate Matching Hub" rev 2.00/0.05 addr 2
vscsi0 at root
scsibus2 at vscsi0: 256 targets
softraid0 at root
scsibus3 at softraid0: 256 targets
softraid0: trying to bring up sd2 degraded
softraid0: sd2 was not shutdown properly
sd2 at scsibus3 targ 1 lun 0: <OPENBSD, SR RAID 1, 006> SCSI2 0/direct fixed
sd2: 57239MB, 512 bytes/sector, 117225713 sectors
softraid0: roaming device  -> sd1a
root on sd2a (bcade8472637be36.a) swap on sd2b dump on sd2b
WARNING: / was not properly unmounted