kernel panic

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

kernel panic

daniel-2
>Synopsis: syncthing with many files crashes the kernel
>Category: kernel
>Environment:
        System      : OpenBSD 6.6
        Details     : OpenBSD 6.6 (GENERIC.MP) #0: Sat Oct 26 08:08:07 MDT 2019
                         [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP

        Architecture: OpenBSD.amd64
        Machine     : amd64
>Description:

        My system constantly crashes overnight, so I have to login
        remotely over VNC to enter the encryption password and restart the machine.

        I have tried to trace the bug in ddb, and I think I have found
        where the bug is, but I don't know how to fix it. When it crashes
        this message is being displayed in ddb:

        /bsd: Stopped at in_delmulti+0xe8:       movq    %rcx,0(%rax)

        which corresponds to this line in sys/netinet/in.c:

        TAILQ_REMOVE(&ifp->if_maddrlist, &inm->inm_ifma, ifma_list);

        I suspect it has something to do with multicast traffic syncthing
        generates, as the same setup doesn't crash my other machines
        which use Linux kernel instead.


>How-To-Repeat:
        I don't know how reproducible the bug is, but it happens on my
        machine just when I use syncthing, if I don't start it, the
        server runs fine without any crashes. Probably worth mentioning that
        my syncthing setup has lots of files (~400k), so I have increased the
        login.conf openfiles-max limits and sysctl kern.maxfiles value.
>Fix:
        I wish I knew...


dmesg:
OpenBSD 6.6 (GENERIC.MP) #0: Sat Oct 26 08:08:07 MDT 2019
    [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 64407580672 (61423MB)
avail mem = 62442901504 (59550MB)
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.8 @ 0xbffffda0 (13 entries)
bios0: vendor SeaBIOS version "rel-1.12.1-0-ga5cab58e9a3f-prebuilt.qemu.org" date 04/01/2014
bios0: QEMU Standard PC (i440FX + PIIX, 1996)
acpi0 at bios0: ACPI 1.0
acpi0: sleep states S3 S4 S5
acpi0: tables DSDT FACP APIC SSDT HPET
acpi0: wakeup devices
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz, 451.76 MHz, 06-4f-01
cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,FSGSBASE,TSC_ADJUST,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,RDSEED,ADX,SMAP,UMIP,MD_CLEAR,IBRS,IBPB,STIBP,SSBD,ARAT,XSAVEOPT,MELTDOWN
cpu0: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 512KB 64b/line 16-way L2 cache
cpu0: ITLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu0: DTLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
cpu0: apic clock running at 1000MHz
cpu1 at mainbus0: apid 1 (application processor)
cpu1: Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz, 501.49 MHz, 06-4f-01
cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,FSGSBASE,TSC_ADJUST,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,RDSEED,ADX,SMAP,UMIP,MD_CLEAR,IBRS,IBPB,STIBP,SSBD,ARAT,XSAVEOPT,MELTDOWN
cpu1: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 512KB 64b/line 16-way L2 cache
cpu1: ITLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu1: DTLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu1: smt 0, core 1, package 0
cpu2 at mainbus0: apid 2 (application processor)
cpu2: Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz, 548.04 MHz, 06-4f-01
cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,FSGSBASE,TSC_ADJUST,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,RDSEED,ADX,SMAP,UMIP,MD_CLEAR,IBRS,IBPB,STIBP,SSBD,ARAT,XSAVEOPT,MELTDOWN
cpu2: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 512KB 64b/line 16-way L2 cache
cpu2: ITLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu2: DTLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu2: smt 0, core 2, package 0
cpu3 at mainbus0: apid 3 (application processor)
cpu3: Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz, 545.05 MHz, 06-4f-01
cpu3: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,FSGSBASE,TSC_ADJUST,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,RDSEED,ADX,SMAP,UMIP,MD_CLEAR,IBRS,IBPB,STIBP,SSBD,ARAT,XSAVEOPT,MELTDOWN
cpu3: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 512KB 64b/line 16-way L2 cache
cpu3: ITLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu3: DTLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu3: smt 0, core 3, package 0
cpu4 at mainbus0: apid 4 (application processor)
cpu4: Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz, 521.22 MHz, 06-4f-01
cpu4: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,FSGSBASE,TSC_ADJUST,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,RDSEED,ADX,SMAP,UMIP,MD_CLEAR,IBRS,IBPB,STIBP,SSBD,ARAT,XSAVEOPT,MELTDOWN
cpu4: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 512KB 64b/line 16-way L2 cache
cpu4: ITLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu4: DTLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu4: smt 0, core 4, package 0
cpu5 at mainbus0: apid 5 (application processor)
cpu5: Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz, 548.26 MHz, 06-4f-01
cpu5: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,FSGSBASE,TSC_ADJUST,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,RDSEED,ADX,SMAP,UMIP,MD_CLEAR,IBRS,IBPB,STIBP,SSBD,ARAT,XSAVEOPT,MELTDOWN
cpu5: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 512KB 64b/line 16-way L2 cache
cpu5: ITLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu5: DTLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu5: smt 0, core 5, package 0
cpu6 at mainbus0: apid 6 (application processor)
cpu6: Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz, 535.31 MHz, 06-4f-01
cpu6: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,FSGSBASE,TSC_ADJUST,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,RDSEED,ADX,SMAP,UMIP,MD_CLEAR,IBRS,IBPB,STIBP,SSBD,ARAT,XSAVEOPT,MELTDOWN
cpu6: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 512KB 64b/line 16-way L2 cache
cpu6: ITLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu6: DTLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu6: smt 0, core 6, package 0
cpu7 at mainbus0: apid 7 (application processor)
cpu7: Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz, 534.40 MHz, 06-4f-01
cpu7: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,FSGSBASE,TSC_ADJUST,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,RDSEED,ADX,SMAP,UMIP,MD_CLEAR,IBRS,IBPB,STIBP,SSBD,ARAT,XSAVEOPT,MELTDOWN
cpu7: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 512KB 64b/line 16-way L2 cache
cpu7: ITLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu7: DTLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu7: smt 0, core 7, package 0
cpu8 at mainbus0: apid 8 (application processor)
cpu8: Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz, 538.85 MHz, 06-4f-01
cpu8: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,FSGSBASE,TSC_ADJUST,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,RDSEED,ADX,SMAP,UMIP,MD_CLEAR,IBRS,IBPB,STIBP,SSBD,ARAT,XSAVEOPT,MELTDOWN
cpu8: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 512KB 64b/line 16-way L2 cache
cpu8: ITLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu8: DTLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu8: smt 0, core 0, package 1
cpu9 at mainbus0: apid 9 (application processor)
cpu9: Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz, 489.50 MHz, 06-4f-01
cpu9: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,FSGSBASE,TSC_ADJUST,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,RDSEED,ADX,SMAP,UMIP,MD_CLEAR,IBRS,IBPB,STIBP,SSBD,ARAT,XSAVEOPT,MELTDOWN
cpu9: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 512KB 64b/line 16-way L2 cache
cpu9: ITLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu9: DTLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu9: smt 0, core 1, package 1
ioapic0 at mainbus0: apid 0 pa 0xfec00000, version 11, 24 pins
acpihpet0 at acpi0: 100000000 Hz
acpiprt0 at acpi0: bus 0 (PCI0)
acpicpu0 at acpi0: C1(@1 halt!)
acpicpu1 at acpi0: C1(@1 halt!)
acpicpu2 at acpi0: C1(@1 halt!)
acpicpu3 at acpi0: C1(@1 halt!)
acpicpu4 at acpi0: C1(@1 halt!)
acpicpu5 at acpi0: C1(@1 halt!)
acpicpu6 at acpi0: C1(@1 halt!)
acpicpu7 at acpi0: C1(@1 halt!)
acpicpu8 at acpi0: C1(@1 halt!)
acpicpu9 at acpi0: C1(@1 halt!)
"ACPI0006" at acpi0 not configured
acpipci0 at acpi0 PCI0: _OSC failed
acpicmos0 at acpi0
"PNP0A06" at acpi0 not configured
"PNP0A06" at acpi0 not configured
"PNP0A06" at acpi0 not configured
"QEMU0002" at acpi0 not configured
"ACPI0010" at acpi0 not configured
"QEMUVGID" at acpi0 not configured
cpu0: using VERW MDS workaround
pvbus0 at mainbus0: KVM
pvclock0 at pvbus0
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel 82441FX" rev 0x02
pcib0 at pci0 dev 1 function 0 "Intel 82371SB ISA" rev 0x00
pciide0 at pci0 dev 1 function 1 "Intel 82371SB IDE" rev 0x00: DMA, channel 0 wired to compatibility, channel 1 wired to compatibility
pciide0: channel 0 disabled (no drives)
pciide0: channel 1 disabled (no drives)
uhci0 at pci0 dev 1 function 2 "Intel 82371SB USB" rev 0x01: apic 0 int 11
piixpm0 at pci0 dev 1 function 3 "Intel 82371AB Power" rev 0x03: apic 0 int 9
iic0 at piixpm0
vga1 at pci0 dev 2 function 0 "VMware SVGA II" rev 0x00
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
virtio0 at pci0 dev 3 function 0 "Qumranet Virtio Memory Balloon" rev 0x00
viomb0 at virtio0
virtio0: apic 0 int 11
virtio1 at pci0 dev 5 function 0 "Qumranet Virtio SCSI" rev 0x00
vioscsi0 at virtio1: qsize 128
scsibus1 at vioscsi0: 255 targets
sd0 at scsibus1 targ 0 lun 0: <QEMU, QEMU HARDDISK, 2.5+>
sd0: 1638400MB, 512 bytes/sector, 3355443200 sectors, thin
virtio1: msix shared
virtio2 at pci0 dev 18 function 0 "Qumranet Virtio Network" rev 0x00
vio0 at virtio2: address 00:50:56:3e:bb:2a
virtio2: msix shared
ppb0 at pci0 dev 30 function 0 "Red Hat Qemu PCI-PCI" rev 0x00
pci1 at ppb0 bus 1
ppb1 at pci0 dev 31 function 0 "Red Hat Qemu PCI-PCI" rev 0x00
pci2 at ppb1 bus 2
isa0 at pcib0
isadma0 at isa0
fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
pckbc0 at isa0 port 0x60/5 irq 1 irq 12
pckbd0 at pckbc0 (kbd slot)
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pms0 at pckbc0 (aux slot)
wsmouse0 at pms0 mux 0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
usb0 at uhci0: USB revision 1.0
uhub0 at usb0 configuration 1 interface 0 "Intel UHCI root hub" rev 1.00/1.00 addr 1
vscsi0 at root
scsibus2 at vscsi0: 256 targets
softraid0 at root
scsibus3 at softraid0: 256 targets
softraid0: sd1 was not shutdown properly
sd1 at scsibus3 targ 1 lun 0: <OPENBSD, SR CRYPTO, 006>
sd1: 1638394MB, 512 bytes/sector, 3355431698 sectors
root on sd1a (967470d09e8aafa5.a) swap on sd1b dump on sd1b
WARNING: / was not properly unmounted
fd0 at fdc0 drive 1: density unknown

usbdevs:
Controller /dev/usb0:
addr 01: 8086:0000 Intel, UHCI root hub
         full speed, self powered, config 1, rev 1.00
         driver: uhub0

Reply | Threaded
Open this post in threaded view
|

Re: kernel panic

Anton Lindqvist-2
On Fri, Nov 01, 2019 at 12:59:45AM +0900, [hidden email] wrote:

> >Synopsis: syncthing with many files crashes the kernel
> >Category: kernel
> >Environment:
> System      : OpenBSD 6.6
> Details     : OpenBSD 6.6 (GENERIC.MP) #0: Sat Oct 26 08:08:07 MDT 2019
> [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
>
> Architecture: OpenBSD.amd64
> Machine     : amd64
> >Description:
>
> My system constantly crashes overnight, so I have to login
> remotely over VNC to enter the encryption password and restart the machine.
>
> I have tried to trace the bug in ddb, and I think I have found
> where the bug is, but I don't know how to fix it. When it crashes
> this message is being displayed in ddb:
>
> /bsd: Stopped at in_delmulti+0xe8:       movq    %rcx,0(%rax)
>
> which corresponds to this line in sys/netinet/in.c:
>
> TAILQ_REMOVE(&ifp->if_maddrlist, &inm->inm_ifma, ifma_list);
>
> I suspect it has something to do with multicast traffic syncthing
> generates, as the same setup doesn't crash my other machines
> which use Linux kernel instead.

Also discovered by syzkaller, with a complete backtrace:

  https://syzkaller.appspot.com/bug?extid=7939d2c4bc9a5dfa707a