panic: vmxnet3_getbuf: buffer has mbuf

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

panic: vmxnet3_getbuf: buffer has mbuf

Stuart Henderson
amd64 MP (2 cores), esxi VM. fairly recent kernel, I *think* it's probably
6.1-release but won't know for sure until I reboot. Running unifi (java,
mongod) and librenms (php, snmpget, rrdtool, mariadb).

Anything else wanted? It's either screen-grabs from vmware console,
or hand-retyped.

panic: vmxnet3_getbuf: buffer has mbuf

TID     PID   UID  PRFLAGS    PFLAGS CPU COMMAND
*420655 71983 755      0x2         0   0 snmpget
23681   1382  732 0x200003 0x4000000   1 mongod

cpu0 :

Debugger+0x9
panic+0xfe
vmxnet3_getbuf+0x123
vmxnet3_rxintr+0x236
vmxnet3_intr+0x61
intr_handler+0x67
Xintr_ioapic_level10+0xcd
-- interrupt --
Xspllower+0xc
mtx_leave+0x34
Xsoftclock+0x14
-- interrupt --
Bad fp: 0xffff800000590000
end trace fr: 0xffff800000590000, count: 5

cpu1 :

x86_ipi_handler+0x76
Xresume_lapic_ipi+0x1c
-- int --
__mp_lock+0x42
syscall+0x2e5
-- syscall (number 78) --
end of kernel
end trace fr: 0x155512f6cf70, count: 10
0x155516ace7fa:

Reply | Threaded
Open this post in threaded view
|

Re: panic: vmxnet3_getbuf: buffer has mbuf

Stuart Henderson
https://marc.info/?l=openbsd-bugs&m=149256099403302&w=2
On 2017/04/19 01:16, Stuart Henderson wrote:
> panic: vmxnet3_getbuf: buffer has mbuf

https://marc.info/?l=openbsd-bugs&m=150028524915195&w=2
On 2017/07/17 10:53, Stuart Henderson wrote:
> I ran into "vmxnet3_getbuf: buffer has mbuf" on an esxi VM (5.5.0 -
> VM version 8) running librenms and unifi (java/mongodb). Lots of snmp
> fetches, lots of disk io from rrdtool, fairly high memory utilisation.

https://marc.info/?l=openbsd-bugs&m=150641651900601&w=2
On 2017/09/26 10:01, Stuart Henderson wrote:
> panic occurred with rsync-over-ssh copying a ~50MB file to the machine.
> active processes according to the auto-listing at panic: sshd, softnet

Still hitting these, latest was on 6.2 + syspatches.

I'm switching to ethernet0.virtualDev = "e1000" to see how that goes.
If anyone has a suggestion for something that would get useful debug
out of it I can switch back.

panic: vmxnet3_getbuf: buffer has mbuf
Stoppped at db_enter+0x5
tid    pid   uid prflags pflags cpu command
344935 91229 0   0x14000 0x200  1   softnet

trace:

db_enter
panic
vmxnet3_getbuf+0x117
vmxnet3_rxintr+0x2bc
vmxnet3_intr+0x70
intr_handler+0x5e
Xintr_ioapic_level10+0xd0
--- interrupt ---
end of kernel
end trace frame: 0xc748faffdc9cbbe9, count: 8
acpi_pdirpa+0x669c78:

OpenBSD 6.2 (GENERIC.MP) #0: Thu Oct 12 19:53:18 CEST 2017
    [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 6425608192 (6127MB)
avail mem = 6223851520 (5935MB)
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.4 @ 0xe0010 (364 entries)
bios0: vendor Phoenix Technologies LTD version "6.00" date 04/14/2014
bios0: VMware, Inc. VMware Virtual Platform
acpi0 at bios0: rev 2
acpi0: sleep states S0 S1 S4 S5
acpi0: tables DSDT FACP BOOT APIC MCFG SRAT HPET WAET
acpi0: wakeup devices PCI0(S3) USB_(S1) P2P0(S3) S1F0(S3) S2F0(S3) S8F0(S3) S16F(S3) S18F(S3) S22F(S3) S23F(S3) S24F(S3) S25F(S3) PE40(S3) S1F0(S3) PE50(S3) S1F0(S3) [...]
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel(R) Xeon(R) CPU E5630 @ 2.53GHz, 2533.39 MHz
cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,SSSE3,CX16,SSE4.1,SSE4.2,POPCNT,AES,HV,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,SENSOR,ARAT
cpu0: 256KB 64b/line 8-way L2 cache
cpu0: TSC frequency 2533385710 Hz
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
cpu0: apic clock running at 65MHz
cpu1 at mainbus0: apid 1 (application processor)
cpu1: Intel(R) Xeon(R) CPU E5630 @ 2.53GHz, 2533.87 MHz
cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,SSSE3,CX16,SSE4.1,SSE4.2,POPCNT,AES,HV,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,SENSOR,ARAT
cpu1: 256KB 64b/line 8-way L2 cache
cpu1: smt 0, core 1, package 0
cpu2 at mainbus0: apid 2 (application processor)
cpu2: Intel(R) Xeon(R) CPU E5630 @ 2.53GHz, 2534.02 MHz
cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,SSSE3,CX16,SSE4.1,SSE4.2,POPCNT,AES,HV,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,SENSOR,ARAT
cpu2: 256KB 64b/line 8-way L2 cache
cpu2: smt 0, core 2, package 0
cpu3 at mainbus0: apid 3 (application processor)
cpu3: Intel(R) Xeon(R) CPU E5630 @ 2.53GHz, 2534.17 MHz
cpu3: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,SSSE3,CX16,SSE4.1,SSE4.2,POPCNT,AES,HV,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,SENSOR,ARAT
cpu3: 256KB 64b/line 8-way L2 cache
cpu3: smt 0, core 3, package 0
ioapic0 at mainbus0: apid 4 pa 0xfec00000, version 11, 24 pins
acpimcfg0 at acpi0 addr 0xf0000000, bus 0-127
acpihpet0 at acpi0: 14318179 Hz
acpiprt0 at acpi0: bus 0 (PCI0)
acpicpu0 at acpi0: C1(@1 halt!)
acpicpu1 at acpi0: C1(@1 halt!)
acpicpu2 at acpi0: C1(@1 halt!)
acpicpu3 at acpi0: C1(@1 halt!)
"PNP0001" at acpi0 not configured
"PNP0F13" at acpi0 not configured
"PNP0A05" at acpi0 not configured
"PNP0400" at acpi0 not configured
"PNP0700" at acpi0 not configured
acpiac0 at acpi0: AC unit online
pvbus0 at mainbus0: VMware
vmt0 at pvbus0
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel 82443BX AGP" rev 0x01
ppb0 at pci0 dev 1 function 0 "Intel 82443BX AGP" rev 0x01
pci1 at ppb0 bus 1
pcib0 at pci0 dev 7 function 0 "Intel 82371AB PIIX4 ISA" rev 0x08
pciide0 at pci0 dev 7 function 1 "Intel 82371AB IDE" rev 0x01: DMA, channel 0 configured to compatibility, channel 1 configured to compatibility
pciide0: channel 0 disabled (no drives)
atapiscsi0 at pciide0 channel 1 drive 0
scsibus1 at atapiscsi0: 2 targets
cd0 at scsibus1 targ 0 lun 0: <NECVMWar, VMware IDE CDR10, 1.00> ATAPI 5/cdrom removable
cd0(pciide0:1:0): using PIO mode 4, Ultra-DMA mode 2
piixpm0 at pci0 dev 7 function 3 "Intel 82371AB Power" rev 0x08: SMBus disabled
"VMware VMCI" rev 0x10 at pci0 dev 7 function 7 not configured
vga1 at pci0 dev 15 function 0 "VMware SVGA II" rev 0x00
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
ppb1 at pci0 dev 17 function 0 "VMware PCI" rev 0x02
pci2 at ppb1 bus 2
ppb2 at pci0 dev 21 function 0 "VMware PCIE" rev 0x01
pci3 at ppb2 bus 3
mpi0 at pci3 dev 0 function 0 "Symbios Logic SAS1068" rev 0x01: apic 4 int 18
mpi0: SAS3444, firmware 1.3.41.32
scsibus2 at mpi0: 256 targets, initiator 16
sd0 at scsibus2 targ 0 lun 0: <VMware, Virtual disk, 1.0> SCSI2 0/direct fixed
sd0: 20480MB, 512 bytes/sector, 41943040 sectors
sd1 at scsibus2 targ 1 lun 0: <VMware, Virtual disk, 1.0> SCSI2 0/direct fixed
sd1: 16384MB, 512 bytes/sector, 33554432 sectors
ppb3 at pci0 dev 21 function 1 "VMware PCIE" rev 0x01
pci4 at ppb3 bus 4
ppb4 at pci0 dev 21 function 2 "VMware PCIE" rev 0x01
pci5 at ppb4 bus 5
ppb5 at pci0 dev 21 function 3 "VMware PCIE" rev 0x01
pci6 at ppb5 bus 6
ppb6 at pci0 dev 21 function 4 "VMware PCIE" rev 0x01
pci7 at ppb6 bus 7
ppb7 at pci0 dev 21 function 5 "VMware PCIE" rev 0x01
pci8 at ppb7 bus 8
ppb8 at pci0 dev 21 function 6 "VMware PCIE" rev 0x01
pci9 at ppb8 bus 9
ppb9 at pci0 dev 21 function 7 "VMware PCIE" rev 0x01
pci10 at ppb9 bus 10
ppb10 at pci0 dev 22 function 0 "VMware PCIE" rev 0x01
pci11 at ppb10 bus 11
vmx0 at pci11 dev 0 function 0 "VMware VMXNET3" rev 0x01: apic 4 int 19, address 00:0c:29:b1:2d:c1
ppb11 at pci0 dev 22 function 1 "VMware PCIE" rev 0x01
pci12 at ppb11 bus 12
ppb12 at pci0 dev 22 function 2 "VMware PCIE" rev 0x01
pci13 at ppb12 bus 13
ppb13 at pci0 dev 22 function 3 "VMware PCIE" rev 0x01
pci14 at ppb13 bus 14
ppb14 at pci0 dev 22 function 4 "VMware PCIE" rev 0x01
pci15 at ppb14 bus 15
ppb15 at pci0 dev 22 function 5 "VMware PCIE" rev 0x01
pci16 at ppb15 bus 16
ppb16 at pci0 dev 22 function 6 "VMware PCIE" rev 0x01
pci17 at ppb16 bus 17
ppb17 at pci0 dev 22 function 7 "VMware PCIE" rev 0x01
pci18 at ppb17 bus 18
ppb18 at pci0 dev 23 function 0 "VMware PCIE" rev 0x01
pci19 at ppb18 bus 19
ppb19 at pci0 dev 23 function 1 "VMware PCIE" rev 0x01
pci20 at ppb19 bus 20
ppb20 at pci0 dev 23 function 2 "VMware PCIE" rev 0x01
pci21 at ppb20 bus 21
ppb21 at pci0 dev 23 function 3 "VMware PCIE" rev 0x01
pci22 at ppb21 bus 22
ppb22 at pci0 dev 23 function 4 "VMware PCIE" rev 0x01
pci23 at ppb22 bus 23
ppb23 at pci0 dev 23 function 5 "VMware PCIE" rev 0x01
pci24 at ppb23 bus 24
ppb24 at pci0 dev 23 function 6 "VMware PCIE" rev 0x01
pci25 at ppb24 bus 25
ppb25 at pci0 dev 23 function 7 "VMware PCIE" rev 0x01
pci26 at ppb25 bus 26
ppb26 at pci0 dev 24 function 0 "VMware PCIE" rev 0x01
pci27 at ppb26 bus 27
ppb27 at pci0 dev 24 function 1 "VMware PCIE" rev 0x01
pci28 at ppb27 bus 28
ppb28 at pci0 dev 24 function 2 "VMware PCIE" rev 0x01
pci29 at ppb28 bus 29
ppb29 at pci0 dev 24 function 3 "VMware PCIE" rev 0x01
pci30 at ppb29 bus 30
ppb30 at pci0 dev 24 function 4 "VMware PCIE" rev 0x01
pci31 at ppb30 bus 31
ppb31 at pci0 dev 24 function 5 "VMware PCIE" rev 0x01
pci32 at ppb31 bus 32
ppb32 at pci0 dev 24 function 6 "VMware PCIE" rev 0x01
pci33 at ppb32 bus 33
ppb33 at pci0 dev 24 function 7 "VMware PCIE" rev 0x01
pci34 at ppb33 bus 34
isa0 at pcib0
isadma0 at isa0
fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
pckbc0 at isa0 port 0x60/5 irq 1 irq 12
pckbd0 at pckbc0 (kbd slot)
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pms0 at pckbc0 (aux slot)
wsmouse0 at pms0 mux 0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
lpt0 at isa0 port 0x378/4 irq 7
vscsi0 at root
scsibus3 at vscsi0: 256 targets
softraid0 at root
scsibus4 at softraid0: 256 targets
root on sd0a (be6f9e4bbacaf9c9.a) swap on sd0b dump on sd0b
WARNING: / was not properly unmounted


Reply | Threaded
Open this post in threaded view
|

Re: panic: vmxnet3_getbuf: buffer has mbuf

keenonkites
Stuart,
did you get any further ?

We're having the same problem here with OpenBSD (6.0 and OpenBSD 6.2) when
taking ESXi snapshots of an OpenBSD machine after some uptime (freshly
rebooted machines do not crash during snapshot).



Any solution/workaround to prevent the crash would be welcomed

Thanks
Patrik




--
Sent from: http://openbsd-archive.7691.n7.nabble.com/openbsd-dev-bugs-f183916.html

Reply | Threaded
Open this post in threaded view
|

Re: panic: vmxnet3_getbuf: buffer has mbuf

Stuart Henderson
On 2018/05/24 02:03, keenonkites wrote:

> Stuart,
> did you get any further ?
>
> We're having the same problem here with OpenBSD (6.0 and OpenBSD 6.2) when
> taking ESXi snapshots of an OpenBSD machine after some uptime (freshly
> rebooted machines do not crash during snapshot).
>
>
>
> Any solution/workaround to prevent the crash would be welcomed

No idea on a fix. Workaround: change the vmx file to use

ethernet0.virtualDev = "e1000"

and OpenBSD config to use em(4).

Reply | Threaded
Open this post in threaded view
|

Re: panic: vmxnet3_getbuf: buffer has mbuf

keenonkites
Thanks. You think it's a bug in vmxnet3 (vm tools ?) or something like
running out of buffers that probably could be fixed by assigning more of
them ? Gesendet: Donnerstag, 24. Mai 2018 um 13:56 Uhr
Von: "Stuart Henderson" <[hidden email]>
An: keenonkites <[hidden email]>
Cc: [hidden email]
Betreff: Re: panic: vmxnet3_getbuf: buffer has mbufOn 2018/05/24 02:03,
keenonkites wrote:
> Stuart,
> did you get any further ?
>
> We're having the same problem here with OpenBSD (6.0 and OpenBSD 6.2)
when
> taking ESXi snapshots of an OpenBSD machine after some uptime (freshly
> rebooted machines do not crash during snapshot).
>
>
>
> Any solution/workaround to prevent the crash would be welcomed

No idea on a fix. Workaround: change the vmx file to use

ethernet0.virtualDev = "e1000"

and OpenBSD config to use em(4).
Reply | Threaded
Open this post in threaded view
|

Re: panic: vmxnet3_getbuf: buffer has mbuf

Stuart Henderson
On 2018/05/24 15:06, [hidden email] wrote:

> Thanks.
>  
> You think it's a bug in vmxnet3 (vm tools ?) or something like running out of buffers that
> probably could be fixed by assigning more of them ?
>  
> Gesendet: Donnerstag, 24. Mai 2018 um 13:56 Uhr
> Von: "Stuart Henderson" <[hidden email]>
> An: keenonkites <[hidden email]>
> Cc: [hidden email]
> Betreff: Re: panic: vmxnet3_getbuf: buffer has mbuf
> On 2018/05/24 02:03, keenonkites wrote:
> > Stuart,
> > did you get any further ?
> >
> > We're having the same problem here with OpenBSD (6.0 and OpenBSD 6.2) when
> > taking ESXi snapshots of an OpenBSD machine after some uptime (freshly
> > rebooted machines do not crash during snapshot).
> >
> >
> >
> > Any solution/workaround to prevent the crash would be welcomed
>
> No idea on a fix. Workaround: change the vmx file to use
>
> ethernet0.virtualDev = "e1000"
>
> and OpenBSD config to use em(4).
>  

"OpenBSD kernel crashes under normal use" -> to my eyes, that's a bug.
But it might be that it's triggered by a VMware bug, too.

Pragmatically:

- it's stable with e1000

- it's going to be a pig to fix: something that takes days of uptime
to trigger makes debugging very hard

- the people who know their way around the kernel well enough to
debug this likely have hundreds of to-dos that will be a bigger
benefit to more people ...

So I have taken the view that switching to e1000 is the sane option
here.

Reply | Threaded
Open this post in threaded view
|

Re: panic: vmxnet3_getbuf: buffer has mbuf

keenonkites
I remember having different strange problems using e1000 in earlier days,
especially with ipsec, gre and ospf over it.... but I'll probably git it
a try again.ThanksPatrik Gesendet: Donnerstag, 24. Mai 2018 um 16:30 Uhr
Von: "Stuart Henderson" <[hidden email]>
An: [hidden email]
Cc: [hidden email]
Betreff: Re: Re: panic: vmxnet3_getbuf: buffer has mbufOn 2018/05/24
15:06, [hidden email] wrote:
> Thanks.
>
> You think it's a bug in vmxnet3 (vm tools ?) or something like running
out of buffers that

> probably could be fixed by assigning more of them ?
>
> Gesendet: Donnerstag, 24. Mai 2018 um 13:56 Uhr
> Von: "Stuart Henderson" <[hidden email]>
> An: keenonkites <[hidden email]>
> Cc: [hidden email]
> Betreff: Re: panic: vmxnet3_getbuf: buffer has mbuf
> On 2018/05/24 02:03, keenonkites wrote:
> > Stuart,
> > did you get any further ?
> >
> > We're having the same problem here with OpenBSD (6.0 and OpenBSD 6.2)
when
> > taking ESXi snapshots of an OpenBSD machine after some uptime
(freshly

> > rebooted machines do not crash during snapshot).
> >
> >
> >
> > Any solution/workaround to prevent the crash would be welcomed
>
> No idea on a fix. Workaround: change the vmx file to use
>
> ethernet0.virtualDev = "e1000"
>
> and OpenBSD config to use em(4).
>

"OpenBSD kernel crashes under normal use" -> to my eyes, that's a bug.
But it might be that it's triggered by a VMware bug, too.

Pragmatically:

- it's stable with e1000

- it's going to be a pig to fix: something that takes days of uptime
to trigger makes debugging very hard

- the people who know their way around the kernel well enough to
debug this likely have hundreds of to-dos that will be a bigger
benefit to more people ...

So I have taken the view that switching to e1000 is the sane option
here.