OpenBSD VM on ESXi: uvn_flush: obj=0xfffffd813ee78298, offset=0x33f000. error during pageout.

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

OpenBSD VM on ESXi: uvn_flush: obj=0xfffffd813ee78298, offset=0x33f000. error during pageout.

Jurjen Oskam-3
Hi,

I'm using VMware ESXi (currently on 6.7 Update 3) to try things out before
running it on bare metal hardware. I've been doing this for a few release
cycles now. With the release of 6.6 I've encountered a phenomenon I
haven't seen before.

After the VM is installed, there is a high probability (~90-95%) that it
will write lots on lots of errors in the dmesg when the kernel is relinked:

uvn_flush: obj=0xfffffd813ee78298, offset=0x33f0000.  error during pageout.
uvn_flush: WARNING: changes to page may be lost!
uvn_flush: obj=0x0, offset=0x33f0000.  error during pageout.
uvn_flush: WARNING: changes to page may be lost!
        [ repeat last two lines many times ]
uvn_flush: obj=0xfffffd813ee78298, offset=0x3400000.  error during pageout.
uvn_flush: WARNING: changes to page may be lost!
uvn_flush: obj=0x0, offset=0x3400000.  error during pageout.
uvn_flush: WARNING: changes to page may be lost!
        [ repeat last two lines many times ]
uvn_flush: obj=0xfffffd813ee78298, offset=0x3410000.  error during pageout.
uvn_flush: WARNING: changes to page may be lost!
uvn_flush: obj=0x0, offset=0x3410000.  error during pageout.
uvn_flush: WARNING: changes to page may be lost!
        [ repeat last two lines many times ]
uvn_flush: obj=0xfffffd813ee78298, offset=0x3420000.  error during pageout.
uvn_flush: WARNING: changes to page may be lost!
uvn_flush: obj=0x0, offset=0x3420000.  error during pageout.
uvn_flush: WARNING: changes to page may be lost!
        [ repeat last two lines many times ]
        [ repeat the above pattern many times ]
uvn_flush: obj=0xfffffd813ee78298, offset=0x5c70000.  error during pageout.
uvn_flush: WARNING: changes to page may be lost!
uvn_flush: obj=0x0, offset=0x5c70000.  error during pageout.
uvn_flush: WARNING: changes to page may be lost!
        [ repeat last two lines many times ]

I can trigger this by running /usr/libexec/reorder_kernel manually. It
happens with 6.6 release and with the latest snapshot. It did not happen
with earlier versions. It also does not happen with library relinking.

The strange thing is that this doesn't happen on all OpenBSD VMs on this
machine. All VMs are configured with "Other 64-bit OS" in VMware.

Given what I found with a web search, a problem such as this is likely to
indicate bad hardware; bad memory in particular. What puzzles me is
that neither the machine (a Supermicro E200-D8 with ECC memory) nor
VMware reports any errors or gives any other indications of trouble.

What can I do from an OpenBSD perspective to investigate this in more
detail?


dmesg of the VM:

OpenBSD 6.6-current (GENERIC.MP) #411: Mon Oct 28 23:16:36 MDT 2019
    [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 4278059008 (4079MB)
avail mem = 4135690240 (3944MB)
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.7 @ 0xe0010 (242 entries)
bios0: vendor Phoenix Technologies LTD version "6.00" date 12/12/2018
bios0: VMware, Inc. VMware Virtual Platform
acpi0 at bios0: ACPI 4.0
acpi0: sleep states S0 S1 S4 S5
acpi0: tables DSDT FACP BOOT APIC MCFG SRAT HPET WAET
acpi0: wakeup devices PCI0(S3) USB_(S1) P2P0(S3) S1F0(S3) S2F0(S3) S8F0(S3) S16F(S3) S18F(S3) S22F(S3) S23F(S3) S24F(S3) S25F(S3) PE40(S3) S1F0(S3) PE50(S3) S1F0(S3) [...]
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel(R) Xeon(R) CPU D-1528 @ 1.90GHz, 1899.84 MHz, 06-56-03
cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,SSE3,PCLMUL,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,RDSEED,ADX,SMAP,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,ARAT,XSAVEOPT,MELTDOWN
cpu0: 256KB 64b/line 8-way L2 cache
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
cpu0: apic clock running at 65MHz
cpu1 at mainbus0: apid 2 (application processor)
cpu1: Intel(R) Xeon(R) CPU D-1528 @ 1.90GHz, 1899.67 MHz, 06-56-03
cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,SSE3,PCLMUL,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,RDSEED,ADX,SMAP,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,ARAT,XSAVEOPT,MELTDOWN
cpu1: 256KB 64b/line 8-way L2 cache
cpu1: smt 0, core 0, package 2
ioapic0 at mainbus0: apid 1 pa 0xfec00000, version 20, 24 pins
acpimcfg0 at acpi0
acpimcfg0: addr 0xf0000000, bus 0-127
acpihpet0 at acpi0: 14318179 Hz
acpiprt0 at acpi0: bus 0 (PCI0)
acpicpu0 at acpi0: C1(@1 halt!)
acpicpu1 at acpi0: C1(@1 halt!)
acpipci0 at acpi0 PCI0: 0x00000000 0x00000011 0x00000001
acpicmos0 at acpi0
"PNP0A05" at acpi0 not configured
acpiac0 at acpi0: AC unit online
cpu0: using VERW MDS workaround
pvbus0 at mainbus0: VMware
vmt0 at pvbus0
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel 82443BX AGP" rev 0x01
ppb0 at pci0 dev 1 function 0 "Intel 82443BX AGP" rev 0x01
pci1 at ppb0 bus 1
pcib0 at pci0 dev 7 function 0 "Intel 82371AB PIIX4 ISA" rev 0x08
pciide0 at pci0 dev 7 function 1 "Intel 82371AB IDE" rev 0x01: DMA, channel 0 configured to compatibility, channel 1 configured to compatibility
pciide0: channel 0 disabled (no drives)
pciide0: channel 1 disabled (no drives)
piixpm0 at pci0 dev 7 function 3 "Intel 82371AB Power" rev 0x08: SMBus disabled
"VMware VMCI" rev 0x10 at pci0 dev 7 function 7 not configured
vga1 at pci0 dev 15 function 0 "VMware SVGA II" rev 0x00
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
ppb1 at pci0 dev 17 function 0 "VMware PCI" rev 0x02
pci2 at ppb1 bus 2
ppb2 at pci0 dev 21 function 0 "VMware PCIE" rev 0x01
pci3 at ppb2 bus 3
vmx0 at pci3 dev 0 function 0 "VMware VMXNET3" rev 0x01: apic 1 int 18, address 00:0c:29:fb:60:eb
ppb3 at pci0 dev 21 function 1 "VMware PCIE" rev 0x01
pci4 at ppb3 bus 4
ppb4 at pci0 dev 21 function 2 "VMware PCIE" rev 0x01
pci5 at ppb4 bus 5
ppb5 at pci0 dev 21 function 3 "VMware PCIE" rev 0x01
pci6 at ppb5 bus 6
ppb6 at pci0 dev 21 function 4 "VMware PCIE" rev 0x01
pci7 at ppb6 bus 7
ppb7 at pci0 dev 21 function 5 "VMware PCIE" rev 0x01
pci8 at ppb7 bus 8
ppb8 at pci0 dev 21 function 6 "VMware PCIE" rev 0x01
pci9 at ppb8 bus 9
ppb9 at pci0 dev 21 function 7 "VMware PCIE" rev 0x01
pci10 at ppb9 bus 10
ppb10 at pci0 dev 22 function 0 "VMware PCIE" rev 0x01
pci11 at ppb10 bus 11
vmx1 at pci11 dev 0 function 0 "VMware VMXNET3" rev 0x01: apic 1 int 19, address 00:0c:29:fb:60:f5
ppb11 at pci0 dev 22 function 1 "VMware PCIE" rev 0x01
pci12 at ppb11 bus 12
ppb12 at pci0 dev 22 function 2 "VMware PCIE" rev 0x01
pci13 at ppb12 bus 13
ppb13 at pci0 dev 22 function 3 "VMware PCIE" rev 0x01
pci14 at ppb13 bus 14
ppb14 at pci0 dev 22 function 4 "VMware PCIE" rev 0x01
pci15 at ppb14 bus 15
ppb15 at pci0 dev 22 function 5 "VMware PCIE" rev 0x01
pci16 at ppb15 bus 16
ppb16 at pci0 dev 22 function 6 "VMware PCIE" rev 0x01
pci17 at ppb16 bus 17
ppb17 at pci0 dev 22 function 7 "VMware PCIE" rev 0x01
pci18 at ppb17 bus 18
ppb18 at pci0 dev 23 function 0 "VMware PCIE" rev 0x01
pci19 at ppb18 bus 19
nvme0 at pci19 dev 0 function 0 "VMware NVMe" rev 0x00: apic 1 int 16, NVMe 1.0
nvme0: VMware Virtual NVMe Disk, firmware 1.0, serial VMWare NVME-0000
scsibus1 at nvme0: 2 targets, initiator 0
sd0 at scsibus1 targ 1 lun 0: <NVMe, VMware Virtual N, 1.0>
sd0: 30720MB, 512 bytes/sector, 62914560 sectors
ppb19 at pci0 dev 23 function 1 "VMware PCIE" rev 0x01
pci20 at ppb19 bus 20
ppb20 at pci0 dev 23 function 2 "VMware PCIE" rev 0x01
pci21 at ppb20 bus 21
ppb21 at pci0 dev 23 function 3 "VMware PCIE" rev 0x01
pci22 at ppb21 bus 22
ppb22 at pci0 dev 23 function 4 "VMware PCIE" rev 0x01
pci23 at ppb22 bus 23
ppb23 at pci0 dev 23 function 5 "VMware PCIE" rev 0x01
pci24 at ppb23 bus 24
ppb24 at pci0 dev 23 function 6 "VMware PCIE" rev 0x01
pci25 at ppb24 bus 25
ppb25 at pci0 dev 23 function 7 "VMware PCIE" rev 0x01
pci26 at ppb25 bus 26
ppb26 at pci0 dev 24 function 0 "VMware PCIE" rev 0x01
pci27 at ppb26 bus 27
ppb27 at pci0 dev 24 function 1 "VMware PCIE" rev 0x01
pci28 at ppb27 bus 28
ppb28 at pci0 dev 24 function 2 "VMware PCIE" rev 0x01
pci29 at ppb28 bus 29
ppb29 at pci0 dev 24 function 3 "VMware PCIE" rev 0x01
pci30 at ppb29 bus 30
ppb30 at pci0 dev 24 function 4 "VMware PCIE" rev 0x01
pci31 at ppb30 bus 31
ppb31 at pci0 dev 24 function 5 "VMware PCIE" rev 0x01
pci32 at ppb31 bus 32
ppb32 at pci0 dev 24 function 6 "VMware PCIE" rev 0x01
pci33 at ppb32 bus 33
ppb33 at pci0 dev 24 function 7 "VMware PCIE" rev 0x01
pci34 at ppb33 bus 34
isa0 at pcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
pckbc0 at isa0 port 0x60/5 irq 1 irq 12
pckbd0 at pckbc0 (kbd slot)
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pms0 at pckbc0 (aux slot)
wsmouse0 at pms0 mux 0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
vscsi0 at root
scsibus2 at vscsi0: 256 targets
softraid0 at root
scsibus3 at softraid0: 256 targets
root on sd0a (afa24b55e438df24.a) swap on sd0b dump on sd0b


Regards,

Jurjen Oskam

Reply | Threaded
Open this post in threaded view
|

Re: OpenBSD VM on ESXi: uvn_flush: obj=0xfffffd813ee78298, offset=0x33f000. error during pageout.

Mike Larkin-2
On Tue, Oct 29, 2019 at 09:16:42PM +0100, Jurjen Oskam wrote:

> Hi,
>
> I'm using VMware ESXi (currently on 6.7 Update 3) to try things out before
> running it on bare metal hardware. I've been doing this for a few release
> cycles now. With the release of 6.6 I've encountered a phenomenon I
> haven't seen before.
>
> After the VM is installed, there is a high probability (~90-95%) that it
> will write lots on lots of errors in the dmesg when the kernel is relinked:
>
> uvn_flush: obj=0xfffffd813ee78298, offset=0x33f0000.  error during pageout.
> uvn_flush: WARNING: changes to page may be lost!
> uvn_flush: obj=0x0, offset=0x33f0000.  error during pageout.
> uvn_flush: WARNING: changes to page may be lost!
> [ repeat last two lines many times ]
> uvn_flush: obj=0xfffffd813ee78298, offset=0x3400000.  error during pageout.
> uvn_flush: WARNING: changes to page may be lost!
> uvn_flush: obj=0x0, offset=0x3400000.  error during pageout.
> uvn_flush: WARNING: changes to page may be lost!
> [ repeat last two lines many times ]
> uvn_flush: obj=0xfffffd813ee78298, offset=0x3410000.  error during pageout.
> uvn_flush: WARNING: changes to page may be lost!
> uvn_flush: obj=0x0, offset=0x3410000.  error during pageout.
> uvn_flush: WARNING: changes to page may be lost!
> [ repeat last two lines many times ]
> uvn_flush: obj=0xfffffd813ee78298, offset=0x3420000.  error during pageout.
> uvn_flush: WARNING: changes to page may be lost!
> uvn_flush: obj=0x0, offset=0x3420000.  error during pageout.
> uvn_flush: WARNING: changes to page may be lost!
> [ repeat last two lines many times ]
> [ repeat the above pattern many times ]
> uvn_flush: obj=0xfffffd813ee78298, offset=0x5c70000.  error during pageout.
> uvn_flush: WARNING: changes to page may be lost!
> uvn_flush: obj=0x0, offset=0x5c70000.  error during pageout.
> uvn_flush: WARNING: changes to page may be lost!
> [ repeat last two lines many times ]
>
> I can trigger this by running /usr/libexec/reorder_kernel manually. It
> happens with 6.6 release and with the latest snapshot. It did not happen
> with earlier versions. It also does not happen with library relinking.
>
> The strange thing is that this doesn't happen on all OpenBSD VMs on this
> machine. All VMs are configured with "Other 64-bit OS" in VMware.
>
> Given what I found with a web search, a problem such as this is likely to
> indicate bad hardware; bad memory in particular. What puzzles me is
> that neither the machine (a Supermicro E200-D8 with ECC memory) nor
> VMware reports any errors or gives any other indications of trouble.
>
> What can I do from an OpenBSD perspective to investigate this in more
> detail?
>
>

See below.

-ml

> dmesg of the VM:
>
> OpenBSD 6.6-current (GENERIC.MP) #411: Mon Oct 28 23:16:36 MDT 2019
>     [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
> real mem = 4278059008 (4079MB)
> avail mem = 4135690240 (3944MB)
> mpath0 at root
> scsibus0 at mpath0: 256 targets
> mainbus0 at root
> bios0 at mainbus0: SMBIOS rev. 2.7 @ 0xe0010 (242 entries)
> bios0: vendor Phoenix Technologies LTD version "6.00" date 12/12/2018
> bios0: VMware, Inc. VMware Virtual Platform
> acpi0 at bios0: ACPI 4.0
> acpi0: sleep states S0 S1 S4 S5
> acpi0: tables DSDT FACP BOOT APIC MCFG SRAT HPET WAET
> acpi0: wakeup devices PCI0(S3) USB_(S1) P2P0(S3) S1F0(S3) S2F0(S3) S8F0(S3) S16F(S3) S18F(S3) S22F(S3) S23F(S3) S24F(S3) S25F(S3) PE40(S3) S1F0(S3) PE50(S3) S1F0(S3) [...]
> acpitimer0 at acpi0: 3579545 Hz, 24 bits
> acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
> cpu0 at mainbus0: apid 0 (boot processor)
> cpu0: Intel(R) Xeon(R) CPU D-1528 @ 1.90GHz, 1899.84 MHz, 06-56-03
> cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,SSE3,PCLMUL,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,RDSEED,ADX,SMAP,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,ARAT,XSAVEOPT,MELTDOWN
> cpu0: 256KB 64b/line 8-way L2 cache
> cpu0: smt 0, core 0, package 0
> mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
> cpu0: apic clock running at 65MHz
> cpu1 at mainbus0: apid 2 (application processor)
> cpu1: Intel(R) Xeon(R) CPU D-1528 @ 1.90GHz, 1899.67 MHz, 06-56-03
> cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,SSE3,PCLMUL,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,RDSEED,ADX,SMAP,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,ARAT,XSAVEOPT,MELTDOWN
> cpu1: 256KB 64b/line 8-way L2 cache
> cpu1: smt 0, core 0, package 2
> ioapic0 at mainbus0: apid 1 pa 0xfec00000, version 20, 24 pins
> acpimcfg0 at acpi0
> acpimcfg0: addr 0xf0000000, bus 0-127
> acpihpet0 at acpi0: 14318179 Hz
> acpiprt0 at acpi0: bus 0 (PCI0)
> acpicpu0 at acpi0: C1(@1 halt!)
> acpicpu1 at acpi0: C1(@1 halt!)
> acpipci0 at acpi0 PCI0: 0x00000000 0x00000011 0x00000001
> acpicmos0 at acpi0
> "PNP0A05" at acpi0 not configured
> acpiac0 at acpi0: AC unit online
> cpu0: using VERW MDS workaround
> pvbus0 at mainbus0: VMware
> vmt0 at pvbus0
> pci0 at mainbus0 bus 0
> pchb0 at pci0 dev 0 function 0 "Intel 82443BX AGP" rev 0x01
> ppb0 at pci0 dev 1 function 0 "Intel 82443BX AGP" rev 0x01
> pci1 at ppb0 bus 1
> pcib0 at pci0 dev 7 function 0 "Intel 82371AB PIIX4 ISA" rev 0x08
> pciide0 at pci0 dev 7 function 1 "Intel 82371AB IDE" rev 0x01: DMA, channel 0 configured to compatibility, channel 1 configured to compatibility
> pciide0: channel 0 disabled (no drives)
> pciide0: channel 1 disabled (no drives)
> piixpm0 at pci0 dev 7 function 3 "Intel 82371AB Power" rev 0x08: SMBus disabled
> "VMware VMCI" rev 0x10 at pci0 dev 7 function 7 not configured
> vga1 at pci0 dev 15 function 0 "VMware SVGA II" rev 0x00
> wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
> wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
> ppb1 at pci0 dev 17 function 0 "VMware PCI" rev 0x02
> pci2 at ppb1 bus 2
> ppb2 at pci0 dev 21 function 0 "VMware PCIE" rev 0x01
> pci3 at ppb2 bus 3
> vmx0 at pci3 dev 0 function 0 "VMware VMXNET3" rev 0x01: apic 1 int 18, address 00:0c:29:fb:60:eb
> ppb3 at pci0 dev 21 function 1 "VMware PCIE" rev 0x01
> pci4 at ppb3 bus 4
> ppb4 at pci0 dev 21 function 2 "VMware PCIE" rev 0x01
> pci5 at ppb4 bus 5
> ppb5 at pci0 dev 21 function 3 "VMware PCIE" rev 0x01
> pci6 at ppb5 bus 6
> ppb6 at pci0 dev 21 function 4 "VMware PCIE" rev 0x01
> pci7 at ppb6 bus 7
> ppb7 at pci0 dev 21 function 5 "VMware PCIE" rev 0x01
> pci8 at ppb7 bus 8
> ppb8 at pci0 dev 21 function 6 "VMware PCIE" rev 0x01
> pci9 at ppb8 bus 9
> ppb9 at pci0 dev 21 function 7 "VMware PCIE" rev 0x01
> pci10 at ppb9 bus 10
> ppb10 at pci0 dev 22 function 0 "VMware PCIE" rev 0x01
> pci11 at ppb10 bus 11
> vmx1 at pci11 dev 0 function 0 "VMware VMXNET3" rev 0x01: apic 1 int 19, address 00:0c:29:fb:60:f5
> ppb11 at pci0 dev 22 function 1 "VMware PCIE" rev 0x01
> pci12 at ppb11 bus 12
> ppb12 at pci0 dev 22 function 2 "VMware PCIE" rev 0x01
> pci13 at ppb12 bus 13
> ppb13 at pci0 dev 22 function 3 "VMware PCIE" rev 0x01
> pci14 at ppb13 bus 14
> ppb14 at pci0 dev 22 function 4 "VMware PCIE" rev 0x01
> pci15 at ppb14 bus 15
> ppb15 at pci0 dev 22 function 5 "VMware PCIE" rev 0x01
> pci16 at ppb15 bus 16
> ppb16 at pci0 dev 22 function 6 "VMware PCIE" rev 0x01
> pci17 at ppb16 bus 17
> ppb17 at pci0 dev 22 function 7 "VMware PCIE" rev 0x01
> pci18 at ppb17 bus 18
> ppb18 at pci0 dev 23 function 0 "VMware PCIE" rev 0x01
> pci19 at ppb18 bus 19
> nvme0 at pci19 dev 0 function 0 "VMware NVMe" rev 0x00: apic 1 int 16, NVMe 1.0
> nvme0: VMware Virtual NVMe Disk, firmware 1.0, serial VMWare NVME-0000

Why did you assign this non-default disk type to the guest VM?

Try assigning mpi(4) (LSI Logic SAS) instead. I've been using that with my
ESXi 6.7U3 box here without problems for weeks.

If that works, it's either an error in our nvme(4) driver or ESXi's emulation
of the NVMe hardware.

-ml

> scsibus1 at nvme0: 2 targets, initiator 0
> sd0 at scsibus1 targ 1 lun 0: <NVMe, VMware Virtual N, 1.0>
> sd0: 30720MB, 512 bytes/sector, 62914560 sectors
> ppb19 at pci0 dev 23 function 1 "VMware PCIE" rev 0x01
> pci20 at ppb19 bus 20
> ppb20 at pci0 dev 23 function 2 "VMware PCIE" rev 0x01
> pci21 at ppb20 bus 21
> ppb21 at pci0 dev 23 function 3 "VMware PCIE" rev 0x01
> pci22 at ppb21 bus 22
> ppb22 at pci0 dev 23 function 4 "VMware PCIE" rev 0x01
> pci23 at ppb22 bus 23
> ppb23 at pci0 dev 23 function 5 "VMware PCIE" rev 0x01
> pci24 at ppb23 bus 24
> ppb24 at pci0 dev 23 function 6 "VMware PCIE" rev 0x01
> pci25 at ppb24 bus 25
> ppb25 at pci0 dev 23 function 7 "VMware PCIE" rev 0x01
> pci26 at ppb25 bus 26
> ppb26 at pci0 dev 24 function 0 "VMware PCIE" rev 0x01
> pci27 at ppb26 bus 27
> ppb27 at pci0 dev 24 function 1 "VMware PCIE" rev 0x01
> pci28 at ppb27 bus 28
> ppb28 at pci0 dev 24 function 2 "VMware PCIE" rev 0x01
> pci29 at ppb28 bus 29
> ppb29 at pci0 dev 24 function 3 "VMware PCIE" rev 0x01
> pci30 at ppb29 bus 30
> ppb30 at pci0 dev 24 function 4 "VMware PCIE" rev 0x01
> pci31 at ppb30 bus 31
> ppb31 at pci0 dev 24 function 5 "VMware PCIE" rev 0x01
> pci32 at ppb31 bus 32
> ppb32 at pci0 dev 24 function 6 "VMware PCIE" rev 0x01
> pci33 at ppb32 bus 33
> ppb33 at pci0 dev 24 function 7 "VMware PCIE" rev 0x01
> pci34 at ppb33 bus 34
> isa0 at pcib0
> isadma0 at isa0
> com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
> pckbc0 at isa0 port 0x60/5 irq 1 irq 12
> pckbd0 at pckbc0 (kbd slot)
> wskbd0 at pckbd0: console keyboard, using wsdisplay0
> pms0 at pckbc0 (aux slot)
> wsmouse0 at pms0 mux 0
> pcppi0 at isa0 port 0x61
> spkr0 at pcppi0
> vscsi0 at root
> scsibus2 at vscsi0: 256 targets
> softraid0 at root
> scsibus3 at softraid0: 256 targets
> root on sd0a (afa24b55e438df24.a) swap on sd0b dump on sd0b
>
>
> Regards,
>
> Jurjen Oskam
>

Reply | Threaded
Open this post in threaded view
|

Re: OpenBSD VM on ESXi: uvn_flush: obj=0xfffffd813ee78298, offset=0x33f000. error during pageout.

Jurjen Oskam-3
On Tue, Oct 29, 2019 at 01:25:10PM -0700, Mike Larkin wrote:

> On Tue, Oct 29, 2019 at 09:16:42PM +0100, Jurjen Oskam wrote:
  [...]
> > uvn_flush: obj=0xfffffd813ee78298, offset=0x33f0000.  error during pageout.
> > uvn_flush: WARNING: changes to page may be lost!
> > uvn_flush: obj=0x0, offset=0x33f0000.  error during pageout.
> > uvn_flush: WARNING: changes to page may be lost!
> > [ repeat last two lines many times ]
  [...]

> > nvme0 at pci19 dev 0 function 0 "VMware NVMe" rev 0x00: apic 1 int 16, NVMe 1.0
> > nvme0: VMware Virtual NVMe Disk, firmware 1.0, serial VMWare NVME-0000
>
> Why did you assign this non-default disk type to the guest VM?
>
> Try assigning mpi(4) (LSI Logic SAS) instead. I've been using that with my
> ESXi 6.7U3 box here without problems for weeks.
>
> If that works, it's either an error in our nvme(4) driver or ESXi's emulation
> of the NVMe hardware.

I forgot to mention that I tried using different controller types, and
nvme(4) happened to be the one I took the dmesg of. The ones I tried were
LSI Logic SAS, LSI Logic Parallel and VMware Paravirtual (the latter
after working around the lost first write problem). All showed the same
symptom.

I have been trying old snapshots (thanks to the snapshot archive at
ftp.hostserver.de), and found the point where the problem started to
occur:

All snapshots I tried up to and including this point did not show the
problem:
OpenBSD 6.6-beta (GENERIC.MP) #202: Mon Aug 12 11:01:21 MDT 2019

All snapshots I tried starting from this point show the problem:
OpenBSD 6.6-beta (GENERIC.MP) #207: Tue Aug 13 11:32:34 MDT 2019


Would it be helpful to start a binary search for the exact commit that
introduced the problem? I've been looking at the commit history around
that time but haven't been able to spot an obvious candidate; but that's
probably because I'm not a programmer.

Regards,

Jurjen Oskam


Reply | Threaded
Open this post in threaded view
|

Re: OpenBSD VM on ESXi: uvn_flush: obj=0xfffffd813ee78298, offset=0x33f000. error during pageout.

Mike Larkin-2
On Wed, Oct 30, 2019 at 12:14:20PM +0100, Jurjen Oskam wrote:

> On Tue, Oct 29, 2019 at 01:25:10PM -0700, Mike Larkin wrote:
>
> > On Tue, Oct 29, 2019 at 09:16:42PM +0100, Jurjen Oskam wrote:
>   [...]
> > > uvn_flush: obj=0xfffffd813ee78298, offset=0x33f0000.  error during pageout.
> > > uvn_flush: WARNING: changes to page may be lost!
> > > uvn_flush: obj=0x0, offset=0x33f0000.  error during pageout.
> > > uvn_flush: WARNING: changes to page may be lost!
> > > [ repeat last two lines many times ]
>   [...]
> > > nvme0 at pci19 dev 0 function 0 "VMware NVMe" rev 0x00: apic 1 int 16, NVMe 1.0
> > > nvme0: VMware Virtual NVMe Disk, firmware 1.0, serial VMWare NVME-0000
> >
> > Why did you assign this non-default disk type to the guest VM?
> >
> > Try assigning mpi(4) (LSI Logic SAS) instead. I've been using that with my
> > ESXi 6.7U3 box here without problems for weeks.
> >
> > If that works, it's either an error in our nvme(4) driver or ESXi's emulation
> > of the NVMe hardware.
>
> I forgot to mention that I tried using different controller types, and
> nvme(4) happened to be the one I took the dmesg of. The ones I tried were
> LSI Logic SAS, LSI Logic Parallel and VMware Paravirtual (the latter
> after working around the lost first write problem). All showed the same
> symptom.
>
> I have been trying old snapshots (thanks to the snapshot archive at
> ftp.hostserver.de), and found the point where the problem started to
> occur:
>
> All snapshots I tried up to and including this point did not show the
> problem:
> OpenBSD 6.6-beta (GENERIC.MP) #202: Mon Aug 12 11:01:21 MDT 2019
>
> All snapshots I tried starting from this point show the problem:
> OpenBSD 6.6-beta (GENERIC.MP) #207: Tue Aug 13 11:32:34 MDT 2019
>
>
> Would it be helpful to start a binary search for the exact commit that
> introduced the problem? I've been looking at the commit history around
> that time but haven't been able to spot an obvious candidate; but that's
> probably because I'm not a programmer.
>
> Regards,
>
> Jurjen Oskam
>
>

yes.

Reply | Threaded
Open this post in threaded view
|

Re: OpenBSD VM on ESXi: uvn_flush: obj=0xfffffd813ee78298, offset=0x33f000. error during pageout.

Stuart Henderson
In reply to this post by Jurjen Oskam-3
On 2019-10-30, Jurjen Oskam <[hidden email]> wrote:

>
> All snapshots I tried up to and including this point did not show the
> problem:
> OpenBSD 6.6-beta (GENERIC.MP) #202: Mon Aug 12 11:01:21 MDT 2019
>
> All snapshots I tried starting from this point show the problem:
> OpenBSD 6.6-beta (GENERIC.MP) #207: Tue Aug 13 11:32:34 MDT 2019
>
>
> Would it be helpful to start a binary search for the exact commit that
> introduced the problem?

Yes, definitely! We usually do this with date-based cvs updates.

>                         I've been looking at the commit history around
> that time but haven't been able to spot an obvious candidate; but that's
> probably because I'm not a programmer.

Sometimes diffs are tested in snapshots before they're committed,
so you might need to look beyond the snapshot dates to find the
commit.