vmd eating lots of memory

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

vmd eating lots of memory

Paul de Weerd
Hi all,

I'm running two vmd(8) VMs:

[weerd@despair] $ cat /etc/vm.conf
vm "undeadly" {
        owner root
        memory 2G
        disk /home/vmm/undeadly.dsk
        disk /storage/vmm/undeadly.dsk
        interface switch "vmmswitch" lladdr "fe:e1:bb:02:6b:bf"
}

vm "testvm" {
        owner root
        memory 2G
        disk /home/vmm/testvm.dsk
        interface switch "vmmswitch" lladdr "fe:e1:bb:02:6b:be"
}

switch "vmmswitch" {
        enable
        interface bridge0
}

Both machines have been up since boot of the host machine:

[weerd@despair] $ vmctl status
   ID   PID VCPUS  MAXMEM  CURMEM     TTY        OWNER    STATE NAME
    2 23133     1    2.0G    1.1G   ttyp1         root  running testvm
    1 42094     1    2.0G    1.5G   ttyp0         root  running undeadly
[weerd@despair] $ uptime
 4:11PM  up 43 days, 57 mins, 12 users, load averages: 0.02, 0.07, 0.07
[weerd@despair] $ ssh undeadly uptime
 4:11PM  up 43 days, 57 mins, 2 users, load averages: 0.28, 0.12, 0.10
[weerd@despair] $ ssh testvm uptime  
 4:11PM  up 43 days, 57 mins, 0 users, load averages: 0.00, 0.00, 0.00

However, the undeadly VM consumes signficantly more memory that the
other VM:

[weerd@despair] $ ps wwwaux | grep -e vm[d] -e [P]ID
USER       PID %CPU %MEM   VSZ   RSS TT  STAT  STARTED       TIME COMMAND
_vmd     42094  3.7 57.9 28769236 19398792 ??  Ip     7Jun19  4741:37.66 vmd: undeadly (vmd)
_vmd      4567  0.0  0.0  1384  1384 ??  Isp    7Jun19    0:00.01 vmd: vmm (vmd)
root     84392  0.0  0.0  1564  1436 ??  Isp    7Jun19    0:00.02 /usr/sbin/vmd
_vmd     55269  0.0  0.0  1304  1468 ??  Isp    7Jun19    0:00.01 vmd: control (vmd)
root     51181  0.0  0.0  1236   928 ??  Is     7Jun19    0:00.01 vmd: priv (vmd)
_vmd     23133  0.0  0.2 2100100 70272 ??  Ip     7Jun19  1886:16.18 vmd: testvm (vmd)

All systems run snaps from around the same time:

[weerd@despair] $ sysctl kern.version
kern.version=OpenBSD 6.5-current (GENERIC.MP) #6: Tue Jun  4 15:05:10 MDT 2019
    [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP

[weerd@despair] $ ssh undeadly sysctl kern.version
kern.version=OpenBSD 6.5-current (GENERIC) #7: Fri Jun  7 00:45:34 MDT 2019
    [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC

[weerd@despair] $ ssh testvm sysctl kern.version
kern.version=OpenBSD 6.5-current (GENERIC) #6: Tue Jun  4 14:57:43 MDT 2019
    [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC

So it looks like there's a memory leak, either only for busy VMs or it
shows more for busier VMs.  Has anyone seen anything similar on their
vmm VMs?

I'll reboot the undeadly VM for now, as other processes are getting
killed for out of memory reasons (bgpd, in my case; see end of below
dmesg of host machine 'despair').

Paul

--- despair dmesg ----------------------------------------------------
OpenBSD 6.5-current (GENERIC.MP) #6: Tue Jun  4 15:05:10 MDT 2019
    [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 34332733440 (32742MB)
avail mem = 33282105344 (31740MB)
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.7 @ 0xe66d0 (57 entries)
bios0: vendor Dell Inc. version "1.2.3" date 07/21/2011
bios0: Dell Inc. PowerEdge R210 II
acpi0 at bios0: rev 2
acpi0: sleep states S0 S4 S5
acpi0: tables DSDT FACP SPMI ASF! HPET APIC MCFG BOOT SSDT ASPT SSDT SSDT SPCR DMAR HEST ERST BERT EINJ
acpi0: wakeup devices P0P1(S4) GLAN(S0) EHC1(S4) EHC2(S4) PXSX(S4) RP01(S5) PXSX(S4) RP02(S5) PXSX(S4) RP03(S5) PXSX(S4) RP04(S5) PXSX(S4) RP05(S5) PXSX(S4) RP06(S5) [...]
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpihpet0 at acpi0: 14318179 Hz
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel(R) Xeon(R) CPU E31260L @ 2.40GHz, 2400.39 MHz, 06-2a-07
cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
cpu0: 256KB 64b/line 8-way L2 cache
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges
cpu0: apic clock running at 100MHz
cpu0: mwait min=64, max=64, C-substates=0.2.1.1, IBE
cpu1 at mainbus0: apid 1 (application processor)
cpu1: Intel(R) Xeon(R) CPU E31260L @ 2.40GHz, 2400.02 MHz, 06-2a-07
cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
cpu1: 256KB 64b/line 8-way L2 cache
cpu1: smt 1, core 0, package 0
cpu2 at mainbus0: apid 2 (application processor)
cpu2: Intel(R) Xeon(R) CPU E31260L @ 2.40GHz, 2400.02 MHz, 06-2a-07
cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
cpu2: 256KB 64b/line 8-way L2 cache
cpu2: smt 0, core 1, package 0
cpu3 at mainbus0: apid 3 (application processor)
cpu3: Intel(R) Xeon(R) CPU E31260L @ 2.40GHz, 2400.02 MHz, 06-2a-07
cpu3: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
cpu3: 256KB 64b/line 8-way L2 cache
cpu3: smt 1, core 1, package 0
cpu4 at mainbus0: apid 4 (application processor)
cpu4: Intel(R) Xeon(R) CPU E31260L @ 2.40GHz, 2400.02 MHz, 06-2a-07
cpu4: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
cpu4: 256KB 64b/line 8-way L2 cache
cpu4: smt 0, core 2, package 0
cpu5 at mainbus0: apid 5 (application processor)
cpu5: Intel(R) Xeon(R) CPU E31260L @ 2.40GHz, 2400.02 MHz, 06-2a-07
cpu5: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
cpu5: 256KB 64b/line 8-way L2 cache
cpu5: smt 1, core 2, package 0
cpu6 at mainbus0: apid 6 (application processor)
cpu6: Intel(R) Xeon(R) CPU E31260L @ 2.40GHz, 2400.02 MHz, 06-2a-07
cpu6: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
cpu6: 256KB 64b/line 8-way L2 cache
cpu6: smt 0, core 3, package 0
cpu7 at mainbus0: apid 7 (application processor)
cpu7: Intel(R) Xeon(R) CPU E31260L @ 2.40GHz, 2400.02 MHz, 06-2a-07
cpu7: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
cpu7: 256KB 64b/line 8-way L2 cache
cpu7: smt 1, core 3, package 0
ioapic0 at mainbus0: apid 0 pa 0xfec00000, version 20, 24 pins
acpimcfg0 at acpi0
acpimcfg0: addr 0xe0000000, bus 0-255
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus 2 (P0P1)
acpiprt2 at acpi0: bus 1 (RP01)
acpiprt3 at acpi0: bus -1 (RP02)
acpiprt4 at acpi0: bus -1 (RP03)
acpiprt5 at acpi0: bus -1 (RP04)
acpiprt6 at acpi0: bus -1 (RP05)
acpiprt7 at acpi0: bus -1 (RP06)
acpiprt8 at acpi0: bus -1 (RP07)
acpiprt9 at acpi0: bus -1 (RP08)
acpiprt10 at acpi0: bus -1 (PEG0)
acpiprt11 at acpi0: bus -1 (PEG1)
acpiprt12 at acpi0: bus -1 (PEG2)
acpiprt13 at acpi0: bus -1 (PEG3)
acpicpu0 at acpi0: C3(350@104 mwait.1@0x20), C2(500@80 mwait.1@0x10), C1(1000@1 mwait.1), PSS
acpicpu1 at acpi0: C3(350@104 mwait.1@0x20), C2(500@80 mwait.1@0x10), C1(1000@1 mwait.1), PSS
acpicpu2 at acpi0: C3(350@104 mwait.1@0x20), C2(500@80 mwait.1@0x10), C1(1000@1 mwait.1), PSS
acpicpu3 at acpi0: C3(350@104 mwait.1@0x20), C2(500@80 mwait.1@0x10), C1(1000@1 mwait.1), PSS
acpicpu4 at acpi0: C3(350@104 mwait.1@0x20), C2(500@80 mwait.1@0x10), C1(1000@1 mwait.1), PSS
acpicpu5 at acpi0: C3(350@104 mwait.1@0x20), C2(500@80 mwait.1@0x10), C1(1000@1 mwait.1), PSS
acpicpu6 at acpi0: C3(350@104 mwait.1@0x20), C2(500@80 mwait.1@0x10), C1(1000@1 mwait.1), PSS
acpicpu7 at acpi0: C3(350@104 mwait.1@0x20), C2(500@80 mwait.1@0x10), C1(1000@1 mwait.1), PSS
acpipwrres0 at acpi0: FN00, resource for FAN0
acpipwrres1 at acpi0: FN01, resource for FAN1
acpipwrres2 at acpi0: FN02, resource for FAN2
acpipwrres3 at acpi0: FN03, resource for FAN3
acpipwrres4 at acpi0: FN04, resource for FAN4
acpitz0 at acpi0: critical temperature is 100 degC
acpipci0 at acpi0 PCI0: 0x00000010 0x00000011 0x00000000
"IPI0001" at acpi0 not configured
acpicmos0 at acpi0
"PNP0A05" at acpi0 not configured
"PNP0A05" at acpi0 not configured
"PNP0C0B" at acpi0 not configured
"PNP0C0B" at acpi0 not configured
"PNP0C0B" at acpi0 not configured
"PNP0C0B" at acpi0 not configured
"PNP0C0B" at acpi0 not configured
ipmi at mainbus0 not configured
cpu0: using VERW MDS workaround (except on vmm entry)
cpu0: Enhanced SpeedStep 2400 MHz: speeds: 2401, 2400, 2300, 2200, 2100, 2000, 1900, 1800, 1700, 1600 MHz
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel Xeon E3-1200 Host" rev 0x09
ehci0 at pci0 dev 26 function 0 "Intel 6 Series USB" rev 0x04: apic 0 int 16
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 configuration 1 interface 0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
ppb0 at pci0 dev 28 function 0 "Intel 6 Series PCIE" rev 0xb4: msi
pci1 at ppb0 bus 1
bnx0 at pci1 dev 0 function 0 "Broadcom BCM5716" rev 0x20: apic 0 int 16
bnx1 at pci1 dev 0 function 1 "Broadcom BCM5716" rev 0x20: apic 0 int 17
ehci1 at pci0 dev 29 function 0 "Intel 6 Series USB" rev 0x04: apic 0 int 23
usb1 at ehci1: USB revision 2.0
uhub1 at usb1 configuration 1 interface 0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
ppb1 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0xa4
pci2 at ppb1 bus 2
2:3:0: mem address conflict 0xffff0000/0x10000
vga1 at pci2 dev 3 function 0 "Matrox MGA G200eW" rev 0x0a
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
pcib0 at pci0 dev 31 function 0 "Intel C202 LPC" rev 0x04
ahci0 at pci0 dev 31 function 2 "Intel 6 Series AHCI" rev 0x04: msi, AHCI 1.3
ahci0: port 0: 3.0Gb/s
ahci0: port 1: 3.0Gb/s
ahci0: port 4: 1.5Gb/s
scsibus1 at ahci0: 32 targets
sd0 at scsibus1 targ 0 lun 0: <ATA, INTEL SSDSC2BW48, DC32> SCSI3 0/direct fixed naa.55cd2e414c807f33
sd0: 457862MB, 512 bytes/sector, 937703088 sectors, thin
sd1 at scsibus1 targ 1 lun 0: <ATA, WDC WD60EFRX-68L, 82.0> SCSI3 0/direct fixed naa.50014ee20dd29e68
sd1: 5723166MB, 512 bytes/sector, 11721045168 sectors
cd0 at scsibus1 targ 4 lun 0: <TSSTcorp, DVD+-RW TS-L633J, D150> ATAPI 5/cdrom removable
ichiic0 at pci0 dev 31 function 3 "Intel 6 Series SMBus" rev 0x04: apic 0 int 19
iic0 at ichiic0
sdtemp0 at iic0 addr 0x18: mcp98243
sdtemp1 at iic0 addr 0x19: mcp98243
sdtemp2 at iic0 addr 0x1a: mcp98243
sdtemp3 at iic0 addr 0x1b: mcp98243
spdmem0 at iic0 addr 0x50: 8GB DDR3 SDRAM ECC PC3-12800 with thermal sensor
spdmem1 at iic0 addr 0x51: 8GB DDR3 SDRAM ECC PC3-12800 with thermal sensor
spdmem2 at iic0 addr 0x52: 8GB DDR3 SDRAM ECC PC3-12800 with thermal sensor
spdmem3 at iic0 addr 0x53: 8GB DDR3 SDRAM ECC PC3-12800 with thermal sensor
isa0 at pcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
com0: console
com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
pckbc0 at isa0 port 0x60/5 irq 1 irq 12
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
vmm0 at mainbus0: VMX/EPT
uhub2 at uhub0 port 1 configuration 1 interface 0 "Intel Rate Matching Hub" rev 2.00/0.00 addr 2
uhub3 at uhub1 port 1 configuration 1 interface 0 "Intel Rate Matching Hub" rev 2.00/0.00 addr 2
uhidev0 at uhub3 port 1 configuration 1 interface 0 "RDing TEMPERHUM1V1.2" rev 2.00/0.01 addr 3
uhidev0: iclass 3/1, 1 report id
ukbd0 at uhidev0 reportid 1: 8 variable keys, 5 key codes
wskbd0 at ukbd0: console keyboard, using wsdisplay0
uhidev1 at uhub3 port 1 configuration 1 interface 1 "RDing TEMPERHUM1V1.2" rev 2.00/0.01 addr 3
uhidev1: iclass 3/1
ugold0 at uhidev1
uhub4 at uhub3 port 5 configuration 1 interface 0 "Standard Microsystems product 0x2514" rev 2.00/0.00 addr 4
vscsi0 at root
scsibus2 at vscsi0: 256 targets
softraid0 at root
scsibus3 at softraid0: 256 targets
sd2 at scsibus3 targ 1 lun 0: <OPENBSD, SR CRYPTO, 006> SCSI2 0/direct fixed
sd2: 409602MB, 512 bytes/sector, 838865577 sectors
root on sd2a (5a5905f3a7206f17.a) swap on sd2b dump on sd2b
bnx0: address bc:30:5b:e8:d6:48
brgphy0 at bnx0 phy 1: BCM5709 10/100/1000baseT PHY, rev. 8
bnx1: address bc:30:5b:e8:d6:49
brgphy1 at bnx1 phy 1: BCM5709 10/100/1000baseT PHY, rev. 8
ugold0: 2 sensors type si7006 (temperature and humidity)
sd3 at scsibus3 targ 2 lun 0: <OPENBSD, SR CRYPTO, 006> SCSI2 0/direct fixed
sd3: 5231654MB, 512 bytes/sector, 10714427745 sectors
UVM: pid 36202 (bgpd), uid 75 killed: out of swap
----------------------------------------------------------------------

--- undeadly dmesg ---------------------------------------------------
OpenBSD 6.5-current (GENERIC) #7: Fri Jun  7 00:45:34 MDT 2019
    [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC
real mem = 2130690048 (2031MB)
avail mem = 2056167424 (1960MB)
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.4 @ 0xf15e0 (10 entries)
bios0: vendor SeaBIOS version "1.11.0p1-OpenBSD-vmm" date 01/01/2011
bios0: OpenBSD VMM
acpi at bios0 not configured
cpu0 at mainbus0: (uniprocessor)
cpu0: Intel(R) Xeon(R) CPU E31260L @ 2.40GHz, 2401.44 MHz, 06-2a-07
cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,CX8,SEP,PGE,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SSE3,PCLMUL,SSSE3,CX16,SSE4.1,SSE4.2,POPCNT,AES,XSAVE,AVX,HV,NXE,LONG,LAHF,ITSC,MD_CLEAR,MELTDOWN
cpu0: 256KB 64b/line 8-way L2 cache
cpu0: smt 0, core 0, package 0
cpu0: using VERW MDS workaround
pvbus0 at mainbus0: OpenBSD
pvclock0 at pvbus0
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "OpenBSD VMM Host" rev 0x00
virtio0 at pci0 dev 1 function 0 "Qumranet Virtio RNG" rev 0x00
viornd0 at virtio0
virtio0: irq 3
virtio1 at pci0 dev 2 function 0 "Qumranet Virtio Network" rev 0x00
vio0 at virtio1: address fe:e1:bb:02:6b:bf
virtio1: irq 5
virtio2 at pci0 dev 3 function 0 "Qumranet Virtio Storage" rev 0x00
vioblk0 at virtio2
scsibus1 at vioblk0: 2 targets
sd0 at scsibus1 targ 0 lun 0: <VirtIO, Block Device, > SCSI3 0/direct fixed
sd0: 6144MB, 512 bytes/sector, 12582912 sectors
virtio2: irq 6
virtio3 at pci0 dev 4 function 0 "Qumranet Virtio Storage" rev 0x00
vioblk1 at virtio3
scsibus2 at vioblk1: 2 targets
sd1 at scsibus2 targ 0 lun 0: <VirtIO, Block Device, > SCSI3 0/direct fixed
sd1: 51200MB, 512 bytes/sector, 104857600 sectors
virtio3: irq 7
virtio4 at pci0 dev 5 function 0 "OpenBSD VMM Control" rev 0x00
vmmci0 at virtio4
virtio4: irq 9
isa0 at mainbus0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns8250, no fifo
com0: console
vscsi0 at root
scsibus3 at vscsi0: 256 targets
softraid0 at root
scsibus4 at softraid0: 256 targets
root on sd0a (112d38627522c4b4.a) swap on sd0b dump on sd0b
----------------------------------------------------------------------

--- testvm dmesg -----------------------------------------------------
OpenBSD 6.5-current (GENERIC) #6: Tue Jun  4 14:57:43 MDT 2019
    [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC
real mem = 2130698240 (2031MB)
avail mem = 2056175616 (1960MB)
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.4 @ 0xf15e0 (10 entries)
bios0: vendor SeaBIOS version "1.11.0p1-OpenBSD-vmm" date 01/01/2011
bios0: OpenBSD VMM
acpi at bios0 not configured
cpu0 at mainbus0: (uniprocessor)
cpu0: Intel(R) Xeon(R) CPU E31260L @ 2.40GHz, 2401.28 MHz, 06-2a-07
cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,CX8,SEP,PGE,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SSE3,PCLMUL,SSSE3,CX16,SSE4.1,SSE4.2,POPCNT,AES,XSAVE,AVX,HV,NXE,LONG,LAHF,ITSC,MD_CLEAR,MELTDOWN
cpu0: 256KB 64b/line 8-way L2 cache
cpu0: smt 0, core 0, package 0
cpu0: using VERW MDS workaround
pvbus0 at mainbus0: OpenBSD
pvclock0 at pvbus0
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "OpenBSD VMM Host" rev 0x00
virtio0 at pci0 dev 1 function 0 "Qumranet Virtio RNG" rev 0x00
viornd0 at virtio0
virtio0: irq 3
virtio1 at pci0 dev 2 function 0 "Qumranet Virtio Network" rev 0x00
vio0 at virtio1: address fe:e1:bb:02:6b:be
virtio1: irq 5
virtio2 at pci0 dev 3 function 0 "Qumranet Virtio Storage" rev 0x00
vioblk0 at virtio2
scsibus1 at vioblk0: 2 targets
sd0 at scsibus1 targ 0 lun 0: <VirtIO, Block Device, > SCSI3 0/direct fixed
sd0: 6144MB, 512 bytes/sector, 12582912 sectors
virtio2: irq 6
virtio3 at pci0 dev 4 function 0 "OpenBSD VMM Control" rev 0x00
vmmci0 at virtio3
virtio3: irq 7
isa0 at mainbus0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns8250, no fifo
com0: console
vscsi0 at root
scsibus2 at vscsi0: 256 targets
softraid0 at root
scsibus3 at softraid0: 256 targets
root on sd0a (10dc99cfac23b3f1.a) swap on sd0b dump on sd0b
----------------------------------------------------------------------

--
>++++++++[<++++++++++>-]<+++++++.>+++[<------>-]<.>+++[<+
+++++++++++>-]<.>++[<------------>-]<+.--------------.[-]
                 http://www.weirdnet.nl/                 

Reply | Threaded
Open this post in threaded view
|

Re: vmd eating lots of memory

Mike Larkin-2
On Sat, Jul 20, 2019 at 04:23:27PM +0200, Paul de Weerd wrote:

> Hi all,
>
> I'm running two vmd(8) VMs:
>
> [weerd@despair] $ cat /etc/vm.conf
> vm "undeadly" {
>         owner root
>         memory 2G
>         disk /home/vmm/undeadly.dsk
>         disk /storage/vmm/undeadly.dsk
>         interface switch "vmmswitch" lladdr "fe:e1:bb:02:6b:bf"
> }
>
> vm "testvm" {
>         owner root
>         memory 2G
>         disk /home/vmm/testvm.dsk
>         interface switch "vmmswitch" lladdr "fe:e1:bb:02:6b:be"
> }
>
> switch "vmmswitch" {
>         enable
>         interface bridge0
> }
>
> Both machines have been up since boot of the host machine:
>
> [weerd@despair] $ vmctl status
>    ID   PID VCPUS  MAXMEM  CURMEM     TTY        OWNER    STATE NAME
>     2 23133     1    2.0G    1.1G   ttyp1         root  running testvm
>     1 42094     1    2.0G    1.5G   ttyp0         root  running undeadly
> [weerd@despair] $ uptime
>  4:11PM  up 43 days, 57 mins, 12 users, load averages: 0.02, 0.07, 0.07
> [weerd@despair] $ ssh undeadly uptime
>  4:11PM  up 43 days, 57 mins, 2 users, load averages: 0.28, 0.12, 0.10
> [weerd@despair] $ ssh testvm uptime  
>  4:11PM  up 43 days, 57 mins, 0 users, load averages: 0.00, 0.00, 0.00
>
> However, the undeadly VM consumes signficantly more memory that the
> other VM:
>
> [weerd@despair] $ ps wwwaux | grep -e vm[d] -e [P]ID
> USER       PID %CPU %MEM   VSZ   RSS TT  STAT  STARTED       TIME COMMAND
> _vmd     42094  3.7 57.9 28769236 19398792 ??  Ip     7Jun19  4741:37.66 vmd: undeadly (vmd)
> _vmd      4567  0.0  0.0  1384  1384 ??  Isp    7Jun19    0:00.01 vmd: vmm (vmd)
> root     84392  0.0  0.0  1564  1436 ??  Isp    7Jun19    0:00.02 /usr/sbin/vmd
> _vmd     55269  0.0  0.0  1304  1468 ??  Isp    7Jun19    0:00.01 vmd: control (vmd)
> root     51181  0.0  0.0  1236   928 ??  Is     7Jun19    0:00.01 vmd: priv (vmd)
> _vmd     23133  0.0  0.2 2100100 70272 ??  Ip     7Jun19  1886:16.18 vmd: testvm (vmd)
>
> All systems run snaps from around the same time:
>
> [weerd@despair] $ sysctl kern.version
> kern.version=OpenBSD 6.5-current (GENERIC.MP) #6: Tue Jun  4 15:05:10 MDT 2019
>     [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
>
> [weerd@despair] $ ssh undeadly sysctl kern.version
> kern.version=OpenBSD 6.5-current (GENERIC) #7: Fri Jun  7 00:45:34 MDT 2019
>     [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC
>
> [weerd@despair] $ ssh testvm sysctl kern.version
> kern.version=OpenBSD 6.5-current (GENERIC) #6: Tue Jun  4 14:57:43 MDT 2019
>     [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC
>
> So it looks like there's a memory leak, either only for busy VMs or it
> shows more for busier VMs.  Has anyone seen anything similar on their
> vmm VMs?
>
> I'll reboot the undeadly VM for now, as other processes are getting
> killed for out of memory reasons (bgpd, in my case; see end of below
> dmesg of host machine 'despair').
>
> Paul
>

Did this just start happening? Nothing relevant has changed in vmd(8) recently
that would cause this, from what I remember.

-ml

> --- despair dmesg ----------------------------------------------------
> OpenBSD 6.5-current (GENERIC.MP) #6: Tue Jun  4 15:05:10 MDT 2019
>     [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
> real mem = 34332733440 (32742MB)
> avail mem = 33282105344 (31740MB)
> mpath0 at root
> scsibus0 at mpath0: 256 targets
> mainbus0 at root
> bios0 at mainbus0: SMBIOS rev. 2.7 @ 0xe66d0 (57 entries)
> bios0: vendor Dell Inc. version "1.2.3" date 07/21/2011
> bios0: Dell Inc. PowerEdge R210 II
> acpi0 at bios0: rev 2
> acpi0: sleep states S0 S4 S5
> acpi0: tables DSDT FACP SPMI ASF! HPET APIC MCFG BOOT SSDT ASPT SSDT SSDT SPCR DMAR HEST ERST BERT EINJ
> acpi0: wakeup devices P0P1(S4) GLAN(S0) EHC1(S4) EHC2(S4) PXSX(S4) RP01(S5) PXSX(S4) RP02(S5) PXSX(S4) RP03(S5) PXSX(S4) RP04(S5) PXSX(S4) RP05(S5) PXSX(S4) RP06(S5) [...]
> acpitimer0 at acpi0: 3579545 Hz, 24 bits
> acpihpet0 at acpi0: 14318179 Hz
> acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
> cpu0 at mainbus0: apid 0 (boot processor)
> cpu0: Intel(R) Xeon(R) CPU E31260L @ 2.40GHz, 2400.39 MHz, 06-2a-07
> cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
> cpu0: 256KB 64b/line 8-way L2 cache
> cpu0: smt 0, core 0, package 0
> mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges
> cpu0: apic clock running at 100MHz
> cpu0: mwait min=64, max=64, C-substates=0.2.1.1, IBE
> cpu1 at mainbus0: apid 1 (application processor)
> cpu1: Intel(R) Xeon(R) CPU E31260L @ 2.40GHz, 2400.02 MHz, 06-2a-07
> cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
> cpu1: 256KB 64b/line 8-way L2 cache
> cpu1: smt 1, core 0, package 0
> cpu2 at mainbus0: apid 2 (application processor)
> cpu2: Intel(R) Xeon(R) CPU E31260L @ 2.40GHz, 2400.02 MHz, 06-2a-07
> cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
> cpu2: 256KB 64b/line 8-way L2 cache
> cpu2: smt 0, core 1, package 0
> cpu3 at mainbus0: apid 3 (application processor)
> cpu3: Intel(R) Xeon(R) CPU E31260L @ 2.40GHz, 2400.02 MHz, 06-2a-07
> cpu3: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
> cpu3: 256KB 64b/line 8-way L2 cache
> cpu3: smt 1, core 1, package 0
> cpu4 at mainbus0: apid 4 (application processor)
> cpu4: Intel(R) Xeon(R) CPU E31260L @ 2.40GHz, 2400.02 MHz, 06-2a-07
> cpu4: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
> cpu4: 256KB 64b/line 8-way L2 cache
> cpu4: smt 0, core 2, package 0
> cpu5 at mainbus0: apid 5 (application processor)
> cpu5: Intel(R) Xeon(R) CPU E31260L @ 2.40GHz, 2400.02 MHz, 06-2a-07
> cpu5: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
> cpu5: 256KB 64b/line 8-way L2 cache
> cpu5: smt 1, core 2, package 0
> cpu6 at mainbus0: apid 6 (application processor)
> cpu6: Intel(R) Xeon(R) CPU E31260L @ 2.40GHz, 2400.02 MHz, 06-2a-07
> cpu6: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
> cpu6: 256KB 64b/line 8-way L2 cache
> cpu6: smt 0, core 3, package 0
> cpu7 at mainbus0: apid 7 (application processor)
> cpu7: Intel(R) Xeon(R) CPU E31260L @ 2.40GHz, 2400.02 MHz, 06-2a-07
> cpu7: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
> cpu7: 256KB 64b/line 8-way L2 cache
> cpu7: smt 1, core 3, package 0
> ioapic0 at mainbus0: apid 0 pa 0xfec00000, version 20, 24 pins
> acpimcfg0 at acpi0
> acpimcfg0: addr 0xe0000000, bus 0-255
> acpiprt0 at acpi0: bus 0 (PCI0)
> acpiprt1 at acpi0: bus 2 (P0P1)
> acpiprt2 at acpi0: bus 1 (RP01)
> acpiprt3 at acpi0: bus -1 (RP02)
> acpiprt4 at acpi0: bus -1 (RP03)
> acpiprt5 at acpi0: bus -1 (RP04)
> acpiprt6 at acpi0: bus -1 (RP05)
> acpiprt7 at acpi0: bus -1 (RP06)
> acpiprt8 at acpi0: bus -1 (RP07)
> acpiprt9 at acpi0: bus -1 (RP08)
> acpiprt10 at acpi0: bus -1 (PEG0)
> acpiprt11 at acpi0: bus -1 (PEG1)
> acpiprt12 at acpi0: bus -1 (PEG2)
> acpiprt13 at acpi0: bus -1 (PEG3)
> acpicpu0 at acpi0: C3(350@104 mwait.1@0x20), C2(500@80 mwait.1@0x10), C1(1000@1 mwait.1), PSS
> acpicpu1 at acpi0: C3(350@104 mwait.1@0x20), C2(500@80 mwait.1@0x10), C1(1000@1 mwait.1), PSS
> acpicpu2 at acpi0: C3(350@104 mwait.1@0x20), C2(500@80 mwait.1@0x10), C1(1000@1 mwait.1), PSS
> acpicpu3 at acpi0: C3(350@104 mwait.1@0x20), C2(500@80 mwait.1@0x10), C1(1000@1 mwait.1), PSS
> acpicpu4 at acpi0: C3(350@104 mwait.1@0x20), C2(500@80 mwait.1@0x10), C1(1000@1 mwait.1), PSS
> acpicpu5 at acpi0: C3(350@104 mwait.1@0x20), C2(500@80 mwait.1@0x10), C1(1000@1 mwait.1), PSS
> acpicpu6 at acpi0: C3(350@104 mwait.1@0x20), C2(500@80 mwait.1@0x10), C1(1000@1 mwait.1), PSS
> acpicpu7 at acpi0: C3(350@104 mwait.1@0x20), C2(500@80 mwait.1@0x10), C1(1000@1 mwait.1), PSS
> acpipwrres0 at acpi0: FN00, resource for FAN0
> acpipwrres1 at acpi0: FN01, resource for FAN1
> acpipwrres2 at acpi0: FN02, resource for FAN2
> acpipwrres3 at acpi0: FN03, resource for FAN3
> acpipwrres4 at acpi0: FN04, resource for FAN4
> acpitz0 at acpi0: critical temperature is 100 degC
> acpipci0 at acpi0 PCI0: 0x00000010 0x00000011 0x00000000
> "IPI0001" at acpi0 not configured
> acpicmos0 at acpi0
> "PNP0A05" at acpi0 not configured
> "PNP0A05" at acpi0 not configured
> "PNP0C0B" at acpi0 not configured
> "PNP0C0B" at acpi0 not configured
> "PNP0C0B" at acpi0 not configured
> "PNP0C0B" at acpi0 not configured
> "PNP0C0B" at acpi0 not configured
> ipmi at mainbus0 not configured
> cpu0: using VERW MDS workaround (except on vmm entry)
> cpu0: Enhanced SpeedStep 2400 MHz: speeds: 2401, 2400, 2300, 2200, 2100, 2000, 1900, 1800, 1700, 1600 MHz
> pci0 at mainbus0 bus 0
> pchb0 at pci0 dev 0 function 0 "Intel Xeon E3-1200 Host" rev 0x09
> ehci0 at pci0 dev 26 function 0 "Intel 6 Series USB" rev 0x04: apic 0 int 16
> usb0 at ehci0: USB revision 2.0
> uhub0 at usb0 configuration 1 interface 0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
> ppb0 at pci0 dev 28 function 0 "Intel 6 Series PCIE" rev 0xb4: msi
> pci1 at ppb0 bus 1
> bnx0 at pci1 dev 0 function 0 "Broadcom BCM5716" rev 0x20: apic 0 int 16
> bnx1 at pci1 dev 0 function 1 "Broadcom BCM5716" rev 0x20: apic 0 int 17
> ehci1 at pci0 dev 29 function 0 "Intel 6 Series USB" rev 0x04: apic 0 int 23
> usb1 at ehci1: USB revision 2.0
> uhub1 at usb1 configuration 1 interface 0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
> ppb1 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0xa4
> pci2 at ppb1 bus 2
> 2:3:0: mem address conflict 0xffff0000/0x10000
> vga1 at pci2 dev 3 function 0 "Matrox MGA G200eW" rev 0x0a
> wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
> wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
> pcib0 at pci0 dev 31 function 0 "Intel C202 LPC" rev 0x04
> ahci0 at pci0 dev 31 function 2 "Intel 6 Series AHCI" rev 0x04: msi, AHCI 1.3
> ahci0: port 0: 3.0Gb/s
> ahci0: port 1: 3.0Gb/s
> ahci0: port 4: 1.5Gb/s
> scsibus1 at ahci0: 32 targets
> sd0 at scsibus1 targ 0 lun 0: <ATA, INTEL SSDSC2BW48, DC32> SCSI3 0/direct fixed naa.55cd2e414c807f33
> sd0: 457862MB, 512 bytes/sector, 937703088 sectors, thin
> sd1 at scsibus1 targ 1 lun 0: <ATA, WDC WD60EFRX-68L, 82.0> SCSI3 0/direct fixed naa.50014ee20dd29e68
> sd1: 5723166MB, 512 bytes/sector, 11721045168 sectors
> cd0 at scsibus1 targ 4 lun 0: <TSSTcorp, DVD+-RW TS-L633J, D150> ATAPI 5/cdrom removable
> ichiic0 at pci0 dev 31 function 3 "Intel 6 Series SMBus" rev 0x04: apic 0 int 19
> iic0 at ichiic0
> sdtemp0 at iic0 addr 0x18: mcp98243
> sdtemp1 at iic0 addr 0x19: mcp98243
> sdtemp2 at iic0 addr 0x1a: mcp98243
> sdtemp3 at iic0 addr 0x1b: mcp98243
> spdmem0 at iic0 addr 0x50: 8GB DDR3 SDRAM ECC PC3-12800 with thermal sensor
> spdmem1 at iic0 addr 0x51: 8GB DDR3 SDRAM ECC PC3-12800 with thermal sensor
> spdmem2 at iic0 addr 0x52: 8GB DDR3 SDRAM ECC PC3-12800 with thermal sensor
> spdmem3 at iic0 addr 0x53: 8GB DDR3 SDRAM ECC PC3-12800 with thermal sensor
> isa0 at pcib0
> isadma0 at isa0
> com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
> com0: console
> com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
> pckbc0 at isa0 port 0x60/5 irq 1 irq 12
> pcppi0 at isa0 port 0x61
> spkr0 at pcppi0
> vmm0 at mainbus0: VMX/EPT
> uhub2 at uhub0 port 1 configuration 1 interface 0 "Intel Rate Matching Hub" rev 2.00/0.00 addr 2
> uhub3 at uhub1 port 1 configuration 1 interface 0 "Intel Rate Matching Hub" rev 2.00/0.00 addr 2
> uhidev0 at uhub3 port 1 configuration 1 interface 0 "RDing TEMPERHUM1V1.2" rev 2.00/0.01 addr 3
> uhidev0: iclass 3/1, 1 report id
> ukbd0 at uhidev0 reportid 1: 8 variable keys, 5 key codes
> wskbd0 at ukbd0: console keyboard, using wsdisplay0
> uhidev1 at uhub3 port 1 configuration 1 interface 1 "RDing TEMPERHUM1V1.2" rev 2.00/0.01 addr 3
> uhidev1: iclass 3/1
> ugold0 at uhidev1
> uhub4 at uhub3 port 5 configuration 1 interface 0 "Standard Microsystems product 0x2514" rev 2.00/0.00 addr 4
> vscsi0 at root
> scsibus2 at vscsi0: 256 targets
> softraid0 at root
> scsibus3 at softraid0: 256 targets
> sd2 at scsibus3 targ 1 lun 0: <OPENBSD, SR CRYPTO, 006> SCSI2 0/direct fixed
> sd2: 409602MB, 512 bytes/sector, 838865577 sectors
> root on sd2a (5a5905f3a7206f17.a) swap on sd2b dump on sd2b
> bnx0: address bc:30:5b:e8:d6:48
> brgphy0 at bnx0 phy 1: BCM5709 10/100/1000baseT PHY, rev. 8
> bnx1: address bc:30:5b:e8:d6:49
> brgphy1 at bnx1 phy 1: BCM5709 10/100/1000baseT PHY, rev. 8
> ugold0: 2 sensors type si7006 (temperature and humidity)
> sd3 at scsibus3 targ 2 lun 0: <OPENBSD, SR CRYPTO, 006> SCSI2 0/direct fixed
> sd3: 5231654MB, 512 bytes/sector, 10714427745 sectors
> UVM: pid 36202 (bgpd), uid 75 killed: out of swap
> ----------------------------------------------------------------------
>
> --- undeadly dmesg ---------------------------------------------------
> OpenBSD 6.5-current (GENERIC) #7: Fri Jun  7 00:45:34 MDT 2019
>     [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC
> real mem = 2130690048 (2031MB)
> avail mem = 2056167424 (1960MB)
> mpath0 at root
> scsibus0 at mpath0: 256 targets
> mainbus0 at root
> bios0 at mainbus0: SMBIOS rev. 2.4 @ 0xf15e0 (10 entries)
> bios0: vendor SeaBIOS version "1.11.0p1-OpenBSD-vmm" date 01/01/2011
> bios0: OpenBSD VMM
> acpi at bios0 not configured
> cpu0 at mainbus0: (uniprocessor)
> cpu0: Intel(R) Xeon(R) CPU E31260L @ 2.40GHz, 2401.44 MHz, 06-2a-07
> cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,CX8,SEP,PGE,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SSE3,PCLMUL,SSSE3,CX16,SSE4.1,SSE4.2,POPCNT,AES,XSAVE,AVX,HV,NXE,LONG,LAHF,ITSC,MD_CLEAR,MELTDOWN
> cpu0: 256KB 64b/line 8-way L2 cache
> cpu0: smt 0, core 0, package 0
> cpu0: using VERW MDS workaround
> pvbus0 at mainbus0: OpenBSD
> pvclock0 at pvbus0
> pci0 at mainbus0 bus 0
> pchb0 at pci0 dev 0 function 0 "OpenBSD VMM Host" rev 0x00
> virtio0 at pci0 dev 1 function 0 "Qumranet Virtio RNG" rev 0x00
> viornd0 at virtio0
> virtio0: irq 3
> virtio1 at pci0 dev 2 function 0 "Qumranet Virtio Network" rev 0x00
> vio0 at virtio1: address fe:e1:bb:02:6b:bf
> virtio1: irq 5
> virtio2 at pci0 dev 3 function 0 "Qumranet Virtio Storage" rev 0x00
> vioblk0 at virtio2
> scsibus1 at vioblk0: 2 targets
> sd0 at scsibus1 targ 0 lun 0: <VirtIO, Block Device, > SCSI3 0/direct fixed
> sd0: 6144MB, 512 bytes/sector, 12582912 sectors
> virtio2: irq 6
> virtio3 at pci0 dev 4 function 0 "Qumranet Virtio Storage" rev 0x00
> vioblk1 at virtio3
> scsibus2 at vioblk1: 2 targets
> sd1 at scsibus2 targ 0 lun 0: <VirtIO, Block Device, > SCSI3 0/direct fixed
> sd1: 51200MB, 512 bytes/sector, 104857600 sectors
> virtio3: irq 7
> virtio4 at pci0 dev 5 function 0 "OpenBSD VMM Control" rev 0x00
> vmmci0 at virtio4
> virtio4: irq 9
> isa0 at mainbus0
> isadma0 at isa0
> com0 at isa0 port 0x3f8/8 irq 4: ns8250, no fifo
> com0: console
> vscsi0 at root
> scsibus3 at vscsi0: 256 targets
> softraid0 at root
> scsibus4 at softraid0: 256 targets
> root on sd0a (112d38627522c4b4.a) swap on sd0b dump on sd0b
> ----------------------------------------------------------------------
>
> --- testvm dmesg -----------------------------------------------------
> OpenBSD 6.5-current (GENERIC) #6: Tue Jun  4 14:57:43 MDT 2019
>     [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC
> real mem = 2130698240 (2031MB)
> avail mem = 2056175616 (1960MB)
> mpath0 at root
> scsibus0 at mpath0: 256 targets
> mainbus0 at root
> bios0 at mainbus0: SMBIOS rev. 2.4 @ 0xf15e0 (10 entries)
> bios0: vendor SeaBIOS version "1.11.0p1-OpenBSD-vmm" date 01/01/2011
> bios0: OpenBSD VMM
> acpi at bios0 not configured
> cpu0 at mainbus0: (uniprocessor)
> cpu0: Intel(R) Xeon(R) CPU E31260L @ 2.40GHz, 2401.28 MHz, 06-2a-07
> cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,CX8,SEP,PGE,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SSE3,PCLMUL,SSSE3,CX16,SSE4.1,SSE4.2,POPCNT,AES,XSAVE,AVX,HV,NXE,LONG,LAHF,ITSC,MD_CLEAR,MELTDOWN
> cpu0: 256KB 64b/line 8-way L2 cache
> cpu0: smt 0, core 0, package 0
> cpu0: using VERW MDS workaround
> pvbus0 at mainbus0: OpenBSD
> pvclock0 at pvbus0
> pci0 at mainbus0 bus 0
> pchb0 at pci0 dev 0 function 0 "OpenBSD VMM Host" rev 0x00
> virtio0 at pci0 dev 1 function 0 "Qumranet Virtio RNG" rev 0x00
> viornd0 at virtio0
> virtio0: irq 3
> virtio1 at pci0 dev 2 function 0 "Qumranet Virtio Network" rev 0x00
> vio0 at virtio1: address fe:e1:bb:02:6b:be
> virtio1: irq 5
> virtio2 at pci0 dev 3 function 0 "Qumranet Virtio Storage" rev 0x00
> vioblk0 at virtio2
> scsibus1 at vioblk0: 2 targets
> sd0 at scsibus1 targ 0 lun 0: <VirtIO, Block Device, > SCSI3 0/direct fixed
> sd0: 6144MB, 512 bytes/sector, 12582912 sectors
> virtio2: irq 6
> virtio3 at pci0 dev 4 function 0 "OpenBSD VMM Control" rev 0x00
> vmmci0 at virtio3
> virtio3: irq 7
> isa0 at mainbus0
> isadma0 at isa0
> com0 at isa0 port 0x3f8/8 irq 4: ns8250, no fifo
> com0: console
> vscsi0 at root
> scsibus2 at vscsi0: 256 targets
> softraid0 at root
> scsibus3 at softraid0: 256 targets
> root on sd0a (10dc99cfac23b3f1.a) swap on sd0b dump on sd0b
> ----------------------------------------------------------------------
>
> --
> >++++++++[<++++++++++>-]<+++++++.>+++[<------>-]<.>+++[<+
> +++++++++++>-]<.>++[<------------>-]<+.--------------.[-]
>                  http://www.weirdnet.nl/                 
>

Reply | Threaded
Open this post in threaded view
|

Re: vmd eating lots of memory

Paul de Weerd
Hi Mike,

On Sat, Jul 20, 2019 at 10:23:02AM -0700, Mike Larkin wrote:
| Did this just start happening? Nothing relevant has changed in vmd(8) recently
| that would cause this, from what I remember.

Prior to this kernel version, I was running

OpenBSD 6.5 (GENERIC.MP) #847: Tue Apr  9 09:12:46 MDT 2019

on both VMs and host.  If the problem was there too, it didn't affect
anything else on the system until I upgraded.

Now that I've found this, I started measuring a bit more.  It looks
like the vmd for undeadly consumes about 8 to 10 MB per hour
(approximately 2MB per 15 minutes, my sample interval).  It varies
quite a bit, and there's a bit spike around when /etc/daily runs of
340 MB and 74 MB.

I'm attaching my samples so far, they're epoch + the VSZ column from
ps for the vmd process per line.

Is there anything else I can collect that can help debug this?

Thanks,

Paul

--
>++++++++[<++++++++++>-]<+++++++.>+++[<------>-]<.>+++[<+
+++++++++++>-]<.>++[<------------>-]<+.--------------.[-]
                 http://www.weirdnet.nl/                 

undeadly.memory (1K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: vmd eating lots of memory

Paul de Weerd
On Sun, Jul 21, 2019 at 10:46:06AM +0200, Paul de Weerd wrote:
| on both VMs and host.  If the problem was there too, it didn't affect
| anything else on the system until I upgraded.

Forgot to make explicit: the other vm remains at a constant memory
footprint; also while running /etc/daily

Paul

--
>++++++++[<++++++++++>-]<+++++++.>+++[<------>-]<.>+++[<+
+++++++++++>-]<.>++[<------------>-]<+.--------------.[-]
                 http://www.weirdnet.nl/                 

Reply | Threaded
Open this post in threaded view
|

Re: vmd eating lots of memory

Mike Larkin-2
In reply to this post by Paul de Weerd
On Sun, Jul 21, 2019 at 10:46:06AM +0200, Paul de Weerd wrote:

> Hi Mike,
>
> On Sat, Jul 20, 2019 at 10:23:02AM -0700, Mike Larkin wrote:
> | Did this just start happening? Nothing relevant has changed in vmd(8) recently
> | that would cause this, from what I remember.
>
> Prior to this kernel version, I was running
>
> OpenBSD 6.5 (GENERIC.MP) #847: Tue Apr  9 09:12:46 MDT 2019
>
> on both VMs and host.  If the problem was there too, it didn't affect
> anything else on the system until I upgraded.
>
> Now that I've found this, I started measuring a bit more.  It looks
> like the vmd for undeadly consumes about 8 to 10 MB per hour
> (approximately 2MB per 15 minutes, my sample interval).  It varies
> quite a bit, and there's a bit spike around when /etc/daily runs of
> 340 MB and 74 MB.
>
> I'm attaching my samples so far, they're epoch + the VSZ column from
> ps for the vmd process per line.
>
> Is there anything else I can collect that can help debug this?
>

Sorry, I nuked the original email that had the vm.conf, can you send that
again?

> Thanks,
>
> Paul
>
> --
> >++++++++[<++++++++++>-]<+++++++.>+++[<------>-]<.>+++[<+
> +++++++++++>-]<.>++[<------------>-]<+.--------------.[-]
>                  http://www.weirdnet.nl/                 

> 1563659689 2207536
> 1563659712 2207568
> 1563660312 2210584
> 1563660912 2213136
> 1563661512 2215732
> 1563662112 2218504
> 1563662713 2220980
> 1563663313 2223704
> 1563663913 2226748
> 1563664513 2228976
> 1563665113 2231824
> 1563665713 2572920
> 1563666313 2647712
> 1563666913 2650232
> 1563667514 2653052
> 1563668114 2655504
> 1563668714 2658308
> 1563669314 2661260
> 1563669914 2663836
> 1563670514 2666324
> 1563671114 2668944
> 1563671714 2671452
> 1563672314 2673780
> 1563672914 2676392
> 1563673514 2678700
> 1563674114 2680980
> 1563674714 2683788
> 1563675314 2685924
> 1563675914 2688648
> 1563676514 2691404
> 1563677114 2693676
> 1563677715 2695844
> 1563678315 2697920
> 1563678915 2699868
> 1563679515 2701944
> 1563680115 2703868
> 1563680715 2705668
> 1563681315 2707488
> 1563681915 2709712
> 1563682515 2711472
> 1563683115 2713196
> 1563683715 2715432
> 1563684315 2717280
> 1563684915 2719156
> 1563685515 2721528
> 1563686115 2723672
> 1563686715 2725828
> 1563687315 2727812
> 1563687915 2729848
> 1563688515 2731488
> 1563689116 2735600
> 1563689716 2737380
> 1563690316 2739648
> 1563690916 2742280
> 1563691516 2744240
> 1563692116 2746120
> 1563692716 2748192
> 1563693316 2750148
> 1563693916 2752156
> 1563694516 2754424
> 1563695116 2757576
> 1563695716 2760144
> 1563696316 2762860
> 1563696916 2764992
> 1563697516 2767408
> 1563698116 2769756
> 1563698716 2771680

Reply | Threaded
Open this post in threaded view
|

Re: vmd eating lots of memory

Paul de Weerd
In reply to this post by Paul de Weerd
A little more follow-up on this vmd-memory-leak issue.

Comparing the two VMs I have running, I started to stress parts where
these two hosts differ.  The testvm hardly does any traffic, while the
undeadly vm sees quite a few visitors on a daily basis, so networking
may be part of the leak.

Running tcpbench against this machine (averaging at ~250Mbit/s)
results in vmd growing by about 80MB to 100MB per minute.  Running
tcpbench against the testvm has similar results.

The undeadly VM also has a second disk configured (on slower storage),
but putting load on that didn't significantly change the memory
consumption (above the 'expected' growth that I've been seeing).

The growth during daily(8) runs still confuses me, as that doesn't do
anything network-related...

Paul

On Sat, Jul 20, 2019 at 04:23:27PM +0200, Paul de Weerd wrote:
| Hi all,
|
| I'm running two vmd(8) VMs:
|
| [weerd@despair] $ cat /etc/vm.conf
| vm "undeadly" {
|         owner root
|         memory 2G
|         disk /home/vmm/undeadly.dsk
|         disk /storage/vmm/undeadly.dsk
|         interface switch "vmmswitch" lladdr "fe:e1:bb:02:6b:bf"
| }
|
| vm "testvm" {
|         owner root
|         memory 2G
|         disk /home/vmm/testvm.dsk
|         interface switch "vmmswitch" lladdr "fe:e1:bb:02:6b:be"
| }
|
| switch "vmmswitch" {
|         enable
|         interface bridge0
| }
|
| Both machines have been up since boot of the host machine:
|
| [weerd@despair] $ vmctl status
|    ID   PID VCPUS  MAXMEM  CURMEM     TTY        OWNER    STATE NAME
|     2 23133     1    2.0G    1.1G   ttyp1         root  running testvm
|     1 42094     1    2.0G    1.5G   ttyp0         root  running undeadly
| [weerd@despair] $ uptime
|  4:11PM  up 43 days, 57 mins, 12 users, load averages: 0.02, 0.07, 0.07
| [weerd@despair] $ ssh undeadly uptime
|  4:11PM  up 43 days, 57 mins, 2 users, load averages: 0.28, 0.12, 0.10
| [weerd@despair] $ ssh testvm uptime  
|  4:11PM  up 43 days, 57 mins, 0 users, load averages: 0.00, 0.00, 0.00
|
| However, the undeadly VM consumes signficantly more memory that the
| other VM:
|
| [weerd@despair] $ ps wwwaux | grep -e vm[d] -e [P]ID
| USER       PID %CPU %MEM   VSZ   RSS TT  STAT  STARTED       TIME COMMAND
| _vmd     42094  3.7 57.9 28769236 19398792 ??  Ip     7Jun19  4741:37.66 vmd: undeadly (vmd)
| _vmd      4567  0.0  0.0  1384  1384 ??  Isp    7Jun19    0:00.01 vmd: vmm (vmd)
| root     84392  0.0  0.0  1564  1436 ??  Isp    7Jun19    0:00.02 /usr/sbin/vmd
| _vmd     55269  0.0  0.0  1304  1468 ??  Isp    7Jun19    0:00.01 vmd: control (vmd)
| root     51181  0.0  0.0  1236   928 ??  Is     7Jun19    0:00.01 vmd: priv (vmd)
| _vmd     23133  0.0  0.2 2100100 70272 ??  Ip     7Jun19  1886:16.18 vmd: testvm (vmd)
|
| All systems run snaps from around the same time:
|
| [weerd@despair] $ sysctl kern.version
| kern.version=OpenBSD 6.5-current (GENERIC.MP) #6: Tue Jun  4 15:05:10 MDT 2019
|     [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
|
| [weerd@despair] $ ssh undeadly sysctl kern.version
| kern.version=OpenBSD 6.5-current (GENERIC) #7: Fri Jun  7 00:45:34 MDT 2019
|     [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC
|
| [weerd@despair] $ ssh testvm sysctl kern.version
| kern.version=OpenBSD 6.5-current (GENERIC) #6: Tue Jun  4 14:57:43 MDT 2019
|     [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC
|
| So it looks like there's a memory leak, either only for busy VMs or it
| shows more for busier VMs.  Has anyone seen anything similar on their
| vmm VMs?
|
| I'll reboot the undeadly VM for now, as other processes are getting
| killed for out of memory reasons (bgpd, in my case; see end of below
| dmesg of host machine 'despair').
|
| Paul
|
| --- despair dmesg ----------------------------------------------------
| OpenBSD 6.5-current (GENERIC.MP) #6: Tue Jun  4 15:05:10 MDT 2019
|     [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
| real mem = 34332733440 (32742MB)
| avail mem = 33282105344 (31740MB)
| mpath0 at root
| scsibus0 at mpath0: 256 targets
| mainbus0 at root
| bios0 at mainbus0: SMBIOS rev. 2.7 @ 0xe66d0 (57 entries)
| bios0: vendor Dell Inc. version "1.2.3" date 07/21/2011
| bios0: Dell Inc. PowerEdge R210 II
| acpi0 at bios0: rev 2
| acpi0: sleep states S0 S4 S5
| acpi0: tables DSDT FACP SPMI ASF! HPET APIC MCFG BOOT SSDT ASPT SSDT SSDT SPCR DMAR HEST ERST BERT EINJ
| acpi0: wakeup devices P0P1(S4) GLAN(S0) EHC1(S4) EHC2(S4) PXSX(S4) RP01(S5) PXSX(S4) RP02(S5) PXSX(S4) RP03(S5) PXSX(S4) RP04(S5) PXSX(S4) RP05(S5) PXSX(S4) RP06(S5) [...]
| acpitimer0 at acpi0: 3579545 Hz, 24 bits
| acpihpet0 at acpi0: 14318179 Hz
| acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
| cpu0 at mainbus0: apid 0 (boot processor)
| cpu0: Intel(R) Xeon(R) CPU E31260L @ 2.40GHz, 2400.39 MHz, 06-2a-07
| cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
| cpu0: 256KB 64b/line 8-way L2 cache
| cpu0: smt 0, core 0, package 0
| mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges
| cpu0: apic clock running at 100MHz
| cpu0: mwait min=64, max=64, C-substates=0.2.1.1, IBE
| cpu1 at mainbus0: apid 1 (application processor)
| cpu1: Intel(R) Xeon(R) CPU E31260L @ 2.40GHz, 2400.02 MHz, 06-2a-07
| cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
| cpu1: 256KB 64b/line 8-way L2 cache
| cpu1: smt 1, core 0, package 0
| cpu2 at mainbus0: apid 2 (application processor)
| cpu2: Intel(R) Xeon(R) CPU E31260L @ 2.40GHz, 2400.02 MHz, 06-2a-07
| cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
| cpu2: 256KB 64b/line 8-way L2 cache
| cpu2: smt 0, core 1, package 0
| cpu3 at mainbus0: apid 3 (application processor)
| cpu3: Intel(R) Xeon(R) CPU E31260L @ 2.40GHz, 2400.02 MHz, 06-2a-07
| cpu3: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
| cpu3: 256KB 64b/line 8-way L2 cache
| cpu3: smt 1, core 1, package 0
| cpu4 at mainbus0: apid 4 (application processor)
| cpu4: Intel(R) Xeon(R) CPU E31260L @ 2.40GHz, 2400.02 MHz, 06-2a-07
| cpu4: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
| cpu4: 256KB 64b/line 8-way L2 cache
| cpu4: smt 0, core 2, package 0
| cpu5 at mainbus0: apid 5 (application processor)
| cpu5: Intel(R) Xeon(R) CPU E31260L @ 2.40GHz, 2400.02 MHz, 06-2a-07
| cpu5: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
| cpu5: 256KB 64b/line 8-way L2 cache
| cpu5: smt 1, core 2, package 0
| cpu6 at mainbus0: apid 6 (application processor)
| cpu6: Intel(R) Xeon(R) CPU E31260L @ 2.40GHz, 2400.02 MHz, 06-2a-07
| cpu6: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
| cpu6: 256KB 64b/line 8-way L2 cache
| cpu6: smt 0, core 3, package 0
| cpu7 at mainbus0: apid 7 (application processor)
| cpu7: Intel(R) Xeon(R) CPU E31260L @ 2.40GHz, 2400.02 MHz, 06-2a-07
| cpu7: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
| cpu7: 256KB 64b/line 8-way L2 cache
| cpu7: smt 1, core 3, package 0
| ioapic0 at mainbus0: apid 0 pa 0xfec00000, version 20, 24 pins
| acpimcfg0 at acpi0
| acpimcfg0: addr 0xe0000000, bus 0-255
| acpiprt0 at acpi0: bus 0 (PCI0)
| acpiprt1 at acpi0: bus 2 (P0P1)
| acpiprt2 at acpi0: bus 1 (RP01)
| acpiprt3 at acpi0: bus -1 (RP02)
| acpiprt4 at acpi0: bus -1 (RP03)
| acpiprt5 at acpi0: bus -1 (RP04)
| acpiprt6 at acpi0: bus -1 (RP05)
| acpiprt7 at acpi0: bus -1 (RP06)
| acpiprt8 at acpi0: bus -1 (RP07)
| acpiprt9 at acpi0: bus -1 (RP08)
| acpiprt10 at acpi0: bus -1 (PEG0)
| acpiprt11 at acpi0: bus -1 (PEG1)
| acpiprt12 at acpi0: bus -1 (PEG2)
| acpiprt13 at acpi0: bus -1 (PEG3)
| acpicpu0 at acpi0: C3(350@104 mwait.1@0x20), C2(500@80 mwait.1@0x10), C1(1000@1 mwait.1), PSS
| acpicpu1 at acpi0: C3(350@104 mwait.1@0x20), C2(500@80 mwait.1@0x10), C1(1000@1 mwait.1), PSS
| acpicpu2 at acpi0: C3(350@104 mwait.1@0x20), C2(500@80 mwait.1@0x10), C1(1000@1 mwait.1), PSS
| acpicpu3 at acpi0: C3(350@104 mwait.1@0x20), C2(500@80 mwait.1@0x10), C1(1000@1 mwait.1), PSS
| acpicpu4 at acpi0: C3(350@104 mwait.1@0x20), C2(500@80 mwait.1@0x10), C1(1000@1 mwait.1), PSS
| acpicpu5 at acpi0: C3(350@104 mwait.1@0x20), C2(500@80 mwait.1@0x10), C1(1000@1 mwait.1), PSS
| acpicpu6 at acpi0: C3(350@104 mwait.1@0x20), C2(500@80 mwait.1@0x10), C1(1000@1 mwait.1), PSS
| acpicpu7 at acpi0: C3(350@104 mwait.1@0x20), C2(500@80 mwait.1@0x10), C1(1000@1 mwait.1), PSS
| acpipwrres0 at acpi0: FN00, resource for FAN0
| acpipwrres1 at acpi0: FN01, resource for FAN1
| acpipwrres2 at acpi0: FN02, resource for FAN2
| acpipwrres3 at acpi0: FN03, resource for FAN3
| acpipwrres4 at acpi0: FN04, resource for FAN4
| acpitz0 at acpi0: critical temperature is 100 degC
| acpipci0 at acpi0 PCI0: 0x00000010 0x00000011 0x00000000
| "IPI0001" at acpi0 not configured
| acpicmos0 at acpi0
| "PNP0A05" at acpi0 not configured
| "PNP0A05" at acpi0 not configured
| "PNP0C0B" at acpi0 not configured
| "PNP0C0B" at acpi0 not configured
| "PNP0C0B" at acpi0 not configured
| "PNP0C0B" at acpi0 not configured
| "PNP0C0B" at acpi0 not configured
| ipmi at mainbus0 not configured
| cpu0: using VERW MDS workaround (except on vmm entry)
| cpu0: Enhanced SpeedStep 2400 MHz: speeds: 2401, 2400, 2300, 2200, 2100, 2000, 1900, 1800, 1700, 1600 MHz
| pci0 at mainbus0 bus 0
| pchb0 at pci0 dev 0 function 0 "Intel Xeon E3-1200 Host" rev 0x09
| ehci0 at pci0 dev 26 function 0 "Intel 6 Series USB" rev 0x04: apic 0 int 16
| usb0 at ehci0: USB revision 2.0
| uhub0 at usb0 configuration 1 interface 0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
| ppb0 at pci0 dev 28 function 0 "Intel 6 Series PCIE" rev 0xb4: msi
| pci1 at ppb0 bus 1
| bnx0 at pci1 dev 0 function 0 "Broadcom BCM5716" rev 0x20: apic 0 int 16
| bnx1 at pci1 dev 0 function 1 "Broadcom BCM5716" rev 0x20: apic 0 int 17
| ehci1 at pci0 dev 29 function 0 "Intel 6 Series USB" rev 0x04: apic 0 int 23
| usb1 at ehci1: USB revision 2.0
| uhub1 at usb1 configuration 1 interface 0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
| ppb1 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0xa4
| pci2 at ppb1 bus 2
| 2:3:0: mem address conflict 0xffff0000/0x10000
| vga1 at pci2 dev 3 function 0 "Matrox MGA G200eW" rev 0x0a
| wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
| wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
| pcib0 at pci0 dev 31 function 0 "Intel C202 LPC" rev 0x04
| ahci0 at pci0 dev 31 function 2 "Intel 6 Series AHCI" rev 0x04: msi, AHCI 1.3
| ahci0: port 0: 3.0Gb/s
| ahci0: port 1: 3.0Gb/s
| ahci0: port 4: 1.5Gb/s
| scsibus1 at ahci0: 32 targets
| sd0 at scsibus1 targ 0 lun 0: <ATA, INTEL SSDSC2BW48, DC32> SCSI3 0/direct fixed naa.55cd2e414c807f33
| sd0: 457862MB, 512 bytes/sector, 937703088 sectors, thin
| sd1 at scsibus1 targ 1 lun 0: <ATA, WDC WD60EFRX-68L, 82.0> SCSI3 0/direct fixed naa.50014ee20dd29e68
| sd1: 5723166MB, 512 bytes/sector, 11721045168 sectors
| cd0 at scsibus1 targ 4 lun 0: <TSSTcorp, DVD+-RW TS-L633J, D150> ATAPI 5/cdrom removable
| ichiic0 at pci0 dev 31 function 3 "Intel 6 Series SMBus" rev 0x04: apic 0 int 19
| iic0 at ichiic0
| sdtemp0 at iic0 addr 0x18: mcp98243
| sdtemp1 at iic0 addr 0x19: mcp98243
| sdtemp2 at iic0 addr 0x1a: mcp98243
| sdtemp3 at iic0 addr 0x1b: mcp98243
| spdmem0 at iic0 addr 0x50: 8GB DDR3 SDRAM ECC PC3-12800 with thermal sensor
| spdmem1 at iic0 addr 0x51: 8GB DDR3 SDRAM ECC PC3-12800 with thermal sensor
| spdmem2 at iic0 addr 0x52: 8GB DDR3 SDRAM ECC PC3-12800 with thermal sensor
| spdmem3 at iic0 addr 0x53: 8GB DDR3 SDRAM ECC PC3-12800 with thermal sensor
| isa0 at pcib0
| isadma0 at isa0
| com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
| com0: console
| com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
| pckbc0 at isa0 port 0x60/5 irq 1 irq 12
| pcppi0 at isa0 port 0x61
| spkr0 at pcppi0
| vmm0 at mainbus0: VMX/EPT
| uhub2 at uhub0 port 1 configuration 1 interface 0 "Intel Rate Matching Hub" rev 2.00/0.00 addr 2
| uhub3 at uhub1 port 1 configuration 1 interface 0 "Intel Rate Matching Hub" rev 2.00/0.00 addr 2
| uhidev0 at uhub3 port 1 configuration 1 interface 0 "RDing TEMPERHUM1V1.2" rev 2.00/0.01 addr 3
| uhidev0: iclass 3/1, 1 report id
| ukbd0 at uhidev0 reportid 1: 8 variable keys, 5 key codes
| wskbd0 at ukbd0: console keyboard, using wsdisplay0
| uhidev1 at uhub3 port 1 configuration 1 interface 1 "RDing TEMPERHUM1V1.2" rev 2.00/0.01 addr 3
| uhidev1: iclass 3/1
| ugold0 at uhidev1
| uhub4 at uhub3 port 5 configuration 1 interface 0 "Standard Microsystems product 0x2514" rev 2.00/0.00 addr 4
| vscsi0 at root
| scsibus2 at vscsi0: 256 targets
| softraid0 at root
| scsibus3 at softraid0: 256 targets
| sd2 at scsibus3 targ 1 lun 0: <OPENBSD, SR CRYPTO, 006> SCSI2 0/direct fixed
| sd2: 409602MB, 512 bytes/sector, 838865577 sectors
| root on sd2a (5a5905f3a7206f17.a) swap on sd2b dump on sd2b
| bnx0: address bc:30:5b:e8:d6:48
| brgphy0 at bnx0 phy 1: BCM5709 10/100/1000baseT PHY, rev. 8
| bnx1: address bc:30:5b:e8:d6:49
| brgphy1 at bnx1 phy 1: BCM5709 10/100/1000baseT PHY, rev. 8
| ugold0: 2 sensors type si7006 (temperature and humidity)
| sd3 at scsibus3 targ 2 lun 0: <OPENBSD, SR CRYPTO, 006> SCSI2 0/direct fixed
| sd3: 5231654MB, 512 bytes/sector, 10714427745 sectors
| UVM: pid 36202 (bgpd), uid 75 killed: out of swap
| ----------------------------------------------------------------------
|
| --- undeadly dmesg ---------------------------------------------------
| OpenBSD 6.5-current (GENERIC) #7: Fri Jun  7 00:45:34 MDT 2019
|     [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC
| real mem = 2130690048 (2031MB)
| avail mem = 2056167424 (1960MB)
| mpath0 at root
| scsibus0 at mpath0: 256 targets
| mainbus0 at root
| bios0 at mainbus0: SMBIOS rev. 2.4 @ 0xf15e0 (10 entries)
| bios0: vendor SeaBIOS version "1.11.0p1-OpenBSD-vmm" date 01/01/2011
| bios0: OpenBSD VMM
| acpi at bios0 not configured
| cpu0 at mainbus0: (uniprocessor)
| cpu0: Intel(R) Xeon(R) CPU E31260L @ 2.40GHz, 2401.44 MHz, 06-2a-07
| cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,CX8,SEP,PGE,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SSE3,PCLMUL,SSSE3,CX16,SSE4.1,SSE4.2,POPCNT,AES,XSAVE,AVX,HV,NXE,LONG,LAHF,ITSC,MD_CLEAR,MELTDOWN
| cpu0: 256KB 64b/line 8-way L2 cache
| cpu0: smt 0, core 0, package 0
| cpu0: using VERW MDS workaround
| pvbus0 at mainbus0: OpenBSD
| pvclock0 at pvbus0
| pci0 at mainbus0 bus 0
| pchb0 at pci0 dev 0 function 0 "OpenBSD VMM Host" rev 0x00
| virtio0 at pci0 dev 1 function 0 "Qumranet Virtio RNG" rev 0x00
| viornd0 at virtio0
| virtio0: irq 3
| virtio1 at pci0 dev 2 function 0 "Qumranet Virtio Network" rev 0x00
| vio0 at virtio1: address fe:e1:bb:02:6b:bf
| virtio1: irq 5
| virtio2 at pci0 dev 3 function 0 "Qumranet Virtio Storage" rev 0x00
| vioblk0 at virtio2
| scsibus1 at vioblk0: 2 targets
| sd0 at scsibus1 targ 0 lun 0: <VirtIO, Block Device, > SCSI3 0/direct fixed
| sd0: 6144MB, 512 bytes/sector, 12582912 sectors
| virtio2: irq 6
| virtio3 at pci0 dev 4 function 0 "Qumranet Virtio Storage" rev 0x00
| vioblk1 at virtio3
| scsibus2 at vioblk1: 2 targets
| sd1 at scsibus2 targ 0 lun 0: <VirtIO, Block Device, > SCSI3 0/direct fixed
| sd1: 51200MB, 512 bytes/sector, 104857600 sectors
| virtio3: irq 7
| virtio4 at pci0 dev 5 function 0 "OpenBSD VMM Control" rev 0x00
| vmmci0 at virtio4
| virtio4: irq 9
| isa0 at mainbus0
| isadma0 at isa0
| com0 at isa0 port 0x3f8/8 irq 4: ns8250, no fifo
| com0: console
| vscsi0 at root
| scsibus3 at vscsi0: 256 targets
| softraid0 at root
| scsibus4 at softraid0: 256 targets
| root on sd0a (112d38627522c4b4.a) swap on sd0b dump on sd0b
| ----------------------------------------------------------------------
|
| --- testvm dmesg -----------------------------------------------------
| OpenBSD 6.5-current (GENERIC) #6: Tue Jun  4 14:57:43 MDT 2019
|     [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC
| real mem = 2130698240 (2031MB)
| avail mem = 2056175616 (1960MB)
| mpath0 at root
| scsibus0 at mpath0: 256 targets
| mainbus0 at root
| bios0 at mainbus0: SMBIOS rev. 2.4 @ 0xf15e0 (10 entries)
| bios0: vendor SeaBIOS version "1.11.0p1-OpenBSD-vmm" date 01/01/2011
| bios0: OpenBSD VMM
| acpi at bios0 not configured
| cpu0 at mainbus0: (uniprocessor)
| cpu0: Intel(R) Xeon(R) CPU E31260L @ 2.40GHz, 2401.28 MHz, 06-2a-07
| cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,CX8,SEP,PGE,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SSE3,PCLMUL,SSSE3,CX16,SSE4.1,SSE4.2,POPCNT,AES,XSAVE,AVX,HV,NXE,LONG,LAHF,ITSC,MD_CLEAR,MELTDOWN
| cpu0: 256KB 64b/line 8-way L2 cache
| cpu0: smt 0, core 0, package 0
| cpu0: using VERW MDS workaround
| pvbus0 at mainbus0: OpenBSD
| pvclock0 at pvbus0
| pci0 at mainbus0 bus 0
| pchb0 at pci0 dev 0 function 0 "OpenBSD VMM Host" rev 0x00
| virtio0 at pci0 dev 1 function 0 "Qumranet Virtio RNG" rev 0x00
| viornd0 at virtio0
| virtio0: irq 3
| virtio1 at pci0 dev 2 function 0 "Qumranet Virtio Network" rev 0x00
| vio0 at virtio1: address fe:e1:bb:02:6b:be
| virtio1: irq 5
| virtio2 at pci0 dev 3 function 0 "Qumranet Virtio Storage" rev 0x00
| vioblk0 at virtio2
| scsibus1 at vioblk0: 2 targets
| sd0 at scsibus1 targ 0 lun 0: <VirtIO, Block Device, > SCSI3 0/direct fixed
| sd0: 6144MB, 512 bytes/sector, 12582912 sectors
| virtio2: irq 6
| virtio3 at pci0 dev 4 function 0 "OpenBSD VMM Control" rev 0x00
| vmmci0 at virtio3
| virtio3: irq 7
| isa0 at mainbus0
| isadma0 at isa0
| com0 at isa0 port 0x3f8/8 irq 4: ns8250, no fifo
| com0: console
| vscsi0 at root
| scsibus2 at vscsi0: 256 targets
| softraid0 at root
| scsibus3 at softraid0: 256 targets
| root on sd0a (10dc99cfac23b3f1.a) swap on sd0b dump on sd0b
| ----------------------------------------------------------------------
|
| --
| >++++++++[<++++++++++>-]<+++++++.>+++[<------>-]<.>+++[<+
| +++++++++++>-]<.>++[<------------>-]<+.--------------.[-]
|                  http://www.weirdnet.nl/                 
|

--
>++++++++[<++++++++++>-]<+++++++.>+++[<------>-]<.>+++[<+
+++++++++++>-]<.>++[<------------>-]<+.--------------.[-]
                 http://www.weirdnet.nl/                 

Reply | Threaded
Open this post in threaded view
|

Re: vmd eating lots of memory

Mike Larkin-2
On Thu, Jul 25, 2019 at 09:54:22PM +0200, Paul de Weerd wrote:

> A little more follow-up on this vmd-memory-leak issue.
>
> Comparing the two VMs I have running, I started to stress parts where
> these two hosts differ.  The testvm hardly does any traffic, while the
> undeadly vm sees quite a few visitors on a daily basis, so networking
> may be part of the leak.
>
> Running tcpbench against this machine (averaging at ~250Mbit/s)
> results in vmd growing by about 80MB to 100MB per minute.  Running
> tcpbench against the testvm has similar results.
>
> The undeadly VM also has a second disk configured (on slower storage),
> but putting load on that didn't significantly change the memory
> consumption (above the 'expected' growth that I've been seeing).
>
> The growth during daily(8) runs still confuses me, as that doesn't do
> anything network-related...
>
> Paul
>

I'll try to look for leaks in that area then. Thanks for the report.

-ml

> On Sat, Jul 20, 2019 at 04:23:27PM +0200, Paul de Weerd wrote:
> | Hi all,
> |
> | I'm running two vmd(8) VMs:
> |
> | [weerd@despair] $ cat /etc/vm.conf
> | vm "undeadly" {
> |         owner root
> |         memory 2G
> |         disk /home/vmm/undeadly.dsk
> |         disk /storage/vmm/undeadly.dsk
> |         interface switch "vmmswitch" lladdr "fe:e1:bb:02:6b:bf"
> | }
> |
> | vm "testvm" {
> |         owner root
> |         memory 2G
> |         disk /home/vmm/testvm.dsk
> |         interface switch "vmmswitch" lladdr "fe:e1:bb:02:6b:be"
> | }
> |
> | switch "vmmswitch" {
> |         enable
> |         interface bridge0
> | }
> |
> | Both machines have been up since boot of the host machine:
> |
> | [weerd@despair] $ vmctl status
> |    ID   PID VCPUS  MAXMEM  CURMEM     TTY        OWNER    STATE NAME
> |     2 23133     1    2.0G    1.1G   ttyp1         root  running testvm
> |     1 42094     1    2.0G    1.5G   ttyp0         root  running undeadly
> | [weerd@despair] $ uptime
> |  4:11PM  up 43 days, 57 mins, 12 users, load averages: 0.02, 0.07, 0.07
> | [weerd@despair] $ ssh undeadly uptime
> |  4:11PM  up 43 days, 57 mins, 2 users, load averages: 0.28, 0.12, 0.10
> | [weerd@despair] $ ssh testvm uptime  
> |  4:11PM  up 43 days, 57 mins, 0 users, load averages: 0.00, 0.00, 0.00
> |
> | However, the undeadly VM consumes signficantly more memory that the
> | other VM:
> |
> | [weerd@despair] $ ps wwwaux | grep -e vm[d] -e [P]ID
> | USER       PID %CPU %MEM   VSZ   RSS TT  STAT  STARTED       TIME COMMAND
> | _vmd     42094  3.7 57.9 28769236 19398792 ??  Ip     7Jun19  4741:37.66 vmd: undeadly (vmd)
> | _vmd      4567  0.0  0.0  1384  1384 ??  Isp    7Jun19    0:00.01 vmd: vmm (vmd)
> | root     84392  0.0  0.0  1564  1436 ??  Isp    7Jun19    0:00.02 /usr/sbin/vmd
> | _vmd     55269  0.0  0.0  1304  1468 ??  Isp    7Jun19    0:00.01 vmd: control (vmd)
> | root     51181  0.0  0.0  1236   928 ??  Is     7Jun19    0:00.01 vmd: priv (vmd)
> | _vmd     23133  0.0  0.2 2100100 70272 ??  Ip     7Jun19  1886:16.18 vmd: testvm (vmd)
> |
> | All systems run snaps from around the same time:
> |
> | [weerd@despair] $ sysctl kern.version
> | kern.version=OpenBSD 6.5-current (GENERIC.MP) #6: Tue Jun  4 15:05:10 MDT 2019
> |     [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
> |
> | [weerd@despair] $ ssh undeadly sysctl kern.version
> | kern.version=OpenBSD 6.5-current (GENERIC) #7: Fri Jun  7 00:45:34 MDT 2019
> |     [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC
> |
> | [weerd@despair] $ ssh testvm sysctl kern.version
> | kern.version=OpenBSD 6.5-current (GENERIC) #6: Tue Jun  4 14:57:43 MDT 2019
> |     [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC
> |
> | So it looks like there's a memory leak, either only for busy VMs or it
> | shows more for busier VMs.  Has anyone seen anything similar on their
> | vmm VMs?
> |
> | I'll reboot the undeadly VM for now, as other processes are getting
> | killed for out of memory reasons (bgpd, in my case; see end of below
> | dmesg of host machine 'despair').
> |
> | Paul
> |
> | --- despair dmesg ----------------------------------------------------
> | OpenBSD 6.5-current (GENERIC.MP) #6: Tue Jun  4 15:05:10 MDT 2019
> |     [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
> | real mem = 34332733440 (32742MB)
> | avail mem = 33282105344 (31740MB)
> | mpath0 at root
> | scsibus0 at mpath0: 256 targets
> | mainbus0 at root
> | bios0 at mainbus0: SMBIOS rev. 2.7 @ 0xe66d0 (57 entries)
> | bios0: vendor Dell Inc. version "1.2.3" date 07/21/2011
> | bios0: Dell Inc. PowerEdge R210 II
> | acpi0 at bios0: rev 2
> | acpi0: sleep states S0 S4 S5
> | acpi0: tables DSDT FACP SPMI ASF! HPET APIC MCFG BOOT SSDT ASPT SSDT SSDT SPCR DMAR HEST ERST BERT EINJ
> | acpi0: wakeup devices P0P1(S4) GLAN(S0) EHC1(S4) EHC2(S4) PXSX(S4) RP01(S5) PXSX(S4) RP02(S5) PXSX(S4) RP03(S5) PXSX(S4) RP04(S5) PXSX(S4) RP05(S5) PXSX(S4) RP06(S5) [...]
> | acpitimer0 at acpi0: 3579545 Hz, 24 bits
> | acpihpet0 at acpi0: 14318179 Hz
> | acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
> | cpu0 at mainbus0: apid 0 (boot processor)
> | cpu0: Intel(R) Xeon(R) CPU E31260L @ 2.40GHz, 2400.39 MHz, 06-2a-07
> | cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
> | cpu0: 256KB 64b/line 8-way L2 cache
> | cpu0: smt 0, core 0, package 0
> | mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges
> | cpu0: apic clock running at 100MHz
> | cpu0: mwait min=64, max=64, C-substates=0.2.1.1, IBE
> | cpu1 at mainbus0: apid 1 (application processor)
> | cpu1: Intel(R) Xeon(R) CPU E31260L @ 2.40GHz, 2400.02 MHz, 06-2a-07
> | cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
> | cpu1: 256KB 64b/line 8-way L2 cache
> | cpu1: smt 1, core 0, package 0
> | cpu2 at mainbus0: apid 2 (application processor)
> | cpu2: Intel(R) Xeon(R) CPU E31260L @ 2.40GHz, 2400.02 MHz, 06-2a-07
> | cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
> | cpu2: 256KB 64b/line 8-way L2 cache
> | cpu2: smt 0, core 1, package 0
> | cpu3 at mainbus0: apid 3 (application processor)
> | cpu3: Intel(R) Xeon(R) CPU E31260L @ 2.40GHz, 2400.02 MHz, 06-2a-07
> | cpu3: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
> | cpu3: 256KB 64b/line 8-way L2 cache
> | cpu3: smt 1, core 1, package 0
> | cpu4 at mainbus0: apid 4 (application processor)
> | cpu4: Intel(R) Xeon(R) CPU E31260L @ 2.40GHz, 2400.02 MHz, 06-2a-07
> | cpu4: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
> | cpu4: 256KB 64b/line 8-way L2 cache
> | cpu4: smt 0, core 2, package 0
> | cpu5 at mainbus0: apid 5 (application processor)
> | cpu5: Intel(R) Xeon(R) CPU E31260L @ 2.40GHz, 2400.02 MHz, 06-2a-07
> | cpu5: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
> | cpu5: 256KB 64b/line 8-way L2 cache
> | cpu5: smt 1, core 2, package 0
> | cpu6 at mainbus0: apid 6 (application processor)
> | cpu6: Intel(R) Xeon(R) CPU E31260L @ 2.40GHz, 2400.02 MHz, 06-2a-07
> | cpu6: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
> | cpu6: 256KB 64b/line 8-way L2 cache
> | cpu6: smt 0, core 3, package 0
> | cpu7 at mainbus0: apid 7 (application processor)
> | cpu7: Intel(R) Xeon(R) CPU E31260L @ 2.40GHz, 2400.02 MHz, 06-2a-07
> | cpu7: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
> | cpu7: 256KB 64b/line 8-way L2 cache
> | cpu7: smt 1, core 3, package 0
> | ioapic0 at mainbus0: apid 0 pa 0xfec00000, version 20, 24 pins
> | acpimcfg0 at acpi0
> | acpimcfg0: addr 0xe0000000, bus 0-255
> | acpiprt0 at acpi0: bus 0 (PCI0)
> | acpiprt1 at acpi0: bus 2 (P0P1)
> | acpiprt2 at acpi0: bus 1 (RP01)
> | acpiprt3 at acpi0: bus -1 (RP02)
> | acpiprt4 at acpi0: bus -1 (RP03)
> | acpiprt5 at acpi0: bus -1 (RP04)
> | acpiprt6 at acpi0: bus -1 (RP05)
> | acpiprt7 at acpi0: bus -1 (RP06)
> | acpiprt8 at acpi0: bus -1 (RP07)
> | acpiprt9 at acpi0: bus -1 (RP08)
> | acpiprt10 at acpi0: bus -1 (PEG0)
> | acpiprt11 at acpi0: bus -1 (PEG1)
> | acpiprt12 at acpi0: bus -1 (PEG2)
> | acpiprt13 at acpi0: bus -1 (PEG3)
> | acpicpu0 at acpi0: C3(350@104 mwait.1@0x20), C2(500@80 mwait.1@0x10), C1(1000@1 mwait.1), PSS
> | acpicpu1 at acpi0: C3(350@104 mwait.1@0x20), C2(500@80 mwait.1@0x10), C1(1000@1 mwait.1), PSS
> | acpicpu2 at acpi0: C3(350@104 mwait.1@0x20), C2(500@80 mwait.1@0x10), C1(1000@1 mwait.1), PSS
> | acpicpu3 at acpi0: C3(350@104 mwait.1@0x20), C2(500@80 mwait.1@0x10), C1(1000@1 mwait.1), PSS
> | acpicpu4 at acpi0: C3(350@104 mwait.1@0x20), C2(500@80 mwait.1@0x10), C1(1000@1 mwait.1), PSS
> | acpicpu5 at acpi0: C3(350@104 mwait.1@0x20), C2(500@80 mwait.1@0x10), C1(1000@1 mwait.1), PSS
> | acpicpu6 at acpi0: C3(350@104 mwait.1@0x20), C2(500@80 mwait.1@0x10), C1(1000@1 mwait.1), PSS
> | acpicpu7 at acpi0: C3(350@104 mwait.1@0x20), C2(500@80 mwait.1@0x10), C1(1000@1 mwait.1), PSS
> | acpipwrres0 at acpi0: FN00, resource for FAN0
> | acpipwrres1 at acpi0: FN01, resource for FAN1
> | acpipwrres2 at acpi0: FN02, resource for FAN2
> | acpipwrres3 at acpi0: FN03, resource for FAN3
> | acpipwrres4 at acpi0: FN04, resource for FAN4
> | acpitz0 at acpi0: critical temperature is 100 degC
> | acpipci0 at acpi0 PCI0: 0x00000010 0x00000011 0x00000000
> | "IPI0001" at acpi0 not configured
> | acpicmos0 at acpi0
> | "PNP0A05" at acpi0 not configured
> | "PNP0A05" at acpi0 not configured
> | "PNP0C0B" at acpi0 not configured
> | "PNP0C0B" at acpi0 not configured
> | "PNP0C0B" at acpi0 not configured
> | "PNP0C0B" at acpi0 not configured
> | "PNP0C0B" at acpi0 not configured
> | ipmi at mainbus0 not configured
> | cpu0: using VERW MDS workaround (except on vmm entry)
> | cpu0: Enhanced SpeedStep 2400 MHz: speeds: 2401, 2400, 2300, 2200, 2100, 2000, 1900, 1800, 1700, 1600 MHz
> | pci0 at mainbus0 bus 0
> | pchb0 at pci0 dev 0 function 0 "Intel Xeon E3-1200 Host" rev 0x09
> | ehci0 at pci0 dev 26 function 0 "Intel 6 Series USB" rev 0x04: apic 0 int 16
> | usb0 at ehci0: USB revision 2.0
> | uhub0 at usb0 configuration 1 interface 0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
> | ppb0 at pci0 dev 28 function 0 "Intel 6 Series PCIE" rev 0xb4: msi
> | pci1 at ppb0 bus 1
> | bnx0 at pci1 dev 0 function 0 "Broadcom BCM5716" rev 0x20: apic 0 int 16
> | bnx1 at pci1 dev 0 function 1 "Broadcom BCM5716" rev 0x20: apic 0 int 17
> | ehci1 at pci0 dev 29 function 0 "Intel 6 Series USB" rev 0x04: apic 0 int 23
> | usb1 at ehci1: USB revision 2.0
> | uhub1 at usb1 configuration 1 interface 0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
> | ppb1 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0xa4
> | pci2 at ppb1 bus 2
> | 2:3:0: mem address conflict 0xffff0000/0x10000
> | vga1 at pci2 dev 3 function 0 "Matrox MGA G200eW" rev 0x0a
> | wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
> | wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
> | pcib0 at pci0 dev 31 function 0 "Intel C202 LPC" rev 0x04
> | ahci0 at pci0 dev 31 function 2 "Intel 6 Series AHCI" rev 0x04: msi, AHCI 1.3
> | ahci0: port 0: 3.0Gb/s
> | ahci0: port 1: 3.0Gb/s
> | ahci0: port 4: 1.5Gb/s
> | scsibus1 at ahci0: 32 targets
> | sd0 at scsibus1 targ 0 lun 0: <ATA, INTEL SSDSC2BW48, DC32> SCSI3 0/direct fixed naa.55cd2e414c807f33
> | sd0: 457862MB, 512 bytes/sector, 937703088 sectors, thin
> | sd1 at scsibus1 targ 1 lun 0: <ATA, WDC WD60EFRX-68L, 82.0> SCSI3 0/direct fixed naa.50014ee20dd29e68
> | sd1: 5723166MB, 512 bytes/sector, 11721045168 sectors
> | cd0 at scsibus1 targ 4 lun 0: <TSSTcorp, DVD+-RW TS-L633J, D150> ATAPI 5/cdrom removable
> | ichiic0 at pci0 dev 31 function 3 "Intel 6 Series SMBus" rev 0x04: apic 0 int 19
> | iic0 at ichiic0
> | sdtemp0 at iic0 addr 0x18: mcp98243
> | sdtemp1 at iic0 addr 0x19: mcp98243
> | sdtemp2 at iic0 addr 0x1a: mcp98243
> | sdtemp3 at iic0 addr 0x1b: mcp98243
> | spdmem0 at iic0 addr 0x50: 8GB DDR3 SDRAM ECC PC3-12800 with thermal sensor
> | spdmem1 at iic0 addr 0x51: 8GB DDR3 SDRAM ECC PC3-12800 with thermal sensor
> | spdmem2 at iic0 addr 0x52: 8GB DDR3 SDRAM ECC PC3-12800 with thermal sensor
> | spdmem3 at iic0 addr 0x53: 8GB DDR3 SDRAM ECC PC3-12800 with thermal sensor
> | isa0 at pcib0
> | isadma0 at isa0
> | com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
> | com0: console
> | com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
> | pckbc0 at isa0 port 0x60/5 irq 1 irq 12
> | pcppi0 at isa0 port 0x61
> | spkr0 at pcppi0
> | vmm0 at mainbus0: VMX/EPT
> | uhub2 at uhub0 port 1 configuration 1 interface 0 "Intel Rate Matching Hub" rev 2.00/0.00 addr 2
> | uhub3 at uhub1 port 1 configuration 1 interface 0 "Intel Rate Matching Hub" rev 2.00/0.00 addr 2
> | uhidev0 at uhub3 port 1 configuration 1 interface 0 "RDing TEMPERHUM1V1.2" rev 2.00/0.01 addr 3
> | uhidev0: iclass 3/1, 1 report id
> | ukbd0 at uhidev0 reportid 1: 8 variable keys, 5 key codes
> | wskbd0 at ukbd0: console keyboard, using wsdisplay0
> | uhidev1 at uhub3 port 1 configuration 1 interface 1 "RDing TEMPERHUM1V1.2" rev 2.00/0.01 addr 3
> | uhidev1: iclass 3/1
> | ugold0 at uhidev1
> | uhub4 at uhub3 port 5 configuration 1 interface 0 "Standard Microsystems product 0x2514" rev 2.00/0.00 addr 4
> | vscsi0 at root
> | scsibus2 at vscsi0: 256 targets
> | softraid0 at root
> | scsibus3 at softraid0: 256 targets
> | sd2 at scsibus3 targ 1 lun 0: <OPENBSD, SR CRYPTO, 006> SCSI2 0/direct fixed
> | sd2: 409602MB, 512 bytes/sector, 838865577 sectors
> | root on sd2a (5a5905f3a7206f17.a) swap on sd2b dump on sd2b
> | bnx0: address bc:30:5b:e8:d6:48
> | brgphy0 at bnx0 phy 1: BCM5709 10/100/1000baseT PHY, rev. 8
> | bnx1: address bc:30:5b:e8:d6:49
> | brgphy1 at bnx1 phy 1: BCM5709 10/100/1000baseT PHY, rev. 8
> | ugold0: 2 sensors type si7006 (temperature and humidity)
> | sd3 at scsibus3 targ 2 lun 0: <OPENBSD, SR CRYPTO, 006> SCSI2 0/direct fixed
> | sd3: 5231654MB, 512 bytes/sector, 10714427745 sectors
> | UVM: pid 36202 (bgpd), uid 75 killed: out of swap
> | ----------------------------------------------------------------------
> |
> | --- undeadly dmesg ---------------------------------------------------
> | OpenBSD 6.5-current (GENERIC) #7: Fri Jun  7 00:45:34 MDT 2019
> |     [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC
> | real mem = 2130690048 (2031MB)
> | avail mem = 2056167424 (1960MB)
> | mpath0 at root
> | scsibus0 at mpath0: 256 targets
> | mainbus0 at root
> | bios0 at mainbus0: SMBIOS rev. 2.4 @ 0xf15e0 (10 entries)
> | bios0: vendor SeaBIOS version "1.11.0p1-OpenBSD-vmm" date 01/01/2011
> | bios0: OpenBSD VMM
> | acpi at bios0 not configured
> | cpu0 at mainbus0: (uniprocessor)
> | cpu0: Intel(R) Xeon(R) CPU E31260L @ 2.40GHz, 2401.44 MHz, 06-2a-07
> | cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,CX8,SEP,PGE,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SSE3,PCLMUL,SSSE3,CX16,SSE4.1,SSE4.2,POPCNT,AES,XSAVE,AVX,HV,NXE,LONG,LAHF,ITSC,MD_CLEAR,MELTDOWN
> | cpu0: 256KB 64b/line 8-way L2 cache
> | cpu0: smt 0, core 0, package 0
> | cpu0: using VERW MDS workaround
> | pvbus0 at mainbus0: OpenBSD
> | pvclock0 at pvbus0
> | pci0 at mainbus0 bus 0
> | pchb0 at pci0 dev 0 function 0 "OpenBSD VMM Host" rev 0x00
> | virtio0 at pci0 dev 1 function 0 "Qumranet Virtio RNG" rev 0x00
> | viornd0 at virtio0
> | virtio0: irq 3
> | virtio1 at pci0 dev 2 function 0 "Qumranet Virtio Network" rev 0x00
> | vio0 at virtio1: address fe:e1:bb:02:6b:bf
> | virtio1: irq 5
> | virtio2 at pci0 dev 3 function 0 "Qumranet Virtio Storage" rev 0x00
> | vioblk0 at virtio2
> | scsibus1 at vioblk0: 2 targets
> | sd0 at scsibus1 targ 0 lun 0: <VirtIO, Block Device, > SCSI3 0/direct fixed
> | sd0: 6144MB, 512 bytes/sector, 12582912 sectors
> | virtio2: irq 6
> | virtio3 at pci0 dev 4 function 0 "Qumranet Virtio Storage" rev 0x00
> | vioblk1 at virtio3
> | scsibus2 at vioblk1: 2 targets
> | sd1 at scsibus2 targ 0 lun 0: <VirtIO, Block Device, > SCSI3 0/direct fixed
> | sd1: 51200MB, 512 bytes/sector, 104857600 sectors
> | virtio3: irq 7
> | virtio4 at pci0 dev 5 function 0 "OpenBSD VMM Control" rev 0x00
> | vmmci0 at virtio4
> | virtio4: irq 9
> | isa0 at mainbus0
> | isadma0 at isa0
> | com0 at isa0 port 0x3f8/8 irq 4: ns8250, no fifo
> | com0: console
> | vscsi0 at root
> | scsibus3 at vscsi0: 256 targets
> | softraid0 at root
> | scsibus4 at softraid0: 256 targets
> | root on sd0a (112d38627522c4b4.a) swap on sd0b dump on sd0b
> | ----------------------------------------------------------------------
> |
> | --- testvm dmesg -----------------------------------------------------
> | OpenBSD 6.5-current (GENERIC) #6: Tue Jun  4 14:57:43 MDT 2019
> |     [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC
> | real mem = 2130698240 (2031MB)
> | avail mem = 2056175616 (1960MB)
> | mpath0 at root
> | scsibus0 at mpath0: 256 targets
> | mainbus0 at root
> | bios0 at mainbus0: SMBIOS rev. 2.4 @ 0xf15e0 (10 entries)
> | bios0: vendor SeaBIOS version "1.11.0p1-OpenBSD-vmm" date 01/01/2011
> | bios0: OpenBSD VMM
> | acpi at bios0 not configured
> | cpu0 at mainbus0: (uniprocessor)
> | cpu0: Intel(R) Xeon(R) CPU E31260L @ 2.40GHz, 2401.28 MHz, 06-2a-07
> | cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,CX8,SEP,PGE,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SSE3,PCLMUL,SSSE3,CX16,SSE4.1,SSE4.2,POPCNT,AES,XSAVE,AVX,HV,NXE,LONG,LAHF,ITSC,MD_CLEAR,MELTDOWN
> | cpu0: 256KB 64b/line 8-way L2 cache
> | cpu0: smt 0, core 0, package 0
> | cpu0: using VERW MDS workaround
> | pvbus0 at mainbus0: OpenBSD
> | pvclock0 at pvbus0
> | pci0 at mainbus0 bus 0
> | pchb0 at pci0 dev 0 function 0 "OpenBSD VMM Host" rev 0x00
> | virtio0 at pci0 dev 1 function 0 "Qumranet Virtio RNG" rev 0x00
> | viornd0 at virtio0
> | virtio0: irq 3
> | virtio1 at pci0 dev 2 function 0 "Qumranet Virtio Network" rev 0x00
> | vio0 at virtio1: address fe:e1:bb:02:6b:be
> | virtio1: irq 5
> | virtio2 at pci0 dev 3 function 0 "Qumranet Virtio Storage" rev 0x00
> | vioblk0 at virtio2
> | scsibus1 at vioblk0: 2 targets
> | sd0 at scsibus1 targ 0 lun 0: <VirtIO, Block Device, > SCSI3 0/direct fixed
> | sd0: 6144MB, 512 bytes/sector, 12582912 sectors
> | virtio2: irq 6
> | virtio3 at pci0 dev 4 function 0 "OpenBSD VMM Control" rev 0x00
> | vmmci0 at virtio3
> | virtio3: irq 7
> | isa0 at mainbus0
> | isadma0 at isa0
> | com0 at isa0 port 0x3f8/8 irq 4: ns8250, no fifo
> | com0: console
> | vscsi0 at root
> | scsibus2 at vscsi0: 256 targets
> | softraid0 at root
> | scsibus3 at softraid0: 256 targets
> | root on sd0a (10dc99cfac23b3f1.a) swap on sd0b dump on sd0b
> | ----------------------------------------------------------------------
> |
> | --
> | >++++++++[<++++++++++>-]<+++++++.>+++[<------>-]<.>+++[<+
> | +++++++++++>-]<.>++[<------------>-]<+.--------------.[-]
> |                  http://www.weirdnet.nl/                 
> |
>
> --
> >++++++++[<++++++++++>-]<+++++++.>+++[<------>-]<.>+++[<+
> +++++++++++>-]<.>++[<------------>-]<+.--------------.[-]
>                  http://www.weirdnet.nl/                 

Reply | Threaded
Open this post in threaded view
|

Re: vmd eating lots of memory

Bryan Linton
On 2019-07-25 13:01:28, Mike Larkin <[hidden email]> wrote:

> On Thu, Jul 25, 2019 at 09:54:22PM +0200, Paul de Weerd wrote:
> > A little more follow-up on this vmd-memory-leak issue.
> >
> > Comparing the two VMs I have running, I started to stress parts where
> > these two hosts differ.  The testvm hardly does any traffic, while the
> > undeadly vm sees quite a few visitors on a daily basis, so networking
> > may be part of the leak.
> >
> > Running tcpbench against this machine (averaging at ~250Mbit/s)
> > results in vmd growing by about 80MB to 100MB per minute.  Running
> > tcpbench against the testvm has similar results.
> >
> > The undeadly VM also has a second disk configured (on slower storage),
> > but putting load on that didn't significantly change the memory
> > consumption (above the 'expected' growth that I've been seeing).
> >
> > The growth during daily(8) runs still confuses me, as that doesn't do
> > anything network-related...
> >
> > Paul
> >
>
> I'll try to look for leaks in that area then. Thanks for the report.
>
> -ml
>

To Paul, is it related to disk activity on the VM?  I.e. Does
doing lots of I/O on the system cause memory usage to increase?
Can you test it with a snapshot/kernel dated May 7th or earlier?

I CCed both of you into a bug report I just submitted to bugs@
because I thought it may possibly be related.

        https://marc.info/?l=openbsd-bugs&m=156412299418191&w=2

In brief, I'm seeing large amounts of memory being consumed
followed by a system hang when files are copied to a vnd(4)
device.

I don't see it with regular disk I/O, only with vnds.

If the VMs are using/accessing memory in a similar way to the
method that vnd(4) does, it might explain why the daily(8) runs
are causing the memory usage to increase.

If this is an unrelated issue, then I apologize for the noise.  I
figured it better to CC both of you in so you could evaluate it on
your own rather than for me to do nothing.

--
Bryan

Reply | Threaded
Open this post in threaded view
|

Re: vmd eating lots of memory

Paul de Weerd
Hi Bryan,

On Fri, Jul 26, 2019 at 04:02:10PM +0900, Bryan Linton wrote:
| On 2019-07-25 13:01:28, Mike Larkin <[hidden email]> wrote:
| > On Thu, Jul 25, 2019 at 09:54:22PM +0200, Paul de Weerd wrote:
| > > A little more follow-up on this vmd-memory-leak issue.
| > >
| > > Comparing the two VMs I have running, I started to stress parts where
| > > these two hosts differ.  The testvm hardly does any traffic, while the
| > > undeadly vm sees quite a few visitors on a daily basis, so networking
| > > may be part of the leak.
| > >
| > > Running tcpbench against this machine (averaging at ~250Mbit/s)
| > > results in vmd growing by about 80MB to 100MB per minute.  Running
| > > tcpbench against the testvm has similar results.
| > >
| > > The undeadly VM also has a second disk configured (on slower storage),
| > > but putting load on that didn't significantly change the memory
| > > consumption (above the 'expected' growth that I've been seeing).
| > >
| > > The growth during daily(8) runs still confuses me, as that doesn't do
| > > anything network-related...
| > >
| > > Paul
| > >
| >
| > I'll try to look for leaks in that area then. Thanks for the report.
| >
| > -ml
| >
|
| To Paul, is it related to disk activity on the VM?  I.e. Does
| doing lots of I/O on the system cause memory usage to increase?

It does not, I specifically tried that (also because the VM with this
issue has two disks, one of them on slower media) but I didn't notice
a difference.

| Can you test it with a snapshot/kernel dated May 7th or earlier?

Not easily on the undeadly production environment.  But I'm building
a test environment at home that I may be able to use for this.

| I CCed both of you into a bug report I just submitted to bugs@
| because I thought it may possibly be related.
|
| https://marc.info/?l=openbsd-bugs&m=156412299418191&w=2

Yep, saw that.  I can definitely try reverting that diff on my test
environment later today.

| In brief, I'm seeing large amounts of memory being consumed
| followed by a system hang when files are copied to a vnd(4)
| device.
|
| I don't see it with regular disk I/O, only with vnds.
|
| If the VMs are using/accessing memory in a similar way to the
| method that vnd(4) does, it might explain why the daily(8) runs
| are causing the memory usage to increase.

I can't really comment on that as I'm not familiar with the
implementation of these parts of the kernel.

| If this is an unrelated issue, then I apologize for the noise.  I
| figured it better to CC both of you in so you could evaluate it on
| your own rather than for me to do nothing.

Cheers,

Paul

--
>++++++++[<++++++++++>-]<+++++++.>+++[<------>-]<.>+++[<+
+++++++++++>-]<.>++[<------------>-]<+.--------------.[-]
                 http://www.weirdnet.nl/                 

Reply | Threaded
Open this post in threaded view
|

Re: vmd eating lots of memory

Paul de Weerd
In reply to this post by Paul de Weerd
Just confirmed I can reproduce this on the latest snapshot (both on
host and VMs) on my home workstation:

OpenBSD 6.5-current (GENERIC.MP) #143: Fri Jul 26 00:37:38 MDT 2019

If I use the host to send traffic with tcpbench, traffic rate peaks
around 1Gbit/s (1055.923 Mbps, according to tcpbench), and memory
consumption is growing at a rate that seems to correlate to the
bandwidth used by the tcpbench traffic.

The test is simply `tcpbench -s -p XXXXX` on the VM and `tcpbench -p
XXXXX vm` on the host.

Paul

On Thu, Jul 25, 2019 at 09:54:22PM +0200, Paul de Weerd wrote:
| A little more follow-up on this vmd-memory-leak issue.
|
| Comparing the two VMs I have running, I started to stress parts where
| these two hosts differ.  The testvm hardly does any traffic, while the
| undeadly vm sees quite a few visitors on a daily basis, so networking
| may be part of the leak.
|
| Running tcpbench against this machine (averaging at ~250Mbit/s)
| results in vmd growing by about 80MB to 100MB per minute.  Running
| tcpbench against the testvm has similar results.
|
| The undeadly VM also has a second disk configured (on slower storage),
| but putting load on that didn't significantly change the memory
| consumption (above the 'expected' growth that I've been seeing).
|
| The growth during daily(8) runs still confuses me, as that doesn't do
| anything network-related...
|
| Paul
|
| On Sat, Jul 20, 2019 at 04:23:27PM +0200, Paul de Weerd wrote:
| | Hi all,
| |
| | I'm running two vmd(8) VMs:
| |
| | [weerd@despair] $ cat /etc/vm.conf
| | vm "undeadly" {
| |         owner root
| |         memory 2G
| |         disk /home/vmm/undeadly.dsk
| |         disk /storage/vmm/undeadly.dsk
| |         interface switch "vmmswitch" lladdr "fe:e1:bb:02:6b:bf"
| | }
| |
| | vm "testvm" {
| |         owner root
| |         memory 2G
| |         disk /home/vmm/testvm.dsk
| |         interface switch "vmmswitch" lladdr "fe:e1:bb:02:6b:be"
| | }
| |
| | switch "vmmswitch" {
| |         enable
| |         interface bridge0
| | }
| |
| | Both machines have been up since boot of the host machine:
| |
| | [weerd@despair] $ vmctl status
| |    ID   PID VCPUS  MAXMEM  CURMEM     TTY        OWNER    STATE NAME
| |     2 23133     1    2.0G    1.1G   ttyp1         root  running testvm
| |     1 42094     1    2.0G    1.5G   ttyp0         root  running undeadly
| | [weerd@despair] $ uptime
| |  4:11PM  up 43 days, 57 mins, 12 users, load averages: 0.02, 0.07, 0.07
| | [weerd@despair] $ ssh undeadly uptime
| |  4:11PM  up 43 days, 57 mins, 2 users, load averages: 0.28, 0.12, 0.10
| | [weerd@despair] $ ssh testvm uptime  
| |  4:11PM  up 43 days, 57 mins, 0 users, load averages: 0.00, 0.00, 0.00
| |
| | However, the undeadly VM consumes signficantly more memory that the
| | other VM:
| |
| | [weerd@despair] $ ps wwwaux | grep -e vm[d] -e [P]ID
| | USER       PID %CPU %MEM   VSZ   RSS TT  STAT  STARTED       TIME COMMAND
| | _vmd     42094  3.7 57.9 28769236 19398792 ??  Ip     7Jun19  4741:37.66 vmd: undeadly (vmd)
| | _vmd      4567  0.0  0.0  1384  1384 ??  Isp    7Jun19    0:00.01 vmd: vmm (vmd)
| | root     84392  0.0  0.0  1564  1436 ??  Isp    7Jun19    0:00.02 /usr/sbin/vmd
| | _vmd     55269  0.0  0.0  1304  1468 ??  Isp    7Jun19    0:00.01 vmd: control (vmd)
| | root     51181  0.0  0.0  1236   928 ??  Is     7Jun19    0:00.01 vmd: priv (vmd)
| | _vmd     23133  0.0  0.2 2100100 70272 ??  Ip     7Jun19  1886:16.18 vmd: testvm (vmd)
| |
| | All systems run snaps from around the same time:
| |
| | [weerd@despair] $ sysctl kern.version
| | kern.version=OpenBSD 6.5-current (GENERIC.MP) #6: Tue Jun  4 15:05:10 MDT 2019
| |     [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
| |
| | [weerd@despair] $ ssh undeadly sysctl kern.version
| | kern.version=OpenBSD 6.5-current (GENERIC) #7: Fri Jun  7 00:45:34 MDT 2019
| |     [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC
| |
| | [weerd@despair] $ ssh testvm sysctl kern.version
| | kern.version=OpenBSD 6.5-current (GENERIC) #6: Tue Jun  4 14:57:43 MDT 2019
| |     [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC
| |
| | So it looks like there's a memory leak, either only for busy VMs or it
| | shows more for busier VMs.  Has anyone seen anything similar on their
| | vmm VMs?
| |
| | I'll reboot the undeadly VM for now, as other processes are getting
| | killed for out of memory reasons (bgpd, in my case; see end of below
| | dmesg of host machine 'despair').
| |
| | Paul
| |
| | --- despair dmesg ----------------------------------------------------
| | OpenBSD 6.5-current (GENERIC.MP) #6: Tue Jun  4 15:05:10 MDT 2019
| |     [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
| | real mem = 34332733440 (32742MB)
| | avail mem = 33282105344 (31740MB)
| | mpath0 at root
| | scsibus0 at mpath0: 256 targets
| | mainbus0 at root
| | bios0 at mainbus0: SMBIOS rev. 2.7 @ 0xe66d0 (57 entries)
| | bios0: vendor Dell Inc. version "1.2.3" date 07/21/2011
| | bios0: Dell Inc. PowerEdge R210 II
| | acpi0 at bios0: rev 2
| | acpi0: sleep states S0 S4 S5
| | acpi0: tables DSDT FACP SPMI ASF! HPET APIC MCFG BOOT SSDT ASPT SSDT SSDT SPCR DMAR HEST ERST BERT EINJ
| | acpi0: wakeup devices P0P1(S4) GLAN(S0) EHC1(S4) EHC2(S4) PXSX(S4) RP01(S5) PXSX(S4) RP02(S5) PXSX(S4) RP03(S5) PXSX(S4) RP04(S5) PXSX(S4) RP05(S5) PXSX(S4) RP06(S5) [...]
| | acpitimer0 at acpi0: 3579545 Hz, 24 bits
| | acpihpet0 at acpi0: 14318179 Hz
| | acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
| | cpu0 at mainbus0: apid 0 (boot processor)
| | cpu0: Intel(R) Xeon(R) CPU E31260L @ 2.40GHz, 2400.39 MHz, 06-2a-07
| | cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
| | cpu0: 256KB 64b/line 8-way L2 cache
| | cpu0: smt 0, core 0, package 0
| | mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges
| | cpu0: apic clock running at 100MHz
| | cpu0: mwait min=64, max=64, C-substates=0.2.1.1, IBE
| | cpu1 at mainbus0: apid 1 (application processor)
| | cpu1: Intel(R) Xeon(R) CPU E31260L @ 2.40GHz, 2400.02 MHz, 06-2a-07
| | cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
| | cpu1: 256KB 64b/line 8-way L2 cache
| | cpu1: smt 1, core 0, package 0
| | cpu2 at mainbus0: apid 2 (application processor)
| | cpu2: Intel(R) Xeon(R) CPU E31260L @ 2.40GHz, 2400.02 MHz, 06-2a-07
| | cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
| | cpu2: 256KB 64b/line 8-way L2 cache
| | cpu2: smt 0, core 1, package 0
| | cpu3 at mainbus0: apid 3 (application processor)
| | cpu3: Intel(R) Xeon(R) CPU E31260L @ 2.40GHz, 2400.02 MHz, 06-2a-07
| | cpu3: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
| | cpu3: 256KB 64b/line 8-way L2 cache
| | cpu3: smt 1, core 1, package 0
| | cpu4 at mainbus0: apid 4 (application processor)
| | cpu4: Intel(R) Xeon(R) CPU E31260L @ 2.40GHz, 2400.02 MHz, 06-2a-07
| | cpu4: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
| | cpu4: 256KB 64b/line 8-way L2 cache
| | cpu4: smt 0, core 2, package 0
| | cpu5 at mainbus0: apid 5 (application processor)
| | cpu5: Intel(R) Xeon(R) CPU E31260L @ 2.40GHz, 2400.02 MHz, 06-2a-07
| | cpu5: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
| | cpu5: 256KB 64b/line 8-way L2 cache
| | cpu5: smt 1, core 2, package 0
| | cpu6 at mainbus0: apid 6 (application processor)
| | cpu6: Intel(R) Xeon(R) CPU E31260L @ 2.40GHz, 2400.02 MHz, 06-2a-07
| | cpu6: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
| | cpu6: 256KB 64b/line 8-way L2 cache
| | cpu6: smt 0, core 3, package 0
| | cpu7 at mainbus0: apid 7 (application processor)
| | cpu7: Intel(R) Xeon(R) CPU E31260L @ 2.40GHz, 2400.02 MHz, 06-2a-07
| | cpu7: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,NXE,RDTSCP,LONG,LAHF,PERF,ITSC,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
| | cpu7: 256KB 64b/line 8-way L2 cache
| | cpu7: smt 1, core 3, package 0
| | ioapic0 at mainbus0: apid 0 pa 0xfec00000, version 20, 24 pins
| | acpimcfg0 at acpi0
| | acpimcfg0: addr 0xe0000000, bus 0-255
| | acpiprt0 at acpi0: bus 0 (PCI0)
| | acpiprt1 at acpi0: bus 2 (P0P1)
| | acpiprt2 at acpi0: bus 1 (RP01)
| | acpiprt3 at acpi0: bus -1 (RP02)
| | acpiprt4 at acpi0: bus -1 (RP03)
| | acpiprt5 at acpi0: bus -1 (RP04)
| | acpiprt6 at acpi0: bus -1 (RP05)
| | acpiprt7 at acpi0: bus -1 (RP06)
| | acpiprt8 at acpi0: bus -1 (RP07)
| | acpiprt9 at acpi0: bus -1 (RP08)
| | acpiprt10 at acpi0: bus -1 (PEG0)
| | acpiprt11 at acpi0: bus -1 (PEG1)
| | acpiprt12 at acpi0: bus -1 (PEG2)
| | acpiprt13 at acpi0: bus -1 (PEG3)
| | acpicpu0 at acpi0: C3(350@104 mwait.1@0x20), C2(500@80 mwait.1@0x10), C1(1000@1 mwait.1), PSS
| | acpicpu1 at acpi0: C3(350@104 mwait.1@0x20), C2(500@80 mwait.1@0x10), C1(1000@1 mwait.1), PSS
| | acpicpu2 at acpi0: C3(350@104 mwait.1@0x20), C2(500@80 mwait.1@0x10), C1(1000@1 mwait.1), PSS
| | acpicpu3 at acpi0: C3(350@104 mwait.1@0x20), C2(500@80 mwait.1@0x10), C1(1000@1 mwait.1), PSS
| | acpicpu4 at acpi0: C3(350@104 mwait.1@0x20), C2(500@80 mwait.1@0x10), C1(1000@1 mwait.1), PSS
| | acpicpu5 at acpi0: C3(350@104 mwait.1@0x20), C2(500@80 mwait.1@0x10), C1(1000@1 mwait.1), PSS
| | acpicpu6 at acpi0: C3(350@104 mwait.1@0x20), C2(500@80 mwait.1@0x10), C1(1000@1 mwait.1), PSS
| | acpicpu7 at acpi0: C3(350@104 mwait.1@0x20), C2(500@80 mwait.1@0x10), C1(1000@1 mwait.1), PSS
| | acpipwrres0 at acpi0: FN00, resource for FAN0
| | acpipwrres1 at acpi0: FN01, resource for FAN1
| | acpipwrres2 at acpi0: FN02, resource for FAN2
| | acpipwrres3 at acpi0: FN03, resource for FAN3
| | acpipwrres4 at acpi0: FN04, resource for FAN4
| | acpitz0 at acpi0: critical temperature is 100 degC
| | acpipci0 at acpi0 PCI0: 0x00000010 0x00000011 0x00000000
| | "IPI0001" at acpi0 not configured
| | acpicmos0 at acpi0
| | "PNP0A05" at acpi0 not configured
| | "PNP0A05" at acpi0 not configured
| | "PNP0C0B" at acpi0 not configured
| | "PNP0C0B" at acpi0 not configured
| | "PNP0C0B" at acpi0 not configured
| | "PNP0C0B" at acpi0 not configured
| | "PNP0C0B" at acpi0 not configured
| | ipmi at mainbus0 not configured
| | cpu0: using VERW MDS workaround (except on vmm entry)
| | cpu0: Enhanced SpeedStep 2400 MHz: speeds: 2401, 2400, 2300, 2200, 2100, 2000, 1900, 1800, 1700, 1600 MHz
| | pci0 at mainbus0 bus 0
| | pchb0 at pci0 dev 0 function 0 "Intel Xeon E3-1200 Host" rev 0x09
| | ehci0 at pci0 dev 26 function 0 "Intel 6 Series USB" rev 0x04: apic 0 int 16
| | usb0 at ehci0: USB revision 2.0
| | uhub0 at usb0 configuration 1 interface 0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
| | ppb0 at pci0 dev 28 function 0 "Intel 6 Series PCIE" rev 0xb4: msi
| | pci1 at ppb0 bus 1
| | bnx0 at pci1 dev 0 function 0 "Broadcom BCM5716" rev 0x20: apic 0 int 16
| | bnx1 at pci1 dev 0 function 1 "Broadcom BCM5716" rev 0x20: apic 0 int 17
| | ehci1 at pci0 dev 29 function 0 "Intel 6 Series USB" rev 0x04: apic 0 int 23
| | usb1 at ehci1: USB revision 2.0
| | uhub1 at usb1 configuration 1 interface 0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
| | ppb1 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0xa4
| | pci2 at ppb1 bus 2
| | 2:3:0: mem address conflict 0xffff0000/0x10000
| | vga1 at pci2 dev 3 function 0 "Matrox MGA G200eW" rev 0x0a
| | wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
| | wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
| | pcib0 at pci0 dev 31 function 0 "Intel C202 LPC" rev 0x04
| | ahci0 at pci0 dev 31 function 2 "Intel 6 Series AHCI" rev 0x04: msi, AHCI 1.3
| | ahci0: port 0: 3.0Gb/s
| | ahci0: port 1: 3.0Gb/s
| | ahci0: port 4: 1.5Gb/s
| | scsibus1 at ahci0: 32 targets
| | sd0 at scsibus1 targ 0 lun 0: <ATA, INTEL SSDSC2BW48, DC32> SCSI3 0/direct fixed naa.55cd2e414c807f33
| | sd0: 457862MB, 512 bytes/sector, 937703088 sectors, thin
| | sd1 at scsibus1 targ 1 lun 0: <ATA, WDC WD60EFRX-68L, 82.0> SCSI3 0/direct fixed naa.50014ee20dd29e68
| | sd1: 5723166MB, 512 bytes/sector, 11721045168 sectors
| | cd0 at scsibus1 targ 4 lun 0: <TSSTcorp, DVD+-RW TS-L633J, D150> ATAPI 5/cdrom removable
| | ichiic0 at pci0 dev 31 function 3 "Intel 6 Series SMBus" rev 0x04: apic 0 int 19
| | iic0 at ichiic0
| | sdtemp0 at iic0 addr 0x18: mcp98243
| | sdtemp1 at iic0 addr 0x19: mcp98243
| | sdtemp2 at iic0 addr 0x1a: mcp98243
| | sdtemp3 at iic0 addr 0x1b: mcp98243
| | spdmem0 at iic0 addr 0x50: 8GB DDR3 SDRAM ECC PC3-12800 with thermal sensor
| | spdmem1 at iic0 addr 0x51: 8GB DDR3 SDRAM ECC PC3-12800 with thermal sensor
| | spdmem2 at iic0 addr 0x52: 8GB DDR3 SDRAM ECC PC3-12800 with thermal sensor
| | spdmem3 at iic0 addr 0x53: 8GB DDR3 SDRAM ECC PC3-12800 with thermal sensor
| | isa0 at pcib0
| | isadma0 at isa0
| | com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
| | com0: console
| | com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
| | pckbc0 at isa0 port 0x60/5 irq 1 irq 12
| | pcppi0 at isa0 port 0x61
| | spkr0 at pcppi0
| | vmm0 at mainbus0: VMX/EPT
| | uhub2 at uhub0 port 1 configuration 1 interface 0 "Intel Rate Matching Hub" rev 2.00/0.00 addr 2
| | uhub3 at uhub1 port 1 configuration 1 interface 0 "Intel Rate Matching Hub" rev 2.00/0.00 addr 2
| | uhidev0 at uhub3 port 1 configuration 1 interface 0 "RDing TEMPERHUM1V1.2" rev 2.00/0.01 addr 3
| | uhidev0: iclass 3/1, 1 report id
| | ukbd0 at uhidev0 reportid 1: 8 variable keys, 5 key codes
| | wskbd0 at ukbd0: console keyboard, using wsdisplay0
| | uhidev1 at uhub3 port 1 configuration 1 interface 1 "RDing TEMPERHUM1V1.2" rev 2.00/0.01 addr 3
| | uhidev1: iclass 3/1
| | ugold0 at uhidev1
| | uhub4 at uhub3 port 5 configuration 1 interface 0 "Standard Microsystems product 0x2514" rev 2.00/0.00 addr 4
| | vscsi0 at root
| | scsibus2 at vscsi0: 256 targets
| | softraid0 at root
| | scsibus3 at softraid0: 256 targets
| | sd2 at scsibus3 targ 1 lun 0: <OPENBSD, SR CRYPTO, 006> SCSI2 0/direct fixed
| | sd2: 409602MB, 512 bytes/sector, 838865577 sectors
| | root on sd2a (5a5905f3a7206f17.a) swap on sd2b dump on sd2b
| | bnx0: address bc:30:5b:e8:d6:48
| | brgphy0 at bnx0 phy 1: BCM5709 10/100/1000baseT PHY, rev. 8
| | bnx1: address bc:30:5b:e8:d6:49
| | brgphy1 at bnx1 phy 1: BCM5709 10/100/1000baseT PHY, rev. 8
| | ugold0: 2 sensors type si7006 (temperature and humidity)
| | sd3 at scsibus3 targ 2 lun 0: <OPENBSD, SR CRYPTO, 006> SCSI2 0/direct fixed
| | sd3: 5231654MB, 512 bytes/sector, 10714427745 sectors
| | UVM: pid 36202 (bgpd), uid 75 killed: out of swap
| | ----------------------------------------------------------------------
| |
| | --- undeadly dmesg ---------------------------------------------------
| | OpenBSD 6.5-current (GENERIC) #7: Fri Jun  7 00:45:34 MDT 2019
| |     [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC
| | real mem = 2130690048 (2031MB)
| | avail mem = 2056167424 (1960MB)
| | mpath0 at root
| | scsibus0 at mpath0: 256 targets
| | mainbus0 at root
| | bios0 at mainbus0: SMBIOS rev. 2.4 @ 0xf15e0 (10 entries)
| | bios0: vendor SeaBIOS version "1.11.0p1-OpenBSD-vmm" date 01/01/2011
| | bios0: OpenBSD VMM
| | acpi at bios0 not configured
| | cpu0 at mainbus0: (uniprocessor)
| | cpu0: Intel(R) Xeon(R) CPU E31260L @ 2.40GHz, 2401.44 MHz, 06-2a-07
| | cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,CX8,SEP,PGE,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SSE3,PCLMUL,SSSE3,CX16,SSE4.1,SSE4.2,POPCNT,AES,XSAVE,AVX,HV,NXE,LONG,LAHF,ITSC,MD_CLEAR,MELTDOWN
| | cpu0: 256KB 64b/line 8-way L2 cache
| | cpu0: smt 0, core 0, package 0
| | cpu0: using VERW MDS workaround
| | pvbus0 at mainbus0: OpenBSD
| | pvclock0 at pvbus0
| | pci0 at mainbus0 bus 0
| | pchb0 at pci0 dev 0 function 0 "OpenBSD VMM Host" rev 0x00
| | virtio0 at pci0 dev 1 function 0 "Qumranet Virtio RNG" rev 0x00
| | viornd0 at virtio0
| | virtio0: irq 3
| | virtio1 at pci0 dev 2 function 0 "Qumranet Virtio Network" rev 0x00
| | vio0 at virtio1: address fe:e1:bb:02:6b:bf
| | virtio1: irq 5
| | virtio2 at pci0 dev 3 function 0 "Qumranet Virtio Storage" rev 0x00
| | vioblk0 at virtio2
| | scsibus1 at vioblk0: 2 targets
| | sd0 at scsibus1 targ 0 lun 0: <VirtIO, Block Device, > SCSI3 0/direct fixed
| | sd0: 6144MB, 512 bytes/sector, 12582912 sectors
| | virtio2: irq 6
| | virtio3 at pci0 dev 4 function 0 "Qumranet Virtio Storage" rev 0x00
| | vioblk1 at virtio3
| | scsibus2 at vioblk1: 2 targets
| | sd1 at scsibus2 targ 0 lun 0: <VirtIO, Block Device, > SCSI3 0/direct fixed
| | sd1: 51200MB, 512 bytes/sector, 104857600 sectors
| | virtio3: irq 7
| | virtio4 at pci0 dev 5 function 0 "OpenBSD VMM Control" rev 0x00
| | vmmci0 at virtio4
| | virtio4: irq 9
| | isa0 at mainbus0
| | isadma0 at isa0
| | com0 at isa0 port 0x3f8/8 irq 4: ns8250, no fifo
| | com0: console
| | vscsi0 at root
| | scsibus3 at vscsi0: 256 targets
| | softraid0 at root
| | scsibus4 at softraid0: 256 targets
| | root on sd0a (112d38627522c4b4.a) swap on sd0b dump on sd0b
| | ----------------------------------------------------------------------
| |
| | --- testvm dmesg -----------------------------------------------------
| | OpenBSD 6.5-current (GENERIC) #6: Tue Jun  4 14:57:43 MDT 2019
| |     [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC
| | real mem = 2130698240 (2031MB)
| | avail mem = 2056175616 (1960MB)
| | mpath0 at root
| | scsibus0 at mpath0: 256 targets
| | mainbus0 at root
| | bios0 at mainbus0: SMBIOS rev. 2.4 @ 0xf15e0 (10 entries)
| | bios0: vendor SeaBIOS version "1.11.0p1-OpenBSD-vmm" date 01/01/2011
| | bios0: OpenBSD VMM
| | acpi at bios0 not configured
| | cpu0 at mainbus0: (uniprocessor)
| | cpu0: Intel(R) Xeon(R) CPU E31260L @ 2.40GHz, 2401.28 MHz, 06-2a-07
| | cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,CX8,SEP,PGE,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SSE3,PCLMUL,SSSE3,CX16,SSE4.1,SSE4.2,POPCNT,AES,XSAVE,AVX,HV,NXE,LONG,LAHF,ITSC,MD_CLEAR,MELTDOWN
| | cpu0: 256KB 64b/line 8-way L2 cache
| | cpu0: smt 0, core 0, package 0
| | cpu0: using VERW MDS workaround
| | pvbus0 at mainbus0: OpenBSD
| | pvclock0 at pvbus0
| | pci0 at mainbus0 bus 0
| | pchb0 at pci0 dev 0 function 0 "OpenBSD VMM Host" rev 0x00
| | virtio0 at pci0 dev 1 function 0 "Qumranet Virtio RNG" rev 0x00
| | viornd0 at virtio0
| | virtio0: irq 3
| | virtio1 at pci0 dev 2 function 0 "Qumranet Virtio Network" rev 0x00
| | vio0 at virtio1: address fe:e1:bb:02:6b:be
| | virtio1: irq 5
| | virtio2 at pci0 dev 3 function 0 "Qumranet Virtio Storage" rev 0x00
| | vioblk0 at virtio2
| | scsibus1 at vioblk0: 2 targets
| | sd0 at scsibus1 targ 0 lun 0: <VirtIO, Block Device, > SCSI3 0/direct fixed
| | sd0: 6144MB, 512 bytes/sector, 12582912 sectors
| | virtio2: irq 6
| | virtio3 at pci0 dev 4 function 0 "OpenBSD VMM Control" rev 0x00
| | vmmci0 at virtio3
| | virtio3: irq 7
| | isa0 at mainbus0
| | isadma0 at isa0
| | com0 at isa0 port 0x3f8/8 irq 4: ns8250, no fifo
| | com0: console
| | vscsi0 at root
| | scsibus2 at vscsi0: 256 targets
| | softraid0 at root
| | scsibus3 at softraid0: 256 targets
| | root on sd0a (10dc99cfac23b3f1.a) swap on sd0b dump on sd0b
| | ----------------------------------------------------------------------
| |
| | --
| | >++++++++[<++++++++++>-]<+++++++.>+++[<------>-]<.>+++[<+
| | +++++++++++>-]<.>++[<------------>-]<+.--------------.[-]
| |                  http://www.weirdnet.nl/                 
| |
|
| --
| >++++++++[<++++++++++>-]<+++++++.>+++[<------>-]<.>+++[<+
| +++++++++++>-]<.>++[<------------>-]<+.--------------.[-]
|                  http://www.weirdnet.nl/                 
|

--
>++++++++[<++++++++++>-]<+++++++.>+++[<------>-]<.>+++[<+
+++++++++++>-]<.>++[<------------>-]<+.--------------.[-]
                 http://www.weirdnet.nl/