VM disk image performance degraded on rewrite of blocks

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

VM disk image performance degraded on rewrite of blocks

openbsd2012
I've recently noticed that VM disk image performance is degraded when blocks are rewritten within a disk image by the guest OS. I suspect this may be related to the thin provisioning of vmctl VM disk images.


On my test machine, first writes to blocks with dd if=/dev/zero are within the range of 175 MB/s. Subsequent writes are anywhere from 25MB/s to 70 MB/s.


Test conditions:

- Jan 24 snapshot

- vmd host running single VM and no other load (dmesg at end of email)

- vmctl create /home/test.vm.fs -s 8G

- booted an existing VM with /home/test.vm.fs attached as sd1 (tested both 6.2-stable and 6.2-current VMs)

- fdisk -iy sd1

- disklabel sd1:

  a:         16771776               64  4.2BSD   2048 16384 12958
- mount /dev/sd1a /mnt


Performance tests:


1. New writes as follows with expected optimal disk performance (NOTE both /dev/zero and /dev/urandom test files):


# dd if=/dev/zero of=/mnt/test1 bs=1m count=1024
1024+0 records in
1024+0 records out
1073741824 bytes transferred in 6.161 secs (174276136 bytes/sec)
# dd if=/dev/urandom of=/mnt/test2 bs=1m count=1024
1024+0 records in
1024+0 records out
1073741824 bytes transferred in 9.838 secs (109132548 bytes/sec)
# dd if=/dev/zero of=/mnt/test3 bs=1m count=1024
1024+0 records in
1024+0 records out
1073741824 bytes transferred in 5.971 secs (179800958 bytes/sec)

2. Overwrite /mnt/test1 and performance is degraded:


# dd if=/dev/zero of=/mnt/test1 bs=1m count=1024
1024+0 records in
1024+0 records out
1073741824 bytes transferred in 25.994 secs (41305958 bytes/sec)

3. Write a new file and performance picks up to expected optimal performance:


# dd if=/dev/zero of=/mnt/test4 bs=1m count=1024
1024+0 records in
1024+0 records out
1073741824 bytes transferred in 6.004 secs (178828580 bytes/sec)

4. Rewrite the fourth file and performance is degraded:


# dd if=/dev/zero of=/mnt/test4 bs=1m count=1024
1024+0 records in
1024+0 records out
1073741824 bytes transferred in 18.498 secs (58044338 bytes/sec)

5. Also degraded performance:

# rm /mnt/test2
# dd if=/dev/zero of=/mnt/test2 bs=1m count=1024
1024+0 records in
1024+0 records out
1073741824 bytes transferred in 17.013 secs (63112746 bytes/sec)

6. New files are written with optimal performance until disk is full (/mnt/test5 assumed to have encountered some pre-written blocks due to rm /mnt/test2):

# dd if=/dev/zero of=/mnt/test5 bs=1m count=1024
1024+0 records in
1024+0 records out
1073741824 bytes transferred in 8.510 secs (126160945 bytes/sec)
# dd if=/dev/zero of=/mnt/test6 bs=1m count=1024
1024+0 records in
1024+0 records out
1073741824 bytes transferred in 5.951 secs (180413534 bytes/sec)
# dd if=/dev/zero of=/mnt/test7 bs=1m count=1024
1024+0 records in
1024+0 records out
1073741824 bytes transferred in 5.988 secs (179291452 bytes/sec)
# dd if=/dev/zero of=/mnt/test8 bs=1m count=1024

/mnt: write failed, file system is full
dd: /mnt/test8: No space left on device
888+0 records in
887+0 records out
930086912 bytes transferred in 5.312 secs (175079201 bytes/sec)
# rm /mnt/*
# dd if=/dev/zero of=/mnt/test9 bs=1m count=1024
1024+0 records in
1024+0 records out
1073741824 bytes transferred in 27.552 secs (38970531 bytes/sec)

7. An issue with cp as well as dd:

# dd if=/dev/urandom of=/mnt/testa bs=1m count=1024
1024+0 records in
1024+0 records out
1073741824 bytes transferred in 27.553 secs (38968811 bytes/sec)
# time cp /mnt/testa /mnt/testb
    0m31.76s real     0m00.00s user     0m11.79s system
# time cp /mnt/testa /mnt/testb
    0m35.91s real     0m00.00s user     0m14.02s system
# time cp /mnt/testb /mnt/testc
    0m36.54s real     0m00.00s user     0m13.98s system

8. From this point forward, all writes to sd1a are degraded.

------------------------- dmesg -------------------------

OpenBSD 6.2-current (GENERIC.MP) #387: Wed Jan 24 12:53:55 MST 2018
    [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 68594307072 (65416MB)
avail mem = 66508509184 (63427MB)
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.8 @ 0xed9b0 (39 entries)
bios0: vendor American Megatrends Inc. version "1.1b" date 07/13/2016
bios0: Supermicro Super Server
acpi0 at bios0: rev 2
acpi0: sleep states S0 S4 S5
acpi0: tables DSDT FACP APIC FPDT FIDT SPMI MCFG UEFI DBG2 HPET WDDT SSDT SSDT SSDT PRAD DMAR HEST BERT ERST EINJ
acpi0: wakeup devices IP2P(S4) XHCI(S4) EHC1(S4) EHC2(S4) RP01(S4) RP02(S4) RP03(S4) RP04(S4) RP05(S4) RP06(S4) RP07(S4) RP08(S4) BR1A(S4) BR1B(S4) BR2A(S4) BR2B(S4) [...]
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz, 2100.32 MHz
cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,SENSOR,ARAT
cpu0: 256KB 64b/line 8-way L2 cache
acpitimer0: recalibrated TSC frequency 2100005999 Hz
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges
cpu0: apic clock running at 99MHz
cpu0: mwait min=64, max=64, C-substates=0.2.1.2, IBE
cpu1 at mainbus0: apid 2 (application processor)
cpu1: Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz, 2100.00 MHz
cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,SENSOR,ARAT
cpu1: 256KB 64b/line 8-way L2 cache
cpu1: smt 0, core 1, package 0
cpu2 at mainbus0: apid 4 (application processor)
cpu2: Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz, 2100.00 MHz
cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,SENSOR,ARAT
cpu2: 256KB 64b/line 8-way L2 cache
cpu2: smt 0, core 2, package 0
cpu3 at mainbus0: apid 6 (application processor)
cpu3: Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz, 2100.00 MHz
cpu3: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,SENSOR,ARAT
cpu3: 256KB 64b/line 8-way L2 cache
cpu3: smt 0, core 3, package 0
cpu4 at mainbus0: apid 8 (application processor)
cpu4: Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz, 2100.00 MHz
cpu4: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,SENSOR,ARAT
cpu4: 256KB 64b/line 8-way L2 cache
cpu4: smt 0, core 4, package 0
cpu5 at mainbus0: apid 10 (application processor)
cpu5: Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz, 2100.00 MHz
cpu5: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,SENSOR,ARAT
cpu5: 256KB 64b/line 8-way L2 cache
cpu5: smt 0, core 5, package 0
cpu6 at mainbus0: apid 12 (application processor)
cpu6: Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz, 2100.00 MHz
cpu6: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,SENSOR,ARAT
cpu6: 256KB 64b/line 8-way L2 cache
cpu6: smt 0, core 6, package 0
cpu7 at mainbus0: apid 14 (application processor)
cpu7: Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz, 2100.00 MHz
cpu7: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,SENSOR,ARAT
cpu7: 256KB 64b/line 8-way L2 cache
cpu7: smt 0, core 7, package 0
cpu8 at mainbus0: apid 1 (application processor)
cpu8: Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz, 2100.00 MHz
cpu8: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,SENSOR,ARAT
cpu8: 256KB 64b/line 8-way L2 cache
cpu8: smt 1, core 0, package 0
cpu9 at mainbus0: apid 3 (application processor)
cpu9: Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz, 2100.00 MHz
cpu9: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,SENSOR,ARAT
cpu9: 256KB 64b/line 8-way L2 cache
cpu9: smt 1, core 1, package 0
cpu10 at mainbus0: apid 5 (application processor)
cpu10: Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz, 2100.00 MHz
cpu10: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,SENSOR,ARAT
cpu10: 256KB 64b/line 8-way L2 cache
cpu10: smt 1, core 2, package 0
cpu11 at mainbus0: apid 7 (application processor)
cpu11: Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz, 2100.00 MHz
cpu11: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,SENSOR,ARAT
cpu11: 256KB 64b/line 8-way L2 cache
cpu11: smt 1, core 3, package 0
cpu12 at mainbus0: apid 9 (application processor)
cpu12: Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz, 2100.00 MHz
cpu12: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,SENSOR,ARAT
cpu12: 256KB 64b/line 8-way L2 cache
cpu12: smt 1, core 4, package 0
cpu13 at mainbus0: apid 11 (application processor)
cpu13: Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz, 2100.00 MHz
cpu13: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,SENSOR,ARAT
cpu13: 256KB 64b/line 8-way L2 cache
cpu13: smt 1, core 5, package 0
cpu14 at mainbus0: apid 13 (application processor)
cpu14: Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz, 2100.00 MHz
cpu14: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,SENSOR,ARAT
cpu14: 256KB 64b/line 8-way L2 cache
cpu14: smt 1, core 6, package 0
cpu15 at mainbus0: apid 15 (application processor)
cpu15: Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz, 2100.00 MHz
cpu15: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,SENSOR,ARAT
cpu15: 256KB 64b/line 8-way L2 cache
cpu15: smt 1, core 7, package 0
ioapic0 at mainbus0: apid 8 pa 0xfec00000, version 20, 24 pins
acpimcfg0 at acpi0 addr 0x80000000, bus 0-255
acpihpet0 at acpi0: 14318179 Hz
acpihpet0: recalibrated TSC frequency 2100010800 Hz
acpiprt0 at acpi0: bus 255 (UNC0)
acpiprt1 at acpi0: bus 0 (PCI0)
acpiprt2 at acpi0: bus 6 (RP01)
acpiprt3 at acpi0: bus 8 (RP05)
acpiprt4 at acpi0: bus 9 (BR39)
acpiprt5 at acpi0: bus 1 (BR1A)
acpiprt6 at acpi0: bus 3 (BR2C)
acpiprt7 at acpi0: bus 5 (BR3A)
acpicpu0 at acpi0: C2(350@41 mwait.3@0x20), C1(1016@1 mwait.1), PSS
acpicpu1 at acpi0: C2(350@41 mwait.3@0x20), C1(1016@1 mwait.1), PSS
acpicpu2 at acpi0: C2(350@41 mwait.3@0x20), C1(1016@1 mwait.1), PSS
acpicpu3 at acpi0: C2(350@41 mwait.3@0x20), C1(1016@1 mwait.1), PSS
acpicpu4 at acpi0: C2(350@41 mwait.3@0x20), C1(1016@1 mwait.1), PSS
acpicpu5 at acpi0: C2(350@41 mwait.3@0x20), C1(1016@1 mwait.1), PSS
acpicpu6 at acpi0: C2(350@41 mwait.3@0x20), C1(1016@1 mwait.1), PSS
acpicpu7 at acpi0: C2(350@41 mwait.3@0x20), C1(1016@1 mwait.1), PSS
acpicpu8 at acpi0: C2(350@41 mwait.3@0x20), C1(1016@1 mwait.1), PSS
acpicpu9 at acpi0: C2(350@41 mwait.3@0x20), C1(1016@1 mwait.1), PSS
acpicpu10 at acpi0: C2(350@41 mwait.3@0x20), C1(1016@1 mwait.1), PSS
acpicpu11 at acpi0: C2(350@41 mwait.3@0x20), C1(1016@1 mwait.1), PSS
acpicpu12 at acpi0: C2(350@41 mwait.3@0x20), C1(1016@1 mwait.1), PSS
acpicpu13 at acpi0: C2(350@41 mwait.3@0x20), C1(1016@1 mwait.1), PSS
acpicpu14 at acpi0: C2(350@41 mwait.3@0x20), C1(1016@1 mwait.1), PSS
acpicpu15 at acpi0: C2(350@41 mwait.3@0x20), C1(1016@1 mwait.1), PSS
"ACPI0004" at acpi0 not configured
"ACPI0007" at acpi0 not configured
"ACPI0007" at acpi0 not configured
"ACPI0007" at acpi0 not configured
"ACPI0007" at acpi0 not configured
"ACPI0007" at acpi0 not configured
"ACPI0007" at acpi0 not configured
"ACPI0007" at acpi0 not configured
"ACPI0007" at acpi0 not configured
"ACPI0007" at acpi0 not configured
"ACPI0007" at acpi0 not configured
"ACPI0007" at acpi0 not configured
"ACPI0007" at acpi0 not configured
"ACPI0007" at acpi0 not configured
"ACPI0007" at acpi0 not configured
"ACPI0007" at acpi0 not configured
"ACPI0007" at acpi0 not configured
"PNP0C33" at acpi0 not configured
"PNP0C14" at acpi0 not configured
"PNP0003" at acpi0 not configured
"PNP0C14" at acpi0 not configured
"IPI0001" at acpi0 not configured
acpibtn0 at acpi0: PWRB
ipmi at mainbus0 not configured
cpu0: Enhanced SpeedStep 2100 MHz: speeds: 2101, 2100, 2000, 1900, 1800, 1700, 1600, 1500, 1400, 1300, 1200, 1100, 1000, 900, 800 MHz
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel Xeon-D Host" rev 0x03
ppb0 at pci0 dev 1 function 0 "Intel Xeon-D PCIE" rev 0x03: msi
pci1 at ppb0 bus 1
ppb1 at pci0 dev 2 function 0 "Intel Xeon-D PCIE" rev 0x03
pci2 at ppb1 bus 2
"Intel Xeon-D QuickData" rev 0x00 at pci2 dev 0 function 0 not configured
"Intel Xeon-D QuickData" rev 0x00 at pci2 dev 0 function 1 not configured
"Intel Xeon-D QuickData" rev 0x00 at pci2 dev 0 function 2 not configured
"Intel Xeon-D QuickData" rev 0x00 at pci2 dev 0 function 3 not configured
ppb2 at pci0 dev 2 function 2 "Intel Xeon-D PCIE" rev 0x03
pci3 at ppb2 bus 3
ix0 at pci3 dev 0 function 0 "Intel X552/X557-AT" rev 0x00: msi, address 0c:c4:7a:c5:59:ce
ix1 at pci3 dev 0 function 1 "Intel X552/X557-AT" rev 0x00: msi, address 0c:c4:7a:c5:59:cf
ppb3 at pci0 dev 3 function 0 "Intel Xeon-D PCIE" rev 0x03: msi
pci4 at ppb3 bus 5
"Intel Xeon-D Address Map" rev 0x03 at pci0 dev 5 function 0 not configured
"Intel Xeon-D Hot Plug" rev 0x03 at pci0 dev 5 function 1 not configured
"Intel Xeon-D RAS" rev 0x03 at pci0 dev 5 function 2 not configured
xhci0 at pci0 dev 20 function 0 "Intel 8 Series xHCI" rev 0x05: msi
usb0 at xhci0: USB revision 3.0
uhub0 at usb0 configuration 1 interface 0 "Intel xHCI root hub" rev 3.00/1.00 addr 1
"Intel 8 Series MEI" rev 0x04 at pci0 dev 22 function 0 not configured
"Intel 8 Series MEI" rev 0x04 at pci0 dev 22 function 1 not configured
ehci0 at pci0 dev 26 function 0 "Intel 8 Series USB" rev 0x05: apic 8 int 18
usb1 at ehci0: USB revision 2.0
uhub1 at usb1 configuration 1 interface 0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
ppb4 at pci0 dev 28 function 0 "Intel 8 Series PCIE" rev 0xd5: msi
pci5 at ppb4 bus 6
em0 at pci5 dev 0 function 0 "Intel I350" rev 0x01: msi, address 0c:c4:7a:c5:57:4a
em1 at pci5 dev 0 function 1 "Intel I350" rev 0x01: msi, address 0c:c4:7a:c5:57:4b
ppb5 at pci0 dev 28 function 4 "Intel 8 Series PCIE" rev 0xd5: msi
pci6 at ppb5 bus 8
ppb6 at pci6 dev 0 function 0 "ASPEED Technology AST1150 PCI" rev 0x03
pci7 at ppb6 bus 9
vga1 at pci7 dev 0 function 0 "ASPEED Technology AST2000" rev 0x30
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
ehci1 at pci0 dev 29 function 0 "Intel 8 Series USB" rev 0x05: apic 8 int 18
usb2 at ehci1: USB revision 2.0
uhub2 at usb2 configuration 1 interface 0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
pcib0 at pci0 dev 31 function 0 "Intel C224 LPC" rev 0x05
ahci0 at pci0 dev 31 function 2 "Intel 8 Series AHCI" rev 0x05: msi, AHCI 1.3
ahci0: port 0: 6.0Gb/s
ahci0: port 1: 6.0Gb/s
ahci0: port 2: 6.0Gb/s
ahci0: port 3: 6.0Gb/s
scsibus1 at ahci0: 32 targets
sd0 at scsibus1 targ 0 lun 0: <ATA, INTEL SSDSC2BB48, G201> SCSI3 0/direct fixed naa.55cd2e414d467cc0
sd0: 457862MB, 512 bytes/sector, 937703088 sectors, thin
sd1 at scsibus1 targ 1 lun 0: <ATA, INTEL SSDSC2BB48, G201> SCSI3 0/direct fixed naa.55cd2e414d467c6c
sd1: 457862MB, 512 bytes/sector, 937703088 sectors, thin
sd2 at scsibus1 targ 2 lun 0: <ATA, HGST HUS726020AL, A5GN> SCSI3 0/direct fixed naa.5000cca245c2f1f7
sd2: 1907729MB, 512 bytes/sector, 3907029168 sectors
sd3 at scsibus1 targ 3 lun 0: <ATA, HGST HUS726020AL, A5GN> SCSI3 0/direct fixed naa.5000cca245c2fae2
sd3: 1907729MB, 512 bytes/sector, 3907029168 sectors
ichiic0 at pci0 dev 31 function 3 "Intel 8 Series SMBus" rev 0x05: apic 8 int 18
iic0 at ichiic0
"Intel 8 Series Thermal" rev 0x05 at pci0 dev 31 function 6 not configured
isa0 at pcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
pckbc0 at isa0 port 0x60/5 irq 1 irq 12
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
pci8 at mainbus0 bus 255
"Intel Xeon-D QPI Link" rev 0x03 at pci8 dev 11 function 0 not configured
"Intel Xeon-D QPI Link" rev 0x03 at pci8 dev 11 function 1 not configured
"Intel Xeon-D QPI Link" rev 0x03 at pci8 dev 11 function 2 not configured
"Intel Xeon-D QPI Debug" rev 0x03 at pci8 dev 11 function 3 not configured
"Intel Xeon-D Cache" rev 0x03 at pci8 dev 12 function 0 not configured
"Intel Xeon-D Cache" rev 0x03 at pci8 dev 12 function 1 not configured
"Intel Xeon-D Cache" rev 0x03 at pci8 dev 12 function 2 not configured
"Intel Xeon-D Cache" rev 0x03 at pci8 dev 12 function 3 not configured
"Intel E5 v4 Cache" rev 0x03 at pci8 dev 12 function 4 not configured
"Intel E5 v4 Cache" rev 0x03 at pci8 dev 12 function 5 not configured
vendor "Intel", unknown product 0x6fe6 (class system subclass miscellaneous, rev 0x03) at pci8 dev 12 function 6 not configured
vendor "Intel", unknown product 0x6fe7 (class system subclass miscellaneous, rev 0x03) at pci8 dev 12 function 7 not configured
"Intel Xeon-D Cache" rev 0x03 at pci8 dev 15 function 0 not configured
"Intel Xeon-D Cache" rev 0x03 at pci8 dev 15 function 4 not configured
"Intel Xeon-D Cache" rev 0x03 at pci8 dev 15 function 5 not configured
"Intel Xeon-D Cache" rev 0x03 at pci8 dev 15 function 6 not configured
"Intel Xeon-D PCIE" rev 0x03 at pci8 dev 16 function 0 not configured
"Intel E5 v4 R2PCIe Agent" rev 0x03 at pci8 dev 16 function 1 not configured
"Intel Xeon-D Ubox" rev 0x03 at pci8 dev 16 function 5 not configured
"Intel Xeon-D Ubox" rev 0x03 at pci8 dev 16 function 6 not configured
"Intel Xeon-D Ubox" rev 0x03 at pci8 dev 16 function 7 not configured
"Intel Xeon-D Home Agent" rev 0x03 at pci8 dev 18 function 0 not configured
"Intel Xeon-D Home Agent" rev 0x03 at pci8 dev 18 function 1 not configured
"Intel Xeon-D Memory" rev 0x03 at pci8 dev 19 function 0 not configured
"Intel Xeon-D Memory" rev 0x03 at pci8 dev 19 function 1 not configured
"Intel Xeon-D Memory" rev 0x03 at pci8 dev 19 function 2 not configured
"Intel Xeon-D Memory" rev 0x03 at pci8 dev 19 function 3 not configured
"Intel Xeon-D Memory" rev 0x03 at pci8 dev 19 function 4 not configured
"Intel Xeon-D Memory" rev 0x03 at pci8 dev 19 function 5 not configured
"Intel Xeon-D Memory" rev 0x03 at pci8 dev 19 function 6 not configured
"Intel Xeon-D Memory" rev 0x03 at pci8 dev 19 function 7 not configured
"Intel Xeon-D Memory" rev 0x03 at pci8 dev 20 function 0 not configured
"Intel Xeon-D Memory" rev 0x03 at pci8 dev 20 function 1 not configured
"Intel Xeon-D Memory" rev 0x03 at pci8 dev 20 function 2 not configured
"Intel Xeon-D Memory" rev 0x03 at pci8 dev 20 function 3 not configured
"Intel Xeon-D Memory" rev 0x03 at pci8 dev 20 function 4 not configured
"Intel Xeon-D Memory" rev 0x03 at pci8 dev 20 function 5 not configured
"Intel Xeon-D Memory" rev 0x03 at pci8 dev 20 function 6 not configured
"Intel Xeon-D Memory" rev 0x03 at pci8 dev 20 function 7 not configured
"Intel Xeon-D Memory" rev 0x03 at pci8 dev 21 function 0 not configured
"Intel Xeon-D Memory" rev 0x03 at pci8 dev 21 function 1 not configured
"Intel Xeon-D Memory" rev 0x03 at pci8 dev 21 function 2 not configured
"Intel Xeon-D Memory" rev 0x03 at pci8 dev 21 function 3 not configured
"Intel Xeon-D PCU" rev 0x03 at pci8 dev 30 function 0 not configured
"Intel Xeon-D PCU" rev 0x03 at pci8 dev 30 function 1 not configured
"Intel Xeon-D PCU" rev 0x03 at pci8 dev 30 function 2 not configured
"Intel Xeon-D PCU" rev 0x03 at pci8 dev 30 function 3 not configured
"Intel Xeon-D PCU" rev 0x03 at pci8 dev 30 function 4 not configured
"Intel Xeon-D PCU" rev 0x03 at pci8 dev 31 function 0 not configured
"Intel Xeon-D PCU" rev 0x03 at pci8 dev 31 function 2 not configured
vmm0 at mainbus0: VMX/EPT
uhub3 at uhub0 port 4 configuration 1 interface 0 "ATEN International product 0x7000" rev 2.00/0.00 addr 2
uhidev0 at uhub3 port 1 configuration 1 interface 0 "ATEN International product 0x2419" rev 1.10/1.00 addr 3
uhidev0: iclass 3/1
ukbd0 at uhidev0: 8 variable keys, 6 key codes
wskbd0 at ukbd0: console keyboard, using wsdisplay0
uhidev1 at uhub3 port 1 configuration 1 interface 1 "ATEN International product 0x2419" rev 1.10/1.00 addr 3
uhidev1: iclass 3/1
ums0 at uhidev1: 3 buttons, Z dir
wsmouse0 at ums0 mux 0
uhub4 at uhub1 port 1 configuration 1 interface 0 "Intel Rate Matching Hub" rev 2.00/0.05 addr 2
uhub5 at uhub2 port 1 configuration 1 interface 0 "Intel Rate Matching Hub" rev 2.00/0.05 addr 2
uhub6 at uhub5 port 3 configuration 1 interface 0 "Genesys Logic USB2.0 Hub" rev 2.00/32.98 addr 3
vscsi0 at root
scsibus2 at vscsi0: 256 targets
softraid0 at root
scsibus3 at softraid0: 256 targets
sd4 at scsibus3 targ 1 lun 0: <OPENBSD, SR RAID 1, 006> SCSI2 0/direct fixed
sd4: 457860MB, 512 bytes/sector, 937697393 sectors
root on sd4a (878cd8842d071086.a) swap on sd4b dump on sd4b

Reply | Threaded
Open this post in threaded view
|

Re: VM disk image performance degraded on rewrite of blocks

Mike Larkin
On Thu, Jan 25, 2018 at 01:58:08PM +0000, openbsd2012 wrote:
> I've recently noticed that VM disk image performance is degraded when blocks are rewritten within a disk image by the guest OS. I suspect this may be related to the thin provisioning of vmctl VM disk images.
>
>
> On my test machine, first writes to blocks with dd if=/dev/zero are within the range of 175 MB/s. Subsequent writes are anywhere from 25MB/s to 70 MB/s.
>

I'm not surprised. There are probably some inefficiencies in the VFS layer.

For what it's worth, vmd(8) doesn't care about how the file was provisioned. It
always uses the same pwrite(2) call every time.

-ml

>
> Test conditions:
>
> - Jan 24 snapshot
>
> - vmd host running single VM and no other load (dmesg at end of email)
>
> - vmctl create /home/test.vm.fs -s 8G
>
> - booted an existing VM with /home/test.vm.fs attached as sd1 (tested both 6.2-stable and 6.2-current VMs)
>
> - fdisk -iy sd1
>
> - disklabel sd1:
>
>   a:         16771776               64  4.2BSD   2048 16384 12958
> - mount /dev/sd1a /mnt
>
>
> Performance tests:
>
>
> 1. New writes as follows with expected optimal disk performance (NOTE both /dev/zero and /dev/urandom test files):
>
>
> # dd if=/dev/zero of=/mnt/test1 bs=1m count=1024
> 1024+0 records in
> 1024+0 records out
> 1073741824 bytes transferred in 6.161 secs (174276136 bytes/sec)
> # dd if=/dev/urandom of=/mnt/test2 bs=1m count=1024
> 1024+0 records in
> 1024+0 records out
> 1073741824 bytes transferred in 9.838 secs (109132548 bytes/sec)
> # dd if=/dev/zero of=/mnt/test3 bs=1m count=1024
> 1024+0 records in
> 1024+0 records out
> 1073741824 bytes transferred in 5.971 secs (179800958 bytes/sec)
>
> 2. Overwrite /mnt/test1 and performance is degraded:
>
>
> # dd if=/dev/zero of=/mnt/test1 bs=1m count=1024
> 1024+0 records in
> 1024+0 records out
> 1073741824 bytes transferred in 25.994 secs (41305958 bytes/sec)
>
> 3. Write a new file and performance picks up to expected optimal performance:
>
>
> # dd if=/dev/zero of=/mnt/test4 bs=1m count=1024
> 1024+0 records in
> 1024+0 records out
> 1073741824 bytes transferred in 6.004 secs (178828580 bytes/sec)
>
> 4. Rewrite the fourth file and performance is degraded:
>
>
> # dd if=/dev/zero of=/mnt/test4 bs=1m count=1024
> 1024+0 records in
> 1024+0 records out
> 1073741824 bytes transferred in 18.498 secs (58044338 bytes/sec)
>
> 5. Also degraded performance:
>
> # rm /mnt/test2
> # dd if=/dev/zero of=/mnt/test2 bs=1m count=1024
> 1024+0 records in
> 1024+0 records out
> 1073741824 bytes transferred in 17.013 secs (63112746 bytes/sec)
>
> 6. New files are written with optimal performance until disk is full (/mnt/test5 assumed to have encountered some pre-written blocks due to rm /mnt/test2):
>
> # dd if=/dev/zero of=/mnt/test5 bs=1m count=1024
> 1024+0 records in
> 1024+0 records out
> 1073741824 bytes transferred in 8.510 secs (126160945 bytes/sec)
> # dd if=/dev/zero of=/mnt/test6 bs=1m count=1024
> 1024+0 records in
> 1024+0 records out
> 1073741824 bytes transferred in 5.951 secs (180413534 bytes/sec)
> # dd if=/dev/zero of=/mnt/test7 bs=1m count=1024
> 1024+0 records in
> 1024+0 records out
> 1073741824 bytes transferred in 5.988 secs (179291452 bytes/sec)
> # dd if=/dev/zero of=/mnt/test8 bs=1m count=1024
>
> /mnt: write failed, file system is full
> dd: /mnt/test8: No space left on device
> 888+0 records in
> 887+0 records out
> 930086912 bytes transferred in 5.312 secs (175079201 bytes/sec)
> # rm /mnt/*
> # dd if=/dev/zero of=/mnt/test9 bs=1m count=1024
> 1024+0 records in
> 1024+0 records out
> 1073741824 bytes transferred in 27.552 secs (38970531 bytes/sec)
>
> 7. An issue with cp as well as dd:
>
> # dd if=/dev/urandom of=/mnt/testa bs=1m count=1024
> 1024+0 records in
> 1024+0 records out
> 1073741824 bytes transferred in 27.553 secs (38968811 bytes/sec)
> # time cp /mnt/testa /mnt/testb
>     0m31.76s real     0m00.00s user     0m11.79s system
> # time cp /mnt/testa /mnt/testb
>     0m35.91s real     0m00.00s user     0m14.02s system
> # time cp /mnt/testb /mnt/testc
>     0m36.54s real     0m00.00s user     0m13.98s system
>
> 8. From this point forward, all writes to sd1a are degraded.
>
> ------------------------- dmesg -------------------------
>
> OpenBSD 6.2-current (GENERIC.MP) #387: Wed Jan 24 12:53:55 MST 2018
>     [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
> real mem = 68594307072 (65416MB)
> avail mem = 66508509184 (63427MB)
> mpath0 at root
> scsibus0 at mpath0: 256 targets
> mainbus0 at root
> bios0 at mainbus0: SMBIOS rev. 2.8 @ 0xed9b0 (39 entries)
> bios0: vendor American Megatrends Inc. version "1.1b" date 07/13/2016
> bios0: Supermicro Super Server
> acpi0 at bios0: rev 2
> acpi0: sleep states S0 S4 S5
> acpi0: tables DSDT FACP APIC FPDT FIDT SPMI MCFG UEFI DBG2 HPET WDDT SSDT SSDT SSDT PRAD DMAR HEST BERT ERST EINJ
> acpi0: wakeup devices IP2P(S4) XHCI(S4) EHC1(S4) EHC2(S4) RP01(S4) RP02(S4) RP03(S4) RP04(S4) RP05(S4) RP06(S4) RP07(S4) RP08(S4) BR1A(S4) BR1B(S4) BR2A(S4) BR2B(S4) [...]
> acpitimer0 at acpi0: 3579545 Hz, 24 bits
> acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
> cpu0 at mainbus0: apid 0 (boot processor)
> cpu0: Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz, 2100.32 MHz
> cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,SENSOR,ARAT
> cpu0: 256KB 64b/line 8-way L2 cache
> acpitimer0: recalibrated TSC frequency 2100005999 Hz
> cpu0: smt 0, core 0, package 0
> mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges
> cpu0: apic clock running at 99MHz
> cpu0: mwait min=64, max=64, C-substates=0.2.1.2, IBE
> cpu1 at mainbus0: apid 2 (application processor)
> cpu1: Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz, 2100.00 MHz
> cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,SENSOR,ARAT
> cpu1: 256KB 64b/line 8-way L2 cache
> cpu1: smt 0, core 1, package 0
> cpu2 at mainbus0: apid 4 (application processor)
> cpu2: Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz, 2100.00 MHz
> cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,SENSOR,ARAT
> cpu2: 256KB 64b/line 8-way L2 cache
> cpu2: smt 0, core 2, package 0
> cpu3 at mainbus0: apid 6 (application processor)
> cpu3: Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz, 2100.00 MHz
> cpu3: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,SENSOR,ARAT
> cpu3: 256KB 64b/line 8-way L2 cache
> cpu3: smt 0, core 3, package 0
> cpu4 at mainbus0: apid 8 (application processor)
> cpu4: Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz, 2100.00 MHz
> cpu4: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,SENSOR,ARAT
> cpu4: 256KB 64b/line 8-way L2 cache
> cpu4: smt 0, core 4, package 0
> cpu5 at mainbus0: apid 10 (application processor)
> cpu5: Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz, 2100.00 MHz
> cpu5: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,SENSOR,ARAT
> cpu5: 256KB 64b/line 8-way L2 cache
> cpu5: smt 0, core 5, package 0
> cpu6 at mainbus0: apid 12 (application processor)
> cpu6: Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz, 2100.00 MHz
> cpu6: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,SENSOR,ARAT
> cpu6: 256KB 64b/line 8-way L2 cache
> cpu6: smt 0, core 6, package 0
> cpu7 at mainbus0: apid 14 (application processor)
> cpu7: Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz, 2100.00 MHz
> cpu7: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,SENSOR,ARAT
> cpu7: 256KB 64b/line 8-way L2 cache
> cpu7: smt 0, core 7, package 0
> cpu8 at mainbus0: apid 1 (application processor)
> cpu8: Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz, 2100.00 MHz
> cpu8: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,SENSOR,ARAT
> cpu8: 256KB 64b/line 8-way L2 cache
> cpu8: smt 1, core 0, package 0
> cpu9 at mainbus0: apid 3 (application processor)
> cpu9: Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz, 2100.00 MHz
> cpu9: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,SENSOR,ARAT
> cpu9: 256KB 64b/line 8-way L2 cache
> cpu9: smt 1, core 1, package 0
> cpu10 at mainbus0: apid 5 (application processor)
> cpu10: Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz, 2100.00 MHz
> cpu10: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,SENSOR,ARAT
> cpu10: 256KB 64b/line 8-way L2 cache
> cpu10: smt 1, core 2, package 0
> cpu11 at mainbus0: apid 7 (application processor)
> cpu11: Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz, 2100.00 MHz
> cpu11: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,SENSOR,ARAT
> cpu11: 256KB 64b/line 8-way L2 cache
> cpu11: smt 1, core 3, package 0
> cpu12 at mainbus0: apid 9 (application processor)
> cpu12: Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz, 2100.00 MHz
> cpu12: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,SENSOR,ARAT
> cpu12: 256KB 64b/line 8-way L2 cache
> cpu12: smt 1, core 4, package 0
> cpu13 at mainbus0: apid 11 (application processor)
> cpu13: Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz, 2100.00 MHz
> cpu13: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,SENSOR,ARAT
> cpu13: 256KB 64b/line 8-way L2 cache
> cpu13: smt 1, core 5, package 0
> cpu14 at mainbus0: apid 13 (application processor)
> cpu14: Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz, 2100.00 MHz
> cpu14: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,SENSOR,ARAT
> cpu14: 256KB 64b/line 8-way L2 cache
> cpu14: smt 1, core 6, package 0
> cpu15 at mainbus0: apid 15 (application processor)
> cpu15: Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz, 2100.00 MHz
> cpu15: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,RDSEED,ADX,SMAP,PT,SENSOR,ARAT
> cpu15: 256KB 64b/line 8-way L2 cache
> cpu15: smt 1, core 7, package 0
> ioapic0 at mainbus0: apid 8 pa 0xfec00000, version 20, 24 pins
> acpimcfg0 at acpi0 addr 0x80000000, bus 0-255
> acpihpet0 at acpi0: 14318179 Hz
> acpihpet0: recalibrated TSC frequency 2100010800 Hz
> acpiprt0 at acpi0: bus 255 (UNC0)
> acpiprt1 at acpi0: bus 0 (PCI0)
> acpiprt2 at acpi0: bus 6 (RP01)
> acpiprt3 at acpi0: bus 8 (RP05)
> acpiprt4 at acpi0: bus 9 (BR39)
> acpiprt5 at acpi0: bus 1 (BR1A)
> acpiprt6 at acpi0: bus 3 (BR2C)
> acpiprt7 at acpi0: bus 5 (BR3A)
> acpicpu0 at acpi0: C2(350@41 mwait.3@0x20), C1(1016@1 mwait.1), PSS
> acpicpu1 at acpi0: C2(350@41 mwait.3@0x20), C1(1016@1 mwait.1), PSS
> acpicpu2 at acpi0: C2(350@41 mwait.3@0x20), C1(1016@1 mwait.1), PSS
> acpicpu3 at acpi0: C2(350@41 mwait.3@0x20), C1(1016@1 mwait.1), PSS
> acpicpu4 at acpi0: C2(350@41 mwait.3@0x20), C1(1016@1 mwait.1), PSS
> acpicpu5 at acpi0: C2(350@41 mwait.3@0x20), C1(1016@1 mwait.1), PSS
> acpicpu6 at acpi0: C2(350@41 mwait.3@0x20), C1(1016@1 mwait.1), PSS
> acpicpu7 at acpi0: C2(350@41 mwait.3@0x20), C1(1016@1 mwait.1), PSS
> acpicpu8 at acpi0: C2(350@41 mwait.3@0x20), C1(1016@1 mwait.1), PSS
> acpicpu9 at acpi0: C2(350@41 mwait.3@0x20), C1(1016@1 mwait.1), PSS
> acpicpu10 at acpi0: C2(350@41 mwait.3@0x20), C1(1016@1 mwait.1), PSS
> acpicpu11 at acpi0: C2(350@41 mwait.3@0x20), C1(1016@1 mwait.1), PSS
> acpicpu12 at acpi0: C2(350@41 mwait.3@0x20), C1(1016@1 mwait.1), PSS
> acpicpu13 at acpi0: C2(350@41 mwait.3@0x20), C1(1016@1 mwait.1), PSS
> acpicpu14 at acpi0: C2(350@41 mwait.3@0x20), C1(1016@1 mwait.1), PSS
> acpicpu15 at acpi0: C2(350@41 mwait.3@0x20), C1(1016@1 mwait.1), PSS
> "ACPI0004" at acpi0 not configured
> "ACPI0007" at acpi0 not configured
> "ACPI0007" at acpi0 not configured
> "ACPI0007" at acpi0 not configured
> "ACPI0007" at acpi0 not configured
> "ACPI0007" at acpi0 not configured
> "ACPI0007" at acpi0 not configured
> "ACPI0007" at acpi0 not configured
> "ACPI0007" at acpi0 not configured
> "ACPI0007" at acpi0 not configured
> "ACPI0007" at acpi0 not configured
> "ACPI0007" at acpi0 not configured
> "ACPI0007" at acpi0 not configured
> "ACPI0007" at acpi0 not configured
> "ACPI0007" at acpi0 not configured
> "ACPI0007" at acpi0 not configured
> "ACPI0007" at acpi0 not configured
> "PNP0C33" at acpi0 not configured
> "PNP0C14" at acpi0 not configured
> "PNP0003" at acpi0 not configured
> "PNP0C14" at acpi0 not configured
> "IPI0001" at acpi0 not configured
> acpibtn0 at acpi0: PWRB
> ipmi at mainbus0 not configured
> cpu0: Enhanced SpeedStep 2100 MHz: speeds: 2101, 2100, 2000, 1900, 1800, 1700, 1600, 1500, 1400, 1300, 1200, 1100, 1000, 900, 800 MHz
> pci0 at mainbus0 bus 0
> pchb0 at pci0 dev 0 function 0 "Intel Xeon-D Host" rev 0x03
> ppb0 at pci0 dev 1 function 0 "Intel Xeon-D PCIE" rev 0x03: msi
> pci1 at ppb0 bus 1
> ppb1 at pci0 dev 2 function 0 "Intel Xeon-D PCIE" rev 0x03
> pci2 at ppb1 bus 2
> "Intel Xeon-D QuickData" rev 0x00 at pci2 dev 0 function 0 not configured
> "Intel Xeon-D QuickData" rev 0x00 at pci2 dev 0 function 1 not configured
> "Intel Xeon-D QuickData" rev 0x00 at pci2 dev 0 function 2 not configured
> "Intel Xeon-D QuickData" rev 0x00 at pci2 dev 0 function 3 not configured
> ppb2 at pci0 dev 2 function 2 "Intel Xeon-D PCIE" rev 0x03
> pci3 at ppb2 bus 3
> ix0 at pci3 dev 0 function 0 "Intel X552/X557-AT" rev 0x00: msi, address 0c:c4:7a:c5:59:ce
> ix1 at pci3 dev 0 function 1 "Intel X552/X557-AT" rev 0x00: msi, address 0c:c4:7a:c5:59:cf
> ppb3 at pci0 dev 3 function 0 "Intel Xeon-D PCIE" rev 0x03: msi
> pci4 at ppb3 bus 5
> "Intel Xeon-D Address Map" rev 0x03 at pci0 dev 5 function 0 not configured
> "Intel Xeon-D Hot Plug" rev 0x03 at pci0 dev 5 function 1 not configured
> "Intel Xeon-D RAS" rev 0x03 at pci0 dev 5 function 2 not configured
> xhci0 at pci0 dev 20 function 0 "Intel 8 Series xHCI" rev 0x05: msi
> usb0 at xhci0: USB revision 3.0
> uhub0 at usb0 configuration 1 interface 0 "Intel xHCI root hub" rev 3.00/1.00 addr 1
> "Intel 8 Series MEI" rev 0x04 at pci0 dev 22 function 0 not configured
> "Intel 8 Series MEI" rev 0x04 at pci0 dev 22 function 1 not configured
> ehci0 at pci0 dev 26 function 0 "Intel 8 Series USB" rev 0x05: apic 8 int 18
> usb1 at ehci0: USB revision 2.0
> uhub1 at usb1 configuration 1 interface 0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
> ppb4 at pci0 dev 28 function 0 "Intel 8 Series PCIE" rev 0xd5: msi
> pci5 at ppb4 bus 6
> em0 at pci5 dev 0 function 0 "Intel I350" rev 0x01: msi, address 0c:c4:7a:c5:57:4a
> em1 at pci5 dev 0 function 1 "Intel I350" rev 0x01: msi, address 0c:c4:7a:c5:57:4b
> ppb5 at pci0 dev 28 function 4 "Intel 8 Series PCIE" rev 0xd5: msi
> pci6 at ppb5 bus 8
> ppb6 at pci6 dev 0 function 0 "ASPEED Technology AST1150 PCI" rev 0x03
> pci7 at ppb6 bus 9
> vga1 at pci7 dev 0 function 0 "ASPEED Technology AST2000" rev 0x30
> wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
> wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
> ehci1 at pci0 dev 29 function 0 "Intel 8 Series USB" rev 0x05: apic 8 int 18
> usb2 at ehci1: USB revision 2.0
> uhub2 at usb2 configuration 1 interface 0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
> pcib0 at pci0 dev 31 function 0 "Intel C224 LPC" rev 0x05
> ahci0 at pci0 dev 31 function 2 "Intel 8 Series AHCI" rev 0x05: msi, AHCI 1.3
> ahci0: port 0: 6.0Gb/s
> ahci0: port 1: 6.0Gb/s
> ahci0: port 2: 6.0Gb/s
> ahci0: port 3: 6.0Gb/s
> scsibus1 at ahci0: 32 targets
> sd0 at scsibus1 targ 0 lun 0: <ATA, INTEL SSDSC2BB48, G201> SCSI3 0/direct fixed naa.55cd2e414d467cc0
> sd0: 457862MB, 512 bytes/sector, 937703088 sectors, thin
> sd1 at scsibus1 targ 1 lun 0: <ATA, INTEL SSDSC2BB48, G201> SCSI3 0/direct fixed naa.55cd2e414d467c6c
> sd1: 457862MB, 512 bytes/sector, 937703088 sectors, thin
> sd2 at scsibus1 targ 2 lun 0: <ATA, HGST HUS726020AL, A5GN> SCSI3 0/direct fixed naa.5000cca245c2f1f7
> sd2: 1907729MB, 512 bytes/sector, 3907029168 sectors
> sd3 at scsibus1 targ 3 lun 0: <ATA, HGST HUS726020AL, A5GN> SCSI3 0/direct fixed naa.5000cca245c2fae2
> sd3: 1907729MB, 512 bytes/sector, 3907029168 sectors
> ichiic0 at pci0 dev 31 function 3 "Intel 8 Series SMBus" rev 0x05: apic 8 int 18
> iic0 at ichiic0
> "Intel 8 Series Thermal" rev 0x05 at pci0 dev 31 function 6 not configured
> isa0 at pcib0
> isadma0 at isa0
> com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
> com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
> pckbc0 at isa0 port 0x60/5 irq 1 irq 12
> pcppi0 at isa0 port 0x61
> spkr0 at pcppi0
> pci8 at mainbus0 bus 255
> "Intel Xeon-D QPI Link" rev 0x03 at pci8 dev 11 function 0 not configured
> "Intel Xeon-D QPI Link" rev 0x03 at pci8 dev 11 function 1 not configured
> "Intel Xeon-D QPI Link" rev 0x03 at pci8 dev 11 function 2 not configured
> "Intel Xeon-D QPI Debug" rev 0x03 at pci8 dev 11 function 3 not configured
> "Intel Xeon-D Cache" rev 0x03 at pci8 dev 12 function 0 not configured
> "Intel Xeon-D Cache" rev 0x03 at pci8 dev 12 function 1 not configured
> "Intel Xeon-D Cache" rev 0x03 at pci8 dev 12 function 2 not configured
> "Intel Xeon-D Cache" rev 0x03 at pci8 dev 12 function 3 not configured
> "Intel E5 v4 Cache" rev 0x03 at pci8 dev 12 function 4 not configured
> "Intel E5 v4 Cache" rev 0x03 at pci8 dev 12 function 5 not configured
> vendor "Intel", unknown product 0x6fe6 (class system subclass miscellaneous, rev 0x03) at pci8 dev 12 function 6 not configured
> vendor "Intel", unknown product 0x6fe7 (class system subclass miscellaneous, rev 0x03) at pci8 dev 12 function 7 not configured
> "Intel Xeon-D Cache" rev 0x03 at pci8 dev 15 function 0 not configured
> "Intel Xeon-D Cache" rev 0x03 at pci8 dev 15 function 4 not configured
> "Intel Xeon-D Cache" rev 0x03 at pci8 dev 15 function 5 not configured
> "Intel Xeon-D Cache" rev 0x03 at pci8 dev 15 function 6 not configured
> "Intel Xeon-D PCIE" rev 0x03 at pci8 dev 16 function 0 not configured
> "Intel E5 v4 R2PCIe Agent" rev 0x03 at pci8 dev 16 function 1 not configured
> "Intel Xeon-D Ubox" rev 0x03 at pci8 dev 16 function 5 not configured
> "Intel Xeon-D Ubox" rev 0x03 at pci8 dev 16 function 6 not configured
> "Intel Xeon-D Ubox" rev 0x03 at pci8 dev 16 function 7 not configured
> "Intel Xeon-D Home Agent" rev 0x03 at pci8 dev 18 function 0 not configured
> "Intel Xeon-D Home Agent" rev 0x03 at pci8 dev 18 function 1 not configured
> "Intel Xeon-D Memory" rev 0x03 at pci8 dev 19 function 0 not configured
> "Intel Xeon-D Memory" rev 0x03 at pci8 dev 19 function 1 not configured
> "Intel Xeon-D Memory" rev 0x03 at pci8 dev 19 function 2 not configured
> "Intel Xeon-D Memory" rev 0x03 at pci8 dev 19 function 3 not configured
> "Intel Xeon-D Memory" rev 0x03 at pci8 dev 19 function 4 not configured
> "Intel Xeon-D Memory" rev 0x03 at pci8 dev 19 function 5 not configured
> "Intel Xeon-D Memory" rev 0x03 at pci8 dev 19 function 6 not configured
> "Intel Xeon-D Memory" rev 0x03 at pci8 dev 19 function 7 not configured
> "Intel Xeon-D Memory" rev 0x03 at pci8 dev 20 function 0 not configured
> "Intel Xeon-D Memory" rev 0x03 at pci8 dev 20 function 1 not configured
> "Intel Xeon-D Memory" rev 0x03 at pci8 dev 20 function 2 not configured
> "Intel Xeon-D Memory" rev 0x03 at pci8 dev 20 function 3 not configured
> "Intel Xeon-D Memory" rev 0x03 at pci8 dev 20 function 4 not configured
> "Intel Xeon-D Memory" rev 0x03 at pci8 dev 20 function 5 not configured
> "Intel Xeon-D Memory" rev 0x03 at pci8 dev 20 function 6 not configured
> "Intel Xeon-D Memory" rev 0x03 at pci8 dev 20 function 7 not configured
> "Intel Xeon-D Memory" rev 0x03 at pci8 dev 21 function 0 not configured
> "Intel Xeon-D Memory" rev 0x03 at pci8 dev 21 function 1 not configured
> "Intel Xeon-D Memory" rev 0x03 at pci8 dev 21 function 2 not configured
> "Intel Xeon-D Memory" rev 0x03 at pci8 dev 21 function 3 not configured
> "Intel Xeon-D PCU" rev 0x03 at pci8 dev 30 function 0 not configured
> "Intel Xeon-D PCU" rev 0x03 at pci8 dev 30 function 1 not configured
> "Intel Xeon-D PCU" rev 0x03 at pci8 dev 30 function 2 not configured
> "Intel Xeon-D PCU" rev 0x03 at pci8 dev 30 function 3 not configured
> "Intel Xeon-D PCU" rev 0x03 at pci8 dev 30 function 4 not configured
> "Intel Xeon-D PCU" rev 0x03 at pci8 dev 31 function 0 not configured
> "Intel Xeon-D PCU" rev 0x03 at pci8 dev 31 function 2 not configured
> vmm0 at mainbus0: VMX/EPT
> uhub3 at uhub0 port 4 configuration 1 interface 0 "ATEN International product 0x7000" rev 2.00/0.00 addr 2
> uhidev0 at uhub3 port 1 configuration 1 interface 0 "ATEN International product 0x2419" rev 1.10/1.00 addr 3
> uhidev0: iclass 3/1
> ukbd0 at uhidev0: 8 variable keys, 6 key codes
> wskbd0 at ukbd0: console keyboard, using wsdisplay0
> uhidev1 at uhub3 port 1 configuration 1 interface 1 "ATEN International product 0x2419" rev 1.10/1.00 addr 3
> uhidev1: iclass 3/1
> ums0 at uhidev1: 3 buttons, Z dir
> wsmouse0 at ums0 mux 0
> uhub4 at uhub1 port 1 configuration 1 interface 0 "Intel Rate Matching Hub" rev 2.00/0.05 addr 2
> uhub5 at uhub2 port 1 configuration 1 interface 0 "Intel Rate Matching Hub" rev 2.00/0.05 addr 2
> uhub6 at uhub5 port 3 configuration 1 interface 0 "Genesys Logic USB2.0 Hub" rev 2.00/32.98 addr 3
> vscsi0 at root
> scsibus2 at vscsi0: 256 targets
> softraid0 at root
> scsibus3 at softraid0: 256 targets
> sd4 at scsibus3 targ 1 lun 0: <OPENBSD, SR RAID 1, 006> SCSI2 0/direct fixed
> sd4: 457860MB, 512 bytes/sector, 937697393 sectors
> root on sd4a (878cd8842d071086.a) swap on sd4b dump on sd4b
>