cannot clean-install KVM/QEMU VM that don't support MSR_TSX_CTRL

classic Classic list List threaded Threaded
4 messages Options
uaa
Reply | Threaded
Open this post in threaded view
|

cannot clean-install KVM/QEMU VM that don't support MSR_TSX_CTRL

uaa
>Synopsis: cannot clean-install KVM/QEMU VM that don't support MSR_TSX_CTRL
>Category: kernel
>Environment:
        System      : OpenBSD 6.7
        Details     : OpenBSD 6.7 (GENERIC.MP) #2: Thu May 21 18:28:46 JST 2020
                         [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP

        Architecture: OpenBSD.amd64
        Machine     : amd64
>Description:
        cpu_tsx_disable() in sys/arch/amd64/amd64/cpu.c tries to set
        MSR_TSX_CTRL register, there is no problem with "real" CPU.
        But under KVM/QEMU, OpenBSD-6.7 will crash if they don't handle
        that MSR register.
        There is no way to by-pass cpu_tsx_disable(), we cannot run
        official binary on old KVM/QEMU host.
>How-To-Repeat:
        simply try to boot OpenBSD-6.7/amd64's bsd.rd, bsd.sp and bsd.mp
>Fix:
        update KVM/QEMU. otherwise upgrade from OpenBSD-6.6 and following
        workaround procedure.

        1) install OpenBSD-6.6
        2) fetch OpenBSD-6.7 kernel source code
        3) modify sys/arch/amd64/amd64/cpu.c to disable cpu_tsx_disable()
        4) build OpenBSD-6.7 GENERIC.MP kernel (as bsd.mp.tmp)
        5) build OpenBSD-6.7 RAMDISK_CD kernel (as bsd.rd.tmp)
        6) add OpenBSD-6.7 installer from bsd.rd to bsd.rd.tmp by rdsetroot
  7) boot bsd.rd.tmp and upgrade
        8) boot bsd.mp.tmp
        9) rebuild OpenBSD-6.7 GENERIC.MP kernel and install

        attached dmesg is modified kernel, by this patch

--- cpu.c~ Thu May 21 20:27:36 2020
+++ cpu.c Thu May 21 18:12:58 2020
@@ -1175,9 +1175,10 @@
     (sefflags_edx & SEFF0EDX_ARCH_CAP)) {
  msr = rdmsr(MSR_ARCH_CAPABILITIES);
  if (msr & ARCH_CAPABILITIES_TSX_CTRL) {
- msr = rdmsr(MSR_TSX_CTRL);
- msr |= TSX_CTRL_RTM_DISABLE | TSX_CTRL_TSX_CPUID_CLEAR;
- wrmsr(MSR_TSX_CTRL, msr);
+ printf("%s: modifying MSR_TSX_CTRL bypassed\n", ci->ci_dev->dv_xname);
+// msr = rdmsr(MSR_TSX_CTRL);
+// msr |= TSX_CTRL_RTM_DISABLE | TSX_CTRL_TSX_CPUID_CLEAR;
+// wrmsr(MSR_TSX_CTRL, msr);
  }
  }
 }


dmesg:
OpenBSD 6.7 (GENERIC.MP) #2: Thu May 21 18:28:46 JST 2020
    [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 1056940032 (1007MB)
avail mem = 1012346880 (965MB)
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.4 @ 0xf6200 (11 entries)
bios0: vendor Seabios version "0.5.1" date 01/01/2011
bios0: Red Hat KVM
acpi0 at bios0: ACPI 1.0
acpi0: sleep states S5
acpi0: tables DSDT FACP SSDT APIC
acpi0: wakeup devices
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel(R) Xeon(R) Gold 6212U CPU @ 2.40GHz, 344.08 MHz, 06-55-07
cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,SSE3,PCLMUL,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,FSGSBASE,TSC_ADJUST,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,AVX512F,AVX512DQ,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,AVX512CD,AVX512BW,AVX512VL,PKU,MD_CLEAR,IBRS,IBPB,STIBP,SSBD,XSAVEOPT,XSAVEC,XGETBV1
cpu0: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 512KB 64b/line 16-way L2 cache
cpu0: ITLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu0: DTLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
cpu0: apic clock running at 1000MHz
cpu1 at mainbus0: apid 1 (application processor)
cpu1: modifying MSR_TSX_CTRL bypassed
cpu1: Intel(R) Xeon(R) Gold 6212U CPU @ 2.40GHz, 673.84 MHz, 06-55-07
cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,SSE3,PCLMUL,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,FSGSBASE,TSC_ADJUST,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,AVX512F,AVX512DQ,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,AVX512CD,AVX512BW,AVX512VL,PKU,MD_CLEAR,IBRS,IBPB,STIBP,SSBD,XSAVEOPT,XSAVEC,XGETBV1
cpu1: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 512KB 64b/line 16-way L2 cache
cpu1: ITLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu1: DTLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu1: smt 0, core 0, package 1
ioapic0 at mainbus0: apid 0 pa 0xfec00000, version 11, 24 pins
acpiprt0 at acpi0: bus 0 (PCI0)
acpicpu0 at acpi0: C1(@1 halt!)
acpicpu1 at acpi0: C1(@1 halt!)
"ACPI0006" at acpi0 not configured
acpipci0 at acpi0 PCI0: _OSC failed
acpicmos0 at acpi0
"QEMU0002" at acpi0 not configured
"ACPI0007" at acpi0 not configured
"ACPI0007" at acpi0 not configured
pvbus0 at mainbus0: KVM
pvclock0 at pvbus0
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel 82441FX" rev 0x02
pcib0 at pci0 dev 1 function 0 "Intel 82371SB ISA" rev 0x00
pciide0 at pci0 dev 1 function 1 "Intel 82371SB IDE" rev 0x00: DMA, channel 0 wired to compatibility, channel 1 wired to compatibility
pciide0: channel 0 disabled (no drives)
atapiscsi0 at pciide0 channel 1 drive 0
scsibus1 at atapiscsi0: 2 targets
cd0 at scsibus1 targ 0 lun 0: <QEMU, QEMU DVD-ROM, 1.5.> removable
cd0(pciide0:1:0): using PIO mode 4, DMA mode 2
uhci0 at pci0 dev 1 function 2 "Intel 82371SB USB" rev 0x01: apic 0 int 11
piixpm0 at pci0 dev 1 function 3 "Intel 82371AB Power" rev 0x03: apic 0 int 9
iic0 at piixpm0
iic0: addr 0x18 00=0c 01=91 02=91 03=91 04=91 05=91 06=91 07=91 08=91 09=85 0a=e2 3e=0c 48=0c 4a=0c 4e=0c fc=0c fe=0c words 00=ffff 01=ffff 02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x19 3e=0c 48=0c 4a=0c 4e=0c fc=0c fe=0c words 00=ffff 01=ffff 02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x1a 00=0c 01=91 02=91 03=91 04=91 05=91 06=91 07=91 08=91 09=85 0a=e2 3e=0c 48=0c 4a=0c 4e=0c fc=0c fe=0c words 00=ffff 01=ffff 02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x1b 3e=0c 48=0c 4a=0c 4e=0c fc=0c fe=0c words 00=ffff 01=ffff 02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x1c 0f=06 3e=0c 48=0c 4a=0c 4e=0c fc=0c fe=0c words 00=ffff 01=ffff 02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x1d 0f=06 3e=0c 48=0c 4a=0c 4e=0c fc=0c fe=0c words 00=ffff 01=ffff 02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x1e 3e=0c 48=0c 4a=0c 4e=0c fc=0c fe=0c words 00=ffff 01=ffff 02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x1f 3e=0c 48=0c 4a=0c 4e=0c fc=0c fe=0c words 00=ffff 01=ffff 02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x20 3e=0c 48=0c 4a=0c 4e=0c fc=0c fe=0c words 00=ffff 01=ffff 02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x21 3e=0c 48=0c 4a=0c 4e=0c fc=0c fe=0c words 00=ffff 01=ffff 02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x22 3e=0c 48=0c 4a=0c 4e=0c fc=0c fe=0c words 00=ffff 01=ffff 02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x23 3e=0c 48=0c 4a=0c 4e=0c fc=0c fe=0c words 00=ffff 01=ffff 02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x24 3e=0c 48=0c 4a=0c 4e=0c fc=0c fe=0c words 00=ffff 01=ffff 02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x25 3e=0c 48=0c 4a=0c 4e=0c fc=0c fe=0c words 00=ffff 01=ffff 02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x26 3e=0c 48=0c 4a=0c 4e=0c fc=0c fe=0c words 00=ffff 01=ffff 02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x27 3e=0c 48=0c 4a=0c 4e=0c fc=0c fe=0c words 00=ffff 01=ffff 02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x28 3e=0c 48=0c 4a=0c 4e=0c 4f=0c fc=0c fe=0c words 00=ffff 01=ffff 02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x29 00=0c 01=91 02=91 03=91 04=91 05=91 06=91 07=91 08=91 09=85 0a=e2 3e=0c 48=0c 4a=0c 4e=0c 4f=0c fc=0c fe=0c words 00=ffff 01=ffff 02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x2a 3e=0c 48=0c 4a=0c 4e=0c 4f=0c fc=0c fe=0c words 00=ffff 01=ffff 02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x2b 00=0c 01=91 02=91 03=91 04=91 05=91 06=91 07=91 08=91 09=85 0a=e2 3e=0c 48=0c 4a=0c 4e=0c 4f=0c fc=0c fe=0c words 00=ffff 01=ffff 02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x2c 00=0c 0e=0c 16=0c 3e=0c 48=0c 4a=0c 4e=0c 4f=0c fc=0c fe=0c words 00=ffff 01=ffff 02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x2d 00=0c 04=0c 0e=0c 16=0c 3e=0c 48=0c 4a=0c 4e=0c 4f=0c fc=0c fe=0c words 00=ffff 01=ffff 02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x2e 00=0c 04=0c 0e=0c 16=0c 3e=0c 48=0c 4a=0c 4e=0c 4f=0c fc=0c fe=0c words 00=ffff 01=ffff 02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x2f 00=0c 0e=0c 3e=0c 48=0c 4a=0c 4e=0c 4f=0c fc=0c fe=0c words 00=ffff 01=ffff 02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x48 3e=0c 48=0c 4a=0c 4e=0c fc=0c fe=0c words 00=ffff 01=ffff 02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x49 3e=0c 48=0c 4a=0c 4e=0c fc=0c fe=0c words 00=ffff 01=ffff 02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x4a 3e=0c 48=0c 4a=0c 4e=0c fc=0c fe=0c words 00=ffff 01=ffff 02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x4b 3e=0c 48=0c 4a=0c 4e=0c fc=0c fe=0c words 00=ffff 01=ffff 02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x4c 00=00 01=00 02=00 03=00 04=00 05=00 06=00 07=00 08=00 3e=0c 48=0c 4a=0c 4e=0c fc=0c fe=0c words 00=ffff 01=ffff 02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x4d 3e=0c 48=0c 4a=0c 4e=0c fc=0c fe=0c words 00=ffff 01=ffff 02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
iic0: addr 0x4e 00=00 01=00 02=00 03=00 04=00 05=00 06=00 07=00 08=00 3e=0c 48=0c 4a=0c 4e=0c fc=0c fe=0c words 00=ffff 01=ffff 02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
vga1 at pci0 dev 2 function 0 "Bochs VGA" rev 0x00
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
virtio0 at pci0 dev 3 function 0 "Qumranet Virtio Network" rev 0x00
vio0 at virtio0: address 9c:a3:ba:02:00:3c
virtio0: msix shared
virtio1 at pci0 dev 4 function 0 "Qumranet Virtio Network" rev 0x00
vio1 at virtio1: address 9c:a3:ba:03:86:dc
virtio1: msix shared
virtio2 at pci0 dev 5 function 0 "Qumranet Virtio Network" rev 0x00
vio2 at virtio2: address 9c:a3:ba:05:0d:7c
virtio2: msix shared
virtio3 at pci0 dev 6 function 0 "Qumranet Virtio Storage" rev 0x00
vioblk0 at virtio3
scsibus2 at vioblk0: 2 targets
sd0 at scsibus2 targ 0 lun 0: <VirtIO, Block Device, >
sd0: 102400MB, 512 bytes/sector, 209715200 sectors
virtio3: msix shared
virtio4 at pci0 dev 7 function 0 "Qumranet Virtio Memory Balloon" rev 0x00
viomb0 at virtio4
virtio4: apic 0 int 11
isa0 at pcib0
isadma0 at isa0
fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
pckbc0 at isa0 port 0x60/5 irq 1 irq 12
pckbd0 at pckbc0 (kbd slot)
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pms0 at pckbc0 (aux slot)
wsmouse0 at pms0 mux 0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
usb0 at uhci0: USB revision 1.0
uhub0 at usb0 configuration 1 interface 0 "Intel UHCI root hub" rev 1.00/1.00 addr 1
uhidev0 at uhub0 port 1 configuration 1 interface 0 "QEMU QEMU USB Tablet" rev 2.00/0.00 addr 2
uhidev0: iclass 3/0
ums0 at uhidev0: 3 buttons, Z dir
wsmouse1 at ums0 mux 0
vscsi0 at root
scsibus3 at vscsi0: 256 targets
softraid0 at root
scsibus4 at softraid0: 256 targets
root on sd0a (a279a6bea6e758ae.a) swap on sd0b dump on sd0b
fd0 at fdc0 drive 1: density unknown

usbdevs:
Controller /dev/usb0:
addr 01: 8086:0000 Intel, UHCI root hub
         full speed, self powered, config 1, rev 1.00
         driver: uhub0
addr 02: 0627:0001 QEMU, QEMU USB Tablet
         full speed, power 100 mA, config 1, rev 0.00, iSerial 42
         driver: uhidev0

Reply | Threaded
Open this post in threaded view
|

Re: cannot clean-install KVM/QEMU VM that don't support MSR_TSX_CTRL

Theo de Raadt-2
[hidden email] wrote:

> >Synopsis: cannot clean-install KVM/QEMU VM that don't support MSR_TSX_CTRL
> >Category: kernel
> >Environment:
> System      : OpenBSD 6.7
> Details     : OpenBSD 6.7 (GENERIC.MP) #2: Thu May 21 18:28:46 JST 2020
> [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
>
> Architecture: OpenBSD.amd64
> Machine     : amd64
> >Description:
> cpu_tsx_disable() in sys/arch/amd64/amd64/cpu.c tries to set
> MSR_TSX_CTRL register, there is no problem with "real" CPU.
> But under KVM/QEMU, OpenBSD-6.7 will crash if they don't handle
> that MSR register.
> There is no way to by-pass cpu_tsx_disable(), we cannot run
> official binary on old KVM/QEMU host.

1) Broken emulator
2) Old broken emulator

A real cpu behaves that way.  The capability bits says the feature
exists, and when it exists, it MUST work.

If any emulator is passing bits on from a real cpu, and then not
handling them, that emulator is *completely broken*.  It is failing
to emulate what it claims to emulate.

Have you have filed a bug with the authors of the emulator?

If newer code emulators have it fixed, then again, how is this our
fault for using the feature as advertised in all real hardware?


Reply | Threaded
Open this post in threaded view
|

Re: cannot clean-install KVM/QEMU VM that don't support MSR_TSX_CTRL

SASANO Takayoshi
> 1) Broken emulator
> 2) Old broken emulator
>
> A real cpu behaves that way.  The capability bits says the feature
> exists, and when it exists, it MUST work.

Yes, I think KVM/QEMU may be malfunction and I found a patch for
QEMU that supports MSR_TSX_CTRL.

https://www.mail-archive.com/qemu-devel@.../msg660546.html

> Have you have filed a bug with the authors of the emulator?

So I didn't write any bug report to them.

> If newer code emulators have it fixed, then again, how is this our
> fault for using the feature as advertised in all real hardware?

It may be no problem that running KVM/QEMU locally (simply install new
emulator by myself), but I found this MSR_TSX_CTRL problem on VPS service.

I understand that I have to consult support desk of the service provider and
ask them to update QEMU.

Regards,
--
SASANO Takayoshi (JG1UAA) <[hidden email]>

Reply | Threaded
Open this post in threaded view
|

Re: cannot clean-install KVM/QEMU VM that don't support MSR_TSX_CTRL

Theo de Raadt-2
Reading that diff, I get a sense they pass on underlying-hardware
cpu flags as-is, and then only write the support code when they feel
like it.

If so, that is ridiculous.  They should immediately mask against a list
of KNOWN and CURRENTLY SUPPORTED bits, and not pass on unknown stuff.

SASANO Takayoshi <[hidden email]> wrote:

> > 1) Broken emulator
> > 2) Old broken emulator
> >
> > A real cpu behaves that way.  The capability bits says the feature
> > exists, and when it exists, it MUST work.
>
> Yes, I think KVM/QEMU may be malfunction and I found a patch for
> QEMU that supports MSR_TSX_CTRL.
>
> https://www.mail-archive.com/qemu-devel@.../msg660546.html
>
> > Have you have filed a bug with the authors of the emulator?
>
> So I didn't write any bug report to them.
>
> > If newer code emulators have it fixed, then again, how is this our
> > fault for using the feature as advertised in all real hardware?
>
> It may be no problem that running KVM/QEMU locally (simply install new
> emulator by myself), but I found this MSR_TSX_CTRL problem on VPS service.
>
> I understand that I have to consult support desk of the service provider and
> ask them to update QEMU.
>
> Regards,
> --
> SASANO Takayoshi (JG1UAA) <[hidden email]>