Unable to boot OpenBSD within QEMU on an Intel Platinum 8176M

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Unable to boot OpenBSD within QEMU on an Intel Platinum 8176M

Brian Rak-2
I have a server with an Intel Platinum CPU:
https://ark.intel.com/products/120505/Intel-Xeon-Platinum-8176M-Processor-38_5M-Cache-2_10-GHz

It's running Fedora 27 Server, kernel version 4.14.8-300.fc27.x86_64,
qemu-system-x86-core-2.10.1-2.fc27.x86_64

I'm starting qemu like this:

/usr/bin/qemu-system-x86_64 -machine accel=kvm -name
guest=test,debug-threads=on -S -object
secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-6-test/master-key.aes
-machine pc-i440fx-2.10,accel=kvm,usb=off,dump-guest-core=off -cpu
Skylake-Client,hypervisor=on -m 32768 -realtime mlock=off -smp
16,sockets=2,cores=8,threads=1 -uuid
6427e485-5aee-4fb6-b5e5-a80c1dc0f4af -no-user-config -nodefaults
-chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-6-test/monitor.sock,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc
base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=delay
-no-shutdown -boot strict=on -device
piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive
file=/var/tmp/cd62.iso,format=raw,if=none,id=drive-ide0-1-0,readonly=on
-device
ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1
-netdev tap,fd=29,id=hostnet0,vhost=on,vhostfd=32 -device
virtio-net-pci,netdev=hostnet0,id=net0,mac=56:00:00:27:d6:3f,bus=pci.0,addr=0x3,rombar=0,bootindex=3
-device usb-tablet,id=input0,bus=usb.0,port=1 -vnc
127.0.0.1:4788,websocket=40688 -device
cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -object
rng-random,id=objrng0,filename=/dev/random -device
virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x6 -msg timestamp=on

I do not have a functional OpenBSD install here, this happens even when
I try to boot off the ISO.

OpenBSD hangs after printing:

pciide0 at pci0 dev 1 function 1 "Intel 82371SB IDE" rev 0x00: DMA,
channel 0 wired to compatibility, channel 1 wired to compatibility

There is a flashing "_" cursor, but I'm unable to interact with it in
any way.

I tried setting this to use an older CPU type, which changed the -cpu
flag to be "-cpu Nehalem,vme=off,x2apic=off,hypervisor=off", but this
did not seem to have any effect.

If I boot the VM with 'boot -c', then 'disable pciide*', I can actually
get to the 'Welcome to the OpenBSD/amd64 6.2 installation program'
prompt, but then the machine hangs whenever I type 'A'. If I choose
another option ('I'), I can get partially through the install before it
hangs.  The hangs at that point seem to be random.

I've attached a screenshot of where it's hanging during the initial boot.

So far we've been able to reproduce this on all of our Intel Scalable
processors, which includes a few other Gold CPUs.  This does work ok on
our older E5 CPUs


2017-12-27 13_06_28-QEMU (test) - TightVNC Viewer.png (97K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Unable to boot OpenBSD within QEMU on an Intel Platinum 8176M

Mike Larkin
On Wed, Dec 27, 2017 at 01:10:13PM -0500, Brian Rak wrote:

> I have a server with an Intel Platinum CPU: https://ark.intel.com/products/120505/Intel-Xeon-Platinum-8176M-Processor-38_5M-Cache-2_10-GHz
>
> It's running Fedora 27 Server, kernel version 4.14.8-300.fc27.x86_64,
> qemu-system-x86-core-2.10.1-2.fc27.x86_64
>
> I'm starting qemu like this:
>
> /usr/bin/qemu-system-x86_64 -machine accel=kvm -name
> guest=test,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-6-test/master-key.aes
> -machine pc-i440fx-2.10,accel=kvm,usb=off,dump-guest-core=off -cpu
> Skylake-Client,hypervisor=on -m 32768 -realtime mlock=off -smp
> 16,sockets=2,cores=8,threads=1 -uuid 6427e485-5aee-4fb6-b5e5-a80c1dc0f4af
> -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-6-test/monitor.sock,server,nowait
> -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew
> -global kvm-pit.lost_tick_policy=delay -no-shutdown -boot strict=on -device
> piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive
> file=/var/tmp/cd62.iso,format=raw,if=none,id=drive-ide0-1-0,readonly=on
> -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1
> -netdev tap,fd=29,id=hostnet0,vhost=on,vhostfd=32 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=56:00:00:27:d6:3f,bus=pci.0,addr=0x3,rombar=0,bootindex=3
> -device usb-tablet,id=input0,bus=usb.0,port=1 -vnc
> 127.0.0.1:4788,websocket=40688 -device
> cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device
> virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -object
> rng-random,id=objrng0,filename=/dev/random -device
> virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x6 -msg timestamp=on

Could you try a simpler config? There's a lot in there that's not needed for
OpenBSD, and I'm wondering if there is some option you've chosen that's
causing problems.

For what it's worth, there have been a number of reports of OpenBSD guests
recently failing when run on KVM (mainly clock related issues but other
things as well). You are using a very new CPU on a very new host OS, I'm not
surprised some things are behaving a bit strangely.

-ml


>
> I do not have a functional OpenBSD install here, this happens even when I
> try to boot off the ISO.
>
> OpenBSD hangs after printing:
>
> pciide0 at pci0 dev 1 function 1 "Intel 82371SB IDE" rev 0x00: DMA, channel
> 0 wired to compatibility, channel 1 wired to compatibility
>
> There is a flashing "_" cursor, but I'm unable to interact with it in any
> way.
>
> I tried setting this to use an older CPU type, which changed the -cpu flag
> to be "-cpu Nehalem,vme=off,x2apic=off,hypervisor=off", but this did not
> seem to have any effect.
>
> If I boot the VM with 'boot -c', then 'disable pciide*', I can actually get
> to the 'Welcome to the OpenBSD/amd64 6.2 installation program' prompt, but
> then the machine hangs whenever I type 'A'. If I choose another option
> ('I'), I can get partially through the install before it hangs.  The hangs
> at that point seem to be random.
>
> I've attached a screenshot of where it's hanging during the initial boot.
>
> So far we've been able to reproduce this on all of our Intel Scalable
> processors, which includes a few other Gold CPUs.  This does work ok on our
> older E5 CPUs
>

Reply | Threaded
Open this post in threaded view
|

Re: Unable to boot OpenBSD within QEMU on an Intel Platinum 8176M

Brian Rak-2


On 12/30/2017 5:13 PM, Mike Larkin wrote:

> On Wed, Dec 27, 2017 at 01:10:13PM -0500, Brian Rak wrote:
>> I have a server with an Intel Platinum CPU: https://ark.intel.com/products/120505/Intel-Xeon-Platinum-8176M-Processor-38_5M-Cache-2_10-GHz
>>
>> It's running Fedora 27 Server, kernel version 4.14.8-300.fc27.x86_64,
>> qemu-system-x86-core-2.10.1-2.fc27.x86_64
>>
>> I'm starting qemu like this:
>>
>> /usr/bin/qemu-system-x86_64 -machine accel=kvm -name
>> guest=test,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-6-test/master-key.aes
>> -machine pc-i440fx-2.10,accel=kvm,usb=off,dump-guest-core=off -cpu
>> Skylake-Client,hypervisor=on -m 32768 -realtime mlock=off -smp
>> 16,sockets=2,cores=8,threads=1 -uuid 6427e485-5aee-4fb6-b5e5-a80c1dc0f4af
>> -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-6-test/monitor.sock,server,nowait
>> -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew
>> -global kvm-pit.lost_tick_policy=delay -no-shutdown -boot strict=on -device
>> piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive
>> file=/var/tmp/cd62.iso,format=raw,if=none,id=drive-ide0-1-0,readonly=on
>> -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1
>> -netdev tap,fd=29,id=hostnet0,vhost=on,vhostfd=32 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=56:00:00:27:d6:3f,bus=pci.0,addr=0x3,rombar=0,bootindex=3
>> -device usb-tablet,id=input0,bus=usb.0,port=1 -vnc
>> 127.0.0.1:4788,websocket=40688 -device
>> cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device
>> virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -object
>> rng-random,id=objrng0,filename=/dev/random -device
>> virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x6 -msg timestamp=on
> Could you try a simpler config? There's a lot in there that's not needed for
> OpenBSD, and I'm wondering if there is some option you've chosen that's
> causing problems.
>
> For what it's worth, there have been a number of reports of OpenBSD guests
> recently failing when run on KVM (mainly clock related issues but other
> things as well). You are using a very new CPU on a very new host OS, I'm not
> surprised some things are behaving a bit strangely.
>
> -ml
The simplest command line I can come up with is:

/usr/bin/qemu-system-x86_64 -machine accel=kvm -name guest=test -machine
pc-i440fx-2.10 -cpu Skylake-Client -m 32768 -drive
file=/var/tmp/cd62.iso,format=raw,if=none,id=drive-ide0-1-0,readonly=on
-device
ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1
-device cirrus-vga,id=video0,bus=pci.0,addr=0x3 -vnc 127.0.0.1:4788

I still see the same hang.  Even using more ancient virtual CPUs (-cpu
pentium3) doesn't seem to help.

Switching to '-machine pc-q35-2.10' doesn't appear to help either.

If I disable KVM, the instance appears to boot normally.  However, I'm
not sure if this points to a KVM issue or if it's just masking a timing
issue (because it runs so much slower emulated)

In this case, we went with Fedora 27 to rule out a problem that would be
fixed by upgrading some part of the host OS.  This isn't an OS we
normally use, and we've seen issues with older versions of qemu.


>
>> I do not have a functional OpenBSD install here, this happens even when I
>> try to boot off the ISO.
>>
>> OpenBSD hangs after printing:
>>
>> pciide0 at pci0 dev 1 function 1 "Intel 82371SB IDE" rev 0x00: DMA, channel
>> 0 wired to compatibility, channel 1 wired to compatibility
>>
>> There is a flashing "_" cursor, but I'm unable to interact with it in any
>> way.
>>
>> I tried setting this to use an older CPU type, which changed the -cpu flag
>> to be "-cpu Nehalem,vme=off,x2apic=off,hypervisor=off", but this did not
>> seem to have any effect.
>>
>> If I boot the VM with 'boot -c', then 'disable pciide*', I can actually get
>> to the 'Welcome to the OpenBSD/amd64 6.2 installation program' prompt, but
>> then the machine hangs whenever I type 'A'. If I choose another option
>> ('I'), I can get partially through the install before it hangs.  The hangs
>> at that point seem to be random.
>>
>> I've attached a screenshot of where it's hanging during the initial boot.
>>
>> So far we've been able to reproduce this on all of our Intel Scalable
>> processors, which includes a few other Gold CPUs.  This does work ok on our
>> older E5 CPUs
>>

Reply | Threaded
Open this post in threaded view
|

Re: Unable to boot OpenBSD within QEMU on an Intel Platinum 8176M

Mike Larkin
On Tue, Jan 02, 2018 at 11:30:47AM -0500, Brian Rak wrote:

>
>
> On 12/30/2017 5:13 PM, Mike Larkin wrote:
> > On Wed, Dec 27, 2017 at 01:10:13PM -0500, Brian Rak wrote:
> > > I have a server with an Intel Platinum CPU: https://ark.intel.com/products/120505/Intel-Xeon-Platinum-8176M-Processor-38_5M-Cache-2_10-GHz
> > >
> > > It's running Fedora 27 Server, kernel version 4.14.8-300.fc27.x86_64,
> > > qemu-system-x86-core-2.10.1-2.fc27.x86_64
> > >
> > > I'm starting qemu like this:
> > >
> > > /usr/bin/qemu-system-x86_64 -machine accel=kvm -name
> > > guest=test,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-6-test/master-key.aes
> > > -machine pc-i440fx-2.10,accel=kvm,usb=off,dump-guest-core=off -cpu
> > > Skylake-Client,hypervisor=on -m 32768 -realtime mlock=off -smp
> > > 16,sockets=2,cores=8,threads=1 -uuid 6427e485-5aee-4fb6-b5e5-a80c1dc0f4af
> > > -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-6-test/monitor.sock,server,nowait
> > > -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew
> > > -global kvm-pit.lost_tick_policy=delay -no-shutdown -boot strict=on -device
> > > piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive
> > > file=/var/tmp/cd62.iso,format=raw,if=none,id=drive-ide0-1-0,readonly=on
> > > -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1
> > > -netdev tap,fd=29,id=hostnet0,vhost=on,vhostfd=32 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=56:00:00:27:d6:3f,bus=pci.0,addr=0x3,rombar=0,bootindex=3
> > > -device usb-tablet,id=input0,bus=usb.0,port=1 -vnc
> > > 127.0.0.1:4788,websocket=40688 -device
> > > cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device
> > > virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -object
> > > rng-random,id=objrng0,filename=/dev/random -device
> > > virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x6 -msg timestamp=on
> > Could you try a simpler config? There's a lot in there that's not needed for
> > OpenBSD, and I'm wondering if there is some option you've chosen that's
> > causing problems.
> >
> > For what it's worth, there have been a number of reports of OpenBSD guests
> > recently failing when run on KVM (mainly clock related issues but other
> > things as well). You are using a very new CPU on a very new host OS, I'm not
> > surprised some things are behaving a bit strangely.
> >
> > -ml
> The simplest command line I can come up with is:
>
> /usr/bin/qemu-system-x86_64 -machine accel=kvm -name guest=test -machine
> pc-i440fx-2.10 -cpu Skylake-Client -m 32768 -drive
> file=/var/tmp/cd62.iso,format=raw,if=none,id=drive-ide0-1-0,readonly=on
> -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1
> -device cirrus-vga,id=video0,bus=pci.0,addr=0x3 -vnc 127.0.0.1:4788
>
> I still see the same hang.  Even using more ancient virtual CPUs (-cpu
> pentium3) doesn't seem to help.
>
> Switching to '-machine pc-q35-2.10' doesn't appear to help either.
>
> If I disable KVM, the instance appears to boot normally.  However, I'm not
> sure if this points to a KVM issue or if it's just masking a timing issue
> (because it runs so much slower emulated)
>
> In this case, we went with Fedora 27 to rule out a problem that would be
> fixed by upgrading some part of the host OS.  This isn't an OS we normally
> use, and we've seen issues with older versions of qemu.
>

The only thing I can say is that recently I've been noticing an uptick in the
quantity of KVM related issues on OpenBSD. Whether this is due to some recent
changes in KVM, or maybe due to more people running OpenBSD on KVM (and thus
increasing the number of reports), I'm not sure. But kettenis@ did note a few
days ago in a reply to a different KVM related issue that it seems their local
APIC emulation code isn't behaving exactly as we expect. But that code hasn't
changed in OpenBSD since, well, forever, so it's likely a KVM issue there.
Whether this is your issue or not I don't know. You might bring this up on
the KVM mailing lists and see if someone can shed light on it. If you search
the tech@/misc@ archives for proxmox related threads, there was a KVM option
reported a week or so back that seemed to fix the issue kettenis@ was commenting
on; perhaps this can help you.

-ml

Reply | Threaded
Open this post in threaded view
|

Re: Unable to boot OpenBSD within QEMU on an Intel Platinum 8176M

Landry Breuil-5
On Sat, Dec 30, 2017 at 09:23:03PM -0800, Mike Larkin wrote:

> On Tue, Jan 02, 2018 at 11:30:47AM -0500, Brian Rak wrote:
> >
> >
> The only thing I can say is that recently I've been noticing an uptick in the
> quantity of KVM related issues on OpenBSD. Whether this is due to some recent
> changes in KVM, or maybe due to more people running OpenBSD on KVM (and thus
> increasing the number of reports), I'm not sure. But kettenis@ did note a few
> days ago in a reply to a different KVM related issue that it seems their local
> APIC emulation code isn't behaving exactly as we expect. But that code hasn't
> changed in OpenBSD since, well, forever, so it's likely a KVM issue there.
> Whether this is your issue or not I don't know. You might bring this up on
> the KVM mailing lists and see if someone can shed light on it. If you search
> the tech@/misc@ archives for proxmox related threads, there was a KVM option
> reported a week or so back that seemed to fix the issue kettenis@ was commenting
> on; perhaps this can help you.

ftr that option was kvm-intel.preemption_timer=0 on the host kernel
commandline.

Reply | Threaded
Open this post in threaded view
|

Re: Unable to boot OpenBSD within QEMU on an Intel Platinum 8176M

Mike Larkin
On Tue, Jan 02, 2018 at 08:37:04PM +0100, Landry Breuil wrote:

> On Sat, Dec 30, 2017 at 09:23:03PM -0800, Mike Larkin wrote:
> > On Tue, Jan 02, 2018 at 11:30:47AM -0500, Brian Rak wrote:
> > >
> > >
> > The only thing I can say is that recently I've been noticing an uptick in the
> > quantity of KVM related issues on OpenBSD. Whether this is due to some recent
> > changes in KVM, or maybe due to more people running OpenBSD on KVM (and thus
> > increasing the number of reports), I'm not sure. But kettenis@ did note a few
> > days ago in a reply to a different KVM related issue that it seems their local
> > APIC emulation code isn't behaving exactly as we expect. But that code hasn't
> > changed in OpenBSD since, well, forever, so it's likely a KVM issue there.
> > Whether this is your issue or not I don't know. You might bring this up on
> > the KVM mailing lists and see if someone can shed light on it. If you search
> > the tech@/misc@ archives for proxmox related threads, there was a KVM option
> > reported a week or so back that seemed to fix the issue kettenis@ was commenting
> > on; perhaps this can help you.
>
> ftr that option was kvm-intel.preemption_timer=0 on the host kernel
> commandline.
>

Thanks Landry, I knew someone would chime in :)

Reply | Threaded
Open this post in threaded view
|

Re: Unable to boot OpenBSD within QEMU on an Intel Platinum 8176M

Matthieu Herrb-7
In reply to this post by Landry Breuil-5
On Tue, Jan 02, 2018 at 08:37:04PM +0100, Landry Breuil wrote:

> On Sat, Dec 30, 2017 at 09:23:03PM -0800, Mike Larkin wrote:
> > On Tue, Jan 02, 2018 at 11:30:47AM -0500, Brian Rak wrote:
> > >
> > >
> > The only thing I can say is that recently I've been noticing an uptick in the
> > quantity of KVM related issues on OpenBSD. Whether this is due to some recent
> > changes in KVM, or maybe due to more people running OpenBSD on KVM (and thus
> > increasing the number of reports), I'm not sure. But kettenis@ did note a few
> > days ago in a reply to a different KVM related issue that it seems their local
> > APIC emulation code isn't behaving exactly as we expect. But that code hasn't
> > changed in OpenBSD since, well, forever, so it's likely a KVM issue there.
> > Whether this is your issue or not I don't know. You might bring this up on
> > the KVM mailing lists and see if someone can shed light on it. If you search
> > the tech@/misc@ archives for proxmox related threads, there was a KVM option
> > reported a week or so back that seemed to fix the issue kettenis@ was commenting
> > on; perhaps this can help you.
>
> ftr that option was kvm-intel.preemption_timer=0 on the host kernel
> commandline.

FWIW, I'm running OpenBSD-current in qemu / KVM using libvirt and
virt-manager for some years now. Current -current still works OK for
me. /var/run/ntpd.drift is 0.47 which doesn't look bad.

Here are dmesg from the VM and the xml file for libvirt.

OpenBSD 6.2-current (GENERIC.MP) #28: Fri Dec 29 18:10:50 CET 2017
    [hidden email]:/usr/obj/GENERIC.MP
real mem = 2130698240 (2031MB)
avail mem = 2059264000 (1963MB)
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.4 @ 0xf1430 (13 entries)
bios0: vendor Bochs version "Bochs" date 01/01/2011
bios0: Bochs Bochs
acpi0 at bios0: rev 0
acpi0: sleep states S3 S4 S5
acpi0: tables DSDT FACP SSDT APIC HPET
acpi0: wakeup devices
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel Xeon E3-12xx v2 (Ivy Bridge), 772.12 MHz
cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,SSE3,PCLMUL,VMX,SSSE3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,RDTSCP,LONG,LAHF,PERF,FSGSBASE,SMEP,ERMS
cpu0: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 512KB 64b/line 16-way L2 cache
cpu0: ITLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu0: DTLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
cpu0: apic clock running at 1000MHz
cpu1 at mainbus0: apid 1 (application processor)
cpu1: Intel Xeon E3-12xx v2 (Ivy Bridge), 919.08 MHz
cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,SSE3,PCLMUL,VMX,SSSE3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,RDTSCP,LONG,LAHF,PERF,FSGSBASE,SMEP,ERMS
cpu1: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 512KB 64b/line 16-way L2 cache
cpu1: ITLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu1: DTLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu1: smt 0, core 0, package 1
cpu2 at mainbus0: apid 2 (application processor)
cpu2: Intel Xeon E3-12xx v2 (Ivy Bridge), 1016.77 MHz
cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,SSE3,PCLMUL,VMX,SSSE3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,RDTSCP,LONG,LAHF,PERF,FSGSBASE,SMEP,ERMS
cpu2: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 512KB 64b/line 16-way L2 cache
cpu2: ITLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu2: DTLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu2: smt 0, core 0, package 2
cpu3 at mainbus0: apid 3 (application processor)
cpu3: Intel Xeon E3-12xx v2 (Ivy Bridge), 968.46 MHz
cpu3: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,SSE3,PCLMUL,VMX,SSSE3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,RDTSCP,LONG,LAHF,PERF,FSGSBASE,SMEP,ERMS
cpu3: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 512KB 64b/line 16-way L2 cache
cpu3: ITLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu3: DTLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu3: smt 0, core 0, package 3
ioapic0 at mainbus0: apid 0 pa 0xfec00000, version 11, 24 pins
acpihpet0 at acpi0: 100000000 Hz
acpiprt0 at acpi0: bus 0 (PCI0)
acpicpu0 at acpi0: C1(@1 halt!)
acpicpu1 at acpi0: C1(@1 halt!)
acpicpu2 at acpi0: C1(@1 halt!)
acpicpu3 at acpi0: C1(@1 halt!)
"ACPI0006" at acpi0 not configured
"ACPI0007" at acpi0 not configured
"ACPI0007" at acpi0 not configured
"ACPI0007" at acpi0 not configured
"ACPI0007" at acpi0 not configured
pvbus0 at mainbus0: KVM
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel 82441FX" rev 0x02
pcib0 at pci0 dev 1 function 0 "Intel 82371SB ISA" rev 0x00
pciide0 at pci0 dev 1 function 1 "Intel 82371SB IDE" rev 0x00: DMA, channel 0 wired to compatibility, channel 1 wired to compatibility
pciide0: channel 0 disabled (no drives)
atapiscsi0 at pciide0 channel 1 drive 0
scsibus1 at atapiscsi0: 2 targets
cd0 at scsibus1 targ 0 lun 0: <QEMU, QEMU DVD-ROM, 1.0> ATAPI 5/cdrom removable
cd0(pciide0:1:0): using PIO mode 4, DMA mode 2
uhci0 at pci0 dev 1 function 2 "Intel 82371SB USB" rev 0x01: apic 0 int 11
piixpm0 at pci0 dev 1 function 3 "Intel 82371AB Power" rev 0x03: apic 0 int 9
iic0 at piixpm0
vga1 at pci0 dev 2 function 0 "Red Hat QXL Video" rev 0x03
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
virtio0 at pci0 dev 3 function 0 "Qumranet Virtio Network" rev 0x00
vio0 at virtio0: address 52:54:00:04:d9:6d
virtio0: msix shared
azalia0 at pci0 dev 4 function 0 "Intel 82801FB HD Audio" rev 0x01: apic 0 int 11
azalia0: No codecs found
virtio1 at pci0 dev 5 function 0 "Qumranet Virtio Memory" rev 0x00
viomb0 at virtio1
virtio1: apic 0 int 10
virtio2 at pci0 dev 6 function 0 "Qumranet Virtio Storage" rev 0x00
vioblk0 at virtio2
scsibus2 at vioblk0: 2 targets
sd0 at scsibus2 targ 0 lun 0: <VirtIO, Block Device, > SCSI3 0/direct fixed
sd0: 20480MB, 512 bytes/sector, 41943040 sectors
virtio2: msix shared
isa0 at pcib0
isadma0 at isa0
fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
com0: console
pckbc0 at isa0 port 0x60/5 irq 1 irq 12
pckbd0 at pckbc0 (kbd slot)
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pms0 at pckbc0 (aux slot)
wsmouse0 at pms0 mux 0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
usb0 at uhci0: USB revision 1.0
uhub0 at usb0 configuration 1 interface 0 "Intel UHCI root hub" rev 1.00/1.00 addr 1
vmm0 at mainbus0: VMX/EPT
vscsi0 at root
scsibus3 at vscsi0: 256 targets
softraid0 at root
scsibus4 at softraid0: 256 targets
root on sd0a (f5535e6a0e453526.a) swap on sd0b dump on sd0b
fd0 at fdc0 drive 1: density unknown

<!--
WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE
OVERWRITTEN AND LOST. Changes to this xml configuration should be made using:
  virsh edit obsd64-current
or other application using the libvirt API.
-->

<domain type='kvm'>
  <name>obsd64-current</name>
  <uuid>dce76d73-7af5-fb02-1389-4cb713dd99b4</uuid>
  <memory unit='KiB'>2097152</memory>
  <currentMemory unit='KiB'>2097152</currentMemory>
  <vcpu placement='static'>4</vcpu>
  <os>
    <type arch='x86_64' machine='pc-1.0'>hvm</type>
    <boot dev='hd'/>
    <boot dev='cdrom'/>
    <boot dev='network'/>
    <bootmenu enable='yes'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <cpu mode='host-model'>
    <model fallback='allow'/>
  </cpu>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/bin/kvm</emulator>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw'/>
      <source dev='/dev/vm/obsd64-current'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <target dev='hdc' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='1' target='0' unit='0'/>
    </disk>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='usb' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <interface type='bridge'>
      <mac address='52:54:00:01:02:03'/>
      <source bridge='br0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes'/>
    <sound model='ich6'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </sound>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </memballoon>
  </devices>
</domain>

--
Matthieu Herrb

Reply | Threaded
Open this post in threaded view
|

Re: Unable to boot OpenBSD within QEMU on an Intel Platinum 8176M

Mike Larkin
In reply to this post by Mike Larkin
On Sat, Dec 30, 2017 at 09:23:03PM -0800, Mike Larkin wrote:

> On Tue, Jan 02, 2018 at 11:30:47AM -0500, Brian Rak wrote:
> >
> >
> > On 12/30/2017 5:13 PM, Mike Larkin wrote:
> > > On Wed, Dec 27, 2017 at 01:10:13PM -0500, Brian Rak wrote:
> > > > I have a server with an Intel Platinum CPU: https://ark.intel.com/products/120505/Intel-Xeon-Platinum-8176M-Processor-38_5M-Cache-2_10-GHz
> > > >
> > > > It's running Fedora 27 Server, kernel version 4.14.8-300.fc27.x86_64,
> > > > qemu-system-x86-core-2.10.1-2.fc27.x86_64
> > > >
> > > > I'm starting qemu like this:
> > > >
> > > > /usr/bin/qemu-system-x86_64 -machine accel=kvm -name
> > > > guest=test,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-6-test/master-key.aes
> > > > -machine pc-i440fx-2.10,accel=kvm,usb=off,dump-guest-core=off -cpu
> > > > Skylake-Client,hypervisor=on -m 32768 -realtime mlock=off -smp
> > > > 16,sockets=2,cores=8,threads=1 -uuid 6427e485-5aee-4fb6-b5e5-a80c1dc0f4af
> > > > -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-6-test/monitor.sock,server,nowait
> > > > -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew
> > > > -global kvm-pit.lost_tick_policy=delay -no-shutdown -boot strict=on -device
> > > > piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive
> > > > file=/var/tmp/cd62.iso,format=raw,if=none,id=drive-ide0-1-0,readonly=on
> > > > -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1
> > > > -netdev tap,fd=29,id=hostnet0,vhost=on,vhostfd=32 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=56:00:00:27:d6:3f,bus=pci.0,addr=0x3,rombar=0,bootindex=3
> > > > -device usb-tablet,id=input0,bus=usb.0,port=1 -vnc
> > > > 127.0.0.1:4788,websocket=40688 -device
> > > > cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device
> > > > virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -object
> > > > rng-random,id=objrng0,filename=/dev/random -device
> > > > virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x6 -msg timestamp=on
> > > Could you try a simpler config? There's a lot in there that's not needed for
> > > OpenBSD, and I'm wondering if there is some option you've chosen that's
> > > causing problems.
> > >
> > > For what it's worth, there have been a number of reports of OpenBSD guests
> > > recently failing when run on KVM (mainly clock related issues but other
> > > things as well). You are using a very new CPU on a very new host OS, I'm not
> > > surprised some things are behaving a bit strangely.
> > >
> > > -ml
> > The simplest command line I can come up with is:
> >
> > /usr/bin/qemu-system-x86_64 -machine accel=kvm -name guest=test -machine
> > pc-i440fx-2.10 -cpu Skylake-Client -m 32768 -drive
> > file=/var/tmp/cd62.iso,format=raw,if=none,id=drive-ide0-1-0,readonly=on
> > -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1
> > -device cirrus-vga,id=video0,bus=pci.0,addr=0x3 -vnc 127.0.0.1:4788
> >
> > I still see the same hang.  Even using more ancient virtual CPUs (-cpu
> > pentium3) doesn't seem to help.
> >
> > Switching to '-machine pc-q35-2.10' doesn't appear to help either.
> >
> > If I disable KVM, the instance appears to boot normally.  However, I'm not
> > sure if this points to a KVM issue or if it's just masking a timing issue
> > (because it runs so much slower emulated)
> >
> > In this case, we went with Fedora 27 to rule out a problem that would be
> > fixed by upgrading some part of the host OS.  This isn't an OS we normally
> > use, and we've seen issues with older versions of qemu.
> >
>
> The only thing I can say is that recently I've been noticing an uptick in the
> quantity of KVM related issues on OpenBSD. Whether this is due to some recent
> changes in KVM, or maybe due to more people running OpenBSD on KVM (and thus
> increasing the number of reports), I'm not sure. But kettenis@ did note a few
> days ago in a reply to a different KVM related issue that it seems their local
> APIC emulation code isn't behaving exactly as we expect. But that code hasn't
> changed in OpenBSD since, well, forever, so it's likely a KVM issue there.
> Whether this is your issue or not I don't know. You might bring this up on
> the KVM mailing lists and see if someone can shed light on it. If you search
> the tech@/misc@ archives for proxmox related threads, there was a KVM option
> reported a week or so back that seemed to fix the issue kettenis@ was commenting
> on; perhaps this can help you.
>
> -ml
>

Following up on old email threads; see the recent message by sf@ regarding
disabling certain KVM features that may help here.

-ml