bhyve OpenBSD guest crash on boot between #61 and #65

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

bhyve OpenBSD guest crash on boot between #61 and #65

Jason Tubnor
Hi,

Following daily snapshots, I have hit a bug that was introduced between
6.5-current #61 and 6.5-current #66 (not sure if it was related to the LLVM
upgrade).

Reference system is bhyve on FreeBSD 11.2 with an OpenBSD guest.  #61 boots
fine and runs as expected.  However, #66 get to the following point and the
guest terminates:

scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 3.0 @ 0xbf957000 (11 entries)
bios0: vendor BHYVE version "1.00" date 03/14/2014
bios0: bhyve BHYVE
acpi0 at bios0: ACPI 4.0
acpi0: sleep states S5
acpi0: tables DSDT FACP HPET APIC MCFG SPCR
acpi0: wakeup devices
acpitimer0 at acpi0: 3579545 Hz, 32 bits
acpihpet0 at acpi0: 16777216 Hz
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel(R) Xeon(R) CPU E3-1230 v6 @ 3.50GHz, 3504.21 MHz, 06-9e-09
cpu0:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,PBE,SSE3,PCLMUL,DTES64,DS-CPL,SSSE3,SDBG,FMA3,CX16,xTPR,PCID,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,LONG,LAHF,ABM,3DNOWP,ITSC,FSGSBASE,BMI1,HLE,AVX2,BMI2,ERMS,INVPCID,RTM,ARAT,XSAVEOPT,MELTDOWN
cpu0: 256KB 64b/line 8-way L2 cache
cpu0: smt 0, core 0, package 0
mtrr: CPU supports MTRRs but not enabled by BIOS

(Note:  The above was text from #61 but was all the console text (as UEFI
is graphics) I could copy/paste).

Below is output from bhyve.log:

rdmsr to register 0xc80 on vcpu 0
Failed to emulate instruction [0xf7 0x04 0x25 0x00 0xd3 0xd1 0x81 0x00 0x10
0x00 0x00 0x74 0x08 0xf3 0x90] at 0xffffffff817648f0

bhyve.log when booting with #61:

rdmsr to register 0xc80 on vcpu 0
rdmsr to register 0xc80 on vcpu 1
rdmsr to register 0xc80 on vcpu 2
rdmsr to register 0xc80 on vcpu 3
rdmsr to register 0xc80 on vcpu 4
rdmsr to register 0xc80 on vcpu 5
rdmsr to register 0xc80 on vcpu 6
rdmsr to register 0xc80 on vcpu 7

RAMDISK #62 is not showing this issue either.

Please advise if there is any further information that is needed.

Jason.
Reply | Threaded
Open this post in threaded view
|

Re: bhyve OpenBSD guest crash on boot between #61 and #65

Bryan Steele-2
On Tue, Jun 25, 2019 at 11:03:24AM +1000, Jason Tubnor wrote:

> Hi,
>
> Following daily snapshots, I have hit a bug that was introduced between
> 6.5-current #61 and 6.5-current #66 (not sure if it was related to the LLVM
> upgrade).
>
> Reference system is bhyve on FreeBSD 11.2 with an OpenBSD guest.  #61 boots
> fine and runs as expected.  However, #66 get to the following point and the
> guest terminates:
>
> scsibus0 at mpath0: 256 targets
> mainbus0 at root
> bios0 at mainbus0: SMBIOS rev. 3.0 @ 0xbf957000 (11 entries)
> bios0: vendor BHYVE version "1.00" date 03/14/2014
> bios0: bhyve BHYVE
> acpi0 at bios0: ACPI 4.0
> acpi0: sleep states S5
> acpi0: tables DSDT FACP HPET APIC MCFG SPCR
> acpi0: wakeup devices
> acpitimer0 at acpi0: 3579545 Hz, 32 bits
> acpihpet0 at acpi0: 16777216 Hz
> acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
> cpu0 at mainbus0: apid 0 (boot processor)
> cpu0: Intel(R) Xeon(R) CPU E3-1230 v6 @ 3.50GHz, 3504.21 MHz, 06-9e-09
> cpu0:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,PBE,SSE3,PCLMUL,DTES64,DS-CPL,SSSE3,SDBG,FMA3,CX16,xTPR,PCID,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,LONG,LAHF,ABM,3DNOWP,ITSC,FSGSBASE,BMI1,HLE,AVX2,BMI2,ERMS,INVPCID,RTM,ARAT,XSAVEOPT,MELTDOWN
> cpu0: 256KB 64b/line 8-way L2 cache
> cpu0: smt 0, core 0, package 0
> mtrr: CPU supports MTRRs but not enabled by BIOS
>
> (Note:  The above was text from #61 but was all the console text (as UEFI
> is graphics) I could copy/paste).
>
> Below is output from bhyve.log:
>
> rdmsr to register 0xc80 on vcpu 0
> Failed to emulate instruction [0xf7 0x04 0x25 0x00 0xd3 0xd1 0x81 0x00 0x10
> 0x00 0x00 0x74 0x08 0xf3 0x90] at 0xffffffff817648f0

Sounds like a bug in bhyve? Perhaps the compiler change may be
exposing the problem, but it's not clear if the new code being
generated is neccesarily wrong. Does the same snapshot/kernel
boot on the bare metal?

>
> bhyve.log when booting with #61:
>
> rdmsr to register 0xc80 on vcpu 0
> rdmsr to register 0xc80 on vcpu 1
> rdmsr to register 0xc80 on vcpu 2
> rdmsr to register 0xc80 on vcpu 3
> rdmsr to register 0xc80 on vcpu 4
> rdmsr to register 0xc80 on vcpu 5
> rdmsr to register 0xc80 on vcpu 6
> rdmsr to register 0xc80 on vcpu 7
>
> RAMDISK #62 is not showing this issue either.
>
> Please advise if there is any further information that is needed.
>
> Jason.
>

Reply | Threaded
Open this post in threaded view
|

Re: bhyve OpenBSD guest crash on boot between #61 and #65

Philip Guenther-3
In reply to this post by Jason Tubnor
On Mon, 24 Jun 2019, Jason Tubnor wrote:
> Following daily snapshots, I have hit a bug that was introduced between
> 6.5-current #61 and 6.5-current #66 (not sure if it was related to the LLVM
> upgrade).
>
> Reference system is bhyve on FreeBSD 11.2 with an OpenBSD guest.  #61 boots
> fine and runs as expected.  However, #66 get to the following point and the
> guest terminates:
...
> Below is output from bhyve.log:
>
> rdmsr to register 0xc80 on vcpu 0
> Failed to emulate instruction [0xf7 0x04 0x25 0x00 0xd3 0xd1 0x81 0x00 0x10
> 0x00 0x00 0x74 0x08 0xf3 0x90] at 0xffffffff817648f0

According to objdump -d, that's:
 f7 04 25 00 d3 d1 81    testl  $0x1000,0xffffffff81d1d300
 00 10 00 00
 74 08                   je     <forward some>
 f3 90                   pause

That's testing the LAPIC ICRLO, a memory-mapped register.  Previously, the
compiler generated code like this:

 8b 0c 25 00 63 cf 81    mov    0xffffffff81cf6300,%ecx
 f7 c1 00 10 00 00       test   $0x1000,%ecx
 74 09                   je     <forward some>
 f3 90                   pause  

where it loaded the LAPIC register into %ecx and then tested that value;
now it combines them and does a direct test.  Congrats, that's legal
according to Intel (reportedly, Windows will use SSE(!) instructions to
read LAPIC registers), so this seems like a bug in Bhyve.


Philip Guenther

Reply | Threaded
Open this post in threaded view
|

Re: bhyve OpenBSD guest crash on boot between #61 and #65

Mike Larkin-2
On Mon, Jun 24, 2019 at 07:16:20PM -0700, [hidden email] wrote:

> On Mon, 24 Jun 2019, Jason Tubnor wrote:
> > Following daily snapshots, I have hit a bug that was introduced between
> > 6.5-current #61 and 6.5-current #66 (not sure if it was related to the LLVM
> > upgrade).
> >
> > Reference system is bhyve on FreeBSD 11.2 with an OpenBSD guest.  #61 boots
> > fine and runs as expected.  However, #66 get to the following point and the
> > guest terminates:
> ...
> > Below is output from bhyve.log:
> >
> > rdmsr to register 0xc80 on vcpu 0
> > Failed to emulate instruction [0xf7 0x04 0x25 0x00 0xd3 0xd1 0x81 0x00 0x10
> > 0x00 0x00 0x74 0x08 0xf3 0x90] at 0xffffffff817648f0
>
> According to objdump -d, that's:
>  f7 04 25 00 d3 d1 81    testl  $0x1000,0xffffffff81d1d300
>  00 10 00 00
>  74 08                   je     <forward some>
>  f3 90                   pause
>
> That's testing the LAPIC ICRLO, a memory-mapped register.  Previously, the
> compiler generated code like this:
>
>  8b 0c 25 00 63 cf 81    mov    0xffffffff81cf6300,%ecx
>  f7 c1 00 10 00 00       test   $0x1000,%ecx
>  74 09                   je     <forward some>
>  f3 90                   pause  
>
> where it loaded the LAPIC register into %ecx and then tested that value;
> now it combines them and does a direct test.  Congrats, that's legal
> according to Intel (reportedly, Windows will use SSE(!) instructions to
> read LAPIC registers), so this seems like a bug in Bhyve.
>
>
> Philip Guenther
>

Yep, please report it to the bhyve guys.

-ml

Reply | Threaded
Open this post in threaded view
|

Re: bhyve OpenBSD guest crash on boot between #61 and #65

Jason Tubnor
On Tue, 25 Jun 2019 at 12:25, Mike Larkin <[hidden email]> wrote:

> On Mon, Jun 24, 2019 at 07:16:20PM -0700, [hidden email] wrote:
>
> > According to objdump -d, that's:
> >  f7 04 25 00 d3 d1 81    testl  $0x1000,0xffffffff81d1d300
> >  00 10 00 00
> >  74 08                   je     <forward some>
> >  f3 90                   pause
> >
> > That's testing the LAPIC ICRLO, a memory-mapped register.  Previously,
> the
> > compiler generated code like this:
> >
> >  8b 0c 25 00 63 cf 81    mov    0xffffffff81cf6300,%ecx
> >  f7 c1 00 10 00 00       test   $0x1000,%ecx
> >  74 09                   je     <forward some>
> >  f3 90                   pause
> >
> > where it loaded the LAPIC register into %ecx and then tested that value;
> > now it combines them and does a direct test.  Congrats, that's legal
> > according to Intel (reportedly, Windows will use SSE(!) instructions to
> > read LAPIC registers), so this seems like a bug in Bhyve.
>
>
> Yep, please report it to the bhyve guys.
>
>
Bug report raised at FreeBSD.  For reference:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=238794

Thanks for the quick response.  Cheers!
Reply | Threaded
Open this post in threaded view
|

Re: bhyve OpenBSD guest crash on boot between #61 and #65

Jason Tubnor
On Tue, 25 Jun 2019 at 13:50, Jason Tubnor <[hidden email]> wrote:

>
> On Tue, 25 Jun 2019 at 12:25, Mike Larkin <[hidden email]> wrote:
>
>> On Mon, Jun 24, 2019 at 07:16:20PM -0700, [hidden email] wrote:
>>
>> > where it loaded the LAPIC register into %ecx and then tested that
>> value;
>> > now it combines them and does a direct test.  Congrats, that's legal
>> > according to Intel (reportedly, Windows will use SSE(!) instructions to
>> > read LAPIC registers), so this seems like a bug in Bhyve.
>>
>>
>> Yep, please report it to the bhyve guys.
>>
>>
> Bug report raised at FreeBSD.  For reference:
> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=238794
>
> Thanks for the quick response.  Cheers!
>
>

This bug has been fixed and patches applied to FreeBSD -HEAD, 12-stable and
11-stable.  Errata patch will be available for 11.3 after it has been
released (about 2 weeks):

Patch review and testing:  https://reviews.freebsd.org/D20755

Commit to source trees:
-HEAD:
https://svnweb.freebsd.org/base/head/sys/amd64/vmm/vmm_instruction_emul.c?revision=349441&view=markup
-12-Stable:
https://svnweb.freebsd.org/base/stable/12/sys/amd64/vmm/vmm_instruction_emul.c?view=log
-11-Stable:
https://svnweb.freebsd.org/base/stable/11/sys/amd64/vmm/vmm_instruction_emul.c?view=log

I'll close the FreeBSD bug report above once a confirmed errata patch has
been released for 11.3

Jason.