Re: Openbsd 6.1 and Current Console Freezes and lockup Proxmox PVE5.0

classic Classic list List threaded Threaded
24 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Re: Openbsd 6.1 and Current Console Freezes and lockup Proxmox PVE5.0

Wolfgang Link
It looks like a KVM bug, which I can cover on a span of commits.

This are commits form the Linux stable repository.
gitk bb3dd056ed1af9b186f0d9fe849eab78c51d14ce..813ae37e6aed72cc457094b6066aa38efd66c9e9

What is certainly safe in any case, it will affect all KVM hosted OpenBSD with a kernel which is newer than 4.9.

Reply | Threaded
Open this post in threaded view
|

Re: Openbsd 6.1 and Current Console Freezes and lockup Proxmox PVE5.0

Oliver Marugg
I’ve got the same freezes/hangs in all of my instances running OpenBSD
6.1-stable and/or 6.1-release syspatched on KVM-based Proxmox 5 as
mentioned by others.

I also tried the change of the serial port as mentioned in this thread
with no positive result.

Badly these freezes/hangs gives us not much chances for a bug reports.
Only once I had a shown reported panic: vioscsi_scsi_cmd:183 vr_state is
2 should be 1. I was able only once to do some screenshots and traces
until it hanged again. Attached to this mail I added my screenshots (7
screenshots of ddb). Or should I send these to bugs@ in relation to this
misc@ thread, also if its not a full bug report?

Due to the fact I use a proxmox cluster for my students I asked Proxmox
about that freezes/hangs of OpenBSD in their own support forum. After
some checks/testrun they answered it could be a possible bug in KVM in
their patches about RTC and apic, but Proxmox doesnt know OpenBSD in
depth. Proxmox also sent with a KVM commit range to this thread.

Unfortunately I am a user and not a dev, from a indepth OpenBSD tech
perspective I am not really able to give advice for a proper operation
of OpenBSD to proxmox/kvm and/or to verify what possible
changes/adjustments are needed in kvm hypervisor for a smooth operation
of OpenBSD in there. Any ideas?

In general: Many thanks for all devs for OpenBSD.

-oliver

pve_openbsd-001.png (166K) Download Attachment
pve_openbsd-007.png (151K) Download Attachment
pve_openbsd-006.png (18K) Download Attachment
pve_openbsd--005.png (49K) Download Attachment
pve_openbsd-004.png (51K) Download Attachment
pve_openbsd-003.png (161K) Download Attachment
pve_openbsd-002.png (151K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Openbsd 6.1 and Current Console Freezes and lockup Proxmox PVE5.0

Mike Larkin
On Fri, Oct 06, 2017 at 05:58:18PM +0200, Oliver Marugg wrote:

> I’ve got the same freezes/hangs in all of my instances running OpenBSD
> 6.1-stable and/or 6.1-release syspatched on KVM-based Proxmox 5 as mentioned
> by others.
>
> I also tried the change of the serial port as mentioned in this thread with
> no positive result.
>
> Badly these freezes/hangs gives us not much chances for a bug reports. Only
> once I had a shown reported panic: vioscsi_scsi_cmd:183 vr_state is 2 should
> be 1. I was able only once to do some screenshots and traces until it hanged
> again. Attached to this mail I added my screenshots (7 screenshots of ddb).
> Or should I send these to bugs@ in relation to this misc@ thread, also if
> its not a full bug report?
>
> Due to the fact I use a proxmox cluster for my students I asked Proxmox
> about that freezes/hangs of OpenBSD in their own support forum. After some
> checks/testrun they answered it could be a possible bug in KVM in their
> patches about RTC and apic, but Proxmox doesnt know OpenBSD in depth.
> Proxmox also sent with a KVM commit range to this thread.
>
> Unfortunately I am a user and not a dev, from a indepth OpenBSD tech
> perspective I am not really able to give advice for a proper operation of
> OpenBSD to proxmox/kvm and/or to verify what possible changes/adjustments
> are needed in kvm hypervisor for a smooth operation of OpenBSD in there. Any
> ideas?
>
> In general: Many thanks for all devs for OpenBSD.
>
> -oliver

Every one of these images shows a panic on halt/reboot.

Does this only happen to you on halt and reboot scenarios?

-ml

Reply | Threaded
Open this post in threaded view
|

Re: Openbsd 6.1 and Current Console Freezes and lockup Proxmox PVE5.0

Oliver Marugg
On 6 Oct 2017, at 20:21, Mike Larkin wrote:
>
> Every one of these images shows a panic on halt/reboot.
>
> Does this only happen to you on halt and reboot scenarios?
>
> -ml

The images show in fact the only freeze/hangs I was able to catch with
ddb and traces during a reboot after freshly builded 6.1-stable
GENERIC.MP kernel (errata patches).

3 Situations where it could freeze:

Using the serial console in proxmox

At my installations also during reboot and shutdown -p now (both showing
syncing disks in serial console), the cpus runs then in proxmox over
100% and nothing happens, also after hours.

Happened a few times during an active ssh session (cvs up or compiling
stable-kernel) or even it hanged somewhen during a day/for days I
noticed later -> no log entries or dmesg gives a hint to any direction
then.
-oliver

Reply | Threaded
Open this post in threaded view
|

Re: Openbsd 6.1 and Current Console Freezes and lockup Proxmox PVE5.0

Oliver Marugg
In reply to this post by Mike Larkin
Just to add a 4th situation of hangs: Login via proxmox (pve)/kvm serial
console (via noVNC), login successful: Vm guest in pve hangs, cpu usage
at above 102%. Only way is to hard stop the Vm guest. -oliver

Reply | Threaded
Open this post in threaded view
|

Re: Openbsd 6.1 and Current Console Freezes and lockup Proxmox PVE5.0

Mike Larkin
On Sat, Oct 07, 2017 at 02:19:58PM +0200, Oliver Marugg wrote:
> Just to add a 4th situation of hangs: Login via proxmox (pve)/kvm serial
> console (via noVNC), login successful: Vm guest in pve hangs, cpu usage at
> above 102%. Only way is to hard stop the Vm guest. -oliver
>

sounds like a kvm bug. Ask your provider to investigate the host side when this
happens.

Reply | Threaded
Open this post in threaded view
|

Re: Openbsd 6.1 and Current Console Freezes and lockup Proxmox PVE5.0

Tom Smyth
In reply to this post by Wolfgang Link
Hello,

I have had this issue in proxmox in 5.0 in all releases
(the beta 1, beta2 and the july 4th release of PRoxmox 5 and the
update in August on intel systems with CPUS newer than the X5460)
Curiously it worked fine as far as I could tell on AMD systems
(Opteron Gen 2/ gen 3 systems)

I have posted in Bugs,
https://marc.info/?l=openbsd-bugs&m=150097397016837&w=2

To be fair to OpenBSD it wasnt a bug in 6.1 (as proxmox 5.0 was not
 released when 6.1 was released,
however Current 6.1+  didnt work either I had opened a Ticket with
Proxmox  and worked on it for about a month
and they couldn't repeat it  which is weird
cause it was just too easy for me to crash openbsd on Proxmox 5.0.
Other operating systems running on Proxmos 5.0 seem to be unaffected

so this issue will only become a bug when OpenBSD 6.2 is released :)

OpenBSD 6.1 & 6.0 work fine in proxmox PVE 4.4 on the same hardware

I hope this helps

Tom Smyth

Reply | Threaded
Open this post in threaded view
|

Re: Openbsd 6.1 and Current Console Freezes and lockup Proxmox PVE5.0

Oliver Marugg
In reply to this post by Mike Larkin
On 7 Oct 2017, at 22:01, Mike Larkin wrote:

> On Sat, Oct 07, 2017 at 02:19:58PM +0200, Oliver Marugg wrote:
>> Just to add a 4th situation of hangs: Login via proxmox (pve)/kvm
>> serial
>> console (via noVNC), login successful: Vm guest in pve hangs, cpu
>> usage at
>> above 102%. Only way is to hard stop the Vm guest. -oliver
>>
>
> sounds like a kvm bug. Ask your provider to investigate the host side
> when this
> happens.

Thanks Mike, will do so. The proxmox guys have also the idea that it
could be a bug in kvm hypervisor (which is the hypervisor part for
proxmox) and will affect OpenBSD since 4.9, they wrote me in their
public forum. As far as I understood they do not know what OpenBSD needs
in kvm or what/where should be fixed in kvm run OpenBSD without that
freezes.

-oliver

Reply | Threaded
Open this post in threaded view
|

Re: Openbsd 6.1 and Current Console Freezes and lockup Proxmox PVE5.0

Mike Larkin
On Sun, Oct 08, 2017 at 11:59:52PM +0200, Oliver Marugg wrote:

> On 7 Oct 2017, at 22:01, Mike Larkin wrote:
>
> > On Sat, Oct 07, 2017 at 02:19:58PM +0200, Oliver Marugg wrote:
> > > Just to add a 4th situation of hangs: Login via proxmox (pve)/kvm
> > > serial
> > > console (via noVNC), login successful: Vm guest in pve hangs, cpu
> > > usage at
> > > above 102%. Only way is to hard stop the Vm guest. -oliver
> > >
> >
> > sounds like a kvm bug. Ask your provider to investigate the host side
> > when this
> > happens.
>
> Thanks Mike, will do so. The proxmox guys have also the idea that it could
> be a bug in kvm hypervisor (which is the hypervisor part for proxmox) and
> will affect OpenBSD since 4.9, they wrote me in their public forum. As far
> as I understood they do not know what OpenBSD needs in kvm or what/where
> should be fixed in kvm run OpenBSD without that freezes.
>
> -oliver

From what I read, the cpu spins to 100%, which means somewhere on the host it's
likely spinning also. Start with systrace/ptrace/ktrace/whatever on the host
qemu-kvm and go from there...

-ml

Reply | Threaded
Open this post in threaded view
|

Re: Openbsd 6.1 and Current Console Freezes and lockup Proxmox PVE5.0

Oliver Marugg
In reply to this post by Oliver Marugg
On 8 Oct 2017, at 23:59, Oliver Marugg wrote:
>>
> Thanks Mike, will do so. The proxmox guys have also the idea that it
> could be a bug in kvm hypervisor (which is the hypervisor part for
> proxmox) and will affect OpenBSD since 4.9, they wrote me in their
> public forum. As far as I understood they do not know what OpenBSD
> needs in kvm or what/where should be fixed in kvm run OpenBSD without
> that freezes.
>
> -oliver

I have to correct my previous post "...will affect OpenBSD since 4.9"
was nonsense and sorry for this noise. Proxmox guys said it will affect
OpenBSD guests under KVM with Linux kernels starting from 4.9 on, I only
saw wrong version numbers.

Currently running tests with OpenBSD 6.2-release and 6.2-stable guests
in proxmox/kvm, but it seems there is no change with freezes to OpenBSD
6.1.

Reply | Threaded
Open this post in threaded view
|

Re: Openbsd 6.1 and Current Console Freezes and lockup Proxmox PVE5.0

Michał Koc
In reply to this post by Mike Larkin
On Sun, Oct 08, 2017 at 11:59:52PM +0200, Oliver Marugg wrote:

>> On 7 Oct 2017, at 22:01, Mike Larkin wrote:
>>
>>> On Sat, Oct 07, 2017 at 02:19:58PM +0200, Oliver Marugg wrote:
>>>> Just to add a 4th situation of hangs: Login via proxmox (pve)/kvm
>>>> serial
>>>> console (via noVNC), login successful: Vm guest in pve hangs, cpu
>>>> usage at
>>>> above 102%. Only way is to hard stop the Vm guest. -oliver
>>>>
>>> sounds like a kvm bug. Ask your provider to investigate the host side
>>> when this
>>> happens.
>> Thanks Mike, will do so. The proxmox guys have also the idea that it could
>> be a bug in kvm hypervisor (which is the hypervisor part for proxmox) and
>> will affect OpenBSD since 4.9, they wrote me in their public forum. As far
>> as I understood they do not know what OpenBSD needs in kvm or what/where
>> should be fixed in kvm run OpenBSD without that freezes.
>>
>> -oliver
> >From what I read, the cpu spins to 100%, which means somewhere on the host it's
> likely spinning also. Start with systrace/ptrace/ktrace/whatever on the host
> qemu-kvm and go from there...
>
> -ml
>
>
>
Hi,

it looks like the cpu process of kvm (CPU 0/KVM) is issuing 1500+ of
ioctl(15, KVM_RUN, 0)  per second while running OpenBSD 6.2 guest.

In case of linux guest the process issues about 15 of those ioctls per
second.

In any case I cannot make openbsd to starve KVM host cpu. OpenBSD uses
at most(when idle) 7% of cpu.

My versions:
- OpenBSD 6.2 amd64
- KVM 2.8.1

BR
M.K.







Reply | Threaded
Open this post in threaded view
|

Re: Openbsd 6.1 and Current Console Freezes and lockup Proxmox PVE5.0

Scott Reese
In reply to this post by Oliver Marugg

> On 8 Oct 2017, at 23:59, Oliver Marugg wrote:
>>>
>> Thanks Mike, will do so. The proxmox guys have also the idea that it
>> could be a bug in kvm hypervisor (which is the hypervisor part for
>> proxmox) and will affect OpenBSD since 4.9, they wrote me in their
>> public forum. As far as I understood they do not know what OpenBSD
>> needs in kvm or what/where should be fixed in kvm run OpenBSD without
>> that freezes.
>>
>> -oliver
>
> I have to correct my previous post "...will affect OpenBSD since 4.9"
> was nonsense and sorry for this noise. Proxmox guys said it will affect
> OpenBSD guests under KVM with Linux kernels starting from 4.9 on, I only
> saw wrong version numbers.
>
> Currently running tests with OpenBSD 6.2-release and 6.2-stable guests
> in proxmox/kvm, but it seems there is no change with freezes to OpenBSD
> 6.1.

Greetings:

Just a data point for you: I run about a dozen OpenBSD VMs, versions 6.0, 6.1,
and 6.2 under KVM on Linux kernel 4.11 without any of the issues that you are
describing. The host system is running Fedora 25, the kernel version is 4.11.3,
and the qemu-kvm package version is 2.7.1-7.

If there's any information I can provide, please let me know.

-Scott

Reply | Threaded
Open this post in threaded view
|

Re: Openbsd 6.1 and Current Console Freezes and lockup Proxmox PVE5.0

Alexis VACHETTE
Dear all,

I tried to run OpenBSD 6.2 on Debian GNU/Linux 9.2 and qemu-kvm 2.8 no
issue so far too.

Proxmox is using Debian GNU/Linux as the core OS.

Next step will be to use the same version of qemu-kvm that Proxmox 5.0
ship by default.

Regards,

On 12/10/2017 13:38, Scott Reese wrote:

>> On 8 Oct 2017, at 23:59, Oliver Marugg wrote:
>>> Thanks Mike, will do so. The proxmox guys have also the idea that it
>>> could be a bug in kvm hypervisor (which is the hypervisor part for
>>> proxmox) and will affect OpenBSD since 4.9, they wrote me in their
>>> public forum. As far as I understood they do not know what OpenBSD
>>> needs in kvm or what/where should be fixed in kvm run OpenBSD without
>>> that freezes.
>>>
>>> -oliver
>> I have to correct my previous post "...will affect OpenBSD since 4.9"
>> was nonsense and sorry for this noise. Proxmox guys said it will affect
>> OpenBSD guests under KVM with Linux kernels starting from 4.9 on, I only
>> saw wrong version numbers.
>>
>> Currently running tests with OpenBSD 6.2-release and 6.2-stable guests
>> in proxmox/kvm, but it seems there is no change with freezes to OpenBSD
>> 6.1.
> Greetings:
>
> Just a data point for you: I run about a dozen OpenBSD VMs, versions 6.0, 6.1,
> and 6.2 under KVM on Linux kernel 4.11 without any of the issues that you are
> describing. The host system is running Fedora 25, the kernel version is 4.11.3,
> and the qemu-kvm package version is 2.7.1-7.
>
> If there's any information I can provide, please let me know.
>
> -Scott
>

Reply | Threaded
Open this post in threaded view
|

Re: Openbsd 6.1 and Current Console Freezes and lockup Proxmox PVE5.0

Mike Larkin
In reply to this post by Michał Koc
On Thu, Oct 12, 2017 at 01:23:36PM +0200, Michał Koc wrote:

> On Sun, Oct 08, 2017 at 11:59:52PM +0200, Oliver Marugg wrote:
> > > On 7 Oct 2017, at 22:01, Mike Larkin wrote:
> > >
> > > > On Sat, Oct 07, 2017 at 02:19:58PM +0200, Oliver Marugg wrote:
> > > > > Just to add a 4th situation of hangs: Login via proxmox (pve)/kvm
> > > > > serial
> > > > > console (via noVNC), login successful: Vm guest in pve hangs, cpu
> > > > > usage at
> > > > > above 102%. Only way is to hard stop the Vm guest. -oliver
> > > > >
> > > > sounds like a kvm bug. Ask your provider to investigate the host side
> > > > when this
> > > > happens.
> > > Thanks Mike, will do so. The proxmox guys have also the idea that it could
> > > be a bug in kvm hypervisor (which is the hypervisor part for proxmox) and
> > > will affect OpenBSD since 4.9, they wrote me in their public forum. As far
> > > as I understood they do not know what OpenBSD needs in kvm or what/where
> > > should be fixed in kvm run OpenBSD without that freezes.
> > >
> > > -oliver
> > >From what I read, the cpu spins to 100%, which means somewhere on the host it's
> > likely spinning also. Start with systrace/ptrace/ktrace/whatever on the host
> > qemu-kvm and go from there...
> >
> > -ml
> >
> >
> >
> Hi,
>
> it looks like the cpu process of kvm (CPU 0/KVM) is issuing 1500+ of
> ioctl(15, KVM_RUN, 0)  per second while running OpenBSD 6.2 guest.
>

What CPU profile is being presented to the OpenBSD guest?

I've seen things like this happen when a vCPU is claimed to have monitor/mwait
support, but the hypervisor implements those as NOPs, which just results in
spinning like this.

In short - try changing the type of CPU presented to the guest and see if that
changes behaviour. At least then you'll have more data points to work with.

-ml

> In case of linux guest the process issues about 15 of those ioctls per
> second.
>
> In any case I cannot make openbsd to starve KVM host cpu. OpenBSD uses at
> most(when idle) 7% of cpu.
>
> My versions:
> - OpenBSD 6.2 amd64
> - KVM 2.8.1
>
> BR
> M.K.
>
>
>
>
>
>
>

Reply | Threaded
Open this post in threaded view
|

Re: Openbsd 6.1 and Current Console Freezes and lockup Proxmox PVE5.0

Michał Koc

> On Thu, Oct 12, 2017 at 01:23:36PM +0200, Michał Koc wrote:
>> On Sun, Oct 08, 2017 at 11:59:52PM +0200, Oliver Marugg wrote:
>>>> On 7 Oct 2017, at 22:01, Mike Larkin wrote:
>>>>
>>>>> On Sat, Oct 07, 2017 at 02:19:58PM +0200, Oliver Marugg wrote:
>>>>>> Just to add a 4th situation of hangs: Login via proxmox (pve)/kvm
>>>>>> serial
>>>>>> console (via noVNC), login successful: Vm guest in pve hangs, cpu
>>>>>> usage at
>>>>>> above 102%. Only way is to hard stop the Vm guest. -oliver
>>>>>>
>>>>> sounds like a kvm bug. Ask your provider to investigate the host side
>>>>> when this
>>>>> happens.
>>>> Thanks Mike, will do so. The proxmox guys have also the idea that it could
>>>> be a bug in kvm hypervisor (which is the hypervisor part for proxmox) and
>>>> will affect OpenBSD since 4.9, they wrote me in their public forum. As far
>>>> as I understood they do not know what OpenBSD needs in kvm or what/where
>>>> should be fixed in kvm run OpenBSD without that freezes.
>>>>
>>>> -oliver
>>> >From what I read, the cpu spins to 100%, which means somewhere on the host it's
>>> likely spinning also. Start with systrace/ptrace/ktrace/whatever on the host
>>> qemu-kvm and go from there...
>>>
>>> -ml
>>>
>>>
>>>
>> Hi,
>>
>> it looks like the cpu process of kvm (CPU 0/KVM) is issuing 1500+ of
>> ioctl(15, KVM_RUN, 0)  per second while running OpenBSD 6.2 guest.
>>
> What CPU profile is being presented to the OpenBSD guest?
>
> I've seen things like this happen when a vCPU is claimed to have monitor/mwait
> support, but the hypervisor implements those as NOPs, which just results in
> spinning like this.
>
> In short - try changing the type of CPU presented to the guest and see if that
> changes behaviour. At least then you'll have more data points to work with.
>
> -ml

Okey,
How would You disable monitor/mwait support in KVM to be presented to
guest ?

changing CPU to pentium or setting <feature policy='disable'
name='monitor'/> does not actually change anything in scope of host cpu
utilization....

BR
M.K.


>
>> In case of linux guest the process issues about 15 of those ioctls per
>> second.
>>
>> In any case I cannot make openbsd to starve KVM host cpu. OpenBSD uses at
>> most(when idle) 7% of cpu.
>>
>> My versions:
>> - OpenBSD 6.2 amd64
>> - KVM 2.8.1
>>
>> BR
>> M.K.
>>
>>
>>
>>
>>
>>
>>

Reply | Threaded
Open this post in threaded view
|

Re: Openbsd 6.1 and Current Console Freezes and lockup Proxmox PVE5.0

Mike Larkin
On Thu, Oct 12, 2017 at 10:36:42PM +0200, Michał Koc wrote:

>
> > On Thu, Oct 12, 2017 at 01:23:36PM +0200, Michał Koc wrote:
> > > On Sun, Oct 08, 2017 at 11:59:52PM +0200, Oliver Marugg wrote:
> > > > > On 7 Oct 2017, at 22:01, Mike Larkin wrote:
> > > > >
> > > > > > On Sat, Oct 07, 2017 at 02:19:58PM +0200, Oliver Marugg wrote:
> > > > > > > Just to add a 4th situation of hangs: Login via proxmox (pve)/kvm
> > > > > > > serial
> > > > > > > console (via noVNC), login successful: Vm guest in pve hangs, cpu
> > > > > > > usage at
> > > > > > > above 102%. Only way is to hard stop the Vm guest. -oliver
> > > > > > >
> > > > > > sounds like a kvm bug. Ask your provider to investigate the host side
> > > > > > when this
> > > > > > happens.
> > > > > Thanks Mike, will do so. The proxmox guys have also the idea that it could
> > > > > be a bug in kvm hypervisor (which is the hypervisor part for proxmox) and
> > > > > will affect OpenBSD since 4.9, they wrote me in their public forum. As far
> > > > > as I understood they do not know what OpenBSD needs in kvm or what/where
> > > > > should be fixed in kvm run OpenBSD without that freezes.
> > > > >
> > > > > -oliver
> > > > >From what I read, the cpu spins to 100%, which means somewhere on the host it's
> > > > likely spinning also. Start with systrace/ptrace/ktrace/whatever on the host
> > > > qemu-kvm and go from there...
> > > >
> > > > -ml
> > > >
> > > >
> > > >
> > > Hi,
> > >
> > > it looks like the cpu process of kvm (CPU 0/KVM) is issuing 1500+ of
> > > ioctl(15, KVM_RUN, 0)  per second while running OpenBSD 6.2 guest.
> > >
> > What CPU profile is being presented to the OpenBSD guest?
> >
> > I've seen things like this happen when a vCPU is claimed to have monitor/mwait
> > support, but the hypervisor implements those as NOPs, which just results in
> > spinning like this.
> >
> > In short - try changing the type of CPU presented to the guest and see if that
> > changes behaviour. At least then you'll have more data points to work with.
> >
> > -ml
>
> Okey,
> How would You disable monitor/mwait support in KVM to be presented to guest
> ?
>

Well, monitor/mwait was just what I recall contributing to something *like*
this.

If you can determine the guest %rip during each ioctl(vm_run) and give me a
kernel or disassembly I may be able to see if it's something obvious.

That, or describe a way I can repro this locally. I have a machine I could
put linux on for an evening to test.

-ml

> changing CPU to pentium or setting <feature policy='disable'
> name='monitor'/> does not actually change anything in scope of host cpu
> utilization....
>
> BR
> M.K.
>
>
> >
> > > In case of linux guest the process issues about 15 of those ioctls per
> > > second.
> > >
> > > In any case I cannot make openbsd to starve KVM host cpu. OpenBSD uses at
> > > most(when idle) 7% of cpu.
> > >
> > > My versions:
> > > - OpenBSD 6.2 amd64
> > > - KVM 2.8.1
> > >
> > > BR
> > > M.K.
> > >
> > >
> > >
> > >
> > >
> > >
> > >
>

Reply | Threaded
Open this post in threaded view
|

Re: Openbsd 6.1 and Current Console Freezes and lockup Proxmox PVE5.0

Mike Larkin
On Thu, Oct 12, 2017 at 03:11:31PM -0700, Mike Larkin wrote:

> On Thu, Oct 12, 2017 at 10:36:42PM +0200, Michał Koc wrote:
> >
> > > On Thu, Oct 12, 2017 at 01:23:36PM +0200, Michał Koc wrote:
> > > > On Sun, Oct 08, 2017 at 11:59:52PM +0200, Oliver Marugg wrote:
> > > > > > On 7 Oct 2017, at 22:01, Mike Larkin wrote:
> > > > > >
> > > > > > > On Sat, Oct 07, 2017 at 02:19:58PM +0200, Oliver Marugg wrote:
> > > > > > > > Just to add a 4th situation of hangs: Login via proxmox (pve)/kvm
> > > > > > > > serial
> > > > > > > > console (via noVNC), login successful: Vm guest in pve hangs, cpu
> > > > > > > > usage at
> > > > > > > > above 102%. Only way is to hard stop the Vm guest. -oliver
> > > > > > > >
> > > > > > > sounds like a kvm bug. Ask your provider to investigate the host side
> > > > > > > when this
> > > > > > > happens.
> > > > > > Thanks Mike, will do so. The proxmox guys have also the idea that it could
> > > > > > be a bug in kvm hypervisor (which is the hypervisor part for proxmox) and
> > > > > > will affect OpenBSD since 4.9, they wrote me in their public forum. As far
> > > > > > as I understood they do not know what OpenBSD needs in kvm or what/where
> > > > > > should be fixed in kvm run OpenBSD without that freezes.
> > > > > >
> > > > > > -oliver
> > > > > >From what I read, the cpu spins to 100%, which means somewhere on the host it's
> > > > > likely spinning also. Start with systrace/ptrace/ktrace/whatever on the host
> > > > > qemu-kvm and go from there...
> > > > >
> > > > > -ml
> > > > >
> > > > >
> > > > >
> > > > Hi,
> > > >
> > > > it looks like the cpu process of kvm (CPU 0/KVM) is issuing 1500+ of
> > > > ioctl(15, KVM_RUN, 0)  per second while running OpenBSD 6.2 guest.
> > > >
> > > What CPU profile is being presented to the OpenBSD guest?
> > >
> > > I've seen things like this happen when a vCPU is claimed to have monitor/mwait
> > > support, but the hypervisor implements those as NOPs, which just results in
> > > spinning like this.
> > >
> > > In short - try changing the type of CPU presented to the guest and see if that
> > > changes behaviour. At least then you'll have more data points to work with.
> > >
> > > -ml
> >
> > Okey,
> > How would You disable monitor/mwait support in KVM to be presented to guest
> > ?
> >
>
> Well, monitor/mwait was just what I recall contributing to something *like*
> this.
>

PS, IIRC qemu -cpu ? will show you a list of recognized cpuid flags, from
which you can subtract off things you don't want.

-ml

> If you can determine the guest %rip during each ioctl(vm_run) and give me a
> kernel or disassembly I may be able to see if it's something obvious.
>
> That, or describe a way I can repro this locally. I have a machine I could
> put linux on for an evening to test.
>
> -ml
>
> > changing CPU to pentium or setting <feature policy='disable'
> > name='monitor'/> does not actually change anything in scope of host cpu
> > utilization....
> >
> > BR
> > M.K.
> >
> >
> > >
> > > > In case of linux guest the process issues about 15 of those ioctls per
> > > > second.
> > > >
> > > > In any case I cannot make openbsd to starve KVM host cpu. OpenBSD uses at
> > > > most(when idle) 7% of cpu.
> > > >
> > > > My versions:
> > > > - OpenBSD 6.2 amd64
> > > > - KVM 2.8.1
> > > >
> > > > BR
> > > > M.K.
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> >
>

Reply | Threaded
Open this post in threaded view
|

Re: Openbsd 6.1 and Current Console Freezes and lockup Proxmox PVE5.0

Michał Koc

> On Thu, Oct 12, 2017 at 03:11:31PM -0700, Mike Larkin wrote:
>> On Thu, Oct 12, 2017 at 10:36:42PM +0200, Michał Koc wrote:
>>>> On Thu, Oct 12, 2017 at 01:23:36PM +0200, Michał Koc wrote:
>>>>> On Sun, Oct 08, 2017 at 11:59:52PM +0200, Oliver Marugg wrote:
>>>>>>> On 7 Oct 2017, at 22:01, Mike Larkin wrote:
>>>>>>>
>>>>>>>> On Sat, Oct 07, 2017 at 02:19:58PM +0200, Oliver Marugg wrote:
>>>>>>>>> Just to add a 4th situation of hangs: Login via proxmox (pve)/kvm
>>>>>>>>> serial
>>>>>>>>> console (via noVNC), login successful: Vm guest in pve hangs, cpu
>>>>>>>>> usage at
>>>>>>>>> above 102%. Only way is to hard stop the Vm guest. -oliver
>>>>>>>>>
>>>>>>>> sounds like a kvm bug. Ask your provider to investigate the host side
>>>>>>>> when this
>>>>>>>> happens.
>>>>>>> Thanks Mike, will do so. The proxmox guys have also the idea that it could
>>>>>>> be a bug in kvm hypervisor (which is the hypervisor part for proxmox) and
>>>>>>> will affect OpenBSD since 4.9, they wrote me in their public forum. As far
>>>>>>> as I understood they do not know what OpenBSD needs in kvm or what/where
>>>>>>> should be fixed in kvm run OpenBSD without that freezes.
>>>>>>>
>>>>>>> -oliver
>>>>>> >From what I read, the cpu spins to 100%, which means somewhere on the host it's
>>>>>> likely spinning also. Start with systrace/ptrace/ktrace/whatever on the host
>>>>>> qemu-kvm and go from there...
>>>>>>
>>>>>> -ml
>>>>>>
>>>>>>
>>>>>>
>>>>> Hi,
>>>>>
>>>>> it looks like the cpu process of kvm (CPU 0/KVM) is issuing 1500+ of
>>>>> ioctl(15, KVM_RUN, 0)  per second while running OpenBSD 6.2 guest.
>>>>>
>>>> What CPU profile is being presented to the OpenBSD guest?
>>>>
>>>> I've seen things like this happen when a vCPU is claimed to have monitor/mwait
>>>> support, but the hypervisor implements those as NOPs, which just results in
>>>> spinning like this.
>>>>
>>>> In short - try changing the type of CPU presented to the guest and see if that
>>>> changes behaviour. At least then you'll have more data points to work with.
>>>>
>>>> -ml
>>> Okey,
>>> How would You disable monitor/mwait support in KVM to be presented to guest
>>> ?
>>>
>> Well, monitor/mwait was just what I recall contributing to something *like*
>> this.
>>
> PS, IIRC qemu -cpu ? will show you a list of recognized cpuid flags, from
> which you can subtract off things you don't want.
Hi Mike,

Guest OpenBSD has those flags presented:
cpu0: FPU,VME,DE,PSE,TSC,MSR,MCE,CX8,APIC,MMX,HV,PERF

What else should I switch off to get desired effect ?

Best regards
M.K.

> -ml
>
>> If you can determine the guest %rip during each ioctl(vm_run) and give me a
>> kernel or disassembly I may be able to see if it's something obvious.
>>
>> That, or describe a way I can repro this locally. I have a machine I could
>> put linux on for an evening to test.
>>
>> -ml
>>
>>> changing CPU to pentium or setting <feature policy='disable'
>>> name='monitor'/> does not actually change anything in scope of host cpu
>>> utilization....
>>>
>>> BR
>>> M.K.
>>>
>>>
>>>>> In case of linux guest the process issues about 15 of those ioctls per
>>>>> second.
>>>>>
>>>>> In any case I cannot make openbsd to starve KVM host cpu. OpenBSD uses at
>>>>> most(when idle) 7% of cpu.
>>>>>
>>>>> My versions:
>>>>> - OpenBSD 6.2 amd64
>>>>> - KVM 2.8.1
>>>>>
>>>>> BR
>>>>> M.K.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>

Reply | Threaded
Open this post in threaded view
|

Re: Openbsd 6.1 and Current Console Freezes and lockup Proxmox PVE5.0

Oliver Marugg
In reply to this post by Mike Larkin
On 13 Oct 2017, at 0:11, Mike Larkin wrote:

> On Thu, Oct 12, 2017 at 10:36:42PM +0200, Michał Koc wrote:
>>
>>> On Thu, Oct 12, 2017 at 01:23:36PM +0200, Michał Koc wrote:
>>>> On Sun, Oct 08, 2017 at 11:59:52PM +0200, Oliver Marugg wrote:
>>>>>> On 7 Oct 2017, at 22:01, Mike Larkin wrote:
>>>>>>
>>>>>>> On Sat, Oct 07, 2017 at 02:19:58PM +0200, Oliver Marugg wrote:
>>>>>>>> Just to add a 4th situation of hangs: Login via proxmox
>>>>>>>> (pve)/kvm
>>>>>>>> serial
>>>>>>>> console (via noVNC), login successful: Vm guest in pve hangs,
>>>>>>>> cpu
>>>>>>>> usage at
>>>>>>>> above 102%. Only way is to hard stop the Vm guest. -oliver
>>>>>>>>
>>>>>>> sounds like a kvm bug. Ask your provider to investigate the host
>>>>>>> side
>>>>>>> when this
>>>>>>> happens.
>>>>>> Thanks Mike, will do so. The proxmox guys have also the idea that
>>>>>> it could
>>>>>> be a bug in kvm hypervisor (which is the hypervisor part for
>>>>>> proxmox) and
>>>>>> will affect OpenBSD since 4.9, they wrote me in their public
>>>>>> forum. As far
>>>>>> as I understood they do not know what OpenBSD needs in kvm or
>>>>>> what/where
>>>>>> should be fixed in kvm run OpenBSD without that freezes.
>>>>>>
>>>>>> -oliver
>>>>>> From what I read, the cpu spins to 100%, which means somewhere on
>>>>>> the host it's
>>>>> likely spinning also. Start with systrace/ptrace/ktrace/whatever
>>>>> on the host
>>>>> qemu-kvm and go from there...
>>>>>
>>>>> -ml
>>>>>
>>>>>
>>>>>
>>>> Hi,
>>>>
>>>> it looks like the cpu process of kvm (CPU 0/KVM) is issuing 1500+
>>>> of
>>>> ioctl(15, KVM_RUN, 0)  per second while running OpenBSD 6.2 guest.
>>>>
>>> What CPU profile is being presented to the OpenBSD guest?
>>>
>>> I've seen things like this happen when a vCPU is claimed to have
>>> monitor/mwait
>>> support, but the hypervisor implements those as NOPs, which just
>>> results in
>>> spinning like this.
>>>
>>> In short - try changing the type of CPU presented to the guest and
>>> see if that
>>> changes behaviour. At least then you'll have more data points to
>>> work with.
>>>
>>> -ml
>>
>> Okey,
>> How would You disable monitor/mwait support in KVM to be presented to
>> guest
>> ?
>>
>
> Well, monitor/mwait was just what I recall contributing to something
> *like*
> this.
>
> If you can determine the guest %rip during each ioctl(vm_run) and give
> me a
> kernel or disassembly I may be able to see if it's something obvious.
>
> That, or describe a way I can repro this locally. I have a machine I
> could
> put linux on for an evening to test.
>
> -ml
>
proxmox is debian 9 based they uses an ubuntu kernel 4.10.17-3-pve
(modified?). It seems that makes it a step harder to compare in this
case.

I tested my OpenBSD Guestswith different processor types like Mike
suggested, stablest fit shows as kvm64 (eq. pentium4 I think) which is
also standard in proxmox kvm. Other processors types like core2duo,
pentium or opteron freeze/hangs somewhen within 30 minutes after
starting, I tested all through.


>> changing CPU to pentium or setting <feature policy='disable'
>> name='monitor'/> does not actually change anything in scope of host
>> cpu
>> utilization....
>>
>> BR
>> M.K.
>>
>>
>>>
>>>> In case of linux guest the process issues about 15 of those ioctls
>>>> per
>>>> second.
>>>>
>>>> In any case I cannot make openbsd to starve KVM host cpu. OpenBSD
>>>> uses at
>>>> most(when idle) 7% of cpu.
>>>>
>>>> My versions:
>>>> - OpenBSD 6.2 amd64
>>>> - KVM 2.8.1
>>>>
>>>> BR
>>>> M.K.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>

Reply | Threaded
Open this post in threaded view
|

Re: Openbsd 6.1 and Current Console Freezes and lockup Proxmox PVE5.0

Mike Larkin
In reply to this post by Michał Koc
On Fri, Oct 13, 2017 at 11:05:12PM +0200, Michał Koc wrote:

>
> > On Thu, Oct 12, 2017 at 03:11:31PM -0700, Mike Larkin wrote:
> > > On Thu, Oct 12, 2017 at 10:36:42PM +0200, Michał Koc wrote:
> > > > > On Thu, Oct 12, 2017 at 01:23:36PM +0200, Michał Koc wrote:
> > > > > > On Sun, Oct 08, 2017 at 11:59:52PM +0200, Oliver Marugg wrote:
> > > > > > > > On 7 Oct 2017, at 22:01, Mike Larkin wrote:
> > > > > > > >
> > > > > > > > > On Sat, Oct 07, 2017 at 02:19:58PM +0200, Oliver Marugg wrote:
> > > > > > > > > > Just to add a 4th situation of hangs: Login via proxmox (pve)/kvm
> > > > > > > > > > serial
> > > > > > > > > > console (via noVNC), login successful: Vm guest in pve hangs, cpu
> > > > > > > > > > usage at
> > > > > > > > > > above 102%. Only way is to hard stop the Vm guest. -oliver
> > > > > > > > > >
> > > > > > > > > sounds like a kvm bug. Ask your provider to investigate the host side
> > > > > > > > > when this
> > > > > > > > > happens.
> > > > > > > > Thanks Mike, will do so. The proxmox guys have also the idea that it could
> > > > > > > > be a bug in kvm hypervisor (which is the hypervisor part for proxmox) and
> > > > > > > > will affect OpenBSD since 4.9, they wrote me in their public forum. As far
> > > > > > > > as I understood they do not know what OpenBSD needs in kvm or what/where
> > > > > > > > should be fixed in kvm run OpenBSD without that freezes.
> > > > > > > >
> > > > > > > > -oliver
> > > > > > > >From what I read, the cpu spins to 100%, which means somewhere on the host it's
> > > > > > > likely spinning also. Start with systrace/ptrace/ktrace/whatever on the host
> > > > > > > qemu-kvm and go from there...
> > > > > > >
> > > > > > > -ml
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > Hi,
> > > > > >
> > > > > > it looks like the cpu process of kvm (CPU 0/KVM) is issuing 1500+ of
> > > > > > ioctl(15, KVM_RUN, 0)  per second while running OpenBSD 6.2 guest.
> > > > > >
> > > > > What CPU profile is being presented to the OpenBSD guest?
> > > > >
> > > > > I've seen things like this happen when a vCPU is claimed to have monitor/mwait
> > > > > support, but the hypervisor implements those as NOPs, which just results in
> > > > > spinning like this.
> > > > >
> > > > > In short - try changing the type of CPU presented to the guest and see if that
> > > > > changes behaviour. At least then you'll have more data points to work with.
> > > > >
> > > > > -ml
> > > > Okey,
> > > > How would You disable monitor/mwait support in KVM to be presented to guest
> > > > ?
> > > >
> > > Well, monitor/mwait was just what I recall contributing to something *like*
> > > this.
> > >
> > PS, IIRC qemu -cpu ? will show you a list of recognized cpuid flags, from
> > which you can subtract off things you don't want.
> Hi Mike,
>
> Guest OpenBSD has those flags presented:
> cpu0: FPU,VME,DE,PSE,TSC,MSR,MCE,CX8,APIC,MMX,HV,PERF
>
> What else should I switch off to get desired effect ?
>

Those flags are completely bizarre. Compare to vmm(4):

cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,SEP,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SSE3,PCLMUL,SSSE3,CX16,SSE4.1,SSE4.2,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,RDTSCP,LONG,LAHF,ITSC,FSGSBASE,SMEP,ERMS

From what you said above, proxmox doesn't even expose PAE or PGE, which means
it's emulating something like a 1990s era 80486 CPU. It doesn't even claim to
support LONG, which means no 64 bit mode either.

It sounds like whatever hypervisor you are using is completely messed up. You
need to take this up with the proxmox or KVM people.

-ml

12