OpenBSD vs Linux KVM Guest Performance and Stability

[Daniel Santos]

I looking for some experience sharing from those who also run OpenBSD
inside KVM on Linux.

I was running 5.2 and now 6.2 inside KVM on a Linux wheezy host with 3.2
kernel.

This system was used as an all in one Zabbix monitoring server, which
included the full stack required to run a central monitoring server:

-Lighttpd
-Zabbix Server
-Zabbix Agent
-Mysql Server
-SNMP utils

First it was using Zabbix 2.x then I have upgraded it to the 3.x series.
The zabbix server was regularily crashing or stopped logging to the DB
and needed to be restarted.
If not that then the db and page load was terribly slow even on LAN and
I only had <15 clients on this server. I started moving the components
off one by one, at the end only the zabbix server engine left on the vm
even that sometimes just stuck and stopped sending data to a standalone
other linux mysql vm. Then I gave up, I recreated the whole environment
with the same Zabbix version on an all in one Debian 9 vm and since then
it runs flawlessly. Another issue was that I kept getting dropped packet
entries in my pflog on the enabled ports (seems like broken
connections).

Anyway my point now is not to configure another Zabbix but another
enterprise software suite (Pydio) with PHP 7.x on OpenBSD 6.3 in the
same KVM environment.

This would include:
-Lighttpd
-php7.x and some modules
-needs cifs mount on one part of the webhosting from an external windows
server
-pydio cifs storage connection to external windows server
-postfix/dovecot server

Were there any improvements made in the 6.3 virtualization code that
would provide me with a brighter outlook? Did anyone run into similar
performance and app crash issues on OBSD in KVM?

[Claudio Jeker]

On Mon, Apr 16, 2018 at 04:15:42PM +0200, Daniel Santos wrote:

> I looking for some experience sharing from those who also run OpenBSD inside
> KVM on Linux.
>
> I was running 5.2 and now 6.2 inside KVM on a Linux wheezy host with 3.2
> kernel.
>
> This system was used as an all in one Zabbix monitoring server, which
> included the full stack required to run a central monitoring server:
>
> -Lighttpd
> -Zabbix Server
> -Zabbix Agent
> -Mysql Server
> -SNMP utils
>
> First it was using Zabbix 2.x then I have upgraded it to the 3.x series. The
> zabbix server was regularily crashing or stopped logging to the DB and
> needed to be restarted.
> If not that then the db and page load was terribly slow even on LAN and I
> only had <15 clients on this server. I started moving the components off one
> by one, at the end only the zabbix server engine left on the vm even that
> sometimes just stuck and stopped sending data to a standalone other linux
> mysql vm. Then I gave up, I recreated the whole environment with the same
> Zabbix version on an all in one Debian 9 vm and since then it runs
> flawlessly. Another issue was that I kept getting dropped packet entries in
> my pflog on the enabled ports (seems like broken connections).
>

Zabbix needs a fast DB which I would not run virtualised. Also the crashes
you have seen may have been fixed in the meantime (between 6.2 and 6.3
there was an libc asr fix that made zabbix reliable for me).

I run a few KVM OpenBSD machines (mainly network / load balancers) they
work reasonably well.

--
:wq Claudio

[sven falempin]

On Mon, Apr 16, 2018 at 10:33 AM, Claudio Jeker <[hidden email]>
wrote:

> On Mon, Apr 16, 2018 at 04:15:42PM +0200, Daniel Santos wrote:
> > I looking for some experience sharing from those who also run OpenBSD
> inside
> > KVM on Linux.
> >
> > I was running 5.2 and now 6.2 inside KVM on a Linux wheezy host with 3.2
> > kernel.
> >
> > This system was used as an all in one Zabbix monitoring server, which
> > included the full stack required to run a central monitoring server:
> >
> > -Lighttpd
> > -Zabbix Server
> > -Zabbix Agent
> > -Mysql Server
> > -SNMP utils
> >
> > First it was using Zabbix 2.x then I have upgraded it to the 3.x series.
> The
> > zabbix server was regularily crashing or stopped logging to the DB and
> > needed to be restarted.
> > If not that then the db and page load was terribly slow even on LAN and I
> > only had <15 clients on this server. I started moving the components off
> one
> > by one, at the end only the zabbix server engine left on the vm even that
> > sometimes just stuck and stopped sending data to a standalone other linux
> > mysql vm. Then I gave up, I recreated the whole environment with the same
> > Zabbix version on an all in one Debian 9 vm and since then it runs
> > flawlessly. Another issue was that I kept getting dropped packet entries
> in
> > my pflog on the enabled ports (seems like broken connections).
> >
>
> Zabbix needs a fast DB which I would not run virtualised. Also the crashes
> you have seen may have been fixed in the meantime (between 6.2 and 6.3
> there was an libc asr fix that made zabbix reliable for me).
>
> I run a few KVM OpenBSD machines (mainly network / load balancers) they
> work reasonably well.
>
> --
> :wq Claudio
>
>

really depends on the KVM/linux version

--
--
---------------------------------------------------------------------------------------------------------------------
Knowing is not enough; we must apply. Willing is not enough; we must do

[Claudio Jeker]

On Mon, Apr 16, 2018 at 11:12:32AM -0400, sven falempin wrote:

> On Mon, Apr 16, 2018 at 10:33 AM, Claudio Jeker <[hidden email]>
> wrote:
>
> > On Mon, Apr 16, 2018 at 04:15:42PM +0200, Daniel Santos wrote:
> > > I looking for some experience sharing from those who also run OpenBSD
> > inside
> > > KVM on Linux.
> > >
> > > I was running 5.2 and now 6.2 inside KVM on a Linux wheezy host with 3.2
> > > kernel.
> > >
> > > This system was used as an all in one Zabbix monitoring server, which
> > > included the full stack required to run a central monitoring server:
> > >
> > > -Lighttpd
> > > -Zabbix Server
> > > -Zabbix Agent
> > > -Mysql Server
> > > -SNMP utils
> > >
> > > First it was using Zabbix 2.x then I have upgraded it to the 3.x series.
> > The
> > > zabbix server was regularily crashing or stopped logging to the DB and
> > > needed to be restarted.
> > > If not that then the db and page load was terribly slow even on LAN and I
> > > only had <15 clients on this server. I started moving the components off
> > one
> > > by one, at the end only the zabbix server engine left on the vm even that
> > > sometimes just stuck and stopped sending data to a standalone other linux
> > > mysql vm. Then I gave up, I recreated the whole environment with the same
> > > Zabbix version on an all in one Debian 9 vm and since then it runs
> > > flawlessly. Another issue was that I kept getting dropped packet entries
> > in
> > > my pflog on the enabled ports (seems like broken connections).
> > >
> >
> > Zabbix needs a fast DB which I would not run virtualised. Also the crashes
> > you have seen may have been fixed in the meantime (between 6.2 and 6.3
> > there was an libc asr fix that made zabbix reliable for me).
> >
> > I run a few KVM OpenBSD machines (mainly network / load balancers) they
> > work reasonably well.
> >
> > --
> > :wq Claudio
> >
> >
> really depends on the KVM/linux version
>

Don't forget to set "options kvm-intel preemption_timer=0" for modprobe on
newer linux kernels. After that it seems to work nicely.

--
:wq Claudio

[Stuart Henderson]

On 2018-04-16, Daniel Santos <[hidden email]> wrote:

> Anyway my point now is not to configure another Zabbix but another
> enterprise software suite (Pydio) with PHP 7.x on OpenBSD 6.3 in the
> same KVM environment.
>
> This would include:
> -Lighttpd
> -php7.x and some modules
> -needs cifs mount on one part of the webhosting from an external windows
> server
> -pydio cifs storage connection to external windows server

Do you mean you need to mount a share from a Windows server via CIFS from
the OpenBSD machine? The only option for that is via FUSE and it's flaky.

[Kapetanakis Giannis]

On 16/04/18 18:40, Claudio Jeker wrote:
>
>> really depends on the KVM/linux version
>>
> Don't forget to set "options kvm-intel preemption_timer=0" for modprobe on
> newer linux kernels. After that it seems to work nicely.
>

This module option (according to lists) is about timing issues with kvm
and obsd.
Does it affect performance as well?

which kernel?

G

[Claudio Jeker]

On Mon, Apr 16, 2018 at 11:10:46PM +0300, Kapetanakis Giannis wrote:

> On 16/04/18 18:40, Claudio Jeker wrote:
> >
> >>really depends on the KVM/linux version
> >>
> >Don't forget to set "options kvm-intel preemption_timer=0" for modprobe on
> >newer linux kernels. After that it seems to work nicely.
> >
>
> This module option (according to lists) is about timing issues with kvm and
> obsd.
> Does it affect performance as well?

It affects stability (sometimes VM hang because the clock interrupts don't
work correctly) which is a big performace killer. Also for me stat clock
was not running so there was no way to know what CPU load the VM has.
 
> which kernel?

I see it for sure on a 4.14 kernel. Guess more affected.

--
:wq Claudio

[Daniel Santos]

On 2018-04-16 23:00, Claudio Jeker wrote:

> On Mon, Apr 16, 2018 at 11:10:46PM +0300, Kapetanakis Giannis wrote:
>> On 16/04/18 18:40, Claudio Jeker wrote:
>> >
>> >>really depends on the KVM/linux version
>> >>
>> >Don't forget to set "options kvm-intel preemption_timer=0" for modprobe on
>> >newer linux kernels. After that it seems to work nicely.
>> >
>>
>> This module option (according to lists) is about timing issues with
>> kvm and
>> obsd.
>> Does it affect performance as well?
>
> It affects stability (sometimes VM hang because the clock interrupts
> don't
> work correctly) which is a big performace killer. Also for me stat
> clock
> was not running so there was no way to know what CPU load the VM has.
>
>> which kernel?
>
> I see it for sure on a 4.14 kernel. Guess more affected.

For me the kernels:

Host: 3.2.0-4-amd64 Debian Wheezy 7.11 (and no I don't want to upgrade
this, it's running production vms over 300 days).

Modules:

kvm_intel             122046  78
kvm                   291951  1 kvm_intel

I don't use any kvm module options on the Linux host. My host also runs
bunch of other VMs (linuxes with different kernels, freeBSD, windows 7).

Guest: OpenBSD 6.2 default


Searching the module option you have recommended, I have run across this
thread:

http://openbsd-archive.7691.n7.nabble.com/Degraded-timing-performance-QEMU-KVM-OpenBSD-6-2-td333611.html

I have done the "while date loop" test on both my old 6.2 and new 6.3
OpenBSD vm for 30 minutes and I don't experience any slowdowns so this
issue might not affect me but there the OP was using Fedora which is a
complex hodgepodge distro compared to Debian.

[Kapetanakis Giannis]

On 17/04/18 10:28, Daniel Santos wrote:

> On 2018-04-16 23:00, Claudio Jeker wrote:
>> On Mon, Apr 16, 2018 at 11:10:46PM +0300, Kapetanakis Giannis wrote:
>>> On 16/04/18 18:40, Claudio Jeker wrote:
>>> >
>>> >>really depends on the KVM/linux version
>>> >>
>>> >Don't forget to set "options kvm-intel preemption_timer=0" for modprobe on
>>> >newer linux kernels. After that it seems to work nicely.
>>> >
>>>
>>> This module option (according to lists) is about timing issues with kvm and
>>> obsd.
>>> Does it affect performance as well?
>>
>> It affects stability (sometimes VM hang because the clock interrupts don't
>> work correctly) which is a big performace killer. Also for me stat clock
>> was not running so there was no way to know what CPU load the VM has.
>>
>>> which kernel?
>>
>> I see it for sure on a 4.14 kernel. Guess more affected.
>
> For me the kernels:
>
> Host: 3.2.0-4-amd64 Debian Wheezy 7.11 (and no I don't want to upgrade this, it's running production vms over 300 days).
>
> Modules:
>
> kvm_intel             122046  78
> kvm                   291951  1 kvm_intel
>
> I don't use any kvm module options on the Linux host. My host also runs bunch of other VMs (linuxes with different kernels, freeBSD, windows 7).

That is why I've asked the kernel version from Claudio.

I also use Centos7 with kernel 3.10.0-693.21.1.el7.x86_64 on supervisor machines
and I don't experience any timing issues or hungs as reported on those threads.
Not using any custom modprobe options either.

Stability is fine and performance is as it is.

G