Performance issues as KVM guest?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Performance issues as KVM guest?

Mark Carroll
Since my hosting provider https://www.bytemark.co.uk/cloud-hosting/
patched for Meltdown last weekend I'm seeing significant performance
issues with an OpenBSD virtual instance there. It seems okay after a
fresh reboot but then progressively returns to being very slow: for
example "sleep 1" may take four seconds, then five, six, seven, then
rather more. Curiously it does tend to be an integral multiplier.

I wondered, is anybody else seeing significant performance problems with
OpenBSD (or other BSDs) virtual instances since Meltdown patching? Is
there anything to tweak at my end or am I reliant on the provider?

-- Mark


OpenBSD 6.1 (GENERIC) #26: Wed Oct  4 18:41:35 CEST 2017
    [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC
real mem = 1055870976 (1006MB)
avail mem = 1019322368 (972MB)
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.4 @ 0xf1c10 (10 entries)
bios0: vendor Bochs version "Bochs" date 01/01/2011
bios0: Bochs Bochs
acpi0 at bios0: rev 0
acpi0: sleep states S3 S4 S5
acpi0: tables DSDT FACP SSDT APIC HPET
acpi0: wakeup devices
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: QEMU Virtual CPU version 2.1.3, 2200.42 MHz
cpu0: FPU,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SSE3,CX16,POPCNT,HV,NXE,LONG,LAHF
cpu0: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 512KB 64b/line 16-way L2 cache
cpu0: ITLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu0: DTLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
cpu0: apic clock running at 1000MHz
ioapic0 at mainbus0: apid 0 pa 0xfec00000, version 11, 24 pins
acpihpet0 at acpi0: 100000000 Hz
acpiprt0 at acpi0: bus 0 (PCI0)
acpicpu0 at acpi0: C1(@1 halt!)
"ACPI0006" at acpi0 not configured
"PNP0303" at acpi0 not configured
"PNP0F13" at acpi0 not configured
"PNP0700" at acpi0 not configured
"PNP0400" at acpi0 not configured
"PNP0501" at acpi0 not configured
"ACPI0007" at acpi0 not configured
pvbus0 at mainbus0: KVM
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel 82441FX" rev 0x02
pcib0 at pci0 dev 1 function 0 "Intel 82371SB ISA" rev 0x00
pciide0 at pci0 dev 1 function 1 "Intel 82371SB IDE" rev 0x00: DMA, channel 0 wired to compatibility, channel 1 wired to compatibility
pciide0: channel 0 disabled (no drives)
atapiscsi0 at pciide0 channel 1 drive 0
scsibus1 at atapiscsi0: 2 targets
cd0 at scsibus1 targ 0 lun 0: <QEMU, QEMU DVD-ROM, 2.1.> ATAPI 5/cdrom removable
cd0(pciide0:1:0): using PIO mode 4, DMA mode 2
uhci0 at pci0 dev 1 function 2 "Intel 82371SB USB" rev 0x01: apic 0 int 11
piixpm0 at pci0 dev 1 function 3 "Intel 82371AB Power" rev 0x03: apic 0 int 9
iic0 at piixpm0
vga1 at pci0 dev 2 function 0 "Cirrus Logic CL-GD5446" rev 0x00
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
virtio0 at pci0 dev 3 function 0 "Qumranet Virtio Network" rev 0x00
vio0 at virtio0: address fe:ff:00:00:4f:1a
virtio0: msix shared
virtio1 at pci0 dev 4 function 0 "Qumranet Virtio Memory" rev 0x00
viomb0 at virtio1
virtio1: apic 0 int 11
virtio2 at pci0 dev 5 function 0 "Qumranet Virtio Storage" rev 0x00
vioblk0 at virtio2
scsibus2 at vioblk0: 2 targets
sd0 at scsibus2 targ 0 lun 0: <VirtIO, Block Device, > SCSI3 0/direct fixed
sd0: 25600MB, 512 bytes/sector, 52428800 sectors
virtio2: msix shared
"Intel 6300ESB WDT" rev 0x00 at pci0 dev 6 function 0 not configured
isa0 at pcib0
isadma0 at isa0
fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
fd0 at fdc0 drive 1: density unknown
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
com0: console
pckbc0 at isa0 port 0x60/5 irq 1 irq 12
pckbd0 at pckbc0 (kbd slot)
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pms0 at pckbc0 (aux slot)
wsmouse0 at pms0 mux 0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
lpt0 at isa0 port 0x378/4 irq 7
usb0 at uhci0: USB revision 1.0
uhub0 at usb0 configuration 1 interface 0 "Intel UHCI root hub" rev 1.00/1.00 addr 1
uhidev0 at uhub0 port 1 configuration 1 interface 0 "QEMU QEMU USB Tablet" rev 1.00/0.00 addr 2
uhidev0: iclass 3/0
ums0 at uhidev0: 3 buttons, Z dir
wsmouse1 at ums0 mux 0
vscsi0 at root
scsibus3 at vscsi0: 256 targets
softraid0 at root
scsibus4 at softraid0: 256 targets
root on sd0a (312946608d4051a2.a) swap on sd0b dump on sd0b

Reply | Threaded
Open this post in threaded view
|

Re: Performance issues as KVM guest?

Mike Larkin
On Wed, Jan 10, 2018 at 03:51:19PM +0000, Mark Carroll wrote:

> Since my hosting provider https://www.bytemark.co.uk/cloud-hosting/
> patched for Meltdown last weekend I'm seeing significant performance
> issues with an OpenBSD virtual instance there. It seems okay after a
> fresh reboot but then progressively returns to being very slow: for
> example "sleep 1" may take four seconds, then five, six, seven, then
> rather more. Curiously it does tend to be an integral multiplier.
>
> I wondered, is anybody else seeing significant performance problems with
> OpenBSD (or other BSDs) virtual instances since Meltdown patching? Is
> there anything to tweak at my end or am I reliant on the provider?
>
> -- Mark
>

There are a ton of threads talking about this issue, and it's not meltdown
specific. Please search the archives.

-ml

>
> OpenBSD 6.1 (GENERIC) #26: Wed Oct  4 18:41:35 CEST 2017
>     [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC
> real mem = 1055870976 (1006MB)
> avail mem = 1019322368 (972MB)
> mpath0 at root
> scsibus0 at mpath0: 256 targets
> mainbus0 at root
> bios0 at mainbus0: SMBIOS rev. 2.4 @ 0xf1c10 (10 entries)
> bios0: vendor Bochs version "Bochs" date 01/01/2011
> bios0: Bochs Bochs
> acpi0 at bios0: rev 0
> acpi0: sleep states S3 S4 S5
> acpi0: tables DSDT FACP SSDT APIC HPET
> acpi0: wakeup devices
> acpitimer0 at acpi0: 3579545 Hz, 24 bits
> acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
> cpu0 at mainbus0: apid 0 (boot processor)
> cpu0: QEMU Virtual CPU version 2.1.3, 2200.42 MHz
> cpu0: FPU,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SSE3,CX16,POPCNT,HV,NXE,LONG,LAHF
> cpu0: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 512KB 64b/line 16-way L2 cache
> cpu0: ITLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
> cpu0: DTLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
> cpu0: smt 0, core 0, package 0
> mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
> cpu0: apic clock running at 1000MHz
> ioapic0 at mainbus0: apid 0 pa 0xfec00000, version 11, 24 pins
> acpihpet0 at acpi0: 100000000 Hz
> acpiprt0 at acpi0: bus 0 (PCI0)
> acpicpu0 at acpi0: C1(@1 halt!)
> "ACPI0006" at acpi0 not configured
> "PNP0303" at acpi0 not configured
> "PNP0F13" at acpi0 not configured
> "PNP0700" at acpi0 not configured
> "PNP0400" at acpi0 not configured
> "PNP0501" at acpi0 not configured
> "ACPI0007" at acpi0 not configured
> pvbus0 at mainbus0: KVM
> pci0 at mainbus0 bus 0
> pchb0 at pci0 dev 0 function 0 "Intel 82441FX" rev 0x02
> pcib0 at pci0 dev 1 function 0 "Intel 82371SB ISA" rev 0x00
> pciide0 at pci0 dev 1 function 1 "Intel 82371SB IDE" rev 0x00: DMA, channel 0 wired to compatibility, channel 1 wired to compatibility
> pciide0: channel 0 disabled (no drives)
> atapiscsi0 at pciide0 channel 1 drive 0
> scsibus1 at atapiscsi0: 2 targets
> cd0 at scsibus1 targ 0 lun 0: <QEMU, QEMU DVD-ROM, 2.1.> ATAPI 5/cdrom removable
> cd0(pciide0:1:0): using PIO mode 4, DMA mode 2
> uhci0 at pci0 dev 1 function 2 "Intel 82371SB USB" rev 0x01: apic 0 int 11
> piixpm0 at pci0 dev 1 function 3 "Intel 82371AB Power" rev 0x03: apic 0 int 9
> iic0 at piixpm0
> vga1 at pci0 dev 2 function 0 "Cirrus Logic CL-GD5446" rev 0x00
> wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
> wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
> virtio0 at pci0 dev 3 function 0 "Qumranet Virtio Network" rev 0x00
> vio0 at virtio0: address fe:ff:00:00:4f:1a
> virtio0: msix shared
> virtio1 at pci0 dev 4 function 0 "Qumranet Virtio Memory" rev 0x00
> viomb0 at virtio1
> virtio1: apic 0 int 11
> virtio2 at pci0 dev 5 function 0 "Qumranet Virtio Storage" rev 0x00
> vioblk0 at virtio2
> scsibus2 at vioblk0: 2 targets
> sd0 at scsibus2 targ 0 lun 0: <VirtIO, Block Device, > SCSI3 0/direct fixed
> sd0: 25600MB, 512 bytes/sector, 52428800 sectors
> virtio2: msix shared
> "Intel 6300ESB WDT" rev 0x00 at pci0 dev 6 function 0 not configured
> isa0 at pcib0
> isadma0 at isa0
> fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
> fd0 at fdc0 drive 1: density unknown
> com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
> com0: console
> pckbc0 at isa0 port 0x60/5 irq 1 irq 12
> pckbd0 at pckbc0 (kbd slot)
> wskbd0 at pckbd0: console keyboard, using wsdisplay0
> pms0 at pckbc0 (aux slot)
> wsmouse0 at pms0 mux 0
> pcppi0 at isa0 port 0x61
> spkr0 at pcppi0
> lpt0 at isa0 port 0x378/4 irq 7
> usb0 at uhci0: USB revision 1.0
> uhub0 at usb0 configuration 1 interface 0 "Intel UHCI root hub" rev 1.00/1.00 addr 1
> uhidev0 at uhub0 port 1 configuration 1 interface 0 "QEMU QEMU USB Tablet" rev 1.00/0.00 addr 2
> uhidev0: iclass 3/0
> ums0 at uhidev0: 3 buttons, Z dir
> wsmouse1 at ums0 mux 0
> vscsi0 at root
> scsibus3 at vscsi0: 256 targets
> softraid0 at root
> scsibus4 at softraid0: 256 targets
> root on sd0a (312946608d4051a2.a) swap on sd0b dump on sd0b
>

Reply | Threaded
Open this post in threaded view
|

Re: Performance issues as KVM guest?

Mark Carroll
On 10 Jan 2018, Mike Larkin wrote:

> On Wed, Jan 10, 2018 at 03:51:19PM +0000, Mark Carroll wrote:
(snip)
>> I wondered, is anybody else seeing significant performance problems with
>> OpenBSD (or other BSDs) virtual instances since Meltdown patching? Is
>> there anything to tweak at my end or am I reliant on the provider?
>
> There are a ton of threads talking about this issue, and it's not meltdown
> specific. Please search the archives.

Ah, perhaps I was wrong to think they're something else -- I'm not their
only customer who found significant issues starting immediately after
last weekend's patching when things had worked fine before. I'll look
back at those threads more carefully then, perhaps they made some other
relevant change at the same time.

-- Mark

Reply | Threaded
Open this post in threaded view
|

Re: Performance issues as KVM guest?

Kent Watsen
In reply to this post by Mike Larkin
On 1/10/18 1:53 PM, Mike Larkin wrote:

> On Wed, Jan 10, 2018 at 03:51:19PM +0000, Mark Carroll wrote:
>> Since my hosting provider https://www.bytemark.co.uk/cloud-hosting/
>> patched for Meltdown last weekend I'm seeing significant performance
>> issues with an OpenBSD virtual instance there. It seems okay after a
>> fresh reboot but then progressively returns to being very slow: for
>> example "sleep 1" may take four seconds, then five, six, seven, then
>> rather more. Curiously it does tend to be an integral multiplier.
>>
>> I wondered, is anybody else seeing significant performance problems with
>> OpenBSD (or other BSDs) virtual instances since Meltdown patching? Is
>> there anything to tweak at my end or am I reliant on the provider?
>>
>> -- Mark
>>
> There are a ton of threads talking about this issue, and it's not meltdown
> specific. Please search the archives.
>
> -ml
>

Really? I just searched the last two years of list email for subject
lines having substrings "virt", "kvm", "perf", and "slow", and didn't
see anything on this specific issue.   Can you provide a link, or the
name of the thread, or some keywords?

Also, Mark, could you say some more about the issue.  For instance, how
long after a reboot does it take until you start to notice the issue,
and how quickly does it get worse?

Thanks,
Kent

Reply | Threaded
Open this post in threaded view
|

Re: Performance issues as KVM guest?

Mike Larkin
On Thu, Jan 11, 2018 at 05:38:18PM +0000, Kent Watsen wrote:

> On 1/10/18 1:53 PM, Mike Larkin wrote:
> > On Wed, Jan 10, 2018 at 03:51:19PM +0000, Mark Carroll wrote:
> > > Since my hosting provider https://www.bytemark.co.uk/cloud-hosting/
> > > patched for Meltdown last weekend I'm seeing significant performance
> > > issues with an OpenBSD virtual instance there. It seems okay after a
> > > fresh reboot but then progressively returns to being very slow: for
> > > example "sleep 1" may take four seconds, then five, six, seven, then
> > > rather more. Curiously it does tend to be an integral multiplier.
> > >
> > > I wondered, is anybody else seeing significant performance problems with
> > > OpenBSD (or other BSDs) virtual instances since Meltdown patching? Is
> > > there anything to tweak at my end or am I reliant on the provider?
> > >
> > > -- Mark
> > >
> > There are a ton of threads talking about this issue, and it's not meltdown
> > specific. Please search the archives.
> >
> > -ml
> >
>
> Really? I just searched the last two years of list email for subject lines
> having substrings "virt", "kvm", "perf", and "slow", and didn't see anything
> on this specific issue.   Can you provide a link, or the name of the thread,
> or some keywords?
>
> Also, Mark, could you say some more about the issue.  For instance, how long
> after a reboot does it take until you start to notice the issue, and how
> quickly does it get worse?
>
> Thanks,
> Kent
>

IIRC several of the threads referred to proxmox.

Reply | Threaded
Open this post in threaded view
|

Re: Performance issues as KVM guest?

Todd C. Miller-2
In reply to this post by Mark Carroll
This sounds like the same issue as was described here:
https://marc.info/?l=openbsd-bugs&m=151430928212450&w=2

 - todd

Reply | Threaded
Open this post in threaded view
|

Re: Performance issues as KVM guest?

Kirill Miazine
In reply to this post by Kent Watsen
* Kent Watsen [2018-01-11 17:38]:
[...]

> > > Since my hosting provider https://www.bytemark.co.uk/cloud-hosting/
> > > patched for Meltdown last weekend I'm seeing significant performance
> > > issues with an OpenBSD virtual instance there. It seems okay after a
> > > fresh reboot but then progressively returns to being very slow: for
> > > example "sleep 1" may take four seconds, then five, six, seven, then
> > > rather more. Curiously it does tend to be an integral multiplier.
> > >
> > > I wondered, is anybody else seeing significant performance problems with
> > > OpenBSD (or other BSDs) virtual instances since Meltdown patching? Is
> > > there anything to tweak at my end or am I reliant on the provider?
> > >
> > > -- Mark
> > >
> > There are a ton of threads talking about this issue, and it's not meltdown
> > specific. Please search the archives.
> >
> > -ml
> >
[...]
> Also, Mark, could you say some more about the issue.  For instance, how long
> after a reboot does it take until you start to notice the issue, and how
> quickly does it get worse?

I'm another customer of Bytemark experiencing the same issue. I'm taking
care of one VM there and I'm primarly noticing it in two situations:
sleep() takes a long time (e.g. sleep(1) might take up to 40 seconds)
and the clock slows down.

Right now, 9 hours after reboot, the clock on VM is 3 hours behind real
clock. And sleep(1) takes 13 secs:

km@buildfarm ~ $ time sleep 1
    0m13.85s real     0m00.00s user     0m00.01s system

This all started after the host was patched and VM rebooted.

Bytemark guys are looking at the issue and doing their own debugging.
Here're findings so far:

    I spun a few OpenBSD VMs up and left them overnight - looks like the
    clock isn't drifting but there's still the 'time sleep 1' issue.
    My testing results seemed to concur with User_4574's, virtio was slowing
    down only a few minutes after a fresh install whereas compatibility
    would stick at 1s, jump to 2s, etc.                                                                                    
>
> Thanks,
> Kent
>

--
    -- Kirill Miazine <[hidden email]>

Reply | Threaded
Open this post in threaded view
|

Re: Performance issues as KVM guest?

Infoomatic
Same problem here.
While we did have significant differences in cpu usage between FreeBSD and OpenBSD (basic OS without configuration: FreeBSD ~ 33min CPU time, OpenBSD ~ 474min CPU time - both started at the same time), with the latest kernel patches for Ubuntu 17.04 (our test environments all run Ubuntu 17.04 for KVM VMs), OpenBSD now becomes practically unusable: as soon as I su or login on the console with su, cpu usage is at 100% - the system freezes. :-/ guess we need some dedicated BSD machines to host some test-VMs ;-)

Regards,
Robert


> Gesendet: Donnerstag, 11. Januar 2018 um 20:32 Uhr
> Von: "Kirill Miazine" <[hidden email]>
> An: [hidden email]
> Betreff: Re: Performance issues as KVM guest?
>
> * Kent Watsen [2018-01-11 17:38]:
> [...]
> > > > Since my hosting provider https://www.bytemark.co.uk/cloud-hosting/
> > > > patched for Meltdown last weekend I'm seeing significant performance
> > > > issues with an OpenBSD virtual instance there. It seems okay after a
> > > > fresh reboot but then progressively returns to being very slow: for
> > > > example "sleep 1" may take four seconds, then five, six, seven, then
> > > > rather more. Curiously it does tend to be an integral multiplier.
> > > >
> > > > I wondered, is anybody else seeing significant performance problems with
> > > > OpenBSD (or other BSDs) virtual instances since Meltdown patching? Is
> > > > there anything to tweak at my end or am I reliant on the provider?
> > > >
> > > > -- Mark
> > > >
> > > There are a ton of threads talking about this issue, and it's not meltdown
> > > specific. Please search the archives.
> > >
> > > -ml
> > >
> [...]
> > Also, Mark, could you say some more about the issue.  For instance, how long
> > after a reboot does it take until you start to notice the issue, and how
> > quickly does it get worse?
>
> I'm another customer of Bytemark experiencing the same issue. I'm taking
> care of one VM there and I'm primarly noticing it in two situations:
> sleep() takes a long time (e.g. sleep(1) might take up to 40 seconds)
> and the clock slows down.
>
> Right now, 9 hours after reboot, the clock on VM is 3 hours behind real
> clock. And sleep(1) takes 13 secs:
>
> km@buildfarm ~ $ time sleep 1
>     0m13.85s real     0m00.00s user     0m00.01s system
>
> This all started after the host was patched and VM rebooted.
>
> Bytemark guys are looking at the issue and doing their own debugging.
> Here're findings so far:
>
>     I spun a few OpenBSD VMs up and left them overnight - looks like the
>     clock isn't drifting but there's still the 'time sleep 1' issue.
>     My testing results seemed to concur with User_4574's, virtio was slowing
>     down only a few minutes after a fresh install whereas compatibility
>     would stick at 1s, jump to 2s, etc.                                                                                    
> >
> > Thanks,
> > Kent
> >
>
> --
>     -- Kirill Miazine <[hidden email]>
>
>

Reply | Threaded
Open this post in threaded view
|

Re: Performance issues as KVM guest?

Stefan Fritsch
Hi,

I don't see this issue on my Debian system, but please try two things:

* disable kvm_intel.preemption_timer on the host
(see /sys/module/kvm_intel/parameters/preemption_timer )
This seems to be buggy in linux 4.10 and newer

* enable hpet in the vm config:
Make sure there is no <timer name='hpet' present='no'/>  in your libvirt
xml (or don't pass -ho-hpet to qemu). Unfortunately, newer libvirt
versions seem to disable hpet by default.


Different issue: If you remove the USB controllers, the CPU load on
the host will reduce by a few percent (~ 3%). Add

<controller type='usb' index='0' model='none'/>

and remove all other usb controller sections. Just removing the usb
controller sections without adding the 'none' makes libvirt add them back
(this is stupid).

Cheers,
Stefan

On Fri, 12 Jan 2018, Infoomatic wrote:

> Same problem here. While we did have significant differences in cpu
> usage between FreeBSD and OpenBSD (basic OS without configuration:
> FreeBSD ~ 33min CPU time, OpenBSD ~ 474min CPU time - both started at
> the same time), with the latest kernel patches for Ubuntu 17.04 (our
> test environments all run Ubuntu 17.04 for KVM VMs), OpenBSD now becomes
> practically unusable: as soon as I su or login on the console with su,
> cpu usage is at 100% - the system freezes. :-/ guess we need some
> dedicated BSD machines to host some test-VMs ;-)
>
> Regards,
> Robert
>
>
> > Gesendet: Donnerstag, 11. Januar 2018 um 20:32 Uhr
> > Von: "Kirill Miazine" <[hidden email]>
> > An: [hidden email]
> > Betreff: Re: Performance issues as KVM guest?
> >
> > * Kent Watsen [2018-01-11 17:38]:
> > [...]
> > > > > Since my hosting provider https://www.bytemark.co.uk/cloud-hosting/
> > > > > patched for Meltdown last weekend I'm seeing significant performance
> > > > > issues with an OpenBSD virtual instance there. It seems okay after a
> > > > > fresh reboot but then progressively returns to being very slow: for
> > > > > example "sleep 1" may take four seconds, then five, six, seven, then
> > > > > rather more. Curiously it does tend to be an integral multiplier.
> > > > >
> > > > > I wondered, is anybody else seeing significant performance problems with
> > > > > OpenBSD (or other BSDs) virtual instances since Meltdown patching? Is
> > > > > there anything to tweak at my end or am I reliant on the provider?
> > > > >
> > > > > -- Mark
> > > > >
> > > > There are a ton of threads talking about this issue, and it's not meltdown
> > > > specific. Please search the archives.
> > > >
> > > > -ml
> > > >
> > [...]
> > > Also, Mark, could you say some more about the issue.  For instance, how long
> > > after a reboot does it take until you start to notice the issue, and how
> > > quickly does it get worse?
> >
> > I'm another customer of Bytemark experiencing the same issue. I'm taking
> > care of one VM there and I'm primarly noticing it in two situations:
> > sleep() takes a long time (e.g. sleep(1) might take up to 40 seconds)
> > and the clock slows down.
> >
> > Right now, 9 hours after reboot, the clock on VM is 3 hours behind real
> > clock. And sleep(1) takes 13 secs:
> >
> > km@buildfarm ~ $ time sleep 1
> >     0m13.85s real     0m00.00s user     0m00.01s system
> >
> > This all started after the host was patched and VM rebooted.
> >
> > Bytemark guys are looking at the issue and doing their own debugging.
> > Here're findings so far:
> >
> >     I spun a few OpenBSD VMs up and left them overnight - looks like the
> >     clock isn't drifting but there's still the 'time sleep 1' issue.
> >     My testing results seemed to concur with User_4574's, virtio was slowing
> >     down only a few minutes after a fresh install whereas compatibility
> >     would stick at 1s, jump to 2s, etc.                                                                                    
> > >
> > > Thanks,
> > > Kent
> > >
> >
> > --
> >     -- Kirill Miazine <[hidden email]>
> >
> >
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Performance issues as KVM guest?

Tom Smyth
In reply to this post by Mark Carroll
Hello Todd,

This issue (Virtual hardware issue happens on latest proxmox5.x  but
not on Proxmox 4.4 ) with 6.2 (and 6.1 for that matter)
It is discussed here
https://marc.info/?l=openbsd-misc&m=151472854021947&w=2

but in recent versions of Proxmox 5.1 (QEMU/KVM)   there were no console freezes
(Proxmox updates fixed this issue (ie virtual  Hardware Fix not an
OpenBSD Software Fix)
https://marc.info/?l=openbsd-misc&m=151467636114177&w=2


So OpenBSD 6.2 Runs Fine on older QEMU and KVM but not on latest  KVM QEMU

My 2 cents is that the issue is a change in Virtual Hardware that is
incompatible with
OpenBSD  as opposed to change in OpenBSD that has caused the issue,


in my humble opinion it is more likely a old driver incompatibility
with newer (Virtual) hardware.



I hope this helps
Tom Smyth

Reply | Threaded
Open this post in threaded view
|

Re: Performance issues as KVM guest?

Infoomatic
In reply to this post by Stefan Fritsch
Hi Stefan,

Thanks a lot, that solved the problem!
However, I still wonder why the difference in cputime consumption between a FreeBSD KVM and a OpenBSD KVM (both just a basic install) is so huge ... now I see 643min on OpenBSD vs 46min on FreeBSD.

Regards,
Robert

> Gesendet: Freitag, 12. Januar 2018 um 12:48 Uhr
> Von: "Stefan Fritsch" <[hidden email]>
> An: Infoomatic <[hidden email]>
> Cc: [hidden email]
> Betreff: Re: Performance issues as KVM guest?
>
> Hi, I don't see this issue on my Debian system, but please try two things: * disable kvm_intel.preemption_timer on the host (see /sys/module/kvm_intel/parameters/preemption_timer ) This seems to be buggy in linux 4.10 and newer * enable hpet in the vm config: Make sure there is no in your libvirt xml (or don't pass -ho-hpet to qemu). Unfortunately, newer libvirt versions seem to disable hpet by default. Different issue: If you remove the USB controllers, the CPU load on the host will reduce by a few percent (~ 3%). Add and remove all other usb controller sections. Just removing the usb controller sections without adding the 'none' makes libvirt add them back (this is stupid). Cheers, Stefan On Fri, 12 Jan 2018, Infoomatic wrote: > Same problem here. While we did have significant differences in cpu > usage between FreeBSD and OpenBSD (basic OS without configuration: > FreeBSD ~ 33min CPU time, OpenBSD ~ 474min CPU time - both started at > the same time), with the latest kernel patches for Ubuntu 17.04 (our > test environments all run Ubuntu 17.04 for KVM VMs), OpenBSD now becomes > practically unusable: as soon as I su or login on the console with su, > cpu usage is at 100% - the system freezes. :-/ guess we need some > dedicated BSD machines to host some test-VMs ;-) > > Regards, > Robert > > > > Gesendet: Donnerstag, 11. Januar 2018 um 20:32 Uhr > > Von: "Kirill Miazine" > > An: [hidden email] > > Betreff: Re: Performance issues as KVM guest? > > > > * Kent Watsen [2018-01-11 17:38]: > > [...] > > > > > Since my hosting provider https://www.bytemark.co.uk/cloud-hosting/ > > > > > patched for Meltdown last weekend I'm seeing significant performance > > > > > issues with an OpenBSD virtual instance there. It seems okay after a > > > > > fresh reboot but then progressively returns to being very slow: for > > > > > example "sleep 1" may take four seconds, then five, six, seven, then > > > > > rather more. Curiously it does tend to be an integral multiplier. > > > > > > > > > > I wondered, is anybody else seeing significant performance problems with > > > > > OpenBSD (or other BSDs) virtual instances since Meltdown patching? Is > > > > > there anything to tweak at my end or am I reliant on the provider? > > > > > > > > > > -- Mark > > > > > > > > > There are a ton of threads talking about this issue, and it's not meltdown > > > > specific. Please search the archives. > > > > > > > > -ml > > > > > > [...] > > > Also, Mark, could you say some more about the issue.  For instance, how long > > > after a reboot does it take until you start to notice the issue, and how > > > quickly does it get worse? > > > > I'm another customer of Bytemark experiencing the same issue. I'm taking > > care of one VM there and I'm primarly noticing it in two situations: > > sleep() takes a long time (e.g. sleep(1) might take up to 40 seconds) > > and the clock slows down. > > > > Right now, 9 hours after reboot, the clock on VM is 3 hours behind real > > clock. And sleep(1) takes 13 secs: > > > > km@buildfarm ~ $ time sleep 1 > > 0m13.85s real 0m00.00s user 0m00.01s system > > > > This all started after the host was patched and VM rebooted. > > > > Bytemark guys are looking at the issue and doing their own debugging. > > Here're findings so far: > > > > I spun a few OpenBSD VMs up and left them overnight - looks like the > > clock isn't drifting but there's still the 'time sleep 1' issue. > > My testing results seemed to concur with User_4574's, virtio was slowing > > down only a few minutes after a fresh install whereas compatibility > > would stick at 1s, jump to 2s, etc. > > > > > > Thanks, > > > Kent > > > > > > > -- > > -- Kirill Miazine > > > > > >

Reply | Threaded
Open this post in threaded view
|

Re: Performance issues as KVM guest?

Tom Smyth
In reply to this post by Tom Smyth
Hello,

Just to clarify  Todds / Stefans Email earlier in the chain, (Thanks
Stefan and Todd,)


disable kvm_intel.preemption_timer on the host (see \
> /sys/module/kvm_intel/parameters/preemption_timer ) This seems to be buggy in linux \
> 4.10 and newer

disabling intel-kvm preemption_timer worked for me and it resolved the
timer drift issue
in openbsd on Proxmox 5.1

so in debian / proxmox  I ran the following command

echo options kvm-intel preemption_timer=N >>/etc/modprobe.d/kvm-intel.conf

then reboot the host,  (the open BSD guest vm default settings seemed
to work fine )


disable kvm_intel.preemption_timer on the host (see \
> /sys/module/kvm_intel/parameters/preemption_timer ) This seems to be buggy in linux \
> 4.10 and newer






On 12 January 2018 at 13:11, Tom Smyth <[hidden email]> wrote:

> Hello Todd,
>
> This issue (Virtual hardware issue happens on latest proxmox5.x  but
> not on Proxmox 4.4 ) with 6.2 (and 6.1 for that matter)
> It is discussed here
> https://marc.info/?l=openbsd-misc&m=151472854021947&w=2
>
> but in recent versions of Proxmox 5.1 (QEMU/KVM)   there were no console freezes
> (Proxmox updates fixed this issue (ie virtual  Hardware Fix not an
> OpenBSD Software Fix)
> https://marc.info/?l=openbsd-misc&m=151467636114177&w=2
>
>
> So OpenBSD 6.2 Runs Fine on older QEMU and KVM but not on latest  KVM QEMU
>
> My 2 cents is that the issue is a change in Virtual Hardware that is
> incompatible with
> OpenBSD  as opposed to change in OpenBSD that has caused the issue,
>
>
> in my humble opinion it is more likely a old driver incompatibility
> with newer (Virtual) hardware.
>
>
>
> I hope this helps
> Tom Smyth



--
Kindest regards,
Tom Smyth

Mobile: +353 87 6193172
The information contained in this E-mail is intended only for the
confidential use of the named recipient. If the reader of this message
is not the intended recipient or the person responsible for
delivering it to the recipient, you are hereby notified that you have
received this communication in error and that any review,
dissemination or copying of this communication is strictly prohibited.
If you have received this in error, please notify the sender
immediately by telephone at the number above and erase the message
You are requested to carry out your own virus check before
opening any attachment.