vmm timer for linux guests

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

vmm timer for linux guests

Sivaram Gowkanapalli
Hello Mike,

From #openbsd, I gather that you are working on the timer issue (time falling behind) that linux guests have in vmm.

I run linux guests in vmm and I am using a linux module just to sync the timer every 5 seconds from the host. I presume that there might be a better solution than that. Just want to check if you might have any test code or material that can help me work around the issue.

I tried this and the timer in the linux guest is kvm-clock and it avoids the timer issue. But, the linux guests hang once in a while. I presume that it might be due to some kvm instruction unsupported by vmm. I got this idea from https://marc.info/?l=openbsd-tech&m=155903945227814&w=2

Index: sys/arch/amd64/amd64/vmm.c
===================================================================
RCS file: /cvs/src/sys/arch/amd64/amd64/vmm.c,v
retrieving revision 1.273
diff -u -p -u -r1.273 vmm.c
--- sys/arch/amd64/amd64/vmm.c  19 Apr 2020 19:29:52 -0000      1.273
+++ sys/arch/amd64/amd64/vmm.c  20 May 2020 17:28:12 -0000
@@ -242,6 +242,7 @@ extern uint64_t tsc_frequency;
 extern int tsc_is_invariant;
 
 const char *vmm_hv_signature = VMM_HV_SIGNATURE;
+const char *kvm_hv_signature = KVM_HV_SIGNATURE;
 
 const struct kmem_pa_mode vmm_kp_contig = {
        .kp_constraint = &no_constraint,
@@ -6830,7 +6831,14 @@ vmm_handle_cpuid(struct vcpu *vcpu)
                *rcx = *((uint32_t *)&vmm_hv_signature[4]);
                *rdx = *((uint32_t *)&vmm_hv_signature[8]);
                break;
+       case 0x40000100: /* KVM CPUID signature */
+               *rax = 0;
+               *rbx = *((uint32_t *)&kvm_hv_signature[0]);
+               *rcx = *((uint32_t *)&kvm_hv_signature[4]);
+               *rdx = *((uint32_t *)&kvm_hv_signature[8]);
+               break;
        case 0x40000001:        /* KVM hypervisor features */
+       case 0x40000101:
                *rax = (1 << KVM_FEATURE_CLOCKSOURCE2) |
                    (1 << KVM_FEATURE_CLOCKSOURCE_STABLE_BIT);
                *rbx = 0;
Index: sys/arch/amd64/include/vmmvar.h
===================================================================
RCS file: /cvs/src/sys/arch/amd64/include/vmmvar.h,v
retrieving revision 1.70
diff -u -p -u -r1.70 vmmvar.h
--- sys/arch/amd64/include/vmmvar.h     8 Apr 2020 07:39:48 -0000       1.70
+++ sys/arch/amd64/include/vmmvar.h     20 May 2020 17:28:12 -0000
@@ -22,6 +22,7 @@
 #define _MACHINE_VMMVAR_H_
 
 #define VMM_HV_SIGNATURE       "OpenBSDVMM58"
+#define KVM_HV_SIGNATURE       "KVMKVMKVM\0\0\0"
 
 #define VMM_MAX_MEM_RANGES     16
 #define VMM_MAX_DISKS_PER_VM   4

Thanks

Reply | Threaded
Open this post in threaded view
|

Re: vmm timer for linux guests

Renato Aguiar-2
Hi Sivaram,

I'm the author of the e-mail thread that you mentioned. After
feedback I got from OpenBSD community, I created a patch for Linux
to enable kvm-clock when booting on VMM. It managed to keep clock
in sync, but I experienced random crashes in vmd after some time
running the VM.

Multiple fixes have been merged to vmm/vmd since then, so I'm
planning to give it another try with OpenBSD 6.7.

If you are interested, I can share the patch with you.

Regards,

--
Renato Aguiar

Reply | Threaded
Open this post in threaded view
|

Re: vmm timer for linux guests

Pratik Vyas
* Renato Aguiar <[hidden email]> [2020-05-21 12:55:45 -0700]:

>Hi Sivaram,
>
>I'm the author of the e-mail thread that you mentioned. After feedback
>I got from OpenBSD community, I created a patch for Linux to enable
>kvm-clock when booting on VMM. It managed to keep clock in sync, but I
>experienced random crashes in vmd after some time running the VM.
>
>Multiple fixes have been merged to vmm/vmd since then, so I'm planning
>to give it another try with OpenBSD 6.7.
>
>If you are interested, I can share the patch with you.
>
>Regards,
>
>--
>Renato Aguiar
>

Hi Sivaram and Renato,

I have been looking at this issue this week with Dave.  (funnily Dave
and I independently also decided to write a linux patch to attach
pvclock and debugging these crashes).

If you have your linux vm with kvm-clock around, can you try this rather
crude patch and see if it still crashes?  This might also fix the '100%
cpu when using alpine via console' problem.

For ref, this is my linux side patch to attach kvm-clock http://ix.io/2lzK

--
Pratik

diff --git a/usr.sbin/vmd/ns8250.c b/usr.sbin/vmd/ns8250.c
index 497e6fad550..33f1a371c16 100644
--- a/usr.sbin/vmd/ns8250.c
+++ b/usr.sbin/vmd/ns8250.c
@@ -36,6 +36,8 @@
 
  extern char *__progname;
  struct ns8250_dev com1_dev;
+struct event read_delay_ev;
+struct timeval read_delay_tv;
 
  static void com_rcv_event(int, short, void *);
  static void com_rcv(struct ns8250_dev *, uint32_t, uint32_t);
@@ -61,6 +63,11 @@ ratelimit(int fd, short type, void *arg)
  vcpu_deassert_pic_irq(com1_dev.vmid, 0, com1_dev.irq);
  }
 
+static void
+arm_read(int fd, short type, void *arg) {
+ event_add(&com1_dev.event, NULL);
+}
+
  void
  ns8250_init(int fd, uint32_t vmid)
  {
@@ -96,7 +103,7 @@ ns8250_init(int fd, uint32_t vmid)
  */
  com1_dev.pause_ct = (com1_dev.baudrate / 8) / 1000 * 10;
 
- event_set(&com1_dev.event, com1_dev.fd, EV_READ | EV_PERSIST,
+ event_set(&com1_dev.event, com1_dev.fd, EV_READ,
     com_rcv_event, (void *)(intptr_t)vmid);
 
  /*
@@ -112,6 +119,9 @@ ns8250_init(int fd, uint32_t vmid)
  timerclear(&com1_dev.rate_tv);
  com1_dev.rate_tv.tv_usec = 10000;
  evtimer_set(&com1_dev.rate, ratelimit, NULL);
+ read_delay_tv.tv_usec = 10000;
+ evtimer_set(&read_delay_ev, arm_read,
+ (void *)(intptr_t)vmid);
  }
 
  static void
@@ -131,6 +141,7 @@ com_rcv_event(int fd, short kind, void *arg)
  */
  if (com1_dev.rcv_pending) {
  mutex_unlock(&com1_dev.mutex);
+ evtimer_add(&read_delay_ev, &read_delay_tv);
  return;
  }
 
@@ -146,6 +157,7 @@ com_rcv_event(int fd, short kind, void *arg)
  vcpu_deassert_pic_irq((uintptr_t)arg, 0, com1_dev.irq);
  }
  }
+ event_add(&com1_dev.event, NULL);
 
  mutex_unlock(&com1_dev.mutex);
  }

Reply | Threaded
Open this post in threaded view
|

Re: vmm timer for linux guests

Sivaram Gowkanapalli
Hello Pratik,

> For ref, this is my linux side patch to attach kvm-clock http://ix.io/2lzK

Thanks, will try the below patches.

Reply | Threaded
Open this post in threaded view
|

Re: vmm timer for linux guests

Sivaram Gowkanapalli
In reply to this post by Renato Aguiar-2
Hello Renato,

Could you please share the linux patches? We can try with them too.

Thanks

________________________________________
From: Renato Aguiar [[hidden email]]
Sent: Thursday, May 21, 2020 3:55 PM
To: [hidden email]
Cc: Sivaram Gowkanapalli
Subject: Re: vmm timer for linux guests

Hi Sivaram,

I'm the author of the e-mail thread that you mentioned. After
feedback I got from OpenBSD community, I created a patch for Linux
to enable kvm-clock when booting on VMM. It managed to keep clock
in sync, but I experienced random crashes in vmd after some time
running the VM.

Multiple fixes have been merged to vmm/vmd since then, so I'm
planning to give it another try with OpenBSD 6.7.

If you are interested, I can share the patch with you.

Regards,

--
Renato Aguiar

Reply | Threaded
Open this post in threaded view
|

Re: vmm timer for linux guests

Dave Voutila-2
In reply to this post by Pratik Vyas
On Fri, May 22, 2020 at 6:15 AM Pratik Vyas <[hidden email]> wrote:
> Hi Sivaram and Renato,
>
> I have been looking at this issue this week with Dave.  (funnily Dave
> and I independently also decided to write a linux patch to attach
> pvclock and debugging these crashes).

My hacky "vmm-clock" implementation is here if you'd like to compare
notes: https://git.sr.ht/~voutilad/linux/log/linux-5.4-obsd

My goal is to strip down the implementation to the bare minimum needed
for an OpenBSD guest similar to what I did with my vmmci(4)
implementation: https://github.com/voutilad/virtio_vmmci

One thing I've found is (at least with busybox's top(1)), the
accounting for cpu utilization is wrong if you just wire up the
kvm-clock clocksource.

>
> If you have your linux vm with kvm-clock around, can you try this rather
> crude patch and see if it still crashes?  This might also fix the '100%
> cpu when using alpine via console' problem.

From my quick testing, with both the standard Linux 5.4 kernel with
Alpine 3.11 and my 5.4.40 fork, it resolves the CPU utilization issue
from the serial console, but doesn't resolve the serial console lockup
when spamming the return key. I haven't dug into why yet.

> For ref, this is my linux side patch to attach kvm-clock http://ix.io/2lzK
>
> @@ -96,7 +103,7 @@ ns8250_init(int fd, uint32_t vmid)
>          */
>         com1_dev.pause_ct = (com1_dev.baudrate / 8) / 1000 * 10;
>
> -       event_set(&com1_dev.event, com1_dev.fd, EV_READ | EV_PERSIST,
> +       event_set(&com1_dev.event, com1_dev.fd, EV_READ,
>             com_rcv_event, (void *)(intptr_t)vmid);

pd@ I think you're missing the analogous code in ns8250_restore().

--
Dave Voutila

Reply | Threaded
Open this post in threaded view
|

Re: vmm timer for linux guests

Sivaram Gowkanapalli
Hello Dave,

Thanks for sharing this. I have been using your hacked up version of virtio_vmmci to keep the timer in sync all this time.

btw, would you recommend a linux distro other than alpine for vmm? I need to run an Oracle Java app which needs glibc, hence, I cannot use alpine. I tried tiny core but had trouble with the install in my initial attempts. I have not dug into the issue yet. Just want to check if there is a better option instead.

I am using devuan and it has been working fine in the vm.

Thanks again
Reply | Threaded
Open this post in threaded view
|

Re: vmm timer for linux guests

Renato Aguiar-2
In reply to this post by Sivaram Gowkanapalli
Hi Sivaram and David,

Here is my patch for Linux 4.19:
https://gitlab.com/renatoaguiar/linux/-/commit/325afccf1e2d156be745f811411327e22bbd4c20

I also tried same patch with Linux 5.4
<https://gitlab.com/renatoaguiar/linux/-/commit/52d04b3eb2405a3034333a820645d239f622bd92>,
but for some reason 4.19 seemed more stable.

Regards,

--
Renato Aguiar

Reply | Threaded
Open this post in threaded view
|

Re: vmm timer for linux guests

Sivaram Gowkanapalli
In reply to this post by Pratik Vyas
Hello Pratik,

Thanks for the patch. I am about to test it out too.

If you do not mind sharing, do you have your patches hosted somewhere?

Thanks

________________________________________
From: Pratik Vyas [[hidden email]]
Sent: Friday, May 22, 2020 6:15 AM
To: Renato Aguiar
Cc: [hidden email]; Sivaram Gowkanapalli; [hidden email]
Subject: Re: vmm timer for linux guests

* Renato Aguiar <[hidden email]> [2020-05-21 12:55:45 -0700]:

>Hi Sivaram,
>
>I'm the author of the e-mail thread that you mentioned. After feedback
>I got from OpenBSD community, I created a patch for Linux to enable
>kvm-clock when booting on VMM. It managed to keep clock in sync, but I
>experienced random crashes in vmd after some time running the VM.
>
>Multiple fixes have been merged to vmm/vmd since then, so I'm planning
>to give it another try with OpenBSD 6.7.
>
>If you are interested, I can share the patch with you.
>
>Regards,
>
>--
>Renato Aguiar
>

Hi Sivaram and Renato,

I have been looking at this issue this week with Dave.  (funnily Dave
and I independently also decided to write a linux patch to attach
pvclock and debugging these crashes).

If you have your linux vm with kvm-clock around, can you try this rather
crude patch and see if it still crashes?  This might also fix the '100%
cpu when using alpine via console' problem.

For ref, this is my linux side patch to attach kvm-clock http://ix.io/2lzK

--
Pratik

diff --git a/usr.sbin/vmd/ns8250.c b/usr.sbin/vmd/ns8250.c
index 497e6fad550..33f1a371c16 100644
--- a/usr.sbin/vmd/ns8250.c
+++ b/usr.sbin/vmd/ns8250.c
@@ -36,6 +36,8 @@

  extern char *__progname;
  struct ns8250_dev com1_dev;
+struct event read_delay_ev;
+struct timeval read_delay_tv;

  static void com_rcv_event(int, short, void *);
  static void com_rcv(struct ns8250_dev *, uint32_t, uint32_t);
@@ -61,6 +63,11 @@ ratelimit(int fd, short type, void *arg)
        vcpu_deassert_pic_irq(com1_dev.vmid, 0, com1_dev.irq);
  }

+static void
+arm_read(int fd, short type, void *arg) {
+       event_add(&com1_dev.event, NULL);
+}
+
  void
  ns8250_init(int fd, uint32_t vmid)
  {
@@ -96,7 +103,7 @@ ns8250_init(int fd, uint32_t vmid)
         */
        com1_dev.pause_ct = (com1_dev.baudrate / 8) / 1000 * 10;

-       event_set(&com1_dev.event, com1_dev.fd, EV_READ | EV_PERSIST,
+       event_set(&com1_dev.event, com1_dev.fd, EV_READ,
            com_rcv_event, (void *)(intptr_t)vmid);

        /*
@@ -112,6 +119,9 @@ ns8250_init(int fd, uint32_t vmid)
        timerclear(&com1_dev.rate_tv);
        com1_dev.rate_tv.tv_usec = 10000;
        evtimer_set(&com1_dev.rate, ratelimit, NULL);
+       read_delay_tv.tv_usec = 10000;
+       evtimer_set(&read_delay_ev, arm_read,
+                       (void *)(intptr_t)vmid);
  }

  static void
@@ -131,6 +141,7 @@ com_rcv_event(int fd, short kind, void *arg)
         */
        if (com1_dev.rcv_pending) {
                mutex_unlock(&com1_dev.mutex);
+               evtimer_add(&read_delay_ev, &read_delay_tv);
                return;
        }

@@ -146,6 +157,7 @@ com_rcv_event(int fd, short kind, void *arg)
                        vcpu_deassert_pic_irq((uintptr_t)arg, 0, com1_dev.irq);
                }
        }
+       event_add(&com1_dev.event, NULL);

        mutex_unlock(&com1_dev.mutex);
  }

Reply | Threaded
Open this post in threaded view
|

Re: vmm timer for linux guests

Sivaram Gowkanapalli
In reply to this post by Pratik Vyas

Hello Pratik,

I thought the serial console has been stable since this patch. I just noticed a hang. So, the issue seems to still persist.

Thanks

Reply | Threaded
Open this post in threaded view
|

Re: vmm timer for linux guests

Sivaram Gowkanapalli
In reply to this post by Dave Voutila-2
Hello Dave, Pratik,

> My hacky "vmm-clock" implementation is here if you'd like to compare
> notes: https://git.sr.ht/~voutilad/linux/log/linux-5.4-obsd

The clock is in sync with these patches. It is keeping the clock in sync with the host.

sudo dmesg | grep -i clock
[    0.034850] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[    0.099077] [vmm_get_wallclock]
[    0.099083] [vmm_get_wallclock] wallclock: 1590341994.140345631
[    0.204807] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x618c5af5a59, max_idle_ns: 440795234522 ns
[    1.214829] clocksource: timekeeping watchdog on CPU0: Marking clocksource 'tsc-early' as unstable because the skew is too large:
[    1.214831] clocksource:                       'vmm-clock' wd_now: 1612079e9ed2e025 wd_last: 1612079e63384222 mask: ffffffffffffffff
[    1.214832] clocksource:                       'tsc-early' cs_now: 824f14258d7a cs_last: 824e48e69b4f mask: ffffffffffffffff
[    1.214833] tsc: Marking TSC unstable due to clocksource watchdog
[    1.241836] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[    1.246960] PTP clock support registered
[    1.254545] clocksource: Switched to clocksource vmm-clock
[    3.412453] rtc_cmos rtc_cmos: setting system clock to 2020-05-24T17:39:57 UTC (1590341997)
[    3.412468] Unstable clock detected, switching default tracing clock to "global"
[    3.412468] If you want to keep using the local clock, then add:
[    3.412468]   "trace_clock=local"

For the next guy with this issue on alpine, http://ix.io/2nfZ is the patch I used against the alpine aports git repo to build the kernel.

Thanks