Severe clock problems with OpenBSD VM on OpenBSD Host

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Severe clock problems with OpenBSD VM on OpenBSD Host

Stefan Arentz
Hi everyone,

I am having an issue where an OpenBSD VM running on vmd is having
serious clock skew issues.

I am relatively new to OpenBSD, so I am not sure how to properly debug
this. What I hope is that I can provide a good amount of data and folks
here can give me some hints and ask me for additional information to
get to the root cause of this.

So first some facts and symptoms:

- Both Host and Guest are running OpenBSD 6.4. The host runs GENERIC.MP
  and the guest GENERIC.
- The host runs 50 guests, all OpenBSD (openbsd.amsterdam)
- Only this VM is having this clock issue (is this correct, or were
  there others?)

- The guest has kern.timecounter.hardware=tsc
- The time on the VM was set with rdate a couple of days ago, and as of
  now the VM is running about 4 hours behind.
- ntpd is running (main process, dns engine, ntp engine)
- when started or restarted, ntpd complains about "pipe write error
  (from main): No such file or directory" but does seem to start

- I just ran rdate nl.pool.ntp.org and the date was properly updated
- One minute after running rdate, the clock is already 7 seconds slow

- The guest also has some severe networking issues. often I cannot type
  more than a few characters before a ~15 second delays happens.
  Interactive typing is difficult.
- I can SSH into the Host and have none of these issues, ruling out
  connectivity issues between me (Toronto) and the Host (Amsterdam)

It would be easy to blame this on NTPd, which does have an unexplained
error message. However, I think even without running NTPd, the clock
skew should not be this extreme.

Somehow I have a gut feeling that the clock issues and the networking
issues are related.

I am root on the VM but I am not on the host. I do have vmctl access.
However, the host admin is friendly (Hi Mischa) and is happy to help to
debug this issue.

I tried to ktrace ntpd to get more insight in the "pipe write error
(from main): No such file or directory" error but I did not get useful
info out of it. This may be because of my unfamiliarity with those
tools.

Help appreciated :-)

 S.
Reply | Threaded
Open this post in threaded view
|

Re: Severe clock problems with OpenBSD VM on OpenBSD Host

trondd-2
On Sat, November 3, 2018 7:10 pm, Stefan Arentz wrote:

> Hi everyone,
>
> I am having an issue where an OpenBSD VM running on vmd is having
> serious clock skew issues.
>
> I am relatively new to OpenBSD, so I am not sure how to properly debug
> this. What I hope is that I can provide a good amount of data and folks
> here can give me some hints and ask me for additional information to
> get to the root cause of this.
>
> So first some facts and symptoms:
>
> - Both Host and Guest are running OpenBSD 6.4. The host runs GENERIC.MP
>   and the guest GENERIC.
> - The host runs 50 guests, all OpenBSD (openbsd.amsterdam)
> - Only this VM is having this clock issue (is this correct, or were
>   there others?)
>
> - The guest has kern.timecounter.hardware=tsc
> - The time on the VM was set with rdate a couple of days ago, and as of
>   now the VM is running about 4 hours behind.
> - ntpd is running (main process, dns engine, ntp engine)
> - when started or restarted, ntpd complains about "pipe write error
>   (from main): No such file or directory" but does seem to start
>
> - I just ran rdate nl.pool.ntp.org and the date was properly updated
> - One minute after running rdate, the clock is already 7 seconds slow
>
> - The guest also has some severe networking issues. often I cannot type
>   more than a few characters before a ~15 second delays happens.
>   Interactive typing is difficult.
> - I can SSH into the Host and have none of these issues, ruling out
>   connectivity issues between me (Toronto) and the Host (Amsterdam)
>
> It would be easy to blame this on NTPd, which does have an unexplained
> error message. However, I think even without running NTPd, the clock
> skew should not be this extreme.
>
> Somehow I have a gut feeling that the clock issues and the networking
> issues are related.
>
> I am root on the VM but I am not on the host. I do have vmctl access.
> However, the host admin is friendly (Hi Mischa) and is happy to help to
> debug this issue.
>
> I tried to ktrace ntpd to get more insight in the "pipe write error
> (from main): No such file or directory" error but I did not get useful
> info out of it. This may be because of my unfamiliarity with those
> tools.
>
> Help appreciated :-)
>
>  S.
>

VMM VMs do have clock issues.  tsc and ntpd should be enough, though (at
least with only a couple VMs it is).  Is ntpd doing anything?  what does
'ntpctl -sa' say?

I think that error is causing ntpd to exit (one of the child procs, if not
the whole thing).

Reply | Threaded
Open this post in threaded view
|

Re: Severe clock problems with OpenBSD VM on OpenBSD Host

oneroi.org.uk
Hi, I can confirm that this is also happening on my VM, also hosted at
openbsd.amsterdam.
Gareth
Reply | Threaded
Open this post in threaded view
|

Re: Severe clock problems with OpenBSD VM on OpenBSD Host

Peter J. Philipp-3
Hi,

I recently built a new vmm with 6.4 and noticed this morning that it had
a clock problem too, however all my other vmm's didn't.

kern.timecounter.tick=1
kern.timecounter.timestepwarnings=0
kern.timecounter.hardware=tsc
kern.timecounter.choice=i8254(0) tsc(-1000) dummy(-1000000)

These are sysctl settings and on my 6.4 vmm i8254 was set, all the
others had tsc set and didn't have this problem.  I set it now to tsc,
it seems to be more accurate.  I don't expect any more tunings.

I'm sorry if this is bad advice but it worked for me.

Regards,

-peter

On 11/7/18 7:40 PM, Gareth Ansell wrote:
> Hi, I can confirm that this is also happening on my VM, also hosted at
> openbsd.amsterdam.
> Gareth

Reply | Threaded
Open this post in threaded view
|

Re: Severe clock problems with OpenBSD VM on OpenBSD Host

j-62
In reply to this post by Stefan Arentz

>    I recently built a new vmm with 6.4 and noticed this morning that it
> had
>    a clock problem too, however all my other vmm's didn't.



FWIW, I don't have an issue with timekeeping on VMM clients on 6.4:

# uptime
  4:40PM  up 4 days, 22:29, 1 user, load averages: 0.00, 0.00, 0.00
# ntpctl -s all
1/1 sensors valid, clock synced, stratum 1

sensor
    wt gd st  next  poll          offset  correction
vmmci0
  *  1  1  0   15s   15s        -0.227ms     0.000ms

I have same kern.timecounter values as you.  However I rely on NTP of
the host
to keep good time.  The clients are simply:

#  ntpd.conf for our VMs
#
sensor *

While ntpd.conf for the host is the default:

#
servers pool.ntp.org
sensor *
constraints from "https://www.google.com"

And keeps time within a millisecond or so:

$ ntpctl -s all
4/4 peers valid, constraint offset 0s, clock synced, stratum 4

peer
    wt tl st  next  poll          offset       delay      jitter
69.17.158.101 from pool pool.ntp.org
     1 10  3  207s  329s        -0.050ms    72.067ms     7.430ms
72.38.129.202 from pool pool.ntp.org
     1 10  2  285s  317s         1.819ms    80.729ms     2.943ms
198.50.139.209 from pool pool.ntp.org
     1 10  2  278s  322s        -5.043ms    87.529ms     8.652ms
207.34.48.31 from pool pool.ntp.org
  *  1 10  3 1524s 1554s        -0.425ms    30.868ms     3.349ms



Host CPU is old AMD:

OpenBSD 6.4 (GENERIC.MP) #364: Thu Oct 11 13:30:23 MDT 2018
     [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: AMD E-450 APU with Radeon(tm) HD Graphics, 6421.62 MHz, 14-02-00
cpu0:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,MWAIT,SSSE3,CX16,POPCNT,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,IBS,SKINIT,ITSC
cpu0: 32KB 64b/line 2-way I-cache, 32KB 64b/line 8-way D-cache, 512KB
64b/line 16-way L2 cache
cpu0: 8 4MB entries fully associative
cpu0: DTLB 40 4KB entries fully associative, 8 4MB entries fully
associative
cpu0: smt 0, core 0, package 0
cpu0: apic clock running at 199MHz
cpu0: mwait min=64, max=64, IBE


--John

Reply | Threaded
Open this post in threaded view
|

Re: Severe clock problems with OpenBSD VM on OpenBSD Host

Stefan Arentz
In reply to this post by Stefan Arentz
Here is an update on the situation:

I installed -current on this VM, clean install, and the ntpd error does not
happen anymore. But the clock issues remain, even with ntpd running.


The ntpd starts without complaints now, and seems to be running with its
regular processes:

_ntp     70093  0.0  0.5   920  2540 ??  S<sp   7:04PM    0:00.02 ntpd: ntp
engine (ntpd)
_ntp     51912  0.0  0.5   736  2464 ??  Isp    7:04PM    0:00.01 ntpd: dns
engine (ntpd)
root     46674  0.0  0.3   792  1640 ??  S<sp   7:04PM    0:00.00
/usr/sbin/ntpd -s


I have set kern.timecounter.hardware to tsc:

# systctl kern.timecounter
kern.timecounter.tick=1
kern.timecounter.timestepwarnings=0
kern.timecounter.hardware=tsc
kern.timecounter.choice=i8254(0) tsc(-1000) dummy(-1000000)


trondd asked for the output of ntpctl -sa, which shows me the following:

# ntpctl -sa
0/4 peers valid, 1/1 sensors valid, constraint offset -3s, clock unsynced,
clock offset is 12771.710ms

peer
   wt tl st  next  poll          offset       delay      jitter
213.154.236.182 from pool pool.ntp.org
    1  2  -  152s  300s             ---- peer not valid ----
83.98.201.134 from pool pool.ntp.org
    1  2  -  152s  300s             ---- peer not valid ----
217.23.3.234 from pool pool.ntp.org
    1  2  -  152s  300s             ---- peer not valid ----
185.92.220.131 from pool pool.ntp.org
    1  2  -  152s  300s             ---- peer not valid ----

sensor
   wt gd st  next  poll          offset  correction
vmmci0
    1  1  0   15s   15s     18001.915ms     0.000ms


I am not sure how to interpret these numbers. I also don't understand the
"peer not valid" messages here. I have another OpenBSD VM which has the
exact same ntpd.conf and it does not complain about any of the peers.


I think my conclusion is that this is not something that can be solved at
the VM level.

 S.


On Sat, Nov 3, 2018 at 8:10 PM Stefan Arentz <[hidden email]>
wrote:

> Hi everyone,
>
> I am having an issue where an OpenBSD VM running on vmd is having
> serious clock skew issues.
>
> I am relatively new to OpenBSD, so I am not sure how to properly debug
> this. What I hope is that I can provide a good amount of data and folks
> here can give me some hints and ask me for additional information to
> get to the root cause of this.
>
> So first some facts and symptoms:
>
> - Both Host and Guest are running OpenBSD 6.4. The host runs GENERIC.MP
>   and the guest GENERIC.
> - The host runs 50 guests, all OpenBSD (openbsd.amsterdam)
> - Only this VM is having this clock issue (is this correct, or were
>   there others?)
>
> - The guest has kern.timecounter.hardware=tsc
> - The time on the VM was set with rdate a couple of days ago, and as of
>   now the VM is running about 4 hours behind.
> - ntpd is running (main process, dns engine, ntp engine)
> - when started or restarted, ntpd complains about "pipe write error
>   (from main): No such file or directory" but does seem to start
>
> - I just ran rdate nl.pool.ntp.org and the date was properly updated
> - One minute after running rdate, the clock is already 7 seconds slow
>
> - The guest also has some severe networking issues. often I cannot type
>   more than a few characters before a ~15 second delays happens.
>   Interactive typing is difficult.
> - I can SSH into the Host and have none of these issues, ruling out
>   connectivity issues between me (Toronto) and the Host (Amsterdam)
>
> It would be easy to blame this on NTPd, which does have an unexplained
> error message. However, I think even without running NTPd, the clock
> skew should not be this extreme.
>
> Somehow I have a gut feeling that the clock issues and the networking
> issues are related.
>
> I am root on the VM but I am not on the host. I do have vmctl access.
> However, the host admin is friendly (Hi Mischa) and is happy to help to
> debug this issue.
>
> I tried to ktrace ntpd to get more insight in the "pipe write error
> (from main): No such file or directory" error but I did not get useful
> info out of it. This may be because of my unfamiliarity with those
> tools.
>
> Help appreciated :-)
>
>  S.
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Severe clock problems with OpenBSD VM on OpenBSD Host

Stefan Arentz
One more update .. when I look at `ntpctl -sa` now, it does not show any
"peer not valid" errors. However, it still rons a good 14 seconds behind.
And it gets worse every minute.

# ntpctl -sa
4/4 peers valid, 1/1 sensors valid, constraint offset -115s (4 errors),
clock unsynced

peer
   wt tl st  next  poll          offset       delay      jitter
213.154.236.182 from pool pool.ntp.org
    1 10  2 3080s 3153s      2298.229ms     3.482ms     1.359ms
83.98.201.134 from pool pool.ntp.org
    1 10  2 3154s 3220s      2764.077ms     2.686ms     0.703ms
217.23.3.234 from pool pool.ntp.org
    1 10  2 2952s 3020s      2682.053ms     2.880ms     0.528ms
185.92.220.131 from pool pool.ntp.org
    1 10  2 2999s 3076s      2266.144ms     2.287ms     0.937ms

sensor
   wt gd st  next  poll          offset  correction
vmmci0
    1  1  0    6s   15s     14607.577ms     0.000ms



On Fri, Nov 9, 2018 at 7:18 PM Stefan Arentz <[hidden email]>
wrote:

> Here is an update on the situation:
>
> I installed -current on this VM, clean install, and the ntpd error does
> not happen anymore. But the clock issues remain, even with ntpd running.
>
>
> The ntpd starts without complaints now, and seems to be running with its
> regular processes:
>
> _ntp     70093  0.0  0.5   920  2540 ??  S<sp   7:04PM    0:00.02 ntpd:
> ntp engine (ntpd)
> _ntp     51912  0.0  0.5   736  2464 ??  Isp    7:04PM    0:00.01 ntpd:
> dns engine (ntpd)
> root     46674  0.0  0.3   792  1640 ??  S<sp   7:04PM    0:00.00
> /usr/sbin/ntpd -s
>
>
> I have set kern.timecounter.hardware to tsc:
>
> # systctl kern.timecounter
> kern.timecounter.tick=1
> kern.timecounter.timestepwarnings=0
> kern.timecounter.hardware=tsc
> kern.timecounter.choice=i8254(0) tsc(-1000) dummy(-1000000)
>
>
> trondd asked for the output of ntpctl -sa, which shows me the following:
>
> # ntpctl -sa
> 0/4 peers valid, 1/1 sensors valid, constraint offset -3s, clock unsynced,
> clock offset is 12771.710ms
>
> peer
>    wt tl st  next  poll          offset       delay      jitter
> 213.154.236.182 from pool pool.ntp.org
>     1  2  -  152s  300s             ---- peer not valid ----
> 83.98.201.134 from pool pool.ntp.org
>     1  2  -  152s  300s             ---- peer not valid ----
> 217.23.3.234 from pool pool.ntp.org
>     1  2  -  152s  300s             ---- peer not valid ----
> 185.92.220.131 from pool pool.ntp.org
>     1  2  -  152s  300s             ---- peer not valid ----
>
> sensor
>    wt gd st  next  poll          offset  correction
> vmmci0
>     1  1  0   15s   15s     18001.915ms     0.000ms
>
>
> I am not sure how to interpret these numbers. I also don't understand the
> "peer not valid" messages here. I have another OpenBSD VM which has the
> exact same ntpd.conf and it does not complain about any of the peers.
>
>
> I think my conclusion is that this is not something that can be solved at
> the VM level.
>
>  S.
>
>
> On Sat, Nov 3, 2018 at 8:10 PM Stefan Arentz <[hidden email]>
> wrote:
>
>> Hi everyone,
>>
>> I am having an issue where an OpenBSD VM running on vmd is having
>> serious clock skew issues.
>>
>> I am relatively new to OpenBSD, so I am not sure how to properly debug
>> this. What I hope is that I can provide a good amount of data and folks
>> here can give me some hints and ask me for additional information to
>> get to the root cause of this.
>>
>> So first some facts and symptoms:
>>
>> - Both Host and Guest are running OpenBSD 6.4. The host runs GENERIC.MP
>>   and the guest GENERIC.
>> - The host runs 50 guests, all OpenBSD (openbsd.amsterdam)
>> - Only this VM is having this clock issue (is this correct, or were
>>   there others?)
>>
>> - The guest has kern.timecounter.hardware=tsc
>> - The time on the VM was set with rdate a couple of days ago, and as of
>>   now the VM is running about 4 hours behind.
>> - ntpd is running (main process, dns engine, ntp engine)
>> - when started or restarted, ntpd complains about "pipe write error
>>   (from main): No such file or directory" but does seem to start
>>
>> - I just ran rdate nl.pool.ntp.org and the date was properly updated
>> - One minute after running rdate, the clock is already 7 seconds slow
>>
>> - The guest also has some severe networking issues. often I cannot type
>>   more than a few characters before a ~15 second delays happens.
>>   Interactive typing is difficult.
>> - I can SSH into the Host and have none of these issues, ruling out
>>   connectivity issues between me (Toronto) and the Host (Amsterdam)
>>
>> It would be easy to blame this on NTPd, which does have an unexplained
>> error message. However, I think even without running NTPd, the clock
>> skew should not be this extreme.
>>
>> Somehow I have a gut feeling that the clock issues and the networking
>> issues are related.
>>
>> I am root on the VM but I am not on the host. I do have vmctl access.
>> However, the host admin is friendly (Hi Mischa) and is happy to help to
>> debug this issue.
>>
>> I tried to ktrace ntpd to get more insight in the "pipe write error
>> (from main): No such file or directory" error but I did not get useful
>> info out of it. This may be because of my unfamiliarity with those
>> tools.
>>
>> Help appreciated :-)
>>
>>  S.
>>
>>
Reply | Threaded
Open this post in threaded view
|

Re: Severe clock problems with OpenBSD VM on OpenBSD Host

Stefan Arentz
In reply to this post by j-62
Oh that is interesting ...


# ntpctl -s all
0/1 sensors valid, clock unsynced

sensor
   wt gd st  next  poll          offset  correction
vmmci0
    1  0  0   13s   15s         - sensor not valid -


sensor not valid - why would that happen?

I have no idea about the inner workings of vmd .. but maybe this host has
so many VMs that it is hitting some limit?

 S.


On Fri, Nov 9, 2018 at 7:43 PM <[hidden email]> wrote:

>
> >    I recently built a new vmm with 6.4 and noticed this morning that it
> > had
> >    a clock problem too, however all my other vmm's didn't.
>
>
>
> FWIW, I don't have an issue with timekeeping on VMM clients on 6.4:
>
> # uptime
>   4:40PM  up 4 days, 22:29, 1 user, load averages: 0.00, 0.00, 0.00
> # ntpctl -s all
> 1/1 sensors valid, clock synced, stratum 1
>
> sensor
>     wt gd st  next  poll          offset  correction
> vmmci0
>   *  1  1  0   15s   15s        -0.227ms     0.000ms
>
> I have same kern.timecounter values as you.  However I rely on NTP of
> the host
> to keep good time.  The clients are simply:
>
> #  ntpd.conf for our VMs
> #
> sensor *
>
> While ntpd.conf for the host is the default:
>
> #
> servers pool.ntp.org
> sensor *
> constraints from "https://www.google.com"
>
> And keeps time within a millisecond or so:
>
> $ ntpctl -s all
> 4/4 peers valid, constraint offset 0s, clock synced, stratum 4
>
> peer
>     wt tl st  next  poll          offset       delay      jitter
> 69.17.158.101 from pool pool.ntp.org
>      1 10  3  207s  329s        -0.050ms    72.067ms     7.430ms
> 72.38.129.202 from pool pool.ntp.org
>      1 10  2  285s  317s         1.819ms    80.729ms     2.943ms
> 198.50.139.209 from pool pool.ntp.org
>      1 10  2  278s  322s        -5.043ms    87.529ms     8.652ms
> 207.34.48.31 from pool pool.ntp.org
>   *  1 10  3 1524s 1554s        -0.425ms    30.868ms     3.349ms
>
>
>
> Host CPU is old AMD:
>
> OpenBSD 6.4 (GENERIC.MP) #364: Thu Oct 11 13:30:23 MDT 2018
>      [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
> cpu0 at mainbus0: apid 0 (boot processor)
> cpu0: AMD E-450 APU with Radeon(tm) HD Graphics, 6421.62 MHz, 14-02-00
> cpu0:
>
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,MWAIT,SSSE3,CX16,POPCNT,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,IBS,SKINIT,ITSC
> cpu0: 32KB 64b/line 2-way I-cache, 32KB 64b/line 8-way D-cache, 512KB
> 64b/line 16-way L2 cache
> cpu0: 8 4MB entries fully associative
> cpu0: DTLB 40 4KB entries fully associative, 8 4MB entries fully
> associative
> cpu0: smt 0, core 0, package 0
> cpu0: apic clock running at 199MHz
> cpu0: mwait min=64, max=64, IBE
>
>
> --John
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Severe clock problems with OpenBSD VM on OpenBSD Host

Jonathan Towne-2
In reply to this post by oneroi.org.uk
On Wed, Nov 07, 2018 at 06:40:20PM +0000, Gareth Ansell scribbled:
# Hi, I can confirm that this is also happening on my VM, also hosted at
# openbsd.amsterdam.
# Gareth

Same here, and none of the tricks I was finding worked.  I was also
unsatisfied with the idea of running rdate or restarting ntpd often
enough to keep things in sync.  Making use of the vmmci(4) timedelta
sensor, I found and modified a piece of existing code by Ted U. to
work around this problem until someone far smarter than I figures out
a proper solution.

Attached, but linked as well in case the attachment doesn't make it.

  https://slite.zenbsd.net/~jontow/vmtimed.c

Neither perfect nor desirable, but workable!  It makes a lot of
adjustments and has been doing an ok job so far.  Doesn't seem to
perfectly keep the clock in a synced state according to "ntpctl -sa".
Here's what it looks like running, and these are the sorts of deltas
it regularly sees and applies.

Nov  6 00:54:43 frea vmtimed: fixing time! -1.152683
Nov  6 00:56:11 frea vmtimed: fixing time! -1.045395
Nov  6 00:58:08 frea vmtimed: fixing time! -1.236222

Here's "ntpctl -sa" as of right now:

4/4 peers valid, 1/1 sensors valid, constraint offset -198s (16 errors), clock synced, stratum 1

peer
   wt tl st  next  poll          offset       delay      jitter
80.127.152.30 from pool pool.ntp.org
    1 10  2  332s 3126s       705.111ms     5.394ms     0.974ms
195.242.98.57 from pool pool.ntp.org
    1 10  2  245s 3183s       657.527ms     2.126ms     0.410ms
93.94.224.67 from pool pool.ntp.org
    1 10  2   53s 3147s       687.997ms     1.931ms     0.532ms
95.211.212.5 from pool pool.ntp.org
    1 10  2 2657s 3118s        73.136ms     2.432ms     0.829ms

sensor
   wt gd st  next  poll          offset  correction
vmmci0  
 *  1  1  0    8s   15s       689.722ms     0.000ms

-- Jonathan Towne

vmtimed.c (2K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Severe clock problems with OpenBSD VM on OpenBSD Host

j-62
In reply to this post by Stefan Arentz
>
> Oh that is interesting ...
>
>
> # ntpctl -s all
> 0/1 sensors valid, clock unsynced
>
> sensor
>    wt gd st  next  poll          offset  correction
> vmmci0
>     1  0  0   13s   15s         - sensor not valid -
>
>
> sensor not valid - why would that happen?
>
> I have no idea about the inner workings of vmd .. but maybe this host
> has
> so many VMs that it is hitting some limit?

Don't forget that ntpd requires numerous polling cycles (at 15 seconds
each)
to convince itself about how reliable it's time sources are.

Give it 5 minutes after restart, or even longer, before deciding for
yourself
that it is not going to sync.

Another reason it may not sync is that the client current time is "too
far away" from
the external ntp time sources.  You need to jump the time manually to
fix that.


--John