Date: Tue, 26 Apr 2016 07:50:59 -0700 From: Neel Natu <neelnatu@gmail.com> To: Ilario Febi <ifebi@schema31.it> Cc: "abrancatelli@schema31.it" <abrancatelli@schema31.it>, "freebsd-virtualization@freebsd.org" <freebsd-virtualization@freebsd.org> Subject: Re: bhyve clock issue Message-ID: <647BCA7C-B915-4FAE-870F-CA914B38B8B5@gmail.com> In-Reply-To: <571F7127.1050100@schema31.it> References: <CANdtQcS9roQV2gi1a-55g7B3j_JyvFrND5s%2BNpyiSMmMfZ3O-w@mail.gmail.com> <8EE098BA-6C8F-41F4-B9CD-88F3D587B10F@gmail.com> <571F7127.1050100@schema31.it>
next in thread | previous in thread | raw e-mail | index | archive | help
What was the timecounter being used when you were seeing the time skew?
Best
Neel
> On Apr 26, 2016, at 6:46 AM, Ilario Febi <ifebi@schema31.it> wrote:
>=20
> Yes,=20
>=20
> I found errors in /var/log/messages and fix adding:=20
>=20
> kern.timecounter.hardware=3DTSC-low
> into /etc/sysctl.conf file
>=20
> Thanks for support.
>=20
>=20
>=20
> Il 14/04/16 18:13, Neel Natu ha scritto:
>> Hi,
>>=20
>> Did you see anything interesting in the ntpd log file around the time the=
large skew happened?
>>=20
>> Best
>> Neel
>>=20
>>> On Apr 13, 2016, at 3:44 AM, Ilario Febi <ifebi@schema31.it> wrote:
>>>=20
>>> Hi All
>>>=20
>>> I have a strange issue about the clock of some FreeBSD Bhyve guests,
>>> maybe someone of you can help me to diagnose it and work on a
>>> solution.
>>>=20
>>> In detail, sometimes (without an apparent regularity) one of four
>>> bhyve guest's clock goes exactly 7 minutes and 10 seconds in the
>>> future.
>>>=20
>>> This clock stays unaligned for about 20 minutes, then, without any
>>> human intervention, the clock realign to proper condition.
>>>=20
>>> My configuration is:
>>>=20
>>> envrm02 (bhyve HOST, 10.2-RELEASE-p14)
>>> -> og-qa (bhyve guest - 10.2-RELEASE-p14)
>>> -> og-ci (bhyve guest - 10.2-RELEASE-p14)
>>> -> og-test (bhyve guest - 10.2-RELEASE-p14)
>>> -> og-prod-rm (bhyve guest - 10.2-RELEASE-p14)
>>> All the og-* machines just have a running copy of tomcat.
>>>=20
>>> I built a simple test to catch the error with python/fabric... the test i=
s:
>>>=20
>>> fab -R ENV,OG,OG-PROD -- date | awk -F ': ' '/2016/{print $2, $1}'
>>>=20
>>> which simply run the command "date" on all hosts.
>>>=20
>>> So, the result is:
>>> @Mac-mini ~/p/f/> ./check_clock.sh
>>> Thu Apr 7 17:38:00 CEST 2016 [envrm02] out
>>> Thu Apr 7 17:38:00 CEST 2016 [og-qa] out
>>> Thu Apr 7 17:38:00 CEST 2016 [og-ci] out
>>> Thu Apr 7 17:38:00 CEST 2016 [og-test] out
>>> Thu Apr 7 17:38:00 CEST 2016 [og-prod-rm] out
>>>=20
>>> @Mac-mini ~/p/f/> ./check_clock.sh
>>> Thu Apr 7 18:38:00 CEST 2016 [envrm02] out
>>> Thu Apr 7 18:45:10 CEST 2016 [og-qa] out
>>> Thu Apr 7 18:38:00 CEST 2016 [og-ci] out
>>> Thu Apr 7 18:38:00 CEST 2016 [og-test] out
>>> Thu Apr 7 18:38:00 CEST 2016 [og-prod-rm] out
>>>=20
>>> @Mac-mini ~/p/f/> ./check_clock.sh
>>> Thu Apr 7 18:41:05 CEST 2016 [envrm02] out
>>> Thu Apr 7 18:48:15 CEST 2016 [og-qa] out
>>> Thu Apr 7 18:41:05 CEST 2016 [og-ci] out
>>> Thu Apr 7 18:41:05 CEST 2016 [og-test] out
>>> Thu Apr 7 18:41:05 CEST 2016 [og-prod-rm] out
>>>=20
>>> @Mac-mini ~/p/f/> ./check_clock.sh
>>> Thu Apr 7 18:46:51 CEST 2016 [envrm02] out
>>> Thu Apr 7 18:46:52 CEST 2016 [og-qa] out
>>> Thu Apr 7 18:46:52 CEST 2016 [og-ci] out
>>> Thu Apr 7 18:46:53 CEST 2016 [og-test] out
>>> Thu Apr 7 18:46:53 CEST 2016 [og-prod-rm] out
>>>=20
>>> The configuration is the same for all hosts and is this:
>>> @Mac-mini ~/p/f/> fab -R ENV,OG,OG-PROD -- cat /etc/rc.conf | grep ntpd
>>> [envrm02] out: ntpd_enable=3D"YES"
>>> [og-qa] out: ntpd_enable=3D"YES"
>>> [og-qa] out: ntpd_sync_on_start=3D"YES"
>>> [og-ci] out: ntpd_enable=3D"YES"
>>> [og-ci] out: ntpd_sync_on_start=3D"YES"
>>> [og-test] out: ntpd_enable=3D"YES"
>>> [og-test] out: ntpd_sync_on_start=3D"YES"
>>> [og-prod-rm] out: ntpd_enable=3D"YES"
>>> [og-prod-rm] out: ntpd_sync_on_start=3D"YES"
>>>=20
>>> @Mac-mini ~/p/f/> fab -R ENV,OG,OG-PROD -- cat /etc/ntp.conf | grep -v '=
out: #'
>>> [envrm02] Executing task '<remainder>'
>>> [envrm02] run: cat /etc/ntp.conf
>>> [envrm02] out:
>>> [envrm02] out: server 0.freebsd.pool.ntp.org iburst
>>> [envrm02] out: server 1.freebsd.pool.ntp.org iburst
>>> [envrm02] out: server 2.freebsd.pool.ntp.org iburst
>>> [envrm02] out:
>>> [envrm02] out: restrict default limited kod nomodify notrap nopeer noque=
ry
>>> [envrm02] out: restrict -6 default limited kod nomodify notrap nopeer no=
query
>>> [envrm02] out: restrict 127.0.0.1
>>> [envrm02] out: restrict -6 ::1
>>> [envrm02] out: restrict 127.127.1.0
>>>=20
>>> [og-qa] Executing task '<remainder>'
>>> [og-qa] run: cat /etc/ntp.conf
>>> [og-qa] out:
>>> [og-qa] out: server 0.freebsd.pool.ntp.org iburst
>>> [og-qa] out: server 1.freebsd.pool.ntp.org iburst
>>> [og-qa] out: server 2.freebsd.pool.ntp.org iburst
>>> [og-qa] out:
>>> [og-qa] out: restrict default limited kod nomodify notrap nopeer noquery=
>>> [og-qa] out: restrict -6 default limited kod nomodify notrap nopeer noqu=
ery
>>> [og-qa] out: restrict 127.0.0.1
>>> [og-qa] out: restrict -6 ::1
>>> [og-qa] out: restrict 127.127.1.0
>>>=20
>>> [og-ci] Executing task '<remainder>'
>>> [og-ci] run: cat /etc/ntp.conf
>>> [og-ci] out:
>>> [og-ci] out: server 0.freebsd.pool.ntp.org iburst
>>> [og-ci] out: server 1.freebsd.pool.ntp.org iburst
>>> [og-ci] out: server 2.freebsd.pool.ntp.org iburst
>>> [og-ci] out:
>>> [og-ci] out: restrict default limited kod nomodify notrap nopeer noquery=
>>> [og-ci] out: restrict -6 default limited kod nomodify notrap nopeer noqu=
ery
>>> [og-ci] out: restrict 127.0.0.1
>>> [og-ci] out: restrict -6 ::1
>>> [og-ci] out: restrict 127.127.1.0
>>>=20
>>> [og-test] Executing task '<remainder>'
>>> [og-test] run: cat /etc/ntp.conf
>>> [og-test] out:
>>> [og-test] out: server 0.freebsd.pool.ntp.org iburst
>>> [og-test] out: server 1.freebsd.pool.ntp.org iburst
>>> [og-test] out: server 2.freebsd.pool.ntp.org iburst
>>> [og-test] out:
>>> [og-test] out: restrict default limited kod nomodify notrap nopeer noque=
ry
>>> [og-test] out: restrict -6 default limited kod nomodify notrap nopeer no=
query
>>> [og-test] out: restrict 127.0.0.1
>>> [og-test] out: restrict -6 ::1
>>> [og-test] out: restrict 127.127.1.0
>>>=20
>>> Some observation:
>>> * The clock of envrm02 (the HOST) always seems to be correct.
>>> * The issue happens on a (apparently) random guest (not always the same o=
ne).
>>> * The issue happens always on one host at a time.
>>>=20
>>> The bhyve command line is, roughly, this one:
>>>=20
>>> /usr/sbin/bhyveload -m 2048 -d /san_storage/VMfs/$machine/$machine.img $=
machine
>>> /usr/sbin/bhyve -c 2 -m 2048 -A -H -P -s 0,hostbridge -s
>>> 2,ahci-hd,/san_storage/VMfs/$machine/$machine.img -s 4,virtio-net,$tap
>>> -s 31,lpc -l com1,stdio $machine
>>>=20
>>> Thanks in advance.
>>> _______________________________________________
>>> freebsd-virtualization@freebsd.org mailing list
>>> https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
>>> To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe@fre=
ebsd.org"
>=20
> --=20
> Ing. Ilario Febi
>=20
> Schema 31 S.p.a. Socio Unico
> System Administrator
>=20
> BO - FI - ROMA - PA=20
> ITALY
> Tel: +39. 06.98.358.472
> Cell: +39. 366.69.45.384
> Fax: +39. 055.71.880.466
> Societ=C3=A0 del Gruppo SC31 ITALIA
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?647BCA7C-B915-4FAE-870F-CA914B38B8B5>