Date: Mon, 7 Apr 2003 21:16:45 +1000 (EST) From: Bruce Evans <bde@zeta.org.au> To: Poul-Henning Kamp <phk@phk.freebsd.dk> Cc: Dag-Erling Smorgrav <des@ofug.org> Subject: Re: cvs commit: src/sys/conf options.i386 src/sys/i386/i386 tsc.c src/sys/i386/conf NOTES Message-ID: <20030407192803.C3990@gamplex.bde.org> In-Reply-To: <4145.1049705887@critter.freebsd.dk> References: <4145.1049705887@critter.freebsd.dk>
next in thread | previous in thread | raw e-mail | index | archive | help
On Mon, 7 Apr 2003, Poul-Henning Kamp wrote: > In message <20030407163148.L3478@gamplex.bde.org>, Bruce Evans writes: > > Sorry Bruce, but this is a fair bit more complicated than that. > > It is true that the temperature coefficient of the RTC crystal and the > crystal which (usually) drives the i8254 and TSC are different, but > we are nowhere close to being able to measure that in a one second > interval with the kind of jitters we suffer in accessing the RTC. I never claimed to measure the differential. What is measured is the relative frequency for the last second. That can be measured to within the precision of a counter read and the RTC read. This precision can be very low for both, since bus accesses can be delayed for a long time (i've measured 170 usec for a single inb()). Howver, the boot-time environment is or can be benign, so then there is no bus contention, and then the jitter is usually very small. If there is any, then there is no way to be sure that it is less for i8254 accesses than for the RTC. Anyway, this was not hard-coded. The default was to not use the RTC except to print a value. Users could read the boot messages to get the value or use CLK_CALIBRATION_LOOP to get more values and analyze the jitter. > If we did many observations or used longer integration times, we might > be able to improve it, but it is certainly not something we can even > contemplate waiting for during bootup. Once we have booted, NTPD > beats anything we can do (see below before protesting about lack > of network connectivity). The point of using the RTC as a reference is that it is the only clock that works if the machine is shut down or off. Machines that aren't connected to a network can do no better than use it as their clock (preferably with a frequency adjustment supplied by a local ntpd and with a daemon to keep the hardware clock reasonably up to date). (See below++.) > >Another thing the old code got right was calibration of the i8254 relative > >to the TSC. They obviously use the same hardware clock on at least > >all of my active machines (BP6, A7V266-E), since the calibrated ratio > >is always the same (as far as ntp running over a long period can tell). > > This is generally true, and while the Xtal is normally the 14.318MHz[1] > neither are universal facts. > > I played with detecting the actual PLL ratios, and this generally is > a viable avenue under the normal circumstances, but sufficient "odd-ball" > systems exist that I am not comfortable with it in general. The point of understanding this point is to learn that there is often no use prefering the TSC over the i8254 or vice versa just to get more accurate timekeeping. Actually, it may be possible to use the i8254 to recover from glitches in the TSC caused by TSC throttling and SMM mistakes, etc. We sort of did this in wollman's pre-timecounter code. The TSC was only used to interpolate between i8254 ticks. But I don't really want to get into this. It is too hard to see the glitches while they are happening or even 1 second too late. This is basically the same problem as adjusting for temperature differences. Too much can happen in 1 second. > >The old calibration code calibrates them relative to another clock so > >the only error in their relative frequencies is from the different time > >that it takes to read their counters. The i8254 counter typically takes > >5 usec longer to read, but for some reason the actual error is less than > >1 i8254 cycle on all of my active systems. > > That is because the i8254 access is synchronized to the "virtual" ISA > bus frequency and your CPU is much faster than that. That must be the reason for the 0-cycle differences but not for the accuracy of the old algorithm. It was essentially: read RTC using rtcin(); wait for it to change (1) read i8254 using getit() (2) read TSC using rdtsc() read RTC using rtcin(); wait for it to change again (1a) read i8254 using getit() (2a) read TSC using rdtsc() i8254 freq = (1a) - (1) TSC freq = (2a) - (1) This is better than I remembered. The only algorithmic problem with it is that the cache state is different for some of the reads, in particular the last 2. The reads of the TSC are delayed by however long it takes to read the i8254. This time is almost constant, so the difference is almost independent of it on fast enough CPUs. Warming up the cache might make it completely independent. Changing this to (2) read TSC DELAY(1000000) (2a) read TSC TSC freq = (2a) - (1) gave 2 new sources of errors although it fixes the RTC source: any error in calibation of DELAY(), plus non-determinism from the loop in DELAY(). The latter may be precisely 0 in much the same cases that the difference in the delays in the old algorithm is precisely 0. > The best result I have had so far, and the only one I have sufficient > faith in to advocate its use in general, takes an entirely different > route: > > The RTC interrupts us at 128Hz for statclock, divide this in software > to get 1Hz and take timestamps and feed them to the NTP kernel-FLL code > and tell NTPD to lock to that at a high stratum. > > This will synchronize the clock to the RTC frequency. > > It may be possible to detect the RTC second roll-over and feed absolute > timestamps to the NTP kernel-PLL code to also synchronize to the RTC > in phase. I've done that of course :-). %%% $ vmstat -i | grep rtc rtc irq8 10763015 128.995709 $ ntpdc -c kerninfo pll offset: -0.0170341 s pll frequency: -2.973 ppm maximum error: 0.6102 s estimated error: 0.023417 s status: 2101 pll ppssignal nano pll time constant: 10 precision: 1e-09 s frequency tolerance: 496 ppm pps frequency: -12.255 ppm pps stability: 0.012 ppm pps jitter: 1.6361e-05 s calibration interval: 256 s calibration cycles: 344 jitter exceeded: 62 stability exceeded: 0 calibration errors: 1 %%% This is on a system with RTC pps enabled in the kernel but pps not configured for nptd, so the pps stuff is informational only. The system actually uses the local clock as a secondary server and an external server over a 56K modem link as the preferred server. (BTW, ntpd is barely usable for my modem link due to huge variance.) The timecounter is the TSC and its frequency is tuned so that the pll frequency is almost 0. IIRC, the pps frequency is the frequency adjustment from local clock to the pps signal. Its value depends mainly on the initial calibrations and how I tuned them. The pps stability is good as far as ntpd can tell (not far). The pps jitter isn't really jitter; it is the frequency adjustment from the external clock to the pps signal. This is the right thing to do unless you object to running 1-2 MB of ntpd plus 0.5K of extra kernel code instead of 1K of extra kernel code to do it. Calibrating the frequency right at the beginning mainly just makes ntpd start up and (completely) sync faster. Note that the calibrated frequency has a mixed chance of being more accurate than the frequency (adjustment) saved in the driftfile in a previous run, depending on whether the ambient temperature changed much since the last run (then the calibrated frequency works better) and on whether the boot-time temperature is a transient (then the one in the driftfile works better). > [1] fun fact: The 14.318MHz is four times the color burst frequency > of NTSC television signals, and was chosen by IBM for the original > IBM PC because it made it possible to generate four different colors > in a purely digital fasion on a television connected to the CGA. Never twice the same TSC. Groan. Bruce
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20030407192803.C3990>