Skip site navigation (1)Skip section navigation (2)
Date:      Wed, 3 Apr 2019 00:27:32 +0200
From:      Michael Tuexen <tuexen@fh-muenster.de>
To:        Konstantin Belousov <kostikbel@gmail.com>
Cc:        Bruce Evans <brde@optusnet.com.au>, freebsd-hackers Hackers <freebsd-hackers@freebsd.org>, FreeBSD PowerPC ML <freebsd-ppc@freebsd.org>
Subject:   Re: powerpc64 head -r344018 stuck sleeping problems: th->th_scale * tc_delta(th) overflows unsigned 64 bits sometimes [patched failed]
Message-ID:  <E0785613-2B6E-4BB3-95CD-03DD96902CD8@fh-muenster.de>
In-Reply-To: <20190324110138.GR1923@kib.kiev.ua>
References:  <20190303111931.GI68879@kib.kiev.ua> <20190303223100.B3572@besplex.bde.org> <20190303161635.GJ68879@kib.kiev.ua> <20190304043416.V5640@besplex.bde.org> <20190304114150.GM68879@kib.kiev.ua> <20190305031010.I4610@besplex.bde.org> <20190306172003.GD2492@kib.kiev.ua> <20190308001005.M2756@besplex.bde.org> <20190307222220.GK2492@kib.kiev.ua> <20190309144844.K1166@besplex.bde.org> <20190324110138.GR1923@kib.kiev.ua>

next in thread | previous in thread | raw e-mail | index | archive | help

--Apple-Mail=_A26B82D4-1AFC-4AF8-A85C-EB10136E809F
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=us-ascii

> On 24. Mar 2019, at 12:01, Konstantin Belousov <kostikbel@gmail.com> =
wrote:
>=20
> On Sat, Mar 09, 2019 at 06:00:14PM +1100, Bruce Evans wrote:
>> I more strongly disclike (sic) the more complete merge.  The central =
APIs
>> have even more parameters and reduced type safety to describe objects =
as
>> (offset, size) pairs.
> I changed the patch to be type-safe.  Now I like it even more.  It =
provides
> 1. internal
> 2. concise
> 3. type-safe
> API to fetch data from timehands.  The implementation needs to be read
> only once.
Hi,

I'm a bit lost... I think this started to fix a problem on G5 PowerMacs.
Do you think this patch solves the problem. Should this be tested?
Or is this still work in progress or a general improvement not necessary
fixing the problem on G5 PowerMacs?

Best regards
Michael
>=20
>> Small delicate loops are ideal for duplicating.  They are easier to
>> understand individually and short enough to compare without using =
diff
>> to see gratuitous and substantive differences.  Multiple instances =
are
>> only hard to write and maintain.  Since these multiple instances are
>> already written, they are only harder to maintain.
> This is a good argument to have bintime_off and getthmember unmerged
> (there are two small but delicate loops). The API is internal, so it =
is
> only matter for maintainer, which means that the absence of =
duplication
> is important. More, all that arguments clearly explain why there =
should
> be not twenty similar loops scattered over the source.
>=20
>>=20
>>>> XX  void
>>>> XX  binuptime(struct bintime *bt)
>>>> XX  {
>>>> XX @@ -361,7 +383,7 @@
>>>> XX  		th =3D timehands;
>>>> XX  		gen =3D atomic_load_acq_int(&th->th_generation);
>>>> XX  		*bt =3D th->th_offset;
>>>> XX -		bintime_addx(bt, th->th_scale * tc_delta(th));
>>>> XX +		bintime_adddelta(bt, th);
>>>> XX  		atomic_thread_fence_acq();
>>>> XX  	} while (gen =3D=3D 0 || gen !=3D th->th_generation);
>>>> XX  }
>>>>=20
>>>> This is the kind of non-churning change that I like.
>>> Ok.  I made all cases where timehands are read, more uniform by
>>> moving calculations after the generation loop.  This makes the
>>> atomic part of the functions easier to see, and loop body has lower
>>> chance to hit generation reset.
>>=20
>> I think this change is slightly worse:
>> - it increases register pressure.  'scale' and 'delta' must be read =
in a
>>   alost program program before the loop exit test.  The above order =
uses
>>   them and stores the results to memory, so more registers are free =
for
>>   the exit test.  i386 certainly runs out of registers.  IIRC, i386 =
now
>>   spills 'gen'.  It would have to spill something to load 'gen' or =
'th'
>>   for the test.
> Which does not matter on any modern architecture anyway.
>=20
>> - it enlarges the window between reading 'scale' and 'delta' and the
>>   caller seeing the results.  Preemption in this window gives results
>>   that may be far in the past.
> My opinion is that quickly exiting the code and avoid retry is more
> important (as in performance) than making an impression that we =
protect
> against preemption. If preemption is important for the caller, then
> the calling place must use some measures like interrupt disabling and
> re-checking the time after the operation. Preemption can occur after =
the
> loop exit with the same consequences.
>=20
>> The 'get' name is another problem.  I would like all the get*time
>> functions and not add new names starting with 'get'.  The library
>> implementation already doesn't bother optimizing the get*time =
functions,
>> but always uses the hardware timecounter.
>>=20
>> getfoo() is a more natural name than foo_get() for the action of =
getting
>> foo, but the latter is better for consistency, especially in code =
that
>> puts the subsystem name first in nearby code.
>>=20
>> The get*time functions would be better if they were more like
>> time_second.  Note that time_second is racy if time_t is too larger
>> for the arch so that accesses to it are not atomic, as happens on
>> 32-bit arches with premature 64-bit time_t.  However, in this 32/64
>> case, the race is only run every 136 years, with the next event
>> scheduled in 2038, so this race is even less important now than other
>> events scheduled in 2038.  Bintimes are 96 or 128 bits, so directly
>> copying a global like time_second for them would race every 1/2**32
>> second on 2-bit arches or every 1 second on 64-bit arches.  Most of
>> the loops on the generation count are for fixing these races, but
>> perhaps a simpler method would work.  On 64-bit arches with atomic
>> 64 accesses on 32-bit boundaries, the following would work:
>> - set the lower 32 bits of the fraction to 0, or ignore them
>> - load the higher 32 bits of the fraction and the lower 32 bits of =
the
>>   seconds
>> - race once every 136 years starting in 2038 reading the higher 32 =
bits
>>   of the seconds non-atomically.
>> - alternatively, break instead of racing in 2038 by setting the =
higher
>>   32 bits to 0.  This is the same as using sbintimes instead of =
bintimes.
>> - drop a few more lower bits by storing a right-shifted value.  Right
>>   shifting by just 1 gives a race frequency of once per 272 years, =
with
>>   the next one in 2006.
> It would make sense if the functions were written from scratch, but =
since
> we already have the generation counts, it is not obvious that such =
change
> is useful.
>=20
> But if we decide to go that route (later), my current patch only
> requires exactly one location getthmember() to experiment with and to
> change after. So you effectively made yet another, and perhaps most
> convincing, argument, for me.
>=20
>>=20
>>> diff --git a/sys/kern/kern_tc.c b/sys/kern/kern_tc.c
>>> index 2656fb4d22f..8d12847f2cd 100644
>>> --- a/sys/kern/kern_tc.c
>>> +++ b/sys/kern/kern_tc.c
>>> @@ -200,20 +202,56 @@ tc_delta(struct timehands *th)
>>> * the comment in <sys/time.h> for a description of these 12 =
functions.
>>> */
>>>=20
>>> -#ifdef FFCLOCK
>>> -void
>>> -fbclock_binuptime(struct bintime *bt)
>>> +static __inline void
>>> +bintime_off(struct bintime *bt, u_int off)
>>> {
>>> 	struct timehands *th;
>>> -	unsigned int gen;
>>> +	struct bintime *btp;
>>> +	uint64_t scale, x;
>>> +	u_int delta, gen, large_delta;
>>>=20
>>> 	do {
>>> 		th =3D timehands;
>>> 		gen =3D atomic_load_acq_int(&th->th_generation);
>>> -		*bt =3D th->th_offset;
>>> -		bintime_addx(bt, th->th_scale * tc_delta(th));
>>=20
>> You didn't fully obfuscate this by combinining this function with
>> getthmember() so as to deduplicate the loop.
>>=20
>>> +		btp =3D (struct bintime *)((vm_offset_t)th + off);
>>=20
>> Ugly conversion to share code.  This is technically incorrect.  =
Improving
>> the casts gives:
>>=20
>> 	btp =3D (void *)(uintptr_t)((uintptr_t)(void *)th + off);
>>=20
>> but this assumes that arithmetic on the intermediate integer does =
what
>> is espected.  uintptr_t is only guaranteed to work when the =
intermediate
>> representation held in it is not adjusted.
> vm_offset_t has the semantic that is needed for the arithmetic.  It is
> better uintptr_t for kernel, where we know that all object pointers =
are
> compatible with vm_offset_t (AKA flat tag-less memory model).
>=20
>>=20
>> Fixing the API gives
>>=20
>>     static __inline void
>>     bintime_off(struct bintime *btp, struct bintime *base_btp)
>>=20
>> where base_btp is &th->th_bintime or &th->th_offset.
>>=20
>> (th_offset and th_bintime are badly named.  th_offset is really a =
base
>> time and the offset is tc_delta().  th_bintime is also a base time.
>> It is the same as th_offset with another actual offset (the =
difference
>> between UTC and local time) already added to it as an optimization.  =
In
>> old versions, th_bintime didn't exist, but the related struct members
>> th_nanotime and th_microtime existed, since these benefit more from
>> not converting on every call.
> How could it be &th->th_offset, when th is calculated inside the call =
?
> But I did modified the API in this spirit, indeed.  It takes the =
member
> name directly as an argument.
>=20
>>=20
>> My old version even documents the struct members, while -current =
still
>> has no comments.  The comments were lost to staticization.  My =
version
>> mostly adds "duh" to the banal comments after recovering them:
>>=20
>> XX /*
>> XX  * XXX rotted comment cloned from <sys/timetc.h>.
>> XX  *
>> XX  * th_counter is undocumented (duh).
>> XX  *
>> XX  * th_adjustment [PPM << 16] which means that the smallest unit of =
correction
>> XX  *     you can apply amounts to 481.5 usec/year.
>> XX  *
>> XX  * th_scale is undocumented (duh).
>> XX  *
>> XX  * th_offset_count is the contents of the counter which =
corresponds to the
>> XX  *
>> XX  *     rest of the offset_* values.
>> XX  *
>> XX  * th_offset is undocumented (duh).
>> XX  *
>> XX  * th_microtime is undocumented (duh).
>> XX  *
>> XX  * th_nanotime is undocumented (duh).
>> XX  *
>> XX  * XXX especially massive bitrot here.  "three" is now "many"...
>> XX  * Each timecounter must supply an array of three timecounters.  =
This is needed
>> XX  * to guarantee atomicity in the code.  Index zero is used to =
transport
>> XX  * modifications, for instance done with sysctl, into the =
timecounter being
>> XX  * used in a safe way.  Such changes may be adopted with a delay =
of up to 1/HZ.
>> XX  * Index one and two are used alternately for the actual =
timekeeping.
>> XX  *
>> XX  * th_generation is undocumented (duh).
>> XX  *
>> XX  * th_next is undocumented (duh).
>> XX  */
>>=20
>>> +		*bt =3D *btp;
>>> +		scale =3D th->th_scale;
>>> +		delta =3D tc_delta(th);
>>> +		large_delta =3D th->th_large_delta;
>>=20
>> I had forgotten that th_scale is so volatile (it may be adjusted on
>> every windup).  th_large_delta is equally volatile.  So moving the
>> calculation outside of the loop gives even more register pressure
>> than I noticed above.
>>=20
>>> 		atomic_thread_fence_acq();
>>> 	} while (gen =3D=3D 0 || gen !=3D th->th_generation);
>>> +
>>> +	if (__predict_false(delta < large_delta)) {
>>> +		/* Avoid overflow for scale * delta. */
>>> +		x =3D (scale >> 32) * delta;
>>> +		bt->sec +=3D x >> 32;
>>> +		bintime_addx(bt, x << 32);
>>> +		bintime_addx(bt, (scale & 0xffffffff) * delta);
>>> +	} else {
>>> +		bintime_addx(bt, scale * delta);
>>> +	}
>>> +}
>>> +
>>> +static __inline void
>>> +getthmember(void *out, size_t out_size, u_int off)
>>> +{
>>> +	struct timehands *th;
>>> +	u_int gen;
>>> +
>>> +	do {
>>> +		th =3D timehands;
>>> +		gen =3D atomic_load_acq_int(&th->th_generation);
>>> +		memcpy(out, (char *)th + off, out_size);
>>=20
>> This isn't so ugly or technically incorrect.  Now the object is =
generic,
>> so the reference to it should be passed as (void *objp, size_t =
objsize)
>> instead of the type-safe (struct bintime *base_bpt).
> _Generic is what gave me a hint how to make the implementation =
type-safe.
> diff --git a/sys/kern/kern_tc.c b/sys/kern/kern_tc.c
> index 2656fb4d22f..4e94f762026 100644
> --- a/sys/kern/kern_tc.c
> +++ b/sys/kern/kern_tc.c
> @@ -72,6 +72,7 @@ struct timehands {
> 	struct timecounter	*th_counter;
> 	int64_t			th_adjustment;
> 	uint64_t		th_scale;
> +	u_int			th_large_delta;
> 	u_int	 		th_offset_count;
> 	struct bintime		th_offset;
> 	struct bintime		th_bintime;
> @@ -90,6 +91,7 @@ static struct timehands th1 =3D {
> static struct timehands th0 =3D {
> 	.th_counter =3D &dummy_timecounter,
> 	.th_scale =3D (uint64_t)-1 / 1000000,
> +	.th_large_delta =3D 1000000,
> 	.th_offset =3D { .sec =3D 1 },
> 	.th_generation =3D 1,
> 	.th_next =3D &th1
> @@ -200,20 +202,72 @@ tc_delta(struct timehands *th)
>  * the comment in <sys/time.h> for a description of these 12 =
functions.
>  */
>=20
> -#ifdef FFCLOCK
> -void
> -fbclock_binuptime(struct bintime *bt)
> +static __inline void
> +bintime_off(struct bintime *bt, u_int off)
> {
> 	struct timehands *th;
> -	unsigned int gen;
> +	struct bintime *btp;
> +	uint64_t scale, x;
> +	u_int delta, gen, large_delta;
>=20
> 	do {
> 		th =3D timehands;
> 		gen =3D atomic_load_acq_int(&th->th_generation);
> -		*bt =3D th->th_offset;
> -		bintime_addx(bt, th->th_scale * tc_delta(th));
> +		btp =3D (struct bintime *)((vm_offset_t)th + off);
> +		*bt =3D *btp;
> +		scale =3D th->th_scale;
> +		delta =3D tc_delta(th);
> +		large_delta =3D th->th_large_delta;
> 		atomic_thread_fence_acq();
> 	} while (gen =3D=3D 0 || gen !=3D th->th_generation);
> +
> +	if (__predict_false(delta >=3D large_delta)) {
> +		/* Avoid overflow for scale * delta. */
> +		x =3D (scale >> 32) * delta;
> +		bt->sec +=3D x >> 32;
> +		bintime_addx(bt, x << 32);
> +		bintime_addx(bt, (scale & 0xffffffff) * delta);
> +	} else {
> +		bintime_addx(bt, scale * delta);
> +	}
> +}
> +#define	GETTHBINTIME(dst, member)					=
	\
> +do {									=
\
> +	_Static_assert(_Generic(((struct timehands *)NULL)->member,	=
\
> +	    struct bintime: 1, default: 0) =3D=3D 1,			=
\
> +	    "struct timehands member is not of struct bintime type");	=
\
> +	bintime_off(dst, __offsetof(struct timehands, member));		=
\
> +} while (0)
> +
> +static __inline void
> +getthmember(void *out, size_t out_size, u_int off)
> +{
> +	struct timehands *th;
> +	u_int gen;
> +
> +	do {
> +		th =3D timehands;
> +		gen =3D atomic_load_acq_int(&th->th_generation);
> +		memcpy(out, (char *)th + off, out_size);
> +		atomic_thread_fence_acq();
> +	} while (gen =3D=3D 0 || gen !=3D th->th_generation);
> +}
> +#define	GETTHMEMBER(dst, member)					=
\
> +do {									=
\
> +	_Static_assert(_Generic(*dst,					=
\
> +	    __typeof(((struct timehands *)NULL)->member): 1,		=
\
> +	    default: 0) =3D=3D 1,						=
\
> +	    "*dst and struct timehands member have different types");	=
\
> +	getthmember(dst, sizeof(*dst), __offsetof(struct timehands,	=
\
> +	    member));							=
\
> +} while (0)
> +
> +#ifdef FFCLOCK
> +void
> +fbclock_binuptime(struct bintime *bt)
> +{
> +
> +	GETTHBINTIME(bt, th_offset);
> }
>=20
> void
> @@ -237,16 +291,8 @@ fbclock_microuptime(struct timeval *tvp)
> void
> fbclock_bintime(struct bintime *bt)
> {
> -	struct timehands *th;
> -	unsigned int gen;
>=20
> -	do {
> -		th =3D timehands;
> -		gen =3D atomic_load_acq_int(&th->th_generation);
> -		*bt =3D th->th_bintime;
> -		bintime_addx(bt, th->th_scale * tc_delta(th));
> -		atomic_thread_fence_acq();
> -	} while (gen =3D=3D 0 || gen !=3D th->th_generation);
> +	GETTHBINTIME(bt, th_bintime);
> }
>=20
> void
> @@ -270,100 +316,55 @@ fbclock_microtime(struct timeval *tvp)
> void
> fbclock_getbinuptime(struct bintime *bt)
> {
> -	struct timehands *th;
> -	unsigned int gen;
>=20
> -	do {
> -		th =3D timehands;
> -		gen =3D atomic_load_acq_int(&th->th_generation);
> -		*bt =3D th->th_offset;
> -		atomic_thread_fence_acq();
> -	} while (gen =3D=3D 0 || gen !=3D th->th_generation);
> +	GETTHMEMBER(bt, th_offset);
> }
>=20
> void
> fbclock_getnanouptime(struct timespec *tsp)
> {
> -	struct timehands *th;
> -	unsigned int gen;
> +	struct bintime bt;
>=20
> -	do {
> -		th =3D timehands;
> -		gen =3D atomic_load_acq_int(&th->th_generation);
> -		bintime2timespec(&th->th_offset, tsp);
> -		atomic_thread_fence_acq();
> -	} while (gen =3D=3D 0 || gen !=3D th->th_generation);
> +	GETTHMEMBER(&bt, th_offset);
> +	bintime2timespec(&bt, tsp);
> }
>=20
> void
> fbclock_getmicrouptime(struct timeval *tvp)
> {
> -	struct timehands *th;
> -	unsigned int gen;
> +	struct bintime bt;
>=20
> -	do {
> -		th =3D timehands;
> -		gen =3D atomic_load_acq_int(&th->th_generation);
> -		bintime2timeval(&th->th_offset, tvp);
> -		atomic_thread_fence_acq();
> -	} while (gen =3D=3D 0 || gen !=3D th->th_generation);
> +	GETTHMEMBER(&bt, th_offset);
> +	bintime2timeval(&bt, tvp);
> }
>=20
> void
> fbclock_getbintime(struct bintime *bt)
> {
> -	struct timehands *th;
> -	unsigned int gen;
>=20
> -	do {
> -		th =3D timehands;
> -		gen =3D atomic_load_acq_int(&th->th_generation);
> -		*bt =3D th->th_bintime;
> -		atomic_thread_fence_acq();
> -	} while (gen =3D=3D 0 || gen !=3D th->th_generation);
> +	GETTHMEMBER(bt, th_bintime);
> }
>=20
> void
> fbclock_getnanotime(struct timespec *tsp)
> {
> -	struct timehands *th;
> -	unsigned int gen;
>=20
> -	do {
> -		th =3D timehands;
> -		gen =3D atomic_load_acq_int(&th->th_generation);
> -		*tsp =3D th->th_nanotime;
> -		atomic_thread_fence_acq();
> -	} while (gen =3D=3D 0 || gen !=3D th->th_generation);
> +	GETTHMEMBER(tsp, th_nanotime);
> }
>=20
> void
> fbclock_getmicrotime(struct timeval *tvp)
> {
> -	struct timehands *th;
> -	unsigned int gen;
>=20
> -	do {
> -		th =3D timehands;
> -		gen =3D atomic_load_acq_int(&th->th_generation);
> -		*tvp =3D th->th_microtime;
> -		atomic_thread_fence_acq();
> -	} while (gen =3D=3D 0 || gen !=3D th->th_generation);
> +	GETTHMEMBER(tvp, th_microtime);
> }
> #else /* !FFCLOCK */
> +
> void
> binuptime(struct bintime *bt)
> {
> -	struct timehands *th;
> -	u_int gen;
>=20
> -	do {
> -		th =3D timehands;
> -		gen =3D atomic_load_acq_int(&th->th_generation);
> -		*bt =3D th->th_offset;
> -		bintime_addx(bt, th->th_scale * tc_delta(th));
> -		atomic_thread_fence_acq();
> -	} while (gen =3D=3D 0 || gen !=3D th->th_generation);
> +	GETTHBINTIME(bt, th_offset);
> }
>=20
> void
> @@ -387,16 +388,8 @@ microuptime(struct timeval *tvp)
> void
> bintime(struct bintime *bt)
> {
> -	struct timehands *th;
> -	u_int gen;
>=20
> -	do {
> -		th =3D timehands;
> -		gen =3D atomic_load_acq_int(&th->th_generation);
> -		*bt =3D th->th_bintime;
> -		bintime_addx(bt, th->th_scale * tc_delta(th));
> -		atomic_thread_fence_acq();
> -	} while (gen =3D=3D 0 || gen !=3D th->th_generation);
> +	GETTHBINTIME(bt, th_bintime);
> }
>=20
> void
> @@ -420,85 +413,47 @@ microtime(struct timeval *tvp)
> void
> getbinuptime(struct bintime *bt)
> {
> -	struct timehands *th;
> -	u_int gen;
>=20
> -	do {
> -		th =3D timehands;
> -		gen =3D atomic_load_acq_int(&th->th_generation);
> -		*bt =3D th->th_offset;
> -		atomic_thread_fence_acq();
> -	} while (gen =3D=3D 0 || gen !=3D th->th_generation);
> +	GETTHMEMBER(bt, th_offset);
> }
>=20
> void
> getnanouptime(struct timespec *tsp)
> {
> -	struct timehands *th;
> -	u_int gen;
> +	struct bintime bt;
>=20
> -	do {
> -		th =3D timehands;
> -		gen =3D atomic_load_acq_int(&th->th_generation);
> -		bintime2timespec(&th->th_offset, tsp);
> -		atomic_thread_fence_acq();
> -	} while (gen =3D=3D 0 || gen !=3D th->th_generation);
> +	GETTHMEMBER(&bt, th_offset);
> +	bintime2timespec(&bt, tsp);
> }
>=20
> void
> getmicrouptime(struct timeval *tvp)
> {
> -	struct timehands *th;
> -	u_int gen;
> +	struct bintime bt;
>=20
> -	do {
> -		th =3D timehands;
> -		gen =3D atomic_load_acq_int(&th->th_generation);
> -		bintime2timeval(&th->th_offset, tvp);
> -		atomic_thread_fence_acq();
> -	} while (gen =3D=3D 0 || gen !=3D th->th_generation);
> +	GETTHMEMBER(&bt, th_offset);
> +	bintime2timeval(&bt, tvp);
> }
>=20
> void
> getbintime(struct bintime *bt)
> {
> -	struct timehands *th;
> -	u_int gen;
>=20
> -	do {
> -		th =3D timehands;
> -		gen =3D atomic_load_acq_int(&th->th_generation);
> -		*bt =3D th->th_bintime;
> -		atomic_thread_fence_acq();
> -	} while (gen =3D=3D 0 || gen !=3D th->th_generation);
> +	GETTHMEMBER(bt, th_bintime);
> }
>=20
> void
> getnanotime(struct timespec *tsp)
> {
> -	struct timehands *th;
> -	u_int gen;
>=20
> -	do {
> -		th =3D timehands;
> -		gen =3D atomic_load_acq_int(&th->th_generation);
> -		*tsp =3D th->th_nanotime;
> -		atomic_thread_fence_acq();
> -	} while (gen =3D=3D 0 || gen !=3D th->th_generation);
> +	GETTHMEMBER(tsp, th_nanotime);
> }
>=20
> void
> getmicrotime(struct timeval *tvp)
> {
> -	struct timehands *th;
> -	u_int gen;
>=20
> -	do {
> -		th =3D timehands;
> -		gen =3D atomic_load_acq_int(&th->th_generation);
> -		*tvp =3D th->th_microtime;
> -		atomic_thread_fence_acq();
> -	} while (gen =3D=3D 0 || gen !=3D th->th_generation);
> +	GETTHMEMBER(tvp, th_microtime);
> }
> #endif /* FFCLOCK */
>=20
> @@ -514,15 +469,8 @@ getboottime(struct timeval *boottime)
> void
> getboottimebin(struct bintime *boottimebin)
> {
> -	struct timehands *th;
> -	u_int gen;
>=20
> -	do {
> -		th =3D timehands;
> -		gen =3D atomic_load_acq_int(&th->th_generation);
> -		*boottimebin =3D th->th_boottime;
> -		atomic_thread_fence_acq();
> -	} while (gen =3D=3D 0 || gen !=3D th->th_generation);
> +	GETTHMEMBER(boottimebin, th_boottime);
> }
>=20
> #ifdef FFCLOCK
> @@ -1038,15 +986,8 @@ getmicrotime(struct timeval *tvp)
> void
> dtrace_getnanotime(struct timespec *tsp)
> {
> -	struct timehands *th;
> -	u_int gen;
>=20
> -	do {
> -		th =3D timehands;
> -		gen =3D atomic_load_acq_int(&th->th_generation);
> -		*tsp =3D th->th_nanotime;
> -		atomic_thread_fence_acq();
> -	} while (gen =3D=3D 0 || gen !=3D th->th_generation);
> +	GETTHMEMBER(tsp, th_nanotime);
> }
>=20
> /*
> @@ -1464,6 +1405,7 @@ tc_windup(struct bintime *new_boottimebin)
> 	scale +=3D (th->th_adjustment / 1024) * 2199;
> 	scale /=3D th->th_counter->tc_frequency;
> 	th->th_scale =3D scale * 2;
> +	th->th_large_delta =3D MIN(((uint64_t)1 << 63) / scale, =
UINT_MAX);
>=20
> 	/*
> 	 * Now that the struct timehands is again consistent, set the =
new
> _______________________________________________
> freebsd-ppc@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-ppc
> To unsubscribe, send any mail to "freebsd-ppc-unsubscribe@freebsd.org"


--Apple-Mail=_A26B82D4-1AFC-4AF8-A85C-EB10136E809F
Content-Disposition: attachment;
	filename=smime.p7s
Content-Type: application/pkcs7-signature;
	name=smime.p7s
Content-Transfer-Encoding: base64

MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgEFADCABgkqhkiG9w0BBwEAAKCCEJAw
ggTVMIIDvaADAgECAghQTsb1PRG0ZDANBgkqhkiG9w0BAQsFADBxMQswCQYDVQQGEwJERTEcMBoG
A1UEChMTRGV1dHNjaGUgVGVsZWtvbSBBRzEfMB0GA1UECxMWVC1UZWxlU2VjIFRydXN0IENlbnRl
cjEjMCEGA1UEAxMaRGV1dHNjaGUgVGVsZWtvbSBSb290IENBIDIwHhcNMTQwNzIyMTIwODI2WhcN
MTkwNzA5MjM1OTAwWjBaMQswCQYDVQQGEwJERTETMBEGA1UEChMKREZOLVZlcmVpbjEQMA4GA1UE
CxMHREZOLVBLSTEkMCIGA1UEAxMbREZOLVZlcmVpbiBQQ0EgR2xvYmFsIC0gRzAxMIIBIjANBgkq
hkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA6ZvDZ4X5Da71jVTDllA1PWLpbkztlNcAW5UidNQg6zSP
1uzAMQQLmYHiphTSUqAoI4SLdIkEXlvg4njBeMsWyyg1OXstkEXQ7aAAeny/Sg4bAMOG6VwrMRF7
DPOCJEOMHDiLamgAmu7cT3ir0sYTm3at7t4m6O8Br3QPwQmi9mvOvdPNFDBP9eXjpMhim4IaAycw
DQJlYE3t0QkjKpY1WCfTdsZxtpAdxO3/NYZ9bzOz2w/FEcKKg6GUXUFr2NIQ9Uz9ylGs2b3vkoO7
2uuLFlZWQ8/h1RM9ph8nMM1JVNvJEzSacXXFbOqnC5j5IZ0nrz6jOTlIaoytyZn7wxLyvQIDAQAB
o4IBhjCCAYIwDgYDVR0PAQH/BAQDAgEGMB0GA1UdDgQWBBRJt8bP6D0ff+pEexMp9/EKcD7eZDAf
BgNVHSMEGDAWgBQxw3kbuvVT1xfgiXotF2wKsyudMzASBgNVHRMBAf8ECDAGAQH/AgECMGIGA1Ud
IARbMFkwEQYPKwYBBAGBrSGCLAEBBAICMBEGDysGAQQBga0hgiwBAQQDADARBg8rBgEEAYGtIYIs
AQEEAwEwDwYNKwYBBAGBrSGCLAEBBDANBgsrBgEEAYGtIYIsHjA+BgNVHR8ENzA1MDOgMaAvhi1o
dHRwOi8vcGtpMDMzNi50ZWxlc2VjLmRlL3JsL0RUX1JPT1RfQ0FfMi5jcmwweAYIKwYBBQUHAQEE
bDBqMCwGCCsGAQUFBzABhiBodHRwOi8vb2NzcDAzMzYudGVsZXNlYy5kZS9vY3NwcjA6BggrBgEF
BQcwAoYuaHR0cDovL3BraTAzMzYudGVsZXNlYy5kZS9jcnQvRFRfUk9PVF9DQV8yLmNlcjANBgkq
hkiG9w0BAQsFAAOCAQEAYyAo/ZwhhnK+OUZZOTIlvKkBmw3Myn1BnIZtCm4ssxNZdbEzkhthJxb/
w7LVNYL7hCoBSb1mu2YvssIGXW4/buMBWlvKQ2NclbbhMacf1QdfTeZlgk4y+cN8ekvNTVx07iHy
dQLsUj7SyWrTkCNuSWc1vn9NVqTszC/Pt6GXqHI+ybxA1lqkCD3WvILDt7cyjrEsjmpttzUCGc/1
OURYY6ckABCwu/xOr24vOLulV0k/2G5QbyyXltwdRpplic+uzPLl2Z9Tsz6hL5Kp2AvGhB8Exuse
6J99tXulAvEkxSRjETTMWpMgKnmIOiVCkKllO3yG0xIVIyn8LNrMOVtUFzCCBaIwggSKoAMCAQIC
BxekJKEJSDMwDQYJKoZIhvcNAQELBQAwWjELMAkGA1UEBhMCREUxEzARBgNVBAoTCkRGTi1WZXJl
aW4xEDAOBgNVBAsTB0RGTi1QS0kxJDAiBgNVBAMTG0RGTi1WZXJlaW4gUENBIEdsb2JhbCAtIEcw
MTAeFw0xNDA1MjcxNDU0MDlaFw0xOTA3MDkyMzU5MDBaMIHGMQswCQYDVQQGEwJERTEcMBoGA1UE
CBMTTm9yZHJoZWluLVdlc3RmYWxlbjERMA8GA1UEBxMITXVlbnN0ZXIxIDAeBgNVBAoTF0ZhY2ho
b2Noc2NodWxlIE11ZW5zdGVyMSMwIQYDVQQLExpEYXRlbnZlcmFyYmVpdHVuZ3N6ZW50cmFsZTEd
MBsGA1UEAxMURkggTXVlbnN0ZXIgQ0EgLSBHMDExIDAeBgkqhkiG9w0BCQEWEWNhQGZoLW11ZW5z
dGVyLmRlMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAuHlsrvBs7CL9IqMH9r//QU9E
pghTV/3skHuQZ3DpNY+lyJWOW5zbtUubgXt7lYHpIE4d4CclTZWqCHwoAI6gqzSSGjUKuX6/0ui/
LhXmlDvCBfwuER+T+3/R59hlLnhI5iYYPQiNywQIa3wJhBLTZrlXw8nDdjI54MAzcVDUX7l21sbo
ZIA6idM7SXmshxoRQ6xsfPHskrceNMcvtHNDhVnVscwRUJQUR55fs0X7Y93PasugWPv3xmgNr1da
Cq94eV+nslNU/GJaT9TQ3uG8pagLXl9NbDNkHIrvFAD5zXO0m/d00I4QhUVQyEtwnTegDqcM+WFh
JXensgnZhWe6bwIDAQABo4IB/jCCAfowEgYDVR0TAQH/BAgwBgEB/wIBATAOBgNVHQ8BAf8EBAMC
AQYwEQYDVR0gBAowCDAGBgRVHSAAMB0GA1UdDgQWBBQK81u85DGA1jVCiabTw8833tHf1zAfBgNV
HSMEGDAWgBRJt8bP6D0ff+pEexMp9/EKcD7eZDAcBgNVHREEFTATgRFjYUBmaC1tdWVuc3Rlci5k
ZTCBiAYDVR0fBIGAMH4wPaA7oDmGN2h0dHA6Ly9jZHAxLnBjYS5kZm4uZGUvZ2xvYmFsLXJvb3Qt
Y2EvcHViL2NybC9jYWNybC5jcmwwPaA7oDmGN2h0dHA6Ly9jZHAyLnBjYS5kZm4uZGUvZ2xvYmFs
LXJvb3QtY2EvcHViL2NybC9jYWNybC5jcmwwgdcGCCsGAQUFBwEBBIHKMIHHMDMGCCsGAQUFBzAB
hidodHRwOi8vb2NzcC5wY2EuZGZuLmRlL09DU1AtU2VydmVyL09DU1AwRwYIKwYBBQUHMAKGO2h0
dHA6Ly9jZHAxLnBjYS5kZm4uZGUvZ2xvYmFsLXJvb3QtY2EvcHViL2NhY2VydC9jYWNlcnQuY3J0
MEcGCCsGAQUFBzAChjtodHRwOi8vY2RwMi5wY2EuZGZuLmRlL2dsb2JhbC1yb290LWNhL3B1Yi9j
YWNlcnQvY2FjZXJ0LmNydDANBgkqhkiG9w0BAQsFAAOCAQEA3kcDNdZKb7kSD7s1ly2qa/2QbQe+
ld3LhZeOcfysdLtN8oweBmgT3MYoZ+D9c+SoUWJAwTKPB15DoGy+fWhelXTpQrqxIGb4ISr1JCjg
slnmMUva0xjwZGxojZ9gE1bi18xfKw3+dMpwCLt6LbLTjr/tyH6otacwr2tZzuuJIUAORnefwTcr
vmB21n/BEQH/ZXruWu8lSO3L9YAmQB6ViaZFCpn2sMmOLACdoWxmUQb3QAjsa327jHUjsz53k9q5
Zrx/g+zOg5s1Wmy2JOlLQMUIZXXf0/6rB5Fr2llx7dBG/Uk7NhZdNy7OzNzci0C4Wnkd8rDVEWHG
hH2gfpcTfjCCBg0wggT1oAMCAQICBxuZiHQ3saMwDQYJKoZIhvcNAQELBQAwgcYxCzAJBgNVBAYT
AkRFMRwwGgYDVQQIExNOb3JkcmhlaW4tV2VzdGZhbGVuMREwDwYDVQQHEwhNdWVuc3RlcjEgMB4G
A1UEChMXRmFjaGhvY2hzY2h1bGUgTXVlbnN0ZXIxIzAhBgNVBAsTGkRhdGVudmVyYXJiZWl0dW5n
c3plbnRyYWxlMR0wGwYDVQQDExRGSCBNdWVuc3RlciBDQSAtIEcwMTEgMB4GCSqGSIb3DQEJARYR
Y2FAZmgtbXVlbnN0ZXIuZGUwHhcNMTYwNzA0MDcwNjEzWhcNMTkwNzA0MDcwNjEzWjB8MQswCQYD
VQQGEwJERTEgMB4GA1UECgwXRmFjaGhvY2hzY2h1bGUgTXVlbnN0ZXIxMjAwBgNVBAsMKUZhY2hi
ZXJlaWNoIEVsZWt0cm90ZWNobmlrIHVuZCBJbmZvcm1hdGlrMRcwFQYDVQQDDA5NaWNoYWVsIFR1
ZXhlbjCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMyaGlBt2ZtuF8QP8zYNrGxXC+es
PMajIPl+hu1LGHnN2BJ3J5ZMN44BOZw3n6LO1FaAgO8D4xU4/AELecX6VxJZ2zOOSD8uTYO4OnUu
24hkjFUQAj13tT644AKUQMMBpgj7wC52V5Jij+mZX/t1S38/WFiCGnirt4xTNi5OmN4K+VNZfG4x
0msDqFjJX70rF1y09/Mylu1M/Y0tu/I9DqhwDQT4LBOvyyaAlhSJ8Jb8m8YTt5xlOzrXlBmj4pKs
74y7C2IKRw4tFozGX1cf1LVEs2eBCb5iUwXrlcMipwm62sJ38GD00EOlRNTpAM5rDAcgWxMCffek
bRv/01whtOkCAwEAAaOCAkcwggJDMEAGA1UdIAQ5MDcwEQYPKwYBBAGBrSGCLAEBBAMFMBEGDysG
AQQBga0hgiwCAQQDATAPBg0rBgEEAYGtIYIsAQEEMAkGA1UdEwQCMAAwDgYDVR0PAQH/BAQDAgXg
MB0GA1UdJQQWMBQGCCsGAQUFBwMCBggrBgEFBQcDBDAdBgNVHQ4EFgQU0B2vaoSoEmYAggD04WZF
2hGif3UwHwYDVR0jBBgwFoAUCvNbvOQxgNY1Qomm08PPN97R39cwIAYDVR0RBBkwF4EVdHVleGVu
QGZoLW11ZW5zdGVyLmRlMIGIBgNVHR8EgYAwfjA9oDugOYY3aHR0cDovL2NkcDEucGNhLmRmbi5k
ZS9maC1tdWVuc3Rlci1jYS9wdWIvY3JsL2NhY3JsLmNybDA9oDugOYY3aHR0cDovL2NkcDIucGNh
LmRmbi5kZS9maC1tdWVuc3Rlci1jYS9wdWIvY3JsL2NhY3JsLmNybDCB1wYIKwYBBQUHAQEEgcow
gccwMwYIKwYBBQUHMAGGJ2h0dHA6Ly9vY3NwLnBjYS5kZm4uZGUvT0NTUC1TZXJ2ZXIvT0NTUDBH
BggrBgEFBQcwAoY7aHR0cDovL2NkcDEucGNhLmRmbi5kZS9maC1tdWVuc3Rlci1jYS9wdWIvY2Fj
ZXJ0L2NhY2VydC5jcnQwRwYIKwYBBQUHMAKGO2h0dHA6Ly9jZHAyLnBjYS5kZm4uZGUvZmgtbXVl
bnN0ZXItY2EvcHViL2NhY2VydC9jYWNlcnQuY3J0MA0GCSqGSIb3DQEBCwUAA4IBAQBI9v+seJM6
AlSIrmmpopz6zh8QAsqGLJkkY2D0KYFucUY/xZaJTtZxvmWddbKk2903Qhg+vZKOf87PHhip7/4t
FSwhxYNSS36WsRJTeUa0f3KkSa28yrIRfWlJATgxfL5X/QQnopjCt34n4221kcsR7LHxBAn37ow+
/2L7WjWDDuOkaM9/ZSCtrN+yFRat1eUVs1Hk7sKT/bfJTsYqzovXitjmCP3YdB40dkuQ6/ZzEdXT
bpa4c45RcRnPqKXnxknK0UfRHNHqk15W7dUPVMzSGFUvjhmWPP2wW6a8F1U5sEqfHcoBFC5CGjGy
7Gk2luk3obi/KLrDyZC+dkjhDYEpMYIEOTCCBDUCAQEwgdIwgcYxCzAJBgNVBAYTAkRFMRwwGgYD
VQQIExNOb3JkcmhlaW4tV2VzdGZhbGVuMREwDwYDVQQHEwhNdWVuc3RlcjEgMB4GA1UEChMXRmFj
aGhvY2hzY2h1bGUgTXVlbnN0ZXIxIzAhBgNVBAsTGkRhdGVudmVyYXJiZWl0dW5nc3plbnRyYWxl
MR0wGwYDVQQDExRGSCBNdWVuc3RlciBDQSAtIEcwMTEgMB4GCSqGSIb3DQEJARYRY2FAZmgtbXVl
bnN0ZXIuZGUCBxuZiHQ3saMwDQYJYIZIAWUDBAIBBQCgggI3MBgGCSqGSIb3DQEJAzELBgkqhkiG
9w0BBwEwHAYJKoZIhvcNAQkFMQ8XDTE5MDQwMjIyMjczMlowLwYJKoZIhvcNAQkEMSIEIGSIIzmA
QlMYcCWNiwVDItNMbp2U1LppxYebTqP1GV1zMIHjBgkrBgEEAYI3EAQxgdUwgdIwgcYxCzAJBgNV
BAYTAkRFMRwwGgYDVQQIExNOb3JkcmhlaW4tV2VzdGZhbGVuMREwDwYDVQQHEwhNdWVuc3RlcjEg
MB4GA1UEChMXRmFjaGhvY2hzY2h1bGUgTXVlbnN0ZXIxIzAhBgNVBAsTGkRhdGVudmVyYXJiZWl0
dW5nc3plbnRyYWxlMR0wGwYDVQQDExRGSCBNdWVuc3RlciBDQSAtIEcwMTEgMB4GCSqGSIb3DQEJ
ARYRY2FAZmgtbXVlbnN0ZXIuZGUCBxuZiHQ3saMwgeUGCyqGSIb3DQEJEAILMYHVoIHSMIHGMQsw
CQYDVQQGEwJERTEcMBoGA1UECBMTTm9yZHJoZWluLVdlc3RmYWxlbjERMA8GA1UEBxMITXVlbnN0
ZXIxIDAeBgNVBAoTF0ZhY2hob2Noc2NodWxlIE11ZW5zdGVyMSMwIQYDVQQLExpEYXRlbnZlcmFy
YmVpdHVuZ3N6ZW50cmFsZTEdMBsGA1UEAxMURkggTXVlbnN0ZXIgQ0EgLSBHMDExIDAeBgkqhkiG
9w0BCQEWEWNhQGZoLW11ZW5zdGVyLmRlAgcbmYh0N7GjMA0GCSqGSIb3DQEBAQUABIIBAIZaQcal
2J/msfopGijjkS2+107hVANwLAQ6CiD93XD/ijJOkTZQLjqzRmSFAFMQGw451uj11sla6SlLLFYI
QR3GJEVfyykRs31RS3HAbU8b96QiB4AYCWStAbulwWbJFHTdKZJsZEc4oHCA3OboIw7gjd7c89PF
3siE2nHFJWUlUevrMqg+xSaqkXV1eOhC8l0foNqzlSYyhYw3wdB3mpZS2kDQ6d5VVO0qskqy5Ua9
RW457EF9AXcW3sCJ6k/B6p78uUUkEkNgBZbRkIykqu/VACIffur+2qF49sKEV4yjhSxFNtJKtqVM
RZNPcwhMe446prrHZbWU3ThPedtIv8QAAAAAAAA=
--Apple-Mail=_A26B82D4-1AFC-4AF8-A85C-EB10136E809F--



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?E0785613-2B6E-4BB3-95CD-03DD96902CD8>