Date: Wed, 28 Jul 2010 06:44:07 +1000 (EST) From: Bruce Evans <brde@optusnet.com.au> To: John Baldwin <jhb@freebsd.org> Cc: jkoshy@freebsd.org, freebsd-amd64@freebsd.org Subject: Re: PERFMON isn't operational on amd64 Message-ID: <20100728054627.T11937@besplex.bde.org> In-Reply-To: <201007271540.30774.jhb@freebsd.org> References: <AANLkTim3m4VX3sy6diTNna2Yb7i%2B4RQZ0kT6U7e_wqTU@mail.gmail.com> <201007270928.24959.jhb@freebsd.org> <20100728044151.Y1330@delplex.bde.org> <201007271540.30774.jhb@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Tue, 27 Jul 2010, John Baldwin wrote:
> On Tuesday, July 27, 2010 3:06:50 pm Bruce Evans wrote:
>> hwpmc(4) is far too featureful for me, yet perfmon(4) still does things
>> for me that hwpmc cannot do:
>> - from userland, without using the library, try undocumented unnamed
>> counters. There are a few useful ones for k7. E.g.:
>> 0xc8 (k8-fr-retired-near-returns) and
>> 0xc9 (k8-fr-retired-near-returns-mispredicted)
>> are k8-only according to hwpmc and amd docs, but also work on my k7
>> (after a 4-line fix for perfmon on kx).
>
> I believe this is not too terribly hard to fix.
Indeed. Just remove restrictions.
> For the two above I think
> this will suffice (manpage would also need to be updated for a full patch):
>
> (I think the misordering in pmc_events.h is required to preserve userland
> ABI.)
>
> Index: hwpmc_amd.c
> ===================================================================
> --- hwpmc_amd.c (revision 210547)
> +++ hwpmc_amd.c (working copy)
> @@ -137,6 +137,8 @@
> { PMC_EV_K7_RETIRED_TAKEN_BRANCHES_MISPREDICTED, 0xC5, 0 },
> { PMC_EV_K7_RETIRED_FAR_CONTROL_TRANSFERS, 0xC6, 0 },
> { PMC_EV_K7_RETIRED_RESYNC_BRANCHES, 0xC7, 0 },
> + { PMC_EV_K7_RETIRED_NEAR_RETURNS, 0xC8, 0 },
> + { PMC_EV_K7_RETIRED_NEAR_RETURNS_MISPREDICTED, 0xC9, 0 },
> { PMC_EV_K7_INTERRUPTS_MASKED_CYCLES, 0xCD, 0 },
> { PMC_EV_K7_INTERRUPTS_MASKED_WHILE_PENDING_CYCLES, 0xCE, 0 },
> { PMC_EV_K7_HARDWARE_INTERRUPTS, 0xCF, 0 },
I think the undocumented ones should require a special flag or something
to use. Maybe they depend on the stepping.
Here is my complete list (54 items), from a case statement that is
supposed to convert magic numbers to the hwpmc/amd names. This needs
to be verified. I don't remember checking all of it. In fact, I
probably didn't, but just duplicated the k8 names with s/k8/kx/ iff
the counter is sometimes nonzero for both k7 and k8, then checked the
more interesting ones. IIRC, checking consisted mainly of running
normal utilites on k7 and k8 to see if the counts were similar. This
wouldn't work if the CPU reasources were too different.
% 0x21) src=kx-ls-microarchitectural-resync-by-self-mod-code;;
% 0x23) src=kx-ls-buffer2-full;;
I couldn't have verified the more arcane and/or lower-level ones like
these if I tried. These are documented for k8, and nonzero for k7.
The 202 cases not in this list are left out because they are either
documented for k7 and identical with k8, or never observed to have nonzero
counts.
% 0x26) src=kx-ls-retired-cflush-instructions;;
% 0x27) src=kx-ls-retired-cpuid-instructions;;
Probably verifiable.
% 0x2a) src=kx-ls-unknown-$ctr1;;
% 0x2b) src=kx-ls-unknown-$ctr1;;
% 0x2e) src=k7-ls-unknown-$ctr1;;
% 0x2f) src=k7-ls-unknown-$ctr1;;
% 0x32) src=kx-ls-unknown-$ctr1;;
% 0x33) src=kx-ls-unknown-$ctr1;;
% 0x36) src=k7-ls-unknown-$ctr1;;
% 0x37) src=k7-ls-unknown-$ctr1;;
% 0x3a) src=kx-ls-unknown-$ctr1;;
% 0x3b) src=kx-ls-unknown-$ctr1;;
% 0x3e) src=k7-ls-unknown-$ctr1;;
% 0x3f) src=k7-ls-unknown-$ctr1;;
Unknown means it's undocumented but observed to have a nonzero count
(kx-*-unkown if nonzero on both, etc.).
% 0x48) src=kx-dc-microarchitectural-late-cancel-of-an-access;;
% 0x49) src=kx-dc-microarchitectural-early-cancel-of-an-access;;
% 0x4d) src=k7-dc-unknown-$ctr1;;
% 0x4e) src=k7-dc-unknown-$ctr1;;
% 0x4f) src=k7-dc-unknown-$ctr1;;
% 0x50) src=kx-dc-unknown-$ctr1;;
% 0x51) src=kx-dc-unknown-$ctr1;;
% 0x55) src=kx-dc-unknown-$ctr1;;
% 0x56) src=kx-dc-unknown-$ctr1;;
% 0x57) src=kx-dc-unknown-$ctr1;;
% 0x58) src=kx-dc-unknown-$ctr1;;
% 0x59) src=kx-dc-unknown-$ctr1;;
% 0x5d) src=k7-dc-unknown-$ctr1;;
% 0x5e) src=k7-dc-unknown-$ctr1;;
% 0x5f) src=k7-dc-unknown-$ctr1;;
% 0x64) src=kx-bu-unknown-$ctr1;;
% 0x68) src=kx-bu-unknown-$ctr1;;
% 0x69) src=kx-bu-unknown-$ctr1;;
% 0x76) src=kx-bu-cpu-clk-unhalted;;
% 0x82) src=kx-ic-refill-from-l2;;
% 0x83) src=kx-ic-refill-from-system;;
% 0x87) src=kx-ic-instruction-fetch-stall;;
% 0x88) src=kx-ic-return-stack-hit;;
% 0x89) src=kx-ic-return-stack-overflow;;
Above 2 are even more interesting than the next 2. I should have checked
these.
% 0xc8) src=kx-fr-retired-near-returns;;
% 0xc9) src=kx-fr-retired-near-returns-mispredicted;;
% 0xca) src=kx-fr-retired-taken-branches-mispred-by-addr-miscompare;;
% 0xd0) src=kx-fr-decoder-empty;;
% 0xd1) src=kx-fr-dispatch-stalls;;
% 0xd2) src=kx-fr-dispatch-stall-from-branch-abort-to-retire;;
% 0xd3) src=kx-fr-dispatch-stall-for-serialization;;
% 0xd4) src=kx-fr-dispatch-stall-for-segment-load;;
% 0xd5) src=kx-fr-dispatch-stall-when-reorder-buffer-is-full;;
% 0xd6) src=kx-fr-dispatch-stall-when-reservation-stations-are-full;;
% 0xd7) src=kx-fr-dispatch-stall-when-fpu-is-full;;
% 0xd8) src=kx-fr-dispatch-stall-when-ls-is-full;;
% 0xd9) src=kx-fr-dispatch-stall-when-waiting-for-all-to-be-quiet;;
% 0xda) src=kx-fr-dispatch-stall-when-far-xfer-or-resync-br-pending;;
>> - in the kernel count, events at the level of individual functions, non-
>> statistically using high resolution kernel profiling. This may result
>> in more time spent counting than doing useful work, but provides
>> high resolution.
>
> I would rather us provide this via hwpmc(4). perfmon(4) does not support
> programmable events on modern Intel CPUs nor does it support non-x86 CPUs.
Sure, but don't remove sort-of-working code until you have a replacement.
Profiling just needs to call non-profiling-routine functions to read
the counters (for perfmon, the whole module is hackishly configued as
profiling-routine so that its normal functions can be called). The
functions need to be callable in the "any" context without deadlocking
or corrupting the context. This is nontrivial if the context is hwpmc
or perfmon. This is especially nontrivial for hwpmc, since it uses
NMIs. For perfmon, problems are supposed to be avoided by acquiring
exclusive access everything in perfmon while profiling. I can only
see it acquiring exclusive access to the counter being used.
Bruce
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20100728054627.T11937>