Date: Wed, 16 Mar 2016 13:29:47 -0700 From: Stanislav Sedov <stas@FreeBSD.org> To: freebsd-hackers@freebsd.org Subject: Peformance issues with r278325 Message-ID: <FA50A68E-7F3D-4361-8A8A-EB7F97EF3D00@FreeBSD.org>
next in thread | raw e-mail | index | archive | help
Hi!
We're seeing weird performance issues on some of our systems on both
FreeBSD stable/10 and HEAD with r278325. In particular, the part of the
change that switched the lapic_ipi_wait to microsecond-based delay seem
to cause slowdowns.
Some background. Out systems mostly run a single large application that
is both I/O heavy and memory allocation heavy. It is doing network =
traffic
processing, but that does not seem to be the culprit. What we noticed =
is that
when running on FreeBSD 10.2+ or HEAD the memory allocations (and, =
particularly,
munmaps) become exponentially slower the bigger the application becomes =
in terms
of the memory space allocated. This does not seem to happen on FreeBSD =
10.1
or stable/10 with the lapic_ipi_wait() change reverted.
The top output when this slowdown happens shows the following:
=
__________________________________________________________________________=
_________
/
| 1585 user -21 r31 20225M 17260M vm map 5 483:20 78.47% =
[beam.smp{beam.smp}]
| 1585 user -21 r31 20225M 17260M *proce 11 483:26 77.29% =
[beam.smp{beam.smp}]
| 1585 user -21 r31 20225M 17260M vm map 15 484:05 76.86% =
[beam.smp{beam.smp}]
| 1585 user -21 r31 20225M 17260M CPU21 21 262:49 76.86% =
[beam.smp{beam.smp}]
| 1585 user -21 r31 20225M 17260M CPU26 26 473:31 76.76% =
[beam.smp{beam.smp}]
| 1585 user -21 r31 20225M 17260M RUN 5 462:01 76.56% =
[beam.smp{beam.smp}]
| 1585 user -21 r31 20225M 17260M vm map 2 282:16 76.37% =
[beam.smp{beam.smp}]
| 1585 user -21 r31 20225M 17260M CPU4 4 475:24 76.27% =
[beam.smp{beam.smp}]
| 1585 user -21 r31 20225M 17260M CPU38 38 366:16 76.17% =
[beam.smp{beam.smp}]
| 1585 user -21 r31 20225M 17260M vm map 16 434:35 76.07% =
[beam.smp{beam.smp}]
=
`-------------------------------------------------------------------------=
-----------
I had a pmcstat output as well somewhere, but can't find it at the =
moment. But it was
indicating that a lot of time has been spend in VM mutexes as far as I =
remember.
I'm not entirely sure why this change would cause any effects like this. =
One theory
I have is that TLB shootdowns become slower due to lapic_ipi_wait taking =
longer than
before to complete (as it was using pause instructions instead of =
explicit delays).
Any ideas? Anything else I can look at to track the reason down? Out =
platform is
Supermicro X9DRi-LN4+ with dual Xeon(R) CPU E5-2690 v2 CPUs. I'm not =
sure if it's
platform specific or not though. I've seen that happen on servers with =
E5-2690 v3
as well.
Thanks!
=
__________________________________________________________________________=
___________
/
| int
| lapic_ipi_wait(int delay)
| {
| - int x;
| + int x, incr;
| =20
| /*
| - * Wait delay microseconds for IPI to be sent. If delay is
| + * Wait delay loops for IPI to be sent. This is highly bogus
| + * since this is sensitive to CPU clock speed. If delay is
| * -1, we wait forever.
| */
| if (delay =3D=3D -1) {
| - while ((lapic->icr_lo & APIC_DELSTAT_MASK) !=3D =
APIC_DELSTAT_IDLE)
| - ia32_pause();
| - return (1);
| - }
| -
| - for (x =3D 0; x < delay; x +=3D 5) {
| + incr =3D 0;
| + delay =3D 1;
| + } else
| + incr =3D 1;
| + for (x =3D 0; x < delay; x +=3D incr) {
| if ((lapic->icr_lo & APIC_DELSTAT_MASK) =3D=3D =
APIC_DELSTAT_IDLE)
| return (1);
| - DELAY(5);
| + ia32_pause();
| }
| return (0);
| }
| @@ -1433,9 +1433,9 @@ lapic_ipi_raw(register_t icrlo, u_int dest)
| intr_restore(saveintr);
| }
| =20
| -#define BEFORE_SPIN 50000
| +#define BEFORE_SPIN 1000000
| #ifdef DETECT_DEADLOCK
| -#define AFTER_SPIN 50
| +#define AFTER_SPIN 1000
| #endif
=
`-------------------------------------------------------------------------=
----------
--
Stanislav Sedov
ST4096-RIPE
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?FA50A68E-7F3D-4361-8A8A-EB7F97EF3D00>