Date: Tue, 11 Nov 2008 12:16:37 -0500 From: John Baldwin <jhb@freebsd.org> To: "Archimedes Gaviola" <archimedes.gaviola@gmail.com> Cc: freebsd-smp@freebsd.org Subject: Re: CPU affinity with ULE scheduler Message-ID: <200811111216.37462.jhb@freebsd.org> In-Reply-To: <42e3d810811102032w7850a1c0t386d80ce747f37d3@mail.gmail.com> References: <42e3d810811100033w172e90dbl209ecbab640cc24f@mail.gmail.com> <200811101733.04547.jhb@freebsd.org> <42e3d810811102032w7850a1c0t386d80ce747f37d3@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Monday 10 November 2008 11:32:55 pm Archimedes Gaviola wrote: > On Tue, Nov 11, 2008 at 6:33 AM, John Baldwin <jhb@freebsd.org> wrote: > > On Monday 10 November 2008 03:33:23 am Archimedes Gaviola wrote: > >> To Whom It May Concerned: > >> > >> Can someone explain or share about ULE scheduler (latest version 2 if > >> I'm not mistaken) dealing with CPU affinity? Is there any existing > >> benchmarks on this with FreeBSD? Because I am currently using 4BSD > >> scheduler and as what I have observed especially on processing high > >> network load traffic on multiple CPU cores, only one CPU were being > >> stressed with network interrupt while the rests are mostly in idle > >> state. This is an AMD-64 (4x) dual-core IBM system with GigE Broadcom > >> network interface cards (bce0 and bce1). Below is the snapshot of the > >> case. > > > > Interrupts are routed to a single CPU. Since bce0 and bce1 are both on the > > same interrupt (irq 23), the CPU that interrupt is routed to is going to end > > up handling all the interrupts for bce0 and bce1. This not something ULE or > > 4BSD have any control over. > > > > -- > > John Baldwin > > > > Hi John, > > I'm sorry for the wrong snapshot. Here's the right one with my concern. > > PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND > 17 root 1 171 52 0K 16K CPU0 0 54:28 95.17% idle: cpu0 > 15 root 1 171 52 0K 16K CPU2 2 55:55 93.65% idle: cpu2 > 14 root 1 171 52 0K 16K CPU3 3 58:53 93.55% idle: cpu3 > 13 root 1 171 52 0K 16K RUN 4 59:14 82.47% idle: cpu4 > 12 root 1 171 52 0K 16K RUN 5 55:42 82.23% idle: cpu5 > 16 root 1 171 52 0K 16K CPU1 1 58:13 77.78% idle: cpu1 > 11 root 1 171 52 0K 16K CPU6 6 54:08 76.17% idle: cpu6 > 36 root 1 -68 -187 0K 16K WAIT 7 8:50 65.53% > irq23: bce0 bce1 > 10 root 1 171 52 0K 16K CPU7 7 48:19 29.79% idle: cpu7 > 43 root 1 171 52 0K 16K pgzero 2 0:35 1.51% pagezero > 1372 root 10 20 0 16716K 5764K kserel 6 58:42 0.00% kmd > 4488 root 1 96 0 30676K 4236K select 2 1:51 0.00% sshd > 18 root 1 -32 -151 0K 16K WAIT 0 1:14 0.00% swi4: clock s > 20 root 1 -44 -163 0K 16K WAIT 0 0:30 0.00% swi1: net > 218 root 1 96 0 3852K 1376K select 0 0:23 0.00% syslogd > 2171 root 1 96 0 30676K 4224K select 6 0:19 0.00% sshd > > Actually I was doing a network performance testing on this system with > FreeBSD-6.2 RELEASE using its default scheduler 4BSD and then I used a > tool to generate big amount of traffic around 600Mbps-700Mbps > traversing the FreeBSD system in bi-direction, meaning both network > interfaces are receiving traffic. What happened was, the CPU (cpu7) > that handles the (irq 23) on both interfaces consumed big amount of > CPU utilization around 65.53% in which it affects other running > applications and services like sshd and httpd. It's no longer > accessible when traffic is bombarded. With the current situation of my > FreeBSD system with only one CPU being stressed, I was thinking of > moving to FreeBSD-7.0 RELEASE with the ULE scheduler because I thought > my concern has something to do with the distributions of load on > multiple CPU cores handled by the scheduler especially at the network > level, processing network load. So, if it is more of interrupt > handling and not on the scheduler, is there a way we can optimize it? > Because if it still routed only to one CPU then for me it's still > inefficient. Who handles interrupt scheduling for bounding CPU in > order to prevent shared IRQ? Is there any improvements with > FreeBSD-7.0 with regards to interrupt handling? It depends. In all likelihood, the interrupts from bce0 and bce1 are both hardwired to the same interrupt pin and so they will always share the same ithread when using the legacy INTx interrupts. However, bce(4) parts do support MSI, and if you try a newer OS snap (6.3 or later) these devices should use MSI in which case each NIC would be assigned to a separate CPU. I would suggest trying 7.0 or a 7.1 release candidate and see if it does better. -- John Baldwin
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200811111216.37462.jhb>