Date: Fri, 26 Apr 2013 09:50:27 -0700 From: Erich Weiler <weiler@soe.ucsc.edu> To: Ryan Stone <rysto32@gmail.com> Cc: Paul Tatarsky <paul@soe.ucsc.edu>, freebsd-net <freebsd-net@freebsd.org> Subject: Re: pf performance? Message-ID: <517AB053.9070505@soe.ucsc.edu> In-Reply-To: <CAFMmRNy3YcddDeOrw0RfZwqqC6_=RwjWr8snwH96QXJa9BSU8g@mail.gmail.com> References: <5176E5C1.9090601@soe.ucsc.edu> <20130426134224.GV76816@FreeBSD.org> <517A93FE.7020209@soe.ucsc.edu> <CAFMmRNy3YcddDeOrw0RfZwqqC6_=RwjWr8snwH96QXJa9BSU8g@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
> In other words, until I see like 100% system usage in one core, I > would have room to grow? > > > Probably not. Mutexes in FreeBSD use "adaptive spinning". This means > that when a thread is unable to acquire a mutex, if the owner of the > mutex is still running on another CPU core then the thread will spin and > wait for the mutex to become free rather than block. This improves the > latency of acquiring the mutex and saves us many expensive calls through > the scheduler, but the downside is that you have one heavily contended > mutex you can have many cores wasted spinning on the lock. In your case > it is likely that your interrupt threads are spinning on the pf lock. > You can partially confirm this by profiling the system. The quick way is: Truly fascinating! So my cores that are sucking up CPU on interrupts may be just "spinning" waiting for a lock and not doing real work? That would explain a lot. > kldload hwpmc > pmcstat -S unhalted-cycles -T > (if unhalted-cycles does not work, try instructions). We'll try this. It doesn't look like the hwpmc module exists on this box but we'll bring it in and try that. > If _mtx_lock_sleep is the top function then you are hitting mutex > contention and more cores are unlikely to help. In this case you might > actually be able to improve performance by *reducing* the number of > threads that are contending in pf, for example by using fewer receive > queues in your NIC. > > If this machine is dedicated for pf then setting sysctl net.isr.direct=0 > might also improve performance, by forcing all packets to go through a > single netisr thread (assuming that net.isr.maxthreads is 1). Note that > this will apply to traffic that does not go through pf, so if this > machine were doing other network things (e.g. serving NFS) then it would > bottleneck your other traffic behind your pf traffic. It's only doing pf stuff so this is something we'll try. Thanks for the enlightening information!!
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?517AB053.9070505>