From owner-freebsd-current@FreeBSD.ORG Fri May 7 11:33:04 2004 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 98F3A16A4CE; Fri, 7 May 2004 11:33:04 -0700 (PDT) Received: from mail.sandvine.com (sandvine.com [199.243.201.138]) by mx1.FreeBSD.org (Postfix) with ESMTP id 7AEF243D31; Fri, 7 May 2004 11:33:03 -0700 (PDT) (envelope-from gnagelhout@sandvine.com) Received: by mail.sandvine.com with Internet Mail Service (5.5.2657.72) id ; Fri, 7 May 2004 14:33:02 -0400 Message-ID: From: Gerrit Nagelhout To: 'Scott Long' , Robert Watson , 'John Baldwin' , 'Bruce Evans' Date: Fri, 7 May 2004 14:32:59 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2657.72) Content-Type: text/plain; charset="iso-8859-1" cc: freebsd-current@freebsd.org Subject: RE: 4.7 vs 5.2.1 SMP/UP bridging performance X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 07 May 2004 18:33:04 -0000 Scott Long wrote: > Robert Watson wrote: > > On Fri, 7 May 2004, Brad Knowles wrote: > > > > > >>At 10:55 PM -0400 2004/05/06, Robert Watson wrote: > >> > >> > >>> On occasion, I've had conversations with Peter Wemm about > providing HAL > >>> modules with optimized versions of various common > routines for specific > >>> hardware platforms. However, that would require us to > make a trade-off > >>> between the performance benefits of inlining and the > performance benefits > >>> of a HAL module... > >> > >> I'm confused. Couldn't you just do this sort of stuff as > >>conditional macros, which would have both benefits? > > > > > > Well, the goal of introducing HAL modules would be that you > don't have to > > recompile the kernel in order to perform local hardware-specific > > optimization of low level routines. I.e., you could > substitute faster > > implementations of zeroing, synchronization, certain math > routines, etc > > based on the CPU discovered at run-time. While you can have switch > > statements, etc, it's faster just to relink the kernel to > use the better > > implementation for the available CPU. However, if you do > that, you still > > end up with the function call cost, which might well out-weight the > > benefits of specialization. > > > > Robert N M Watson FreeBSD Core Team, TrustedBSD Projects > > robert@fledge.watson.org Senior Research Scientist, > McAfee Research > > > > It really depends on how you link the HAL module in. Calling > indirectly > through function pointers is pretty darn slow, and I suspect that the > long pipeline of a P4 makes this even worse. Switching to a better > instruction might save you 20 cycles, but the indirect call to do it > might cost you 30 and that assumes that the branched > instruction stream > is still in the L1 cache and that twiddling %esp and %ebp gives no > pipeline stalls themselves. Even without the indirect jump, > all of the > housekeeping that goes into making a function call might > drown out most > benefits. The only way that this might make sense is if you move the > abstraction upwards and have it encompass more common code, or do some > sort of self-modifying code scheme early in the boot. The alternative > might be to have the HAL be a compile-time option like Brad hinted at. > > Scott The biggest problem I still see with all of this, is that even if I could compile the kernel for the P4, under SMP there is still no fast locking mechanism in place (that I am aware of, although I am researching that). I ran a few more tests and did some more calculations to determine the impact of (removing) mutexes, and here is what I found: for UP, I was able to get 850kpps, which is 3294 cycles/packet (at 2.8Ghz) for SMP, it was 500kpps, which is 5600 cycles/packet, or an additional 2306 cycles/packet, which presumably goes mostly towards the atomic locked operations. At ~120 cycles/lock extra for SMP, this means that there should be around 19 atomic operations per packet. After getting rid of one mutex (from IF_DEQUEUE, this is not safe, but fun to try), the performance went to 530kpps, or 5283 cycles/packet. This is a savings of ~317 cycles per packet. After a quick look through the bridge code path, I found the following atomic operations (I probably missed some, and might have some that don't always lock, but the total seems about right) em_process_receive_interrupts (EM_LOCK) bus_dma ? mb_alloc ? (MBP_PERSISTENT flag is set, where is this first locked?) bridge_in (BDG_LOCK) if_handoff (IF_LOCK) em_start (EM_LOCK) IF_DEQUEUE (IF_LOCK) m_free (atomic_cmpset_int) m_free (atomic_subtract_int) At 2 locks/mutex this adds up to about 16 atomic operations per packet. I think that some of the changes that Robert mentioned before about putting mbufs in a list before releasing the lock should help a lot for the Xeons. I am willing to try out some of these changes (both testing for performance, and making the actual code changes) because we can't switch over to 5.x until the performance is back up to where 4.7 was. Most of my experience with FreeBSD (from the last 1/2 year or so of looking at (and changing a few things) the code) is in the area of the low level network drivers (em) and some of the lower stack layers. This is why I have focused on the bridging data path to compare the performance. I must admit that I don't know exactly what code changes are going on in the stack, but if fine-grained locking means a (large) increase in the number of mutexes throughout the stack, I am quite concerned about the performance of the whole system on P4/Xeons. With fine-grained locking I think that the cost of individual functions will go up (a lot in the Xeons :( ), but the overal performance may still be better because multiple threads can do work simultaneously if there is nothing else for the other processors to do. What I am concerned about is that if you have a dual-xeon system with enough kernel (stack) work to keep one processor busy, and enough user-space work to keep the other 3 processors busy on 4.7, what will happen on 5.x? Thanks, Gerrit