From owner-freebsd-hackers@FreeBSD.ORG Mon Oct 3 21:40:00 2011 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B8EC61065672; Mon, 3 Oct 2011 21:40:00 +0000 (UTC) (envelope-from lev@FreeBSD.org) Received: from onlyone.friendlyhosting.spb.ru (onlyone.friendlyhosting.spb.ru [IPv6:2a01:4f8:131:60a2::2]) by mx1.freebsd.org (Postfix) with ESMTP id 4F0058FC12; Mon, 3 Oct 2011 21:40:00 +0000 (UTC) Received: from lion.home.serebryakov.spb.ru (unknown [IPv6:2001:470:923f:1:906c:6af3:5301:18c6]) (Authenticated sender: lev@serebryakov.spb.ru) by onlyone.friendlyhosting.spb.ru (Postfix) with ESMTPA id 5376B4AC1C; Tue, 4 Oct 2011 01:39:58 +0400 (MSD) Date: Tue, 4 Oct 2011 01:39:50 +0400 From: Lev Serebryakov Organization: FreeBSD X-Priority: 3 (Normal) Message-ID: <176698417.20111004013950@serebryakov.spb.ru> To: mdf@FreeBSD.org In-Reply-To: References: <1317653757588-4865200.post@n5.nabble.com> MIME-Version: 1.0 Content-Type: text/plain; charset=windows-1251 Content-Transfer-Encoding: quoted-printable Cc: freebsd-hackers@freebsd.org, satish kondapalli , Arnaud Lacombe Subject: Re: NUMA Support is there in FreeBSD. X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: lev@FreeBSD.org List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Oct 2011 21:40:00 -0000 Hello, Mdf. You wrote 3 =EE=EA=F2=FF=E1=F0=FF 2011 =E3., 21:34:29: > Your statement isn't incorrect. What I'm saying is that there's no > KPI for requesting bound memory because, while the netstat example is > a fine one for where local memory is desired, the majority [1] of > processing is not bound to a CPU and so round-robin allocations will > produce uniform performance results -- that is, not the best possible, > but not wildly fluctuating as scheduling decisions over different runs > give different remote memory penalties. We have exactly the same config at ${WORK}, as Arnaud describes. And we need to process huge (4Gbit+ wire speed in small -- 100-1000 bytes -- packets) UDP traffic. Without fixed affinity of "netisr" threads our system drops some packets on the way between DMA-mapped network card buffers and kernel structures. One big difference: we use Solaris and it have all needed API, KPI and userland control utilities to tune system, both kernel-side and userland-side. Even Solaris, though, could no process such traffic "automagically". We didn't try FreeBSD, as our ops knows nothing about it (I'm only FreeBSD fan in team and I'm developer, not operations)... I wrote this as example, that for some tasks system NEEDS all these NUMA-specific knobs. BTW, NUMA-aware allocator in HotSpot (Sun's, errr, sorry, Oracle's Java VM), added between Java6 and Java7, increased performance for some workloads up to 300% on 72-way system (SunFire 15K), and gives about 3% performance drop on worst situations :) And it was allocator in virtual machine! But it would have been impossible without kernel API, so this changes works well only on HotSpot/Solaris ;-) Again, I wrote this all to show, that NUMA-awareness could be very useful on big iron. --=20 // Black Lion AKA Lev Serebryakov