From owner-freebsd-net@FreeBSD.ORG Sun Jul 20 02:28:06 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 99E7A106566B for ; Sun, 20 Jul 2008 02:28:06 +0000 (UTC) (envelope-from kip.macy@gmail.com) Received: from rv-out-0506.google.com (rv-out-0506.google.com [209.85.198.228]) by mx1.freebsd.org (Postfix) with ESMTP id 600E48FC12 for ; Sun, 20 Jul 2008 02:28:06 +0000 (UTC) (envelope-from kip.macy@gmail.com) Received: by rv-out-0506.google.com with SMTP id b25so1020714rvf.43 for ; Sat, 19 Jul 2008 19:28:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:cc:in-reply-to:mime-version:content-type :content-transfer-encoding:content-disposition:references; bh=mSU3BTn0mvS2K91/nfJ/xPFfPMh30MpFuZ10yFA7EuE=; b=m4mFyR5008p7SgjDRqiVZrvhZ+rzByreMDCuMbfwZHPL0KAUPAq3DVBUSi6F7K0qj7 vxy1IAnNHacRXJR2MMFcl2GEeU9e3dcZmkCejyQv0r8PG4244ZwcaR9CEoomhLXOqOTY C+EfKj0b6Q+URAJm5WBr8wMmCM9c5vp+PeHSk= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version :content-type:content-transfer-encoding:content-disposition :references; b=byHFqZK5Gt3m7SUleSs56pGclZRrWVF3G5WJW5BNnOXKQfzuOn/SkEOqdBUHj/AAe6 SlttxyzMTJP8NnZSCrUmLjYF0OjRYyrQdkW+pxNJlQelDQys5g04Ci6GaYaPNNirgdWf 2nPimSfAAcN50slZ/9kbBPRH7GTueAw9T6/Ww= Received: by 10.140.201.15 with SMTP id y15mr1035220rvf.33.1216520885887; Sat, 19 Jul 2008 19:28:05 -0700 (PDT) Received: by 10.141.123.2 with HTTP; Sat, 19 Jul 2008 19:28:05 -0700 (PDT) Message-ID: Date: Sat, 19 Jul 2008 19:28:05 -0700 From: "Kip Macy" To: "Brian McGinty" In-Reply-To: <601bffc40807191917g131bacao6485376365304f55@mail.gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <4867420D.7090406@gtcomm.net> <4871E85C.8090907@freebsd.org> <48726422.7050703@gtcomm.net> <200807080107.m6817XxO021966@lava.sentex.ca> <601bffc40807081346q454c1f40td47a0f54806d8a8c@mail.gmail.com> <601bffc40807112344n7a683f81y516f540e24d87389@mail.gmail.com> <601bffc40807191917g131bacao6485376365304f55@mail.gmail.com> Cc: FreeBSD Net , Paul Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 20 Jul 2008 02:28:06 -0000 On Sat, Jul 19, 2008 at 7:17 PM, Brian McGinty wrote: > G'day Kip, > >> I'm hoping to get to it some time in August. I'm a bit behind in my >> contracts at the moment. > > A few weeks ago, I did a quick comparison of the driver between > FreeBSD and Linux, and found quite a few differences that's worth > pulling over. The guy from Intel working on FreeBSD, Jack?, is he the > one that does this sort of sync-up of the drivers between the two > distribution, or you? There's been a lot of changes recently, > including full support for multiple Rx/Tx queues that significantly > ups the ante on performance. FreeBSD doesn't support multiple Rx/Tx, > or does something half arsed. This is on a variant of RELENG_6 FreeBSD with a recent version of ULE and running the Checkpoint firewall. It also uses the full number of queues available to igb (4) and #queues == #cores (8 in this case) for ixgbe. The drivers in CVS have some bugs that I have fixed in this FreeBSD variant. FreeBSD's CVS version of the Intel drivers definitely lags Linux in terms of some optimizations. Even my version doesn't have some of the linux optimizations. >> FYI: I'm actually able to forward 2.3Mpps between 2 10Gig interfaces >> on an 8-core system. I'm hoping to push it up to 3Mpps. This is testing with an IXIA I don't currently have zero loss numbers. This is not fully loaded. However, ixgbe spews out pause frames when rx gets backed up so losses never get much above 0.1%. > Is this no-loss number, and how did you test it? I don't have > throughput numbers for the Oplin. I'm waiting to get some time on the > Ixia at work to generate performance numbers for 1G and 10G for all > packet sizes, on FreeBSD and Linux, on a 16 core system, and blast it > to the list. I expect Linux to do 2-3 times better :-) Sure, if you don't care about packet reordering. On their own box Checkpoint claims that Linux is currently able to do 20% better than we are seeing. Even they don't claim 200% - 300%. I know people who are switching off of Linux for memcache because they simply can't make it perform. So you're mileage really varies depending on the workload. I'm not sure where you get your numbers from. I would really like to get a hold of this magical Linux distribution to do a side by side comparison on the same workload. A 200% - 300% performance delta would definitely justify switching. Thanks, Kip