From owner-freebsd-net@FreeBSD.ORG Sun Aug 18 18:42:16 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id E23C2440 for ; Sun, 18 Aug 2013 18:42:16 +0000 (UTC) (envelope-from barney_cordoba@yahoo.com) Received: from nm27-vm3.bullet.mail.ne1.yahoo.com (nm27-vm3.bullet.mail.ne1.yahoo.com [98.138.91.157]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 8FB5225BE for ; Sun, 18 Aug 2013 18:42:16 +0000 (UTC) Received: from [98.138.226.176] by nm27.bullet.mail.ne1.yahoo.com with NNFMP; 18 Aug 2013 18:39:13 -0000 Received: from [98.138.89.163] by tm11.bullet.mail.ne1.yahoo.com with NNFMP; 18 Aug 2013 18:39:13 -0000 Received: from [127.0.0.1] by omp1019.mail.ne1.yahoo.com with NNFMP; 18 Aug 2013 18:39:13 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 35399.84936.bm@omp1019.mail.ne1.yahoo.com Received: (qmail 3379 invoked by uid 60001); 18 Aug 2013 18:39:13 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1376851152; bh=rIqqUuQBuT+kZJpyuiy3klO6plhWRZ+orxuAqq5wLaI=; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type; b=v8O9V/iDBQG9gmLBdj5hxMxU9coMyoh2aUrWtstuxD66Dge4+l7ZK75uRhSGiUs9rFUABRHZgziSeIQr1YDM7mpx/NfrL1sTQqGVLb8GsYWXGZUsWQtqRV+wfES8IR6hB3ketgdZz3JjSuWNFFkyxGN+POVEms+wHYQei5gW4gY= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type; b=mznyS3KJ98fL7s8ltWeIxz9zR6VgymdS7gJvJ+5fbs83bsQ9Wt9l2WFh4Ed1xP8wjARUg0TGOzPUOVbu1QzunKocwN3J1ND3LrzCdDYrxHSriEnYXcmlRgQrIRhTlDgAXV5FPpzt75TziPBJsZ/me6nsOQ3kEWATO5hO26rRlPg=; X-YMail-OSG: w_jyoZwVM1nTkwtSIM.pUPxptPLz6dj7r8JplXkKCVFvRrY ep7aWnNLXsQzYBsL_lnLw85kqXniiI4zbWC9q7zmmv8FtfyKIE9ILd1XB86g Da97A1dwfLEel3_cTUStLAhnXJXx.5o4nZ2htU8b_.FwsBfC9zp5xO3PVEUi XzgZsY3EWTRaNij9l05Zya1X78lOrEo7S8SgzK2BgKMo6I.8PGHIqq5SusRs MsB6vlMKkphWXfU_BeTjPzQokSjpo5Tfa7y21UefiR71uDtkHgCtxrGEiCyB o.oLEQThU9jgiOqcRABKTJhU32loAw045GVqRlshJLuBDwJWlhhB6DQR8Wwn 3MtMXwY4OLqFQewvis.J.qVio21EKX9E.lZyWqCimUqKPDTxkmNAY11F00Wv mNn7.mfiK5xDxgrSunpstxp7NbjHTNXG1_5dFnAz3nBnjAnLELh3J7pdhTOU TqSLRggrXzvSnGd8WM4QgAPwsMHIEq8i7er4cnG2MUTgfTHymFV94NP4FyMu 5dnOkCjL3LFZxp_JFIjGUdcfG55Xkhs6kDQ91XjhBJYFMqHs5h.qZ5djqGzh gyPRUaD.IWUW9ffFlTlMt7b6_o2gfKVDM6f8JKxOGvUz5UWLjKThSMoGFifX DUlF3ilhdwaODPaEqzDxvF5pdr_LTFpL2Vi5. Received: from [98.203.118.124] by web121606.mail.ne1.yahoo.com via HTTP; Sun, 18 Aug 2013 11:39:12 PDT X-Rocket-MIMEInfo: 002.001, R3JlYXQuIE5ldmVyIGhhcyB0aGUgYmVlbiBhIGJldHRlciBleHBsYW5hdGlvbiBmb3IgdGhlIHdvcmQgS2x1ZGdlIHRoYW4gbmV0bWFwLgoKCl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fCiBGcm9tOiBBZHJpYW4gQ2hhZGQgPGFkcmlhbkBmcmVlYnNkLm9yZz4KVG86IEppbSBUaG9tcHNvbiA8amltQG5ldGdhdGUuY29tPiAKQ2M6IEJhcm5leSBDb3Jkb2JhIDxiYXJuZXlfY29yZG9iYUB5YWhvby5jb20.OyBGcmVlQlNEIE5ldCA8bmV0QGZyZWVic2Qub3JnPjsgTHVpZ2kgUml6em8gPHJpenpvQGkBMAEBAQE- X-Mailer: YahooMailWebService/0.8.154.571 References: <520A6D07.5080106@freebsd.org> <520AFBE8.1090109@freebsd.org> <520B24A0.4000706@freebsd.org> <520B3056.1000804@freebsd.org> <20130814102109.GA63246@onelab2.iet.unipi.it> <1376745244.6575.YahooMailNeo@web121606.mail.ne1.yahoo.com> <1376748170.66110.YahooMailNeo@web121601.mail.ne1.yahoo.com> <1376833738.94737.YahooMailNeo@web121605.mail.ne1.yahoo.com> <71EA3DFB-B410-432D-98E0-B6341556BE6D@netgate.com> Message-ID: <1376851152.3322.YahooMailNeo@web121606.mail.ne1.yahoo.com> Date: Sun, 18 Aug 2013 11:39:12 -0700 (PDT) From: Barney Cordoba Subject: Re: it's the output, not ack coalescing (Re: TSO and FreeBSD vs Linux) To: Adrian Chadd In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: "freebsd-net@freebsd.org" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: Barney Cordoba List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 18 Aug 2013 18:42:16 -0000 Great. Never has the been a better explanation for the word Kludge than net= map.=0A=0A=0A________________________________=0A From: Adrian Chadd =0ATo: Jim Thompson =0ACc: Barney Cordoba ; FreeBSD Net ; Luigi Rizzo ; Lawrence Stewart =0ASent: Sunday, Au= gust 18, 2013 11:57 AM=0ASubject: Re: it's the output, not ack coalescing (= Re: TSO and FreeBSD vs Linux)=0A =0A=0ARight. Well, post some profiling dat= a, let's figure this out sometime.=0A=0ALuigi can do bridging with 2 cores = using netmap. So it's technically=0Apossible. There's just a lot of kernel = gunk in the way of doing it ye olde=0Away.=0A=0A=0A=0A-adrian=0A=0A=0AOn 18= August 2013 07:25, Jim Thompson wrote:=0A=0A>=0A> On Aug= 18, 2013, at 8:48 AM, Barney Cordoba =0A> wrote:= =0A>=0A> > I could fill a tx queue with 10gb of traffic with=A0 yesteryear'= s cpus.=0A> It's not an achievement. Being able to bridge=0A> > real traffi= c at 10gb/s with 2 cores is=0A>=0A> Or forward at layer 3.=0A>=0A> Or filte= r packets.=0A>=0A> Or IPSEC.=0A>=0A> Or...=0A______________________________= _________________ From owner-freebsd-net@FreeBSD.ORG Sun Aug 18 19:18:37 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 08972B9A for ; Sun, 18 Aug 2013 19:18:37 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-wg0-x236.google.com (mail-wg0-x236.google.com [IPv6:2a00:1450:400c:c00::236]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 8D3962739 for ; Sun, 18 Aug 2013 19:18:36 +0000 (UTC) Received: by mail-wg0-f54.google.com with SMTP id e12so2959771wgh.9 for ; Sun, 18 Aug 2013 12:18:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=eDhvPd4JNvPgrVF2aFlH7WoRmqJCI5w7gkx6nODeUy4=; b=W2zVqVlu/sArglET4jknTc1sRBvGAX+K5EnD9GA3ns580rcMJq/KX744umETGSSsO/ kGduWmEfG2P2qO8gq1petSERqK/Z1VPQWHiTfWAAO0h6PIie+3TxUyzoEpZ3FRWPHxqe GLepEuKHz3hFH3QI4xhZ/Gk9Uey8AKQ3uhF9flvYnxo1q+mz0mE7IhnbdE6xyfFlVVdk pQwzSUELkYqXLNBfkj5CpfEDCM8jun+KC2v2qyT7NAbjA84fDQk2kMORWbHNeVnJufZa IDUg3YLKOiZ3j7pZSb5gsKbUALjmpR7BhLU2SzBEBMGealoPwBpK4DemoIlLRLyAFJVS N+3g== MIME-Version: 1.0 X-Received: by 10.180.37.164 with SMTP id z4mr5470283wij.30.1376853514182; Sun, 18 Aug 2013 12:18:34 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.217.116.136 with HTTP; Sun, 18 Aug 2013 12:18:34 -0700 (PDT) In-Reply-To: <1376851152.3322.YahooMailNeo@web121606.mail.ne1.yahoo.com> References: <520A6D07.5080106@freebsd.org> <520AFBE8.1090109@freebsd.org> <520B24A0.4000706@freebsd.org> <520B3056.1000804@freebsd.org> <20130814102109.GA63246@onelab2.iet.unipi.it> <1376745244.6575.YahooMailNeo@web121606.mail.ne1.yahoo.com> <1376748170.66110.YahooMailNeo@web121601.mail.ne1.yahoo.com> <1376833738.94737.YahooMailNeo@web121605.mail.ne1.yahoo.com> <71EA3DFB-B410-432D-98E0-B6341556BE6D@netgate.com> <1376851152.3322.YahooMailNeo@web121606.mail.ne1.yahoo.com> Date: Sun, 18 Aug 2013 12:18:34 -0700 X-Google-Sender-Auth: -ac0bum_uBW0cJfLWgHjDga6TdQ Message-ID: Subject: Re: it's the output, not ack coalescing (Re: TSO and FreeBSD vs Linux) From: Adrian Chadd To: Barney Cordoba Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: "freebsd-net@freebsd.org" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 18 Aug 2013 19:18:37 -0000 On 18 August 2013 11:39, Barney Cordoba wrote: > Great. Never has the been a better explanation for the word Kludge than > netmap. > Nah. Netmap is a reimplementation of some reasonably well known ways of pushing bits. Luigi just pushed it up to eleven and demonstrated what current hardware is capable of. I have never bought the "We need eleventy cores just to push 10ge of real traffic!" before. Luigi did note down where the per-packet inefficiencies were. What we have to do now is sit down and for each of those, figure out what the root causes are and how to mitigate it. There's some architectural things that need tidying up (read: CPU pinning, queue handling, some locking hilarity) but if they're solved, we'll end up having dual core boxes push line rate packets for routing. So the gauntlet has been thrown. Let's fix this shit up. -adrian