From owner-freebsd-net@FreeBSD.ORG Thu Nov 8 04:53:21 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id F3274E0E; Thu, 8 Nov 2012 04:53:20 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-pb0-f54.google.com (mail-pb0-f54.google.com [209.85.160.54]) by mx1.freebsd.org (Postfix) with ESMTP id B86F78FC08; Thu, 8 Nov 2012 04:53:20 +0000 (UTC) Received: by mail-pb0-f54.google.com with SMTP id wz17so99809pbc.13 for ; Wed, 07 Nov 2012 20:53:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=f/DGVy/ZIitPcRqTGq2rF/a4iXJspF2SS3isMtqv5+U=; b=wsU7zKMVW85CdYHfxbiC6aGCbnJ0j4puH7LJ9wm1SVRbufrTEStKj1+AutVtFKDDWW sR60AB76gCgOv+NKSiwgGON3wwUJRD9cn7GTuhFQft43ZX/v53AcX/sB+/sPQk2pyzNq ficz2SV4TbM47EBrB/snEmWn/fKA2hV0JhRNrrBtUvWRj6M3SS0kPrhgoE2gYMOkI47E oOFK7DYsD+uQ+m4DvBfy1+JFTnPgrrKa2BY25ikTRZ4vMXESu/4uE58FrCfgITxb0/vV pl7TZogs0E6zUa7pan6OAVXJGQEouBHfWlY90TZrvFd/715+vCf9G3FOKScGd6zbLSQ6 Z5XQ== MIME-Version: 1.0 Received: by 10.68.193.228 with SMTP id hr4mr13983003pbc.95.1352350394785; Wed, 07 Nov 2012 20:53:14 -0800 (PST) Sender: adrian.chadd@gmail.com Received: by 10.68.124.130 with HTTP; Wed, 7 Nov 2012 20:53:14 -0800 (PST) In-Reply-To: <20121108023858.GA3127@michelle.cdnetworks.com> References: <201211080206.qA826RiN054539@svn.freebsd.org> <20121108023858.GA3127@michelle.cdnetworks.com> Date: Wed, 7 Nov 2012 20:53:14 -0800 X-Google-Sender-Auth: QPBcv0wSdhb1ckrDWcts1xvUv20 Message-ID: Subject: Re: svn commit: r242739 - stable/9/sys/dev/ti From: Adrian Chadd To: pyunyh@gmail.com Content-Type: text/plain; charset=ISO-8859-1 Cc: FreeBSD Net , Pyun YongHyeon X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Nov 2012 04:53:21 -0000 On 7 November 2012 18:38, YongHyeon PYUN wrote: > On Wed, Nov 07, 2012 at 06:15:30PM -0800, Adrian Chadd wrote: >> So I am curious - did this give a real benefit? > > In 3.x/4.x days it surely have had helped a lot, I guess mainly > because the CPU was not fast enough to saturate the link with > software checksum(i.e. NFS over UDP). Right. > Generally I prefer correctness to performance and it seems there > is no easy way to get full advantage of TCP/UDP checksum offloading > of controller on fragmented IP packets on FreeBSD 8+. So I disabled > it to reduce the chance of generating corrupted packets. Right. The problem here is that if_transmit pushes the locking to the driver, but there's no way to guarantee that the frames aren't interleaved with other parallel invocations of if_transmit(). Since SMP in 3.x and 4.x didn't really support multiple CPUs in kernel space, this wouldn't have been a problem. Ie, the network layer didn't explicitly define the behaviour with concurrent kernel senders, because that concept didn't exist yet. >> If so, may I suggest we perhaps accelerate discussing if_transmit() of >> multiple frames per call? > > Hmm, actually I'm still not a fan of if_transmit() at this moment. > Honestly I don't have good queuing code in driver to handle queue > full condition. Interactions with altq(9) is also one of my > concern as well as packet reordering issue of drbr(9) interface. > Well, can you provide me with some embedded hardware that has this particular NIC and multiple CPUs? I'd like to tinker with this but the SMP systems I have contain em NICs and I don't really feel up to hacking up this idea on the em(4) driver. :-) I have a vague idea of making if_transmit() as an interface be able to send multiple frames at once - so the TCP/UDP code could generate all the fragments and fire them off at once to the NIC. This should fix your issue and begin allowing devices to take advantage of multiple-dispatch rather than one if_transmit() call at a time. We could create a device or interface flag that indicates whether the driver can handle multiple mbufs chained via m_nextpkt through if_transmit(), and then teach one or two drivers that particular logic. What do you think? Adrian