From owner-freebsd-net@FreeBSD.ORG Fri Sep 5 22:24:58 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4F7DCC4; Fri, 5 Sep 2014 22:24:58 +0000 (UTC) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id E6263182C; Fri, 5 Sep 2014 22:24:57 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AqwEAKg3ClSDaFve/2dsb2JhbABag2BXBIJ4xhAKhnlTAYEbd4QDAQEBAwEBAQEgKyALBRYYAgINGQIpAQkmBggCBQQBGgIEiBkIDalylToBF4EsjUoGAQEbNAeCeYFTBZVug3uEX5NKgWcegXghLweBAAgXIoEHAQEB X-IronPort-AV: E=Sophos;i="5.04,476,1406606400"; d="scan'208";a="151957905" Received: from muskoka.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.222]) by esa-jnhn.mail.uoguelph.ca with ESMTP; 05 Sep 2014 18:24:56 -0400 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id D6526B4061; Fri, 5 Sep 2014 18:24:56 -0400 (EDT) Date: Fri, 5 Sep 2014 18:24:56 -0400 (EDT) From: Rick Macklem To: Hans Petter Selasky Message-ID: <575724560.33015896.1409955896874.JavaMail.root@uoguelph.ca> In-Reply-To: <540A0301.9040701@selasky.org> Subject: Re: [RFC] Patch to improve TSO limitation formula in general MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.91.203] X-Mailer: Zimbra 7.2.6_GA_2926 (ZimbraWebClient - FF3.0 (Win)/7.2.6_GA_2926) Cc: freebsd-net@freebsd.org, FreeBSD Current , Scott Long X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Sep 2014 22:24:58 -0000 Hans Petter Selasky wrote: > Hi, > > I've tested the attached patch with success and would like to have > some > feedback from other FreeBSD network developers. The problem is that > the > current TSO limitation only limits the number of bytes that can be > transferred in a TSO packet and not the number of mbuf's. > > The current solution is to have a quick and dirty custom m_dup() in > the > TX path to re-allocate the mbuf chains into 4K ones to make it > simple. > All of this hack can be avoided if the definition of the TSO limit > can > be changed a bit, like shown here: > > > /* > + * Structure defining hardware TSO limits. > + */ > +struct if_tso_limit { > + u_int raw_value[0]; /* access all fields as one */ > + u_char frag_count; /* maximum number of fragments: > 1..255 */ > + u_char frag_size_log2; /* maximum fragment size: 2 ** > (12..16) */ > + u_char hdr_size_log2; /* maximum header size: 2 ** (2..8) > */ > + u_char reserved; /* zero */ > +}; > > > First we need to know the maximum fragment count. Typical value is > 32. > Second we need to know the maximum fragment size. Typical value is > 4K. > Last we need to know of any headers that should be subtracted from > the > maximum. Hence this code is running in the fast path, I would like to > use "u_char" for all fields and allow copy-only access as a "u_int" > as > an optimization. This avoids cludges and messing with additional > header > files. > > I would like to push this patch after some more testing to -current > and > then to 10-stable hopefully before the coming 10-release, because the > current solution is affecting performance of the Mellanox based > network > adapters in an unfair way. For example by setting the current TSO > limit > to 32KBytes which will be OK for all-2K fragments, we see a severe > degradation in performance. Even though the hardware is fully capable > of > transmitting 16 4K mbufs. > Ok, I didn't see this until now, but I will take a look at the patch. My main comment is that I tried using a mix of 2K and 4K mbuf clusters in NFS and was able (with some effort) get the UMA allocator all messed up, where it would basically be stuck because it couldn't allocate boundary tags. As such, until this issue w.r.t. UMA is rssolved, mixing MCLBYTES and MPAGESIZE clusters is very dangerous imho. (alc@ did send me a simple patch related to this UMA problem, but I haven't been able to test it yet.) rick ps: For the M_WAITOK case, the allocator problem shows up as threads sleeping on "btallo" which happens in vmem_bt_alloc() in kern/subr_vmem.c. It may never happen on 64bit arches, but it can definitely happen on i386. > Comments and reviews are welcome! > > --HPS > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to > "freebsd-net-unsubscribe@freebsd.org"