From owner-freebsd-current@FreeBSD.ORG Fri Sep 5 21:19:59 2014 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4A3E5203; Fri, 5 Sep 2014 21:19:59 +0000 (UTC) Received: from mail-pa0-x229.google.com (mail-pa0-x229.google.com [IPv6:2607:f8b0:400e:c03::229]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 156051EBA; Fri, 5 Sep 2014 21:19:59 +0000 (UTC) Received: by mail-pa0-f41.google.com with SMTP id lf10so1021817pab.14 for ; Fri, 05 Sep 2014 14:19:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type; bh=rjaOasgCmEznfIHhnkzXm+3EVP16SeM7c2lurW0ryY4=; b=dq5u+3lXvb8XTAfWPSWzr+5uJe0F/RZneCmjpSozYWTKkM5YwUZ7+JjM16pcy5LkYN 7TRGymZjnDqG+8vDb+lNHjzNPKPAyuroADnoDc8ZmssttZQXPZVdeb67z/v7/yRuv9Pd OX0SkrC4Pu0uCQ6h5zR3EmOLQ7vMlTUidQDe5EtsEoqnNpZLYDXCh+n7NyOhKU3hEb2t ISI/eMcw4jXhYGB06k+wOOvSE1IVwh6Fy1pCVWIpH0kCgCIetfq2GsNWqBzdZIQsSIyR keDfDrfKZQzSIw9V3WiY4x3+Uc2SXjx2k/1gnQLuxW0/tvXo6eBJA7BQaNp90CNIQtp5 FHdA== X-Received: by 10.70.52.199 with SMTP id v7mr25947614pdo.49.1409951998448; Fri, 05 Sep 2014 14:19:58 -0700 (PDT) MIME-Version: 1.0 Received: by 10.70.55.162 with HTTP; Fri, 5 Sep 2014 14:19:18 -0700 (PDT) In-Reply-To: <540A0301.9040701@selasky.org> References: <540A0301.9040701@selasky.org> From: Eric Joyner Date: Fri, 5 Sep 2014 14:19:18 -0700 Message-ID: Subject: Re: [RFC] Patch to improve TSO limitation formula in general To: Hans Petter Selasky Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: "freebsd-net@freebsd.org" , FreeBSD Current , Scott Long , Jack F Vogel X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Sep 2014 21:19:59 -0000 There are some concerns if we use this with devices that ixl supports: - The maximum fragment size is 16KB-1, which isn't a power of 2. - You can't get the maximum TSO size for ixl devices by multiplying the maximum number of fragments by the maximum size. Instead the number of fragments is AFAIK unlimited, but a segment can only span 8 mbufs (including the [up to 3] mbufs containing the header), and the maximum TSO size is 256KB. And one question: - Is hdr_size_log2 supposed to be the length of the L2 header? We can fit 254 L2 bytes in our hardware during a TSO, so if that's the value, I guess that's fine, barring the it-not-being-a-power-of-2 issue. With all that said, the 8 mbuf limit per segment issue is a TSO limitation that we'd like to notify the stack about, so I wonder if that could be incorporated along with this. Right now, our driver checks to see if a segment in a TSO spans more than six mbufs and then m_defrag()'s the entire chain if one exists; it's less than optimal but necessary to prevent errors. - Eric --- - Eric Joyner On Fri, Sep 5, 2014 at 11:37 AM, Hans Petter Selasky wrote: > Hi, > > I've tested the attached patch with success and would like to have some > feedback from other FreeBSD network developers. The problem is that the > current TSO limitation only limits the number of bytes that can be > transferred in a TSO packet and not the number of mbuf's. > > The current solution is to have a quick and dirty custom m_dup() in the TX > path to re-allocate the mbuf chains into 4K ones to make it simple. All of > this hack can be avoided if the definition of the TSO limit can be changed > a bit, like shown here: > > > /* > + * Structure defining hardware TSO limits. > + */ > +struct if_tso_limit { > + u_int raw_value[0]; /* access all fields as one */ > + u_char frag_count; /* maximum number of fragments: 1..255 */ > + u_char frag_size_log2; /* maximum fragment size: 2 ** (12..16) */ > + u_char hdr_size_log2; /* maximum header size: 2 ** (2..8) */ > + u_char reserved; /* zero */ > +}; > > > First we need to know the maximum fragment count. Typical value is 32. > Second we need to know the maximum fragment size. Typical value is 4K. > Last we need to know of any headers that should be subtracted from the > maximum. Hence this code is running in the fast path, I would like to use > "u_char" for all fields and allow copy-only access as a "u_int" as an > optimization. This avoids cludges and messing with additional header files. > > I would like to push this patch after some more testing to -current and > then to 10-stable hopefully before the coming 10-release, because the > current solution is affecting performance of the Mellanox based network > adapters in an unfair way. For example by setting the current TSO limit to > 32KBytes which will be OK for all-2K fragments, we see a severe degradation > in performance. Even though the hardware is fully capable of transmitting > 16 4K mbufs. > > Comments and reviews are welcome! > > --HPS > > _______________________________________________ > freebsd-current@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" >