Date: Thu, 26 Apr 2018 08:43:01 +0200 From: Ben RUBSON <ben.rubson@gmail.com> To: Rick Macklem <rmacklem@uoguelph.ca> Cc: Ryan Stone <rysto32@gmail.com>, Konstantin Belousov <kostikbel@gmail.com>, FreeBSD Current <current@freebsd.org>, FreeBSD Net <net@freebsd.org>, Meny Yossefi <menyy@mellanox.com> Subject: Re: mlx5(4) jumbo receive Message-ID: <E0387477-F62E-4D54-B086-7C1F0210DE40@gmail.com> In-Reply-To: <YQBPR0101MB104231911340A77D6DDC360EDD8F0@YQBPR0101MB1042.CANPRD01.PROD.OUTLOOK.COM> References: <20180424085553.GA6887@kib.kiev.ua> <CAFMmRNwKXCxGbz7zmOsiCMWbY3%2BrBzGwJJo_HHkAbbhEz2UfLQ@mail.gmail.com> <YQBPR0101MB104231911340A77D6DDC360EDD8F0@YQBPR0101MB1042.CANPRD01.PROD.OUTLOOK.COM>
next in thread | previous in thread | raw e-mail | index | archive | help
On 26 Apr 2018, Rick Macklem wrote: > Ryan Stone wrote: >> On Tue, Apr 24, 2018 at 4:55 AM, Konstantin Belousov >> >><kostikbel@gmail.com>wrote: >>> +#ifndef MLX5E_MAX_RX_BYTES >>> +#define MLX5E_MAX_RX_BYTES MCLBYTES >>> +#endif >> >> Why do you use a 2KB buffer rather than a PAGE_SIZE'd buffer? >> MJUMPAGESIZE should offer significantly better performance for jumbo >> frames without increasing the risk of memory fragmentation. > Actually, when I was playing with using jumbo mbuf clusters for NFS, I > was able > to get it to fragment to the point where allocations failed when mixing > 2K and > 4K mbuf clusters. > Admittedly I was using a 256Mbyte i386 and it wasn't easily reproduced, but > it was possible. > --> Using a mix of 2K and 4K mbuf clusters can result in fragmentation, > although > I suspect that it isn't nearly as serious as what can happen when using 9K > mbuf clusters. I used to face the fragmentation issue easily with MTU set to 9000 (x86_64 / 64GB / Connect-X/3). I then decreased the MTU until 9K mbufs are not more used, to 4072 bytes then. The other used interface of this 2-ports card is set with a 1500 MTU. Until now (about a year later), no more issue. # vmstat -z | grep mbuf ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP mbuf_packet: 256, 26080155, 16400, 9652, 999757417, 0, 0 mbuf: 256, 26080155, 16419, 11349, 85322301444, 0, 0 mbuf_cluster: 2048, 4075022, 26052, 550, 1059817, 0, 0 mbuf_jumbo_page: 4096, 2037511, 16400, 9522, 45863599682, 0, 0 mbuf_jumbo_9k: 9216, 603707, 0, 0, 0, 0, 0 mbuf_jumbo_16k: 16384, 339585, 0, 0, 0, 0, 0 Here's my experience :) Ben
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?E0387477-F62E-4D54-B086-7C1F0210DE40>