From owner-freebsd-net@FreeBSD.ORG Thu Jan 23 06:49:30 2014 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 898CE6F4; Thu, 23 Jan 2014 06:49:30 +0000 (UTC) Received: from mail-pa0-x234.google.com (mail-pa0-x234.google.com [IPv6:2607:f8b0:400e:c03::234]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 56D331770; Thu, 23 Jan 2014 06:49:30 +0000 (UTC) Received: by mail-pa0-f52.google.com with SMTP id bj1so1479340pad.11 for ; Wed, 22 Jan 2014 22:49:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:date:from:to:cc:subject:message-id:mail-followup-to :references:mime-version:content-type:content-disposition :in-reply-to:user-agent; bh=8nu8C46jFUizQcoaDjLJhLMlEUal0qiexfe+nx02CSg=; b=vOh+r/KiVMkiGp8MHmVkpahCaviq9KUH/QiKjmJdqjWA+GcZih7TIK4v54ZCARtloO g/nI6zGxGb+omp1i2zmv39E3A30Jcz/Uc0ZGOJ7ma+iNYMytTqFgrEadV5Y2MyoQybVC 5BnpPddEThU6iU22FfY4HfZCTwCSqb+T/M1VI0lQBXf5HREWvkBSgV8xKgkmwaDRXcdJ uz842eEccHeRv9SjbGNEzmFWbdJchCpz4VPqSHjFeuU+ZFqqSIqTkr0AnemvJT6oC3+2 BbUWvgTsZOFlO7gDJD+nd8PyN2N6raD5C0ckXNk255PQJaHSxArrykhoRZMlfaKxWa0Z CFhg== X-Received: by 10.66.139.8 with SMTP id qu8mr6299850pab.157.1390459769370; Wed, 22 Jan 2014 22:49:29 -0800 (PST) Received: from ox (c-24-6-44-228.hsd1.ca.comcast.net. [24.6.44.228]) by mx.google.com with ESMTPSA id tu3sm57821624pab.1.2014.01.22.22.49.28 for (version=TLSv1.2 cipher=RC4-SHA bits=128/128); Wed, 22 Jan 2014 22:49:28 -0800 (PST) Sender: Navdeep Parhar Date: Wed, 22 Jan 2014 22:49:23 -0800 From: Navdeep Parhar To: Garrett Wollman Subject: Re: Use of contiguous physical memory in cxgbe driver Message-ID: <20140123064923.GA6501@ox> Mail-Followup-To: Garrett Wollman , net@freebsd.org References: <21216.22944.314697.179039@hergotha.csail.mit.edu> <52E06104.70600@FreeBSD.org> <21216.36928.132606.318491@hergotha.csail.mit.edu> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <21216.36928.132606.318491@hergotha.csail.mit.edu> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Jan 2014 06:49:30 -0000 On Wed, Jan 22, 2014 at 10:45:04PM -0500, Garrett Wollman wrote: > < said: > > > On 01/22/14 15:52, Garrett Wollman wrote: > >> At this point everyone is well aware that requiring contiguous > >> physical page when the hardware can do scatter-gather is a very bad > >> idea. > > > I wouldn't put it this way. Using buffers with size > PAGE_SIZE has its > > advantages. > > These advantages do not come close to balancing out the disadvantage > of "your server eventually falls off the network due to physmem > fragmentation, better hope you can reset it remotely because driving > in to work at 3 AM sucks." This seems to imply that the only alternate to a successful allocation of the preferred size (which may be > PAGE_SIZE) is to not allocate any rx buffer at all and thus fall off the network eventually. This is a false choice. The driver can always fall back to allocating PAGE_SIZE sized buffers iff larger allocations fail. I'm considering implementing something along these lines in cxgbe(4), with exponential backoffs to avoid repeated allocation attempts from jumbo zones that are depleted. Regards, Navdeep > If any free pages are available at all, the allocation of a 4k jumbo > mbuf will succeed. A 9k jumbo mbuf requires three physically > contiguous pages, and it's very, very easy for physical memory to get > fragmented to the point where that is impossible. > > -GAWollman