From owner-freebsd-current Thu Jul 4 21:21:23 2002 Delivered-To: freebsd-current@freebsd.org Received: from mx1.FreeBSD.org (mx1.FreeBSD.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E594137B400; Thu, 4 Jul 2002 21:21:14 -0700 (PDT) Received: from angelica.unixdaemons.com (angelica.unixdaemons.com [209.148.64.135]) by mx1.FreeBSD.org (Postfix) with ESMTP id 4162A43E42; Thu, 4 Jul 2002 21:21:14 -0700 (PDT) (envelope-from bmilekic@angelica.unixdaemons.com) Received: from angelica.unixdaemons.com (bmilekic@localhost.unixdaemons.com [127.0.0.1]) by angelica.unixdaemons.com (8.12.5/8.12.1) with ESMTP id g654KvdT006762; Fri, 5 Jul 2002 00:20:57 -0400 (EDT) X-Authentication-Warning: angelica.unixdaemons.com: Host bmilekic@localhost.unixdaemons.com [127.0.0.1] claimed to be angelica.unixdaemons.com Received: (from bmilekic@localhost) by angelica.unixdaemons.com (8.12.5/8.12.1/Submit) id g654KubO006761; Fri, 5 Jul 2002 00:20:56 -0400 (EDT) (envelope-from bmilekic) Date: Fri, 5 Jul 2002 00:20:56 -0400 From: Bosko Milekic To: Andrew Gallatin Cc: "Kenneth D. Merry" , current@FreeBSD.ORG, net@FreeBSD.ORG Subject: Re: virtually contig jumbo mbufs (was Re: new zero copy sockets snapshot) Message-ID: <20020705002056.A5365@unixdaemons.com> References: <20020619090046.A2063@panzer.kdm.org> <20020619120641.A18434@unixdaemons.com> <15633.17238.109126.952673@grasshopper.cs.duke.edu> <20020619233721.A30669@unixdaemons.com> <15633.62357.79381.405511@grasshopper.cs.duke.edu> <20020620114511.A22413@unixdaemons.com> <15634.534.696063.241224@grasshopper.cs.duke.edu> <20020620134723.A22954@unixdaemons.com> <15652.46870.463359.853754@grasshopper.cs.duke.edu> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5.1i In-Reply-To: <15652.46870.463359.853754@grasshopper.cs.duke.edu>; from gallatin@cs.duke.edu on Thu, Jul 04, 2002 at 04:59:02PM -0400 Sender: owner-freebsd-current@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On Thu, Jul 04, 2002 at 04:59:02PM -0400, Andrew Gallatin wrote: > > I believe that the Intel chips do "virtual page caching" and that the > > logic that does the virtual -> physical address translation sits between > > the L2 cache and RAM. If that is indeed the case, then your idea of > > testing with virtually contiguous pages is a good one. > > Unfortunately, I don't know if the PIII is doing speculative > > cache-loads, but it could very well be the case. If it is and if in > > fact the chip does caching based on virtual addresses, then providing it > > with virtually contiguous address space may yield better results. If > > you try this, please let me know. I'm extremely interested in seeing > > the results! > > contigmalloc'ed private jumbo mbufs (same as bge, if_ti, etc): > > % iperf -c ugly-my -l 32k -fm > ------------------------------------------------------------ > Client connecting to ugly-my, TCP port 5001 > TCP window size: 0.2 MByte (default) > ------------------------------------------------------------ > [ 3] local 192.168.1.3 port 1031 connected with 192.168.1.4 port 5001 > [ ID] Interval Transfer Bandwidth > [ 3] 0.0-10.0 sec 2137 MBytes 1792 Mbits/sec > > > > malloc'ed, physically discontigous private jumbo mbufs: > > % iperf -c ugly-my -l 32k -fm > ------------------------------------------------------------ > Client connecting to ugly-my, TCP port 5001 > TCP window size: 0.2 MByte (default) > ------------------------------------------------------------ > [ 3] local 192.168.1.3 port 1029 connected with 192.168.1.4 port 5001 > [ ID] Interval Transfer Bandwidth > [ 3] 0.0-10.0 sec 2131 MBytes 1788 Mbits/sec > > > So I'd be willing to believe that the 4Mb/sec loss was due to > the extra overhead of setting up 2 additional DMAs. > > > So it looks like this idea would work. Yes, it certainly confirms the virtual-based caching assumptions. I would like to provide virtually contiguous large buffers and believe I can do that via mb_alloc... however, they would be several wired-down pages. Would this be in line with the requirements that these buffers would have, in your mind? (wired-down means that your buffers will come out exactly as they would out of malloc(), so if you were using malloc() already, I'm assuming that wired-down is OK). I think I can allocate the jumbo buffers via mb_alloc from the same map as I allocate clusters from - the clust_map - and keep them in buckets/slabs in per-CPU caches, like I do for mbufs and regular clusters right now. Luigi is in the process of doing some optimisation work around mb_alloc and I'll probably be doing the SMP-specific stuff after he's done so once that's taken care of, we can take a stab at this if you think it's worth it. > Drew Regards, -- Bosko Milekic bmilekic@unixdaemons.com bmilekic@FreeBSD.org To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-current" in the body of the message