From owner-freebsd-current@FreeBSD.ORG Sun Nov 4 12:14:08 2007 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 598A816A421 for ; Sun, 4 Nov 2007 12:14:08 +0000 (UTC) (envelope-from kris@FreeBSD.org) Received: from weak.local (pointyhat.freebsd.org [IPv6:2001:4f8:fff6::2b]) by mx1.freebsd.org (Postfix) with ESMTP id 96D9B13C4B8; Sun, 4 Nov 2007 12:14:07 +0000 (UTC) (envelope-from kris@FreeBSD.org) Message-ID: <472DB78E.5030103@FreeBSD.org> Date: Sun, 04 Nov 2007 13:14:06 +0100 From: Kris Kennaway User-Agent: Thunderbird 2.0.0.6 (Macintosh/20070728) MIME-Version: 1.0 To: freebsd-current@freebsd.org, Peter Schuller References: <200711021208.25913.Thomas.Sparrevohn@btinternet.com> <20071103164231.GB23714@outcold.yadt.co.uk> <200711040948.25732.peter.schuller@infidyne.com> <20071104110521.GA12145@outcold.yadt.co.uk> In-Reply-To: <20071104110521.GA12145@outcold.yadt.co.uk> Content-Type: text/plain; charset=ISO-8859-15; format=flowed Content-Transfer-Encoding: 7bit Cc: Subject: Re: ZFS slowness (not using cache?) X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 04 Nov 2007 12:14:08 -0000 David Taylor wrote: > On Sun, 04 Nov 2007, Peter Schuller wrote: > >>> For example, pkg_delete seems to be _extremely_ slow and ^T reports that >>> it is stuck waiting on zfs:(&zio->io_cv) for an unreasonable (IMO) amount >>> of time. >> FWIW, I have seen pkg_install (and possibly other pkg_* tools) being extremely >> slow seemingly as a result of the active set of files it touches exceededing >> the amount cached. In particular I had this problem after converting to ZFS, >> but prior to switching to amd64 and more RAM. >> >> It would sit and churn on disk I/O forever, entirely seek bound. Tracing the >> processes showed it traversing the package database over and over (presumably >> recursively following dependencies or some such). So the same files were >> touched any number of times. As a result, with too little cached, runtime >> exploded (it took hours and hours upgrading my desktop using *binary* >> pre-built packages because the larger packages with a lot of dependencies >> would take forever to install and delete). > > Hmm. That prompted me to have a look at the arcstats, and I'm now rather > confused. It seems to have plenty of cache free, but not actually using > it properly. > > I'm running i386 for now (on an amd64 motherboard), with 4GB of RAM > (~3.5Gb usable) and the following settings in loader.conf: > > geom_stripe_load="YES" > geom_label_load="YES" > zfs_load="YES" > snd_driver_load="YES" > nvidia_load="YES" > > hw.ata.atapi_dma="1" > kern.maxfiles="25000" > kern.ipc.shmmax=67108864 > kern.ipc.shmall=32768 > > kern.maxdsiz="900M" > vm.kmem_size=1450M > vfs.zfs.arc_max=500M > vfs.root.mountfrom="zfs:tank/i386_root" > > This results in (sysctl): > > vm.kmem_size_scale: 3 > vm.kmem_size_max: 335544320 # I have just noticed this is about 300MB, > # far lower than vm.kmem_size. Is that a problem? No but you can't set kmem to 1500MB without first increasing the kernel address space (1GB on i386) to some suitably larger value via KVA_PAGES. > vm.kmem_size_min: 0 > vm.kmem_size: 1520435200 > > vfs.zfs.arc_min: 47513600 > vfs.zfs.arc_max: 524288000 > kstat.zfs.misc.arcstats.c_min: 47513600 > kstat.zfs.misc.arcstats.c_max: 524288000 > kstat.zfs.misc.arcstats.size: 86967808 > > Regardless of what I do, I can't seem to get arcstats.size to exceed 100MB. > It initially goes up with disk usage, but then starts to drop again, causing > it to hold steady around 70-90MB. Something seems to be agressively pushing > data out of the cache, despite it being <20% full, which seems rather fishy. > Does anyone know what's going on? Try increasing kern.maxvnodes. Kris >