From owner-freebsd-questions@FreeBSD.ORG Thu Apr 14 15:23:53 2011 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C1BA91065674 for ; Thu, 14 Apr 2011 15:23:53 +0000 (UTC) (envelope-from ml@my.gd) Received: from mail-ww0-f42.google.com (mail-ww0-f42.google.com [74.125.82.42]) by mx1.freebsd.org (Postfix) with ESMTP id 607D68FC12 for ; Thu, 14 Apr 2011 15:23:53 +0000 (UTC) Received: by wwk4 with SMTP id 4so6134523wwk.1 for ; Thu, 14 Apr 2011 08:23:52 -0700 (PDT) Received: by 10.227.131.195 with SMTP id y3mr993498wbs.38.1302794413549; Thu, 14 Apr 2011 08:20:13 -0700 (PDT) Received: from dfleuriot.local ([83.167.62.196]) by mx.google.com with ESMTPS id w25sm1063420wbd.5.2011.04.14.08.20.11 (version=SSLv3 cipher=OTHER); Thu, 14 Apr 2011 08:20:12 -0700 (PDT) Message-ID: <4DA710AA.2030501@my.gd> Date: Thu, 14 Apr 2011 17:20:10 +0200 From: Damien Fleuriot User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.15) Gecko/20110303 Thunderbird/3.1.9 MIME-Version: 1.0 To: freebsd-questions@freebsd.org References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Subject: Re: ZFS performance strangeness X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 14 Apr 2011 15:23:53 -0000 On 4/12/11 1:33 PM, Lars Wilke wrote: > Hi, > > There are quite a few threads about ZFS and performance difficulties, > but i did not find anything that really helped :) > Therefor any advice would be highly appreciated. > I started to use ZFS with 8.1R, only tuning i did was setting > > vm.kmem_size_scale="1" > vfs.zfs.arc_max="40000M" > > The machines are supermicro boards with 48 GB ECC RAM and 15k RPM SAS > drives. Local read/write performance was and is great. > But exporting via NFS was a mixed bag in 8.1R. > Generally r/w speed over NFS was ok, but large reads or writes took > ages. Most of the reads and writes were small, so i did not bother. > > Now i upgraded one machine to 8.2R and i get very good write performance > over NFS but read performance drops to a ridiciously low value, around > 1-2 MB/s. While writes are around 100MB/s. The network is a dedicated > 1GB Ethernet. The zpool uses RAIDZ1 over 7 drives, one vdev. > The filesystem has compression enabled. Turning it off made no > difference AFAICT > > Now i tried a few of the suggested tunables and my last try was this > > vfs.zfs.txg.timeout="5" > vfs.zfs.prefetch_disable="1" > vfs.zfs.txg.synctime=2 > fs.zfs.vdev.min_pending="1" > fs.zfs.vdev.max_pending="1" > > still no luck. Writting is fast, reading is not. Even with enabled > prefetching. The only thing i noticed is, that reading for example 10MB > is fast (on a freshly mounted fs) but when reading larger amounts, i.e. > couple hundred MBs, the performance drops and zpool iostat or iostat -x > show that there is not much activity on the zpool/hdds. > > It seems as if ZFS does not care that someone wants to read data, also idle > time of the reading process happily ticks up and gets higher and higher!? > When trying to access the file during this time, the process blocks and > sometimes is difficult to kill, i.e. ls -la on the file. > > I read and write with dd and before read tests i umount and mount the > NFS share again. > > dd if=/dev/zero of=/mnt/bla size=1M count=X > dd if=/mnt/bla of=/dev/null size=1M count=Y > > mount is done with this options from two centos 5 boxes: > rw,noatime,tcp,bg,intr,hard,nfsvers=3,noacl,nocto > > thanks > --lars > > _______________________________________________ > freebsd-questions@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-questions > To unsubscribe, send any mail to "freebsd-questions-unsubscribe@freebsd.org" I refer you to this post by Jeremy Chadwick with tuning values *AND* their actual explanation. http://lists.freebsd.org/pipermail/freebsd-stable/2011-February/061642.html