From owner-freebsd-questions@FreeBSD.ORG Tue Apr 12 12:15:08 2011 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3DB3A106564A for ; Tue, 12 Apr 2011 12:15:08 +0000 (UTC) (envelope-from freebsd-questions@m.gmane.org) Received: from lo.gmane.org (lo.gmane.org [80.91.229.12]) by mx1.freebsd.org (Postfix) with ESMTP id BF0FD8FC08 for ; Tue, 12 Apr 2011 12:15:07 +0000 (UTC) Received: from list by lo.gmane.org with local (Exim 4.69) (envelope-from ) id 1Q9cUr-0007cz-6M for freebsd-questions@freebsd.org; Tue, 12 Apr 2011 14:15:05 +0200 Received: from 85.183.14.177 ([85.183.14.177]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Tue, 12 Apr 2011 14:15:05 +0200 Received: from lw by 85.183.14.177 with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Tue, 12 Apr 2011 14:15:05 +0200 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-questions@freebsd.org From: Lars Wilke Date: Tue, 12 Apr 2011 13:33:02 +0200 Lines: 53 Message-ID: X-Complaints-To: usenet@dough.gmane.org X-Gmane-NNTP-Posting-Host: 85.183.14.177 User-Agent: slrn/pre1.0.0-18 (Linux) Subject: ZFS performance strangeness X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Apr 2011 12:15:08 -0000 Hi, There are quite a few threads about ZFS and performance difficulties, but i did not find anything that really helped :) Therefor any advice would be highly appreciated. I started to use ZFS with 8.1R, only tuning i did was setting vm.kmem_size_scale="1" vfs.zfs.arc_max="40000M" The machines are supermicro boards with 48 GB ECC RAM and 15k RPM SAS drives. Local read/write performance was and is great. But exporting via NFS was a mixed bag in 8.1R. Generally r/w speed over NFS was ok, but large reads or writes took ages. Most of the reads and writes were small, so i did not bother. Now i upgraded one machine to 8.2R and i get very good write performance over NFS but read performance drops to a ridiciously low value, around 1-2 MB/s. While writes are around 100MB/s. The network is a dedicated 1GB Ethernet. The zpool uses RAIDZ1 over 7 drives, one vdev. The filesystem has compression enabled. Turning it off made no difference AFAICT Now i tried a few of the suggested tunables and my last try was this vfs.zfs.txg.timeout="5" vfs.zfs.prefetch_disable="1" vfs.zfs.txg.synctime=2 fs.zfs.vdev.min_pending="1" fs.zfs.vdev.max_pending="1" still no luck. Writting is fast, reading is not. Even with enabled prefetching. The only thing i noticed is, that reading for example 10MB is fast (on a freshly mounted fs) but when reading larger amounts, i.e. couple hundred MBs, the performance drops and zpool iostat or iostat -x show that there is not much activity on the zpool/hdds. It seems as if ZFS does not care that someone wants to read data, also idle time of the reading process happily ticks up and gets higher and higher!? When trying to access the file during this time, the process blocks and sometimes is difficult to kill, i.e. ls -la on the file. I read and write with dd and before read tests i umount and mount the NFS share again. dd if=/dev/zero of=/mnt/bla size=1M count=X dd if=/mnt/bla of=/dev/null size=1M count=Y mount is done with this options from two centos 5 boxes: rw,noatime,tcp,bg,intr,hard,nfsvers=3,noacl,nocto thanks --lars