From owner-freebsd-fs@FreeBSD.ORG Sun Mar 6 09:04:58 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2ADF6106566C for ; Sun, 6 Mar 2011 09:04:58 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta06.emeryville.ca.mail.comcast.net (qmta06.emeryville.ca.mail.comcast.net [76.96.30.56]) by mx1.freebsd.org (Postfix) with ESMTP id 0F1338FC0C for ; Sun, 6 Mar 2011 09:04:57 +0000 (UTC) Received: from omta02.emeryville.ca.mail.comcast.net ([76.96.30.19]) by qmta06.emeryville.ca.mail.comcast.net with comcast id Fl3j1g0020QkzPwA6l4xq8; Sun, 06 Mar 2011 09:04:57 +0000 Received: from koitsu.dyndns.org ([98.248.33.18]) by omta02.emeryville.ca.mail.comcast.net with comcast id Fl4w1g0010PUQVN8Nl4wbh; Sun, 06 Mar 2011 09:04:56 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id DEE459B422; Sun, 6 Mar 2011 01:04:55 -0800 (PST) Date: Sun, 6 Mar 2011 01:04:55 -0800 From: Jeremy Chadwick To: Joshua Boyd Message-ID: <20110306090455.GA87055@icarus.home.lan> References: <1299232133.18671.3.camel@pc286.embl.fr> <20110304100517.GA23249@icarus.home.lan> <20110304105608.GA23887@icarus.home.lan> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-fs@freebsd.org Subject: Re: kmem_map too small with ZFS and 8.2-RELEASE X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 Mar 2011 09:04:58 -0000 On Sun, Mar 06, 2011 at 02:59:43AM -0500, Joshua Boyd wrote: > On Fri, Mar 4, 2011 at 5:56 AM, Jeremy Chadwick > wrote: > > If you get better performance -- really, truly, honestly -- with > > prefetch enabled on your system, then I strongly recommend you keep it > > enabled. ?However, for what it's worth (probably not much), this is the > > first I've ever heard of a FreeBSD system performing better with > > prefetch enabled. > > I just recently turned it on after having it turned off for a long > time ... my speeds went from ~300MB/s to 600+MB/s in bonnie++. This is > a dual core AM3 system with 8GB of ram, and 15 disks in a striped > raidz configuration (3 sets striped). Here are some numbers for you. This is from a 8.2-STABLE (RELENG_8) system built Thu Feb 24 22:06:45 PST 2011, type amd64. /boot/loader.conf ====================== ahci_load="yes" vm.kmem_size="8192M" vfs.zfs.arc_max="6144M" vfs.zfs.prefetch_disable="1" vfs.zfs.txg.timeout="5" /etc/sysctl.conf ====================== kern.maxvnodes=250000 vfs.zfs.txg.write_limit_override=1073741824 ZFS details ====================== # zpool list NAME SIZE USED AVAIL CAP HEALTH ALTROOT data 2.72T 780G 1.96T 28% ONLINE - # zfs list NAME USED AVAIL REFER MOUNTPOINT data 519G 1.28T 28.0K none data/backups 519G 1.28T 519G /backups data/home 7.19M 1.28T 7.19M /home # zpool status data pool: data state: ONLINE scrub: none requested config: NAME STATE READ WRITE CKSUM data ONLINE 0 0 0 raidz1 ONLINE 0 0 0 ada1 ONLINE 0 0 0 ada2 ONLINE 0 0 0 ada3 ONLINE 0 0 0 Pool and filesystem attributes are all defaults (ZFS v15), except for mountpoint. Disk details ====================== ada1: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C) ada2 at ahcich2 bus 0 scbus2 target 0 lun 0 ada2: ATA-8 SATA 2.x device ada2: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada2: Command Queueing enabled ada2: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C) ada3 at ahcich3 bus 0 scbus3 target 0 lun 0 ada3: ATA-8 SATA 2.x device ada3: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada3: Command Queueing enabled ada3: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C) Benchmark commands ====================== shutdown -r now cd /backups && bonnie++ -s 16g -u root Benchmark results #1 ====================== Version 1.96 ------Sequential Output------ --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP XXXXX 16G 92 99 127026 42 62634 18 244 99 177522 22 207.0 5 Latency 144ms 1277ms 1616ms 47845us 111ms 639ms Version 1.96 ------Sequential Create------ --------Random Create-------- XXXXX -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 18945 96 +++++ +++ 17552 98 18975 94 +++++ +++ 18286 98 Latency 18018us 85us 131us 17862us 38us 80us Benchmark results #2 ====================== Results with only one change made to /boot/loader.conf: vfs.zfs.prefetch_disable="0" Version 1.96 ------Sequential Output------ --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP XXXXX 16G 95 99 126656 40 74678 24 229 97 187918 26 219.8 5 Latency 159ms 1813ms 2506ms 330ms 308ms 657ms Version 1.96 ------Sequential Create------ --------Random Create-------- XXXXX -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 20140 96 +++++ +++ 16814 96 10431 54 +++++ +++ 14846 95 Latency 21010us 158us 1242us 146ms 721us 11662us -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP 4BD6C0CB |