From owner-freebsd-stable@FreeBSD.ORG Mon Feb 28 20:16:37 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 32891106566B for ; Mon, 28 Feb 2011 20:16:37 +0000 (UTC) (envelope-from bartosz.stec@it4pro.pl) Received: from mainframe.kkip.pl (kkip.pl [87.105.164.78]) by mx1.freebsd.org (Postfix) with ESMTP id 8B5768FC08 for ; Mon, 28 Feb 2011 20:16:36 +0000 (UTC) Received: from static-78-8-144-74.ssp.dialog.net.pl ([78.8.144.74] helo=[192.168.0.2]) by mainframe.kkip.pl with esmtpsa (TLSv1:CAMELLIA256-SHA:256) (Exim 4.73 (FreeBSD)) (envelope-from ) id 1Pu9W5-000P5U-Mn; Mon, 28 Feb 2011 21:16:34 +0100 Message-ID: <4D6C0297.5020402@it4pro.pl> Date: Mon, 28 Feb 2011 21:16:23 +0100 From: Bartosz Stec Organization: IT4Pro User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; pl; rv:1.9.2.13) Gecko/20101207 Lightning/1.0b2 Thunderbird/3.1.7 MIME-Version: 1.0 To: Jeremy Chadwick References: <4D660909.6090202@my.gd> <20110224075517.GA18146@icarus.home.lan> In-Reply-To: <20110224075517.GA18146@icarus.home.lan> X-Authenticated-User: bartosz.stec@it4pro.pl X-Authenticator: plain X-Sender-Verify: SUCCEEDED (sender exists & accepts mail) X-Spam-Score: -8.1 X-Spam-Score-Int: -80 X-Exim-Version: 4.73 (build at 10-Jan-2011 16:29:01) X-Date: 2011-02-28 21:16:34 X-Connected-IP: 78.8.144.74:53901 X-Message-Linecount: 499 X-Body-Linecount: 485 X-Message-Size: 17375 X-Body-Size: 16687 X-Received-Count: 1 X-Recipient-Count: 2 X-Local-Recipient-Count: 2 X-Local-Recipient-Defer-Count: 0 X-Local-Recipient-Fail-Count: 0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: "freebsd-stable@freebsd.org" Subject: Re: ZFS - abysmal performance with samba since upgrade to 8.2-RELEASE X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Feb 2011 20:16:37 -0000 W dniu 2011-02-24 08:55, Jeremy Chadwick pisze: > (...snip...) > Samba > ======================= > Rebuild the port (ports/net/samba35) with AIO_SUPPORT enabled. To use > AIO you will need to load the aio.ko kernel module (kldload aio) first. > > Relevant smb.conf tunings: > > [global] > socket options = TCP_NODELAY SO_SNDBUF=131072 SO_RCVBUF=131072 > use sendfile = no > min receivefile size = 16384 > aio read size = 16384 > aio write size = 16384 > aio write behind = yes > > > > ZFS pools > ======================= > pool: backups > state: ONLINE > scrub: none requested > config: > > NAME STATE READ WRITE CKSUM > backups ONLINE 0 0 0 > ada2 ONLINE 0 0 0 > > errors: No known data errors > > pool: data > state: ONLINE > scrub: none requested > config: > > NAME STATE READ WRITE CKSUM > data ONLINE 0 0 0 > ada1 ONLINE 0 0 0 > > errors: No known data errors > > > > ZFS tunings > ======================= > Your tunings here are "wild" (meaning all over the place). Your use > of vfs.zfs.txg.synctime="1" is probably hurting you quite badly, in > addition to your choice to enable prefetching (every ZFS FreeBSD system > I've used has benefit tremendously from having prefetching disabled, > even on systems with 8GB RAM and more). You do not need to specify > vm.kmem_size_max, so please remove that. Keeping vm.kmem_size is fine. > Also get rid of your vdev tunings, I'm not sure why you have those. > > My relevant /boot/loader.conf tunings for 8.2-RELEASE (note to readers: > the version of FreeBSD you're running, and build date, matters greatly > here so do not just blindly apply these without thinking first): > > # We use Samba built with AIO support; we need this module! > aio_load="yes" > > # Increase vm.kmem_size to allow for ZFS ARC to utilise more memory. > vm.kmem_size="8192M" > vfs.zfs.arc_max="6144M" > > # Disable ZFS prefetching > # http://southbrain.com/south/2008/04/the-nightmare-comes-slowly-zfs.html > # Increases overall speed of ZFS, but when disk flushing/writes occur, > # system is less responsive (due to extreme disk I/O). > # NOTE: Systems with 8GB of RAM or more have prefetch enabled by > # default. > vfs.zfs.prefetch_disable="1" > > # Decrease ZFS txg timeout value from 30 (default) to 5 seconds. This > # should increase throughput and decrease the "bursty" stalls that > # happen during immense I/O with ZFS. > # http://lists.freebsd.org/pipermail/freebsd-fs/2009-December/007343.html > # http://lists.freebsd.org/pipermail/freebsd-fs/2009-December/007355.html > vfs.zfs.txg.timeout="5" > > > > sysctl tunings > ======================= > Please note that the below kern.maxvnodes tuning is based on my system > usage, and yours may vary, so you can remove or comment out this option > if you wish. The same goes for vfs.ufs.dirhash_maxmem. As for > vfs.zfs.txg.write_limit_override, I strongly suggest you keep this > commented out for starters; it effectively "rate limits" ZFS I/O, and > this smooths out overall performance (otherwise I was seeing what > appeared to be incredible network transfer speed, then the system would > churn hard for quite some time on physical I/O, then fast network speed, > physical I/O, etc... very "bursty", which I didn't want). > > # Increase send/receive buffer maximums from 256KB to 16MB. > # FreeBSD 7.x and later will auto-tune the size, but only up to the max. > net.inet.tcp.sendbuf_max=16777216 > net.inet.tcp.recvbuf_max=16777216 > > # Double send/receive TCP datagram memory allocation. This defines the > # amount of memory taken up by default *per socket*. > net.inet.tcp.sendspace=65536 > net.inet.tcp.recvspace=131072 > > # dirhash_maxmem defaults to 2097152 (2048KB). dirhash_mem has reached > # this limit a few times, so we should increase dirhash_maxmem to > # something like 16MB (16384*1024). > vfs.ufs.dirhash_maxmem=16777216 > > # > # ZFS tuning parameters > # NOTE: Be sure to see /boot/loader.conf for additional tunings > # > > # Increase number of vnodes; we've seen vfs.numvnodes reach 115,000 > # at times. Default max is a little over 200,000. Playing it safe... > kern.maxvnodes=250000 > > # Set TXG write limit to a lower threshold. This helps "level out" > # the throughput rate (see "zpool iostat"). A value of 256MB works well > # for systems with 4GB of RAM, while 1GB works well for us w/ 8GB on > # disks which have 64MB cache. > vfs.zfs.txg.write_limit_override=1073741824 > > > > Good luck. > Jeremy, you're just invaluable! :) In short - I applied tips suggested above (only difference was vfs.zfs.txg.write_limit_override set to 128MB, and sendfile, which I still have enabled) and it's first time _ever_ I see samba performing so fast on FreeBSD (on 100Mb link)! long story: I'm using old, crappy, low memory desktop PC as home router/test server/(very little) storage: FreeBSD 9.0-CURRENT #2 r219090: Mon Feb 28 03:06:13 CET 2011 CPU: mobile AMD Athlon(tm) XP 2200+ (1800.10-MHz 686-class CPU) real memory = 1610612736 (1536 MB) avail memory = 1562238976 (1489 MB) ad0: 39205MB at ata0-master UDMA133 ad1: 38166MB at ata0-slave UDMA100 ad2: 39205MB at ata1-master UDMA133 xl0: <3Com 3c905B-TX Fast Etherlink XL> It's ZFS only (just updated to v28) system in RAIDZ1 configuration, attached to cheap belkin 100Mb switch used for home network. From couple of months I experienced pathetic SMB transfer - from 20kB/s to 200kB/s. Especially when system was idle, because the most funny thing about that - transfer was much better when system was busy (csup or make world for instance). SMB throughput jumped to 2-4MB/s then (well, from time to time at least). I've been using following settings and tunings while I was experiencing this issue: smb.conf: [global] socket options = TCP_NODELAY SO_SNDBUF=65536 SO_RCVBUF=65536 use sendfile = yes min receivefile size = 16384 aio read size = 16384 aio write size = 16384 aio write behind = true loader.conf: vm.kmem_size="1536M" vm.kmem_size_max="1536M" vfs.zfs.arc_max="1024M" aio_load="YES" sysctl.conf: kern.ipc.maxsockbuf=2097152 net.inet.tcp.recvspace=262144 net.inet.tcp.recvspace=262144 net.inet.tcp.mssdflt=1452 net.inet.udp.recvspace=65535 net.inet.udp.maxdgram=65535 net.local.stream.recvspace=65535 net.local.stream.sendspace=65535 After applying tunables from Jeremy my configs looks like this: smb.conf: [global] socket options = TCP_NODELAY SO_SNDBUF=131072 SO_RCVBUF=131072 use sendfile = yes min receivefile size = 16384 aio read size = 16384 aio write size = 16384 aio write behind = yes loader.conf: vm.kmem_size="1536M" vm.kmem_size_max="1536M" vfs.zfs.arc_max="1024M" vfs.zfs.txg.timeout="5" aio_load="YES" sysctl.conf: net.inet.tcp.sendbuf_max=16777216 net.inet.tcp.recvbuf_max=16777216 net.inet.tcp.sendspace=65536 net.inet.tcp.recvspace=131072 vfs.ufs.dirhash_maxmem=16777216 kern.maxvnodes=250000 vfs.zfs.txg.write_limit_override=134217728 Test: copying 1GB file both sides. Results: stable 8MB/s both sides! Thank you very much! -- Bartosz Stec