From owner-freebsd-questions@freebsd.org Thu Jan 21 21:35:33 2016 Return-Path: Delivered-To: freebsd-questions@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 236CCA8C1A2; Thu, 21 Jan 2016 21:35:33 +0000 (UTC) (envelope-from dweimer@dweimer.net) Received: from webmail.dweimer.net (24-240-198-187.static.stls.mo.charter.com [24.240.198.187]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 035DA116A; Thu, 21 Jan 2016 21:35:31 +0000 (UTC) (envelope-from dweimer@dweimer.net) Received: from webmail.dweimer.local (localhost [192.168.5.2]) by webmail.dweimer.net (8.15.2/8.15.2) with ESMTPS id u0LLZOWj009926 (version=TLSv1.2 cipher=DHE-RSA-CHACHA20-POLY1305 bits=256 verify=NO); Thu, 21 Jan 2016 15:35:24 -0600 (CST) (envelope-from dweimer@dweimer.net) Received: (from www@localhost) by webmail.dweimer.local (8.15.2/8.15.2/Submit) id u0LLZNxc009925; Thu, 21 Jan 2016 15:35:23 -0600 (CST) (envelope-from dweimer@dweimer.net) X-Authentication-Warning: webmail.dweimer.local: www set sender to dweimer@dweimer.net using -f To: Mason Loring Bliss Subject: Re: ZFS performance help sought X-PHP-Script: www.dweimer.net/webmail/index.php for 71.86.41.122, 192.168.5.3 MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit Date: Thu, 21 Jan 2016 15:35:23 -0600 From: dweimer Cc: freebsd-questions@freebsd.org, owner-freebsd-questions@freebsd.org Organization: dweimer.net Reply-To: dweimer@dweimer.net Mail-Reply-To: dweimer@dweimer.net In-Reply-To: <20160121205139.GG4538@blisses.org> References: <20160121205139.GG4538@blisses.org> Message-ID: X-Sender: dweimer@dweimer.net User-Agent: Roundcube Webmail/1.1.4 X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Jan 2016 21:35:33 -0000 On 2016-01-21 2:51 pm, Mason Loring Bliss wrote: > Hi all. > > I've bounced back and forth between FreeBSD and Linux, and one of the > reasons > why I tend to part with FreeBSD is frustration with ZFS performance. > I'm > using my desktop for a couple roles at home. One of the roles is that > it > aggregates back-ups from across my machines and gathers them onto a > back-up > pool. > > Running FreeBSD, a zfs send/receive from one pool to another makes my > system > almost unusably slow and even begins to dig me into swap a little. As a > test > a little while back, I capped the ARC at half RAM, but that didn't > matter a > bit. I started looking into scheduler tweaking when I decided to take > the > path of least resistance and just install Linux instead. ZFS on Linux > on > literally the same hardware, dealing with the same pools, handles this > same > disk I/O without a hiccough. > > I've moved back to FreeBSD on this box now, and I'd like to resolve > this > issue. I don't know if it's a matter of fixing something broken in > FreeBSD's > scheduling or tuning ZFS somehow such that it's friendlier. (For what > it's > worth, renice'd zfs processes don't make a bit of difference, just as > capping > ARC didn't.) > > The box has FreeBSD 10.2, eight gigs of RAM, and I'm dealing with pools > 1TB > or smaller. No deduplication. I'm not enough of a ZFS guru to have a > strong > notion of what needs to change. I've not seen anything that seems > particularly relevant in tuning guides. I have precious little > diagnostic > data to share. That said, here's a quick idea of what FreeBSD is doing, > captured last night, with my box doing precious little else beyond the > transfer: > > last pid: 2631; load averages: 9.44, 8.94, 7.28 up 0+00:29:03 > 22:04:12 > 58 processes: 2 running, 56 sleeping > CPU: 0.8% user, 0.0% nice, 95.8% system, 0.1% interrupt, 3.3% idle > Mem: 136M Active, 14M Inact, 6915M Wired, 8784K Cache, 855M Free > ARC: 6234M Total, 248M MFU, 5233M MRU, 626M Anon, 45M Header, 82M Other > Swap: 8192M Total, 884K Used, 8191M Free > > PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU > COMMAND > 1334 root 1 52 15 42248K 3164K RUN 0 1:27 7.57% > zfs > 1333 root 1 38 15 42248K 3232K pipewr 1 1:04 4.98% > zfs > > Not that this matters in grand scheme of things, but I'm hoping to get > a > handle on what's happening here before frustration drives me back to > Linux. > I'd be happy to gather diagnostics given some pointers on what would be > useful. It seems unlikely that FreeBSD is this desperately inferior to > Linux > in terms of the competency of its scheduler, but I'm not sure what to > tune to > bring it up to the generally usable state I see on the same hardware > under > Linux. > > Thanks in advance! Try Starting here: https://wiki.freebsd.org/ZFSTuningGuide#General_Tuning I have these settings set on my system running on an AMD FX(tm)-6300 Six-Core Processor, with 16G of RAM, running 7 jails, and A Debian Linux Virtual Box VM, that runs Ubiquiti Unifi-Video software doing recording on 5 IP cameras. Running 4 1T SATA II drives in a raidz and a 2T SATA III drive with a zpool ontop of GELI for Bacula backups. ZFS Boot, with GPT partitions allocating a portion of each of the 1T drives as swap outside of the zpool. System works like a champ, its all standard desktop hardware, using on board SATA III ports All the drives are WD Green drives, so they are towards the low performance side. # ZFS Options zfs_load="YES" vfs.zfs.arc_max="4096M" vfs.zfs.arc_min="2048M" vfs.zfs.prefetch_disable="1" vfs.zfs.txg.timeout="5" vfs.zfs.write_limit_override=2147483648 -- Thanks, Dean E. Weimer http://www.dweimer.net/