From owner-freebsd-current@FreeBSD.ORG Wed Nov 9 19:39:36 2011 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C4213106564A for ; Wed, 9 Nov 2011 19:39:36 +0000 (UTC) (envelope-from yanegomi@gmail.com) Received: from mail-dy0-f54.google.com (mail-dy0-f54.google.com [209.85.220.54]) by mx1.freebsd.org (Postfix) with ESMTP id 4A4D48FC08 for ; Wed, 9 Nov 2011 19:39:36 +0000 (UTC) Received: by dyk29 with SMTP id 29so124876dyk.13 for ; Wed, 09 Nov 2011 11:39:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=Z0K1tusVeHf5hcqlWendAwerzc/QeV1Rh8cDSLACbRk=; b=Qh8mpDZ8C2/xWEeQlIrKxc0jTvIWHngTA/N328OaGGjbxCwWmFraEVV1llzMVJdjTs kkKTsHYuYfA3/thpeXmCgfoKQkoN5CRRMhFdrc6JvKZ0LcXwI/ceh6/opMs/dOpn3zO9 D2wYk9plYa/DVGLxZRosejVXlMXo0y62bb79c= MIME-Version: 1.0 Received: by 10.182.41.69 with SMTP id d5mr1184287obl.47.1320867574524; Wed, 09 Nov 2011 11:39:34 -0800 (PST) Received: by 10.182.122.33 with HTTP; Wed, 9 Nov 2011 11:39:34 -0800 (PST) In-Reply-To: <82C85C01-62C4-4E75-B3F2-59D703CA5D78@gsoft.com.au> References: <82C85C01-62C4-4E75-B3F2-59D703CA5D78@gsoft.com.au> Date: Wed, 9 Nov 2011 11:39:34 -0800 Message-ID: From: Garrett Cooper To: "Daniel O'Connor" Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Cc: Kurt Touet , "freebsd-current@freebsd.org" , Dan The Man Subject: Re: samba+zfs X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Nov 2011 19:39:36 -0000 On Nov 8, 2011, at 11:07 PM, "Daniel O'Connor" wrot= e: > > On 09/11/2011, at 17:32, Garrett Cooper wrote >>> dd's of large files (spooled backups going to tape) to /dev/null are as= slow as Samba. >> >> - Dedupe? > > Nope. > >> - Compression? > > On the mail spool & ports, but not on the tape spool. > >> - How much RAM? > > 8GB. >> - What debug options do you have enabled in the kernel? > > It is 8.2-GENERIC so.. no WITNESS (for example) Ok. 8.2 release or stable? >> I've been noticing a slowdown in some respects with NFS/SMB, but I >> suspected it was because I have an re(4) based NIC. ZFS has also wired >> down a lot of my system memory for the L2ARC=85 > > > re isn't great but I wouldn't expect it to slow down over time.. Unless b= ounce buffers got used more and more or something. > > I have an em0 card in this system - but in any case it is slow locally (i= .e. dd a large file with 64k block size). Good point. Simple base cases help isolate the root cause. That being said, my disk speed(s) are a lot better than my network + samba speeds (zfs:store is mfid0 backed with write cache enabled; zfs:sac is a single ada(4) backed disk with write cache enabled -- err... it shouldn't be like that), but I suspect that's misconfiguration on my part: $ sysctl hw.model hw.physmem hw.model: Intel(R) Xeon(R) CPU W3520 @ 2.67GHz hw.physmem: 12863094784 $ sudo mfiutil show volumes mfi0 Volumes: Id Size Level Stripe State Cache Name mfid0 ( 1860G) RAID-6 64k OPTIMAL Enabled $ zpool status pool: sac state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM sac ONLINE 0 0 0 ada0p3 ONLINE 0 0 0 errors: No known data errors pool: store state: ONLINE scan: scrub repaired 0 in 10h9m with 0 errors on Mon Nov 7 18:58:01 2011 config: NAME STATE READ WRITE CKSUM store ONLINE 0 0 0 mfid0p1 ONLINE 0 0 0 errors: No known data errors $ zfs list -o name,mountpoint,atime,sync,compression,dedup NAME MOUNTPOINT ATIME SYNC COMPRESS DEDUP sac legacy on standard off off sac/root / on standard off off sac/scratch /scratch on standard off off sac/scratch/freenas /scratch/freenas off standard on off sac/scratch/freenas/FreeBSD /scratch/freenas/FreeBSD off standard on off sac/usr /usr on standard off off sac/var /var on standard off off store /store on standard off off store/freebsd /store/freebsd on standard on on store/home /usr/home on standard off off $ dd if=3D/dev/zero of=3Dfoo bs=3D1m count=3D1024 1024+0 records out 1073741824 bytes transferred in 13.426620 secs (79971119 bytes/sec) $ cd /store $ dd if=3D/dev/zero of=3Dfoo bs=3D1m count=3D1024 1024+0 records in 1024+0 records out 1073741824 bytes transferred in 7.565117 secs (141933276 bytes/sec) $ cat /usr/local/etc/smb.conf [global] workgroup =3D WORKGROUP server string =3D BAYONETTA security =3D user load printers =3D no max log size =3D 50 preferred master =3D yes local master =3D yes socket options =3D SO_RCVBUF=3D16384 SO_SNDBUF=3D16384 nt acl support =3D yes inherit acls =3D yes map acl inherit =3D yes aio read size =3D 16384 aio write size =3D 16384 [scratch] path =3D /scratch writeable =3D yes [store] path =3D /store writeable =3D yes $ I'll have to: 1. Recheck what Windows 7 says when transferring out to my box with a large file. 2. Use nc to quickly measure network performance. 3. Try transferring over NFS with both my Macbook and setup FreeBSD or Linux on the other workstation for testing out NFS transfers (64kB rsize/wsize of course). Wash, rinse, repeat with samba. The last I remember the transfer speeds were pitiful with samba36 (somewhere around 45MBps to my 'store' zpool). I've been conservative with the socket settings, but it might be time to bump that up along with the mbuf cluster count (for some odd reason I haven't changed it from the system default), reboot, and retest. Thanks, -Garrett