Skip site navigation (1)Skip section navigation (2)
Date:      Fri, 4 Jan 2013 15:06:06 +0100
From:      Fleuriot Damien <ml@my.gd>
To:        "O. Hartmann" <ohartman@zedat.fu-berlin.de>
Cc:        Current FreeBSD <freebsd-current@FreeBSD.org>
Subject:   Re: ZFS/RAIDZ and SAMBA: abyssimal performance
Message-ID:  <1ADC2ECB-70FF-4DDD-9D62-16E2EEECDD8B@my.gd>
In-Reply-To: <50E6DE91.7010404@zedat.fu-berlin.de>
References:  <50E6DE91.7010404@zedat.fu-berlin.de>

next in thread | previous in thread | raw e-mail | index | archive | help

On Jan 4, 2013, at 2:52 PM, "O. Hartmann" <ohartman@zedat.fu-berlin.de> =
wrote:

> I use a small testing server. The hardware is most modern Intel =
hardware
> (i3-3220, Z77 chipset), 16GB RAM. The OS is FreeBSD 10.0-CURRENT #1
> r245036M: Fri Jan  4 12:48:53 CET 2013.
>=20
> The ZFS subsystem is comprised by 3 Western Digital 3 TB harddrives =
(WDC
> WD30EZRX-00DC0B0 80.00A80> ATA-9 SATA 3.x device), setup as a ZFS =
RAIDZ:
>=20
> ---
> root@gate [etc] zpool status
>  pool: ASGARD00
> state: ONLINE
>  scan: scrub repaired 0 in 1h45m with 0 errors on Sat Dec  1 20:59:44 =
2012
> config:
>=20
>        NAME                                            STATE     READ
> WRITE CKSUM
>        ASGARD00                                        ONLINE       0
>   0     0
>          raidz1-0                                      ONLINE       0
>   0     0
>            gptid/1e716118-1492-11e2-b828-90f6526a24d6  ONLINE       0
>   0     0
>            gptid/294a6798-1492-11e2-b828-90f6526a24d6  ONLINE       0
>   0     0
>            gptid/30c813f8-1492-11e2-b828-90f6526a24d6  ONLINE       0
>   0     0
>        logs
>          ada0p1                                        ONLINE       0
>   0     0
>        cache
>          ada0p2                                        ONLINE       0
>   0     0
>=20
> errors: No known data errors
> ---
>=20
> The "logs" and "cache" device is a single SAMSUNG 830 SSD, 60 GB
> capacity, GPT partinioned, logs (ZIL) has 5GB, cache has ~55 GB.
>=20
> I think its not the optimal setup using the very same SSD for both
> caching/L2ARC and ZIL, but without the cache device the performance
> doen't differ much at the moment. Luckliy, with ZFS I can change the
> arrangement as I like.
>=20
> The ZFS volumes created on the pool named ASGARD00 are standard, only
> options sharenfs/sharesmb/checksum are set to yes. Everthing elese is
> set to the defaults.
>=20
> In /boot/loader.conf I set the following parameters according to many
> (and confusing!) help and suggestions on the web:
>=20
> # ZFS
> #vfs.zfs.cache_flush_disable=3D1
> #
> #vfs.zfs.write_limit_override=3D1073741824 # 1GB
> vfs.zfs.l2arc_noprefetch=3D0
> vfs.zfs.l2arc_headroom=3D6
>=20
> The NFSv4 performance (client is also FreeBSD 10.0-CURRENT of the same
> date) is moderate to disapointing and doesn't exceed 45 - 55 MB/s
> sustained, but here are sometimes "spikes" I can watch with "systat =
-vm
> 1" reporting 120 MB/s per drive (ada2/ada3/ada4, the 3x 3TB WD drives =
in
> RAIDZ). I still benchmark via iozone. Both server and client use JUMBO
> frames (MTU=3D6120), which gives better throughput compared to the
> standard MTU=3D1500.
>=20
> The local performance on the server itself is slightly better, but
> iozone reports some strange numbers. The benchmark "writes" (using 4
> threads, 4k blocksizes, writing four times files of size 1G to the ZFS
> volume reports sometimes 150 MB/s throughput, and then 70 MB/s and
> re-writes is then 1/10 of the "write" throughput and according to the
> manual of iozone, re-write is considered to have higher values due to
> the lack of writing the meta data again. But I'm still testing this =
case.
>=20
> Well, the ZFS volumes are also shared as SAMBA CIFS volumes and here I
> experience something that is simply described as "abyssimal"
> performance! =46rom both a dedicated Windows 7 Pro client and a =
VirtualBox
> 4.2.6-client access to folders in a share, say my local home, can take
> ages! Opening files takes eons, if possible, in most cases windows
> reports "can not open ...". Copying files from Windows to the SAMBA
> share doesn't work or take ages, the throughput visible on the server
> side watched by "systat -vm 1" reports spiking 0.48 MB/s, with a =
hiatus
> of several seconds.
>=20
> Well, the SAMBA setup is straightforward, for two weeks now I have
> permutated nearly every parameter suggested on all the web's help =
sites
> and I simply took the well configuration from one of our lab's FreeBSD
> 9.1-STABLE SAMBA servers and changed the local settings for IP and
> domain names etc. The working server (FreeBSD 9.1-STABLE) in question
> has a single ZFS drive and is exporting this also via NFSv4. It =
doesn't
> have RAIDZ setup!
>=20
> Before I start benchmarking further with iozone I need to know whether
> there is an unresolved problem in FreeBSD 10.0 with ZFS/RAIDZ and =
SAMBA
> or whether I'm mislead and have overseen an important setup option.
> Before exposing all of my setups here I need to clearify.
>=20
> I didn't find so far any issues on the web regarding SAMBA, NFSv4 and
> ZFS/RAIDZ.
>=20
> Thanks in advance,
> Oliver
>=20
>=20



I experienced the same performance problem, then followed Jeremy =
Chadwick's advice and tuned variables a bit, I'm getting excellent =
performance now.


/boot/loader.conf

# Tune ZFS somewhat aye ?
vm.kmem_size=3D"3072M"
vfs.zfs.arc_min=3D"128M"
vfs.zfs.arc_max=3D"2048M"

# Decrease ZFS txg timeout value from 30 (default) to 5 seconds.  This
# should increase throughput and decrease the "bursty" stalls that
# happen during immense I/O with ZFS.
# =
http://lists.freebsd.org/pipermail/freebsd-fs/2009-December/007343.html
# =
http://lists.freebsd.org/pipermail/freebsd-fs/2009-December/007355.html
vfs.zfs.txg.timeout=3D"5"



And network cards:
# Up a bit our intel cards parameters
hw.em.txd=3D4096
hw.em.rxd=3D4096
hw.em.tx_int_delay=3D512
hw.em.rx_int_delay=3D512
hw.em.tx_abs_int_delay=3D1024
hw.em.rx_abs_int_delay=3D1024



And this is under [global] in /usr/local/etc/smb.conf:
   min receivefile size =3D 16384
   aio read size =3D 16384
   aio write size =3D 16384
   aio write behind =3D yes




Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1ADC2ECB-70FF-4DDD-9D62-16E2EEECDD8B>