From owner-freebsd-current@FreeBSD.ORG Wed Nov 9 02:16:30 2011 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C4558106566B for ; Wed, 9 Nov 2011 02:16:30 +0000 (UTC) (envelope-from dan@sunsaturn.com) Received: from sunsaturn.com (mail1.sunsaturn.com [IPv6:2001:49f0:4004::2]) by mx1.freebsd.org (Postfix) with ESMTP id 7731F8FC13 for ; Wed, 9 Nov 2011 02:16:30 +0000 (UTC) Received: by sunsaturn.com (Postfix, from userid 1001) id 04DF2119C75; Tue, 8 Nov 2011 20:16:29 -0600 (CST) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=sunsaturn.com; s=gamma; t=1320804990; bh=6p0bAEmSF7kIH28vnBsGahfztzzyk0H1kw9aKeiLdFg=; h=Date:From:To:cc:Subject:In-Reply-To:Message-ID:References: MIME-Version:Content-Type; b=HDYdlY/H3lqW68olvWaBmHPAcIaD5jo+E85hNKX5BEWEY71vzcb2nq3GB0cj42T44 obONlErvjBg97gpEfXghHjC5Hw0D/Ox1qn1tMwOhDYda2/50mDYGCmBW22zDyOtfJw XWwV5tzutjE6nUqm28tNz5huWxrUsN3b1EJYcfiQ= Received: from localhost (localhost [127.0.0.1]) by sunsaturn.com (Postfix) with ESMTP id EFD80119C6B; Tue, 8 Nov 2011 20:16:29 -0600 (CST) Date: Tue, 8 Nov 2011 20:16:29 -0600 (CST) From: Dan The Man To: Kurt Touet In-Reply-To: Message-ID: References: User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) MIME-Version: 1.0 Content-Type: MULTIPART/MIXED; BOUNDARY="2338054437-237842630-1320804989=:93923" Cc: Garrett Cooper , freebsd-current@freebsd.org Subject: Re: samba+zfs X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Nov 2011 02:16:30 -0000 This message is in MIME format. The first part should be readable text, while the remaining parts are likely unreadable without MIME-aware tools. --2338054437-237842630-1320804989=:93923 Content-Type: TEXT/PLAIN; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8BIT On Tue, 8 Nov 2011, Kurt Touet wrote: > On Tue, Nov 8, 2011 at 3:32 AM, Dan The Man wrote: >> >> >> Sorry I meant it was running fine on beta3 and 8.2 stable, and NOT RC1: >> asterisk:~# uname -a >> FreeBSD asterisk.sunsaturn.com 9.0-RC1 FreeBSD 9.0-RC1 #0: Mon Oct 31 >> 19:46:53 CDT 2011 droot@asterisk.sunsaturn.com:/usr/obj/usr/src/sys/MYKERNEL >>  amd64 >> asterisk:~# >> >> >> >> Dan. >> >> >> -- >> Dan The Man >> CTO/ Senior System Administrator >> Websites, Domains and Everything else >> http://www.SunSaturn.com >> Email: Dan@SunSaturn.com >> >> On Tue, 8 Nov 2011, Dan The Man wrote: >> >>> >>> Ok here is some specs: this been running fine on 8.2 stable and i was sure >>> it was running fine on RC1 as well. I did some testing against samba 34 35 >>> and 36 in the ports collection all with the same slow write problems. >>> >>> I did further testing mounting drive in question with NFS and it did not >>> suffer the same problem, so it seems just samba related here, where samba >>> would actually outperform my NFS mount before, now its taking 10x as long >>> to write anything. >>> >>> This is really most simplistic setup I have, all I want to do is map a >>> network drive at the house read/write so my laptop, desktop etc all have >>> access to it. I have played with all the smb.conf options, and can't seem >>> to find where the issue is, further research suggests others are >>> experiencing same problems with beta3 from following forum post: >>> >>> http://forums.freebsd.org/showthread.php?t=27300 >>> >>> Hardware this is running on: I beleive a 4 year old amd chip and board, >>> with 2 gigs of ram, this is a home PC that serves as a NAS, it has 1 SSD as >>> UFS boot OS filesystem, and uses ZFS in raid0 with 3 3terrabyte cheap >>> hitachi drives for my storage space,which is mirrored nightly with rsync >>> with another duplicate machine(cause I know someone is going to say why not >>> use raid5-raidz) >>> >>> Network specs: machine currently has dedicated IPV4 and gif0 tunneled IPV6 >>> addresses to he.net. >>> >>> I've played with nearly every option in smb.conf disabling, enabling etc >>> and can't seem to find the issue: here are my current config file settings >>> on machine that could apply to samba: >>> >>> asterisk:~# cat /boot/loader.conf >>> autoboot_delay="5" >>> accf_data_load="YES" >>> aio_load="YES" >>> zfs_load="YES" >>> kern.maxbcache=64M >>> kern.ipc.maxpipekva=4M >>> >>> vfs.zfs.prefetch_disable=1 >>> vm.kmem_size="1844M" >>> vfs.zfs.arc_min="1024M" >>> vfs.zfs.arc_max="1536M" >>> vfs.zfs.vdev.min_pending=2 >>> vfs.zfs.vdev.max_pending=8 >>> vfs.zfs.txg.timeout=5 >>> vfs.zfs.zil_disable="1" >>> ahci_load="YES" >>> asterisk:~# >>> >>> asterisk:~# cat /usr/local/etc/smb.conf >>> # Global parameters >>> [global] >>>       workgroup = HOME >>>       netbios name = ASTERISK >>>       server string = "Primary backups" >>>       interfaces = sk0 >>>       #smb ports = 139 >>>       #security = USER >>>       security = SHARE >>>       encrypt passwords = Yes >>>       #socket options = TCP_NODELAY SO_RCVBUF=65536 SO_SNDBUF=65536 >>>       domain master = no >>>       wins support = yes >>>       guest account = root >>>       socket options=SO_RCVBUF=131072 SO_SNDBUF=131072 TCP_NODELAY >>>       use sendfile = no >>>       level2 oplocks = True >>>       read raw = no >>>       write cache size = 262144 >>>       min receivefile size = 16384 >>>       aio read size = 16384 >>>       aio write size = 16384 >>>       aio write behind = yes >>>       dns proxy = no >>>       max log size = 50 >>>       #log file = /dev/null >>>       log file = /var/log/samba.log >>>       debug level = 1 >>>       syslog = 0 >>> >>> [data] >>>       comment = "Primary backups" >>>       path = /data/public >>>       read only = No >>>       guest ok = Yes >>>       valid users = root >>> asterisk:~# asterisk:~# cat /etc/sysctl.conf >>> # $FreeBSD: src/etc/sysctl.conf,v 1.8.34.1 2009/08/03 08:13:06 kensmith >>> Exp $ >>> # >>> #  This file is read when going to multi-user and its contents piped thru >>> #  ``sysctl'' to adjust kernel values.  ``man 5 sysctl.conf'' for details. >>> # >>> >>> # Uncomment this to prevent users from seeing information about processes >>> that >>> # are being run under another UID. >>> #security.bsd.see_other_uids=0 >>> >>> #raise file descriptors on the system >>> kern.maxfiles=204916 >>> kern.maxfilesperproc=204916 >>> >>> #raise sockets we can accept >>> kern.ipc.somaxconn=32768 >>> >>> #http://www-didc.lbl.gov/TCP-tuning/FreeBSD.html >>> kern.ipc.maxsockbuf=16777216 >>> net.inet.tcp.rfc1323=1 >>> net.inet.tcp.sendbuf_max=16777216 >>> net.inet.tcp.recvbuf_max=16777216 >>> net.inet.tcp.sendspace=65536 >>> net.inet.tcp.recvspace=131072 >>> >>> #NGINX webserver http://wiki.nginx.org/FreeBSDOptimizations >>> net.inet.icmp.icmplim=500 >>> kern.ipc.nmbjumbop=192000 >>> kern.ipc.nmbclusters=229376 >>> kern.ipc.maxsockets=204800 >>> net.inet.tcp.maxtcptw=163840 >>> #also add following to /boot/loader.conf >>> #vm.kmem_size=1844M >>> #kern.maxbcache=64M >>> #kern.ipc.maxpipekva=4M >>> >>> #default setting of net.inet.ip.portrange.first is to low, causing us >>> problems with bind >>> net.inet.ip.portrange.last=65535 >>> net.inet.ip.portrange.first=1024 >>> >>> #DOS protection >>> net.inet.tcp.msl=7500 >>> net.inet.tcp.blackhole=2 >>> net.inet.udp.blackhole=1 >>> net.inet.icmp.icmplim=50 >>> net.inet.ip.accept_sourceroute=0 >>> net.inet.ip.sourceroute=0 >>> >>> #some stuff for samba >>> kern.ipc.nmbclusters=32768 >>> kern.maxvnodes=800000 >>> net.inet.tcp.delayed_ack=0 >>> net.inet.tcp.inflight.enable=0 >>> net.inet.tcp.path_mtu_discovery=0 >>> net.inet.tcp.recvbuf_auto=1 >>> net.inet.tcp.recvbuf_inc=524288 >>> net.inet.tcp.sendbuf_auto=1 >>> net.inet.tcp.sendbuf_inc=524288 >>> net.inet.udp.maxdgram=57344 >>> net.inet.udp.recvspace=65535 >>> net.local.stream.recvspace=65535 >>> net.local.stream.sendspace=65535 >>> net.inet.tcp.mssdflt=1460 >>> >>> #IPSEC >>> net.inet.ip.forwarding=1 >>> net.inet6.ip6.forwarding=1 >>> kern.module_path=/boot/kernel;/boot/modules;/usr/local/modules >>> >>> >>> #NFS--not concerned with data integrity when playing mostly already stored >>> movies >>> vfs.nfsrv.async=1 >>> >>> #JAIL >>> #i like to use ping etc inside jail! >>> security.jail.allow_raw_sockets=1 >>> asterisk:~# >>> >>> >>> Here are logs of me trying to mux a DTS mkv file from samba.log on debug >>> level 10, I get the following over and over again: >>> >>> [2011/11/08 03:24:00.067974, 10] ../lib/util/util.c:415(dump_data) >>> [2011/11/08 03:24:00.067981,  3] smbd/process.c:1466(switch_message) >>>  switch message SMBwriteX (pid 64308) conn 0x805008450 >>> [2011/11/08 03:24:00.067990,  4] smbd/uid.c:345(change_to_user) >>>  Skipping user change - already user >>> [2011/11/08 03:24:00.068001, 10] >>> locking/locking.c:120(strict_lock_default) >>>  is_locked: optimisation - exclusive oplock on file >>> torrent_downloads_finished/Point.Break.1991.720p (1).mkv >>> [2011/11/08 03:24:00.068010, 10] >>> locking/locking.c:162(strict_lock_default) >>>  strict_lock_default: flavour = WINDOWS_LOCK brl start=83431665 len=65536 >>> unlocked for fnum 49966 file >>> torrent_downloads_finished/Point.Break.1991.720p (1).mkv >>> [2011/11/08 03:24:00.068021, 10] lib/recvfile.c:65(default_sys_recvfile) >>>  default_sys_recvfile: from = 33, to = 39, offset=83431665, count = 65536 >>> [2011/11/08 03:24:00.068995, 10] smbd/fileio.c:143(real_write_file) >>>  real_write_file (torrent_downloads_finished/Point.Break.1991.720p >>> (1).mkv): pos = 83431665, size = 65536, returned 65536 >>> [2011/11/08 03:24:00.069013,  3] smbd/reply.c:4639(reply_write_and_X) >>>  writeX fnum=49966 num=65536 wrote=65536 >>> [2011/11/08 03:24:00.069038, 10] >>> lib/util_sock.c:516(read_smb_length_return_keepalive) >>>  got smb length of 65600 >>> [2011/11/08 03:24:00.069052, 10] smbd/reply.c:4459(is_valid_writeX_buffer) >>>  is_valid_writeX_buffer: true len = 65600, doff = 64, numtowrite = 65536 >>> [2011/11/08 03:24:00.069062,  6] smbd/process.c:1659(process_smb) >>>  got message type 0x0 of len 0x3f >>> [2011/11/08 03:24:00.069072,  3] smbd/process.c:1661(process_smb) >>>  Transaction 15398 of length 67 (65536 toread) >>> [2011/11/08 03:24:00.069081,  5] lib/util.c:332(show_msg) >>> [2011/11/08 03:24:00.069087,  5] lib/util.c:342(show_msg) >>>  size=63 >>>  smb_com=0x2f >>>  smb_rcls=0 >>>  smb_reh=0 >>>  smb_err=0 >>>  smb_flg=24 >>>  smb_flg2=51207 >>>  smb_tid=1 >>>  smb_pid=65279 >>>  smb_uid=0 >>>  smb_mid=36032 >>>  smt_wct=14 >>>  smb_vwv[ 0]=  255 (0xFF) >>>  smb_vwv[ 1]=57054 (0xDEDE) >>>  smb_vwv[ 2]=49966 (0xC32E) >>>  smb_vwv[ 3]= 4337 (0x10F1) >>>  smb_vwv[ 4]= 1274 (0x4FA) >>>  smb_vwv[ 5]=65535 (0xFFFF) >>>  smb_vwv[ 6]=65535 (0xFFFF) >>>  smb_vwv[ 7]=    0 (0x0) >>>  smb_vwv[ 8]=    0 (0x0) >>>  smb_vwv[ 9]=    1 (0x1) >>>  smb_vwv[10]=    0 (0x0) >>>  smb_vwv[11]=   64 (0x40) >>>  smb_vwv[12]=    0 (0x0) >>>  smb_vwv[13]=    0 (0x0) >>>  smb_bcc=0 >>> [2011/11/08 03:24:00.069163, 10] ../lib/util/util.c:415(dump_data) >>> [2011/11/08 03:24:00.069170,  3] smbd/process.c:1466(switch_message) >>>  switch message SMBwriteX (pid 64308) conn 0x805008450 >>> [2011/11/08 03:24:00.069179,  4] smbd/uid.c:345(change_to_user) >>>  Skipping user change - already user >>> [2011/11/08 03:24:00.069188, 10] >>> locking/locking.c:120(strict_lock_default) >>>  is_locked: optimisation - exclusive oplock on file >>> torrent_downloads_finished/Point.Break.1991.720p (1).mkv >>> [2011/11/08 03:24:00.069197, 10] >>> locking/locking.c:162(strict_lock_default) >>>  strict_lock_default: flavour = WINDOWS_LOCK brl start=83497201 len=65536 >>> unlocked for fnum 49966 file >>> torrent_downloads_finished/Point.Break.1991.720p (1).mkv >>> [2011/11/08 03:24:00.069221, 10] lib/recvfile.c:65(default_sys_recvfile) >>>  default_sys_recvfile: from = 33, to = 39, offset=83497201, count = 65536 >>> [2011/11/08 03:24:00.069987, 10] smbd/fileio.c:143(real_write_file) >>>  real_write_file (torrent_downloads_finished/Point.Break.1991.720p >>> (1).mkv): pos = 83497201, size = 65536, returned 65536 >>> [2011/11/08 03:24:00.070004,  3] smbd/reply.c:4639(reply_write_and_X) >>>  writeX fnum=49966 num=65536 wrote=65536 >>> [2011/11/08 03:24:00.070030, 10] >>> lib/util_sock.c:516(read_smb_length_return_keepalive) >>>  got smb length of 65600 >>> [2011/11/08 03:24:00.070044, 10] smbd/reply.c:4459(is_valid_writeX_buffer) >>>  is_valid_writeX_buffer: true len = 65600, doff = 64, numtowrite = 65536 >>> [2011/11/08 03:24:00.070053,  6] smbd/process.c:1659(process_smb) >>>  got message type 0x0 of len 0x3f >>> [2011/11/08 03:24:00.070063,  3] smbd/process.c:1661(process_smb) >>>  Transaction 15399 of length 67 (65536 toread) >>> [2011/11/08 03:24:00.070072,  5] lib/util.c:332(show_msg) >>> [2011/11/08 03:24:00.070077,  5] lib/util.c:342(show_msg) >>>  size=63 >>>  smb_com=0x2f >>>  smb_rcls=0 >>>  smb_reh=0 >>>  smb_err=0 >>>  smb_flg=24 >>>  smb_flg2=51207 >>>  smb_tid=1 >>>  smb_pid=65279 >>>  smb_uid=0 >>>  smb_mid=36102 >>>  smt_wct=14 >>>  smb_vwv[ 0]=  255 (0xFF) >>>  smb_vwv[ 1]=57054 (0xDEDE) >>>  smb_vwv[ 2]=49966 (0xC32E) >>>  smb_vwv[ 3]= 4337 (0x10F1) >>>  smb_vwv[ 4]= 1275 (0x4FB) >>>  smb_vwv[ 5]=65535 (0xFFFF) >>>  smb_vwv[ 6]=65535 (0xFFFF) >>>  smb_vwv[ 7]=    0 (0x0) >>>  smb_vwv[ 8]=    0 (0x0) >>>  smb_vwv[ 9]=    1 (0x1) >>>  smb_vwv[10]=    0 (0x0) >>>  smb_vwv[11]=   64 (0x40) >>>  smb_vwv[12]=    0 (0x0) >>>  smb_vwv[13]=    0 (0x0) >>>  smb_bcc=0 >>> >>> >>> Hopefully maybe someone can shine some light on this.... >>> >>> >>> Dan. >>> >>> >>> -- >>> Dan The Man >>> CTO/ Senior System Administrator >>> Websites, Domains and Everything else >>> http://www.SunSaturn.com >>> Email: Dan@SunSaturn.com >>> >>> On Fri, 28 Oct 2011, Garrett Cooper wrote: >>> >>>> On Thu, Oct 27, 2011 at 6:42 PM, Dan wrote: >>>>> >>>>> >>>>> Updated from 9.0 beta3 to RC1 and using mkvmerge over samba/zfs >>>>> its taking over an hour to just mux in things like DTS english, where it >>>>> was >>>>> 15 minutes on beta3. >>>> >>>> Hi Dan, >>>> - Can you do more deterministic / scientific benchmarks? >>>> - Did you upgrade Samba? >>>> - What is your system's operating hardware profile? >>>> Thanks! >>>> -Garrett >>>> > > I had noticed a similar problem with doing large writes from my > windows workstation to my freebsd ZFS array. Previously I had been > able to write at around 30MB/s (limited by a slow SATA controller), > and those speeds had dropped to 3-5MB/s with long pauses between > writes to the array (monitoring with zpool iostat). However, I just > updated to stable/9 r227357 and the issue seems to have gone away; I'm > back up to 30MB/s writes. > > Hope you find the same sort of thing, > -kurt > Yep your right, on RC2 my samba ZFS write speeds are back. Thanks for heads up Kurt. Dan. -- Dan The Man CTO/ Senior System Administrator Websites, Domains and Everything else http://www.SunSaturn.com Email: Dan@SunSaturn.com --2338054437-237842630-1320804989=:93923--