Date: Tue, 8 Nov 2011 23:59:54 -0600 From: Kurt Touet <ktouet@gmail.com> To: Dan The Man <dan@sunsaturn.com> Cc: freebsd-current@freebsd.org Subject: Re: samba+zfs Message-ID: <CALM%2B6aJiM%2BNfikax9vNiYzBJqo3B8WLEXiZgrjUUQa9ngCQaKg@mail.gmail.com> In-Reply-To: <alpine.BSF.2.00.1111082015260.93923@sunsaturn.com> References: <alpine.BSF.2.00.1110272039500.50739@sunsaturn.com> <CAGH67wRZZx0hG9ug2k-5ohCOPJ9sZOU9iFVKg7hv9WM=R761GA@mail.gmail.com> <alpine.BSF.2.00.1111080259270.89703@sunsaturn.com> <alpine.BSF.2.00.1111080328010.89703@sunsaturn.com> <CALM%2B6aJkF=CFq8LA3FrSMYo8La-8txK4h2p4yZtdHshskBU6Vw@mail.gmail.com> <alpine.BSF.2.00.1111082015260.93923@sunsaturn.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Tue, Nov 8, 2011 at 8:16 PM, Dan The Man <dan@sunsaturn.com> wrote: > > On Tue, 8 Nov 2011, Kurt Touet wrote: > >> On Tue, Nov 8, 2011 at 3:32 AM, Dan The Man <dan@sunsaturn.com> wrote: >>> >>> >>> Sorry I meant it was running fine on beta3 and 8.2 stable, and NOT RC1: >>> asterisk:~# uname -a >>> FreeBSD asterisk.sunsaturn.com 9.0-RC1 FreeBSD 9.0-RC1 #0: Mon Oct 31 >>> 19:46:53 CDT 2011 >>> droot@asterisk.sunsaturn.com:/usr/obj/usr/src/sys/MYKERNEL >>> =A0amd64 >>> asterisk:~# >>> >>> >>> >>> Dan. >>> >>> >>> -- >>> Dan The Man >>> CTO/ Senior System Administrator >>> Websites, Domains and Everything else >>> http://www.SunSaturn.com >>> Email: Dan@SunSaturn.com >>> >>> On Tue, 8 Nov 2011, Dan The Man wrote: >>> >>>> >>>> Ok here is some specs: this been running fine on 8.2 stable and i was >>>> sure >>>> it was running fine on RC1 as well. I did some testing against samba 3= 4 >>>> 35 >>>> and 36 in the ports collection all with the same slow write problems. >>>> >>>> I did further testing mounting drive in question with NFS and it did n= ot >>>> suffer the same problem, so it seems just samba related here, where >>>> samba >>>> would actually outperform my NFS mount before, now its taking 10x as >>>> long >>>> to write anything. >>>> >>>> This is really most simplistic setup I have, all I want to do is map a >>>> network drive at the house read/write so my laptop, desktop etc all ha= ve >>>> access to it. I have played with all the smb.conf options, and can't >>>> seem >>>> to find where the issue is, further research suggests others are >>>> experiencing same problems with beta3 from following forum post: >>>> >>>> http://forums.freebsd.org/showthread.php?t=3D27300 >>>> >>>> Hardware this is running on: I beleive a 4 year old amd chip and board= , >>>> with 2 gigs of ram, this is a home PC that serves as a NAS, it has 1 S= SD >>>> as >>>> UFS boot OS filesystem, and uses ZFS in raid0 with 3 3terrabyte cheap >>>> hitachi drives for my storage space,which is mirrored nightly with rsy= nc >>>> with another duplicate machine(cause I know someone is going to say wh= y >>>> not >>>> use raid5-raidz) >>>> >>>> Network specs: machine currently has dedicated IPV4 and gif0 tunneled >>>> IPV6 >>>> addresses to he.net. >>>> >>>> I've played with nearly every option in smb.conf disabling, enabling e= tc >>>> and can't seem to find the issue: here are my current config file >>>> settings >>>> on machine that could apply to samba: >>>> >>>> asterisk:~# cat /boot/loader.conf >>>> autoboot_delay=3D"5" >>>> accf_data_load=3D"YES" >>>> aio_load=3D"YES" >>>> zfs_load=3D"YES" >>>> kern.maxbcache=3D64M >>>> kern.ipc.maxpipekva=3D4M >>>> >>>> vfs.zfs.prefetch_disable=3D1 >>>> vm.kmem_size=3D"1844M" >>>> vfs.zfs.arc_min=3D"1024M" >>>> vfs.zfs.arc_max=3D"1536M" >>>> vfs.zfs.vdev.min_pending=3D2 >>>> vfs.zfs.vdev.max_pending=3D8 >>>> vfs.zfs.txg.timeout=3D5 >>>> vfs.zfs.zil_disable=3D"1" >>>> ahci_load=3D"YES" >>>> asterisk:~# >>>> >>>> asterisk:~# cat /usr/local/etc/smb.conf >>>> # Global parameters >>>> [global] >>>> =A0 =A0 =A0 workgroup =3D HOME >>>> =A0 =A0 =A0 netbios name =3D ASTERISK >>>> =A0 =A0 =A0 server string =3D "Primary backups" >>>> =A0 =A0 =A0 interfaces =3D sk0 >>>> =A0 =A0 =A0 #smb ports =3D 139 >>>> =A0 =A0 =A0 #security =3D USER >>>> =A0 =A0 =A0 security =3D SHARE >>>> =A0 =A0 =A0 encrypt passwords =3D Yes >>>> =A0 =A0 =A0 #socket options =3D TCP_NODELAY SO_RCVBUF=3D65536 SO_SNDBU= F=3D65536 >>>> =A0 =A0 =A0 domain master =3D no >>>> =A0 =A0 =A0 wins support =3D yes >>>> =A0 =A0 =A0 guest account =3D root >>>> =A0 =A0 =A0 socket options=3DSO_RCVBUF=3D131072 SO_SNDBUF=3D131072 TCP= _NODELAY >>>> =A0 =A0 =A0 use sendfile =3D no >>>> =A0 =A0 =A0 level2 oplocks =3D True >>>> =A0 =A0 =A0 read raw =3D no >>>> =A0 =A0 =A0 write cache size =3D 262144 >>>> =A0 =A0 =A0 min receivefile size =3D 16384 >>>> =A0 =A0 =A0 aio read size =3D 16384 >>>> =A0 =A0 =A0 aio write size =3D 16384 >>>> =A0 =A0 =A0 aio write behind =3D yes >>>> =A0 =A0 =A0 dns proxy =3D no >>>> =A0 =A0 =A0 max log size =3D 50 >>>> =A0 =A0 =A0 #log file =3D /dev/null >>>> =A0 =A0 =A0 log file =3D /var/log/samba.log >>>> =A0 =A0 =A0 debug level =3D 1 >>>> =A0 =A0 =A0 syslog =3D 0 >>>> >>>> [data] >>>> =A0 =A0 =A0 comment =3D "Primary backups" >>>> =A0 =A0 =A0 path =3D /data/public >>>> =A0 =A0 =A0 read only =3D No >>>> =A0 =A0 =A0 guest ok =3D Yes >>>> =A0 =A0 =A0 valid users =3D root >>>> asterisk:~# asterisk:~# cat /etc/sysctl.conf >>>> # $FreeBSD: src/etc/sysctl.conf,v 1.8.34.1 2009/08/03 08:13:06 kensmit= h >>>> Exp $ >>>> # >>>> # =A0This file is read when going to multi-user and its contents piped >>>> thru >>>> # =A0``sysctl'' to adjust kernel values. =A0``man 5 sysctl.conf'' for >>>> details. >>>> # >>>> >>>> # Uncomment this to prevent users from seeing information about >>>> processes >>>> that >>>> # are being run under another UID. >>>> #security.bsd.see_other_uids=3D0 >>>> >>>> #raise file descriptors on the system >>>> kern.maxfiles=3D204916 >>>> kern.maxfilesperproc=3D204916 >>>> >>>> #raise sockets we can accept >>>> kern.ipc.somaxconn=3D32768 >>>> >>>> #http://www-didc.lbl.gov/TCP-tuning/FreeBSD.html >>>> kern.ipc.maxsockbuf=3D16777216 >>>> net.inet.tcp.rfc1323=3D1 >>>> net.inet.tcp.sendbuf_max=3D16777216 >>>> net.inet.tcp.recvbuf_max=3D16777216 >>>> net.inet.tcp.sendspace=3D65536 >>>> net.inet.tcp.recvspace=3D131072 >>>> >>>> #NGINX webserver http://wiki.nginx.org/FreeBSDOptimizations >>>> net.inet.icmp.icmplim=3D500 >>>> kern.ipc.nmbjumbop=3D192000 >>>> kern.ipc.nmbclusters=3D229376 >>>> kern.ipc.maxsockets=3D204800 >>>> net.inet.tcp.maxtcptw=3D163840 >>>> #also add following to /boot/loader.conf >>>> #vm.kmem_size=3D1844M >>>> #kern.maxbcache=3D64M >>>> #kern.ipc.maxpipekva=3D4M >>>> >>>> #default setting of net.inet.ip.portrange.first is to low, causing us >>>> problems with bind >>>> net.inet.ip.portrange.last=3D65535 >>>> net.inet.ip.portrange.first=3D1024 >>>> >>>> #DOS protection >>>> net.inet.tcp.msl=3D7500 >>>> net.inet.tcp.blackhole=3D2 >>>> net.inet.udp.blackhole=3D1 >>>> net.inet.icmp.icmplim=3D50 >>>> net.inet.ip.accept_sourceroute=3D0 >>>> net.inet.ip.sourceroute=3D0 >>>> >>>> #some stuff for samba >>>> kern.ipc.nmbclusters=3D32768 >>>> kern.maxvnodes=3D800000 >>>> net.inet.tcp.delayed_ack=3D0 >>>> net.inet.tcp.inflight.enable=3D0 >>>> net.inet.tcp.path_mtu_discovery=3D0 >>>> net.inet.tcp.recvbuf_auto=3D1 >>>> net.inet.tcp.recvbuf_inc=3D524288 >>>> net.inet.tcp.sendbuf_auto=3D1 >>>> net.inet.tcp.sendbuf_inc=3D524288 >>>> net.inet.udp.maxdgram=3D57344 >>>> net.inet.udp.recvspace=3D65535 >>>> net.local.stream.recvspace=3D65535 >>>> net.local.stream.sendspace=3D65535 >>>> net.inet.tcp.mssdflt=3D1460 >>>> >>>> #IPSEC >>>> net.inet.ip.forwarding=3D1 >>>> net.inet6.ip6.forwarding=3D1 >>>> kern.module_path=3D/boot/kernel;/boot/modules;/usr/local/modules >>>> >>>> >>>> #NFS--not concerned with data integrity when playing mostly already >>>> stored >>>> movies >>>> vfs.nfsrv.async=3D1 >>>> >>>> #JAIL >>>> #i like to use ping etc inside jail! >>>> security.jail.allow_raw_sockets=3D1 >>>> asterisk:~# >>>> >>>> >>>> Here are logs of me trying to mux a DTS mkv file from samba.log on deb= ug >>>> level 10, I get the following over and over again: >>>> >>>> [2011/11/08 03:24:00.067974, 10] ../lib/util/util.c:415(dump_data) >>>> [2011/11/08 03:24:00.067981, =A03] smbd/process.c:1466(switch_message) >>>> =A0switch message SMBwriteX (pid 64308) conn 0x805008450 >>>> [2011/11/08 03:24:00.067990, =A04] smbd/uid.c:345(change_to_user) >>>> =A0Skipping user change - already user >>>> [2011/11/08 03:24:00.068001, 10] >>>> locking/locking.c:120(strict_lock_default) >>>> =A0is_locked: optimisation - exclusive oplock on file >>>> torrent_downloads_finished/Point.Break.1991.720p (1).mkv >>>> [2011/11/08 03:24:00.068010, 10] >>>> locking/locking.c:162(strict_lock_default) >>>> =A0strict_lock_default: flavour =3D WINDOWS_LOCK brl start=3D83431665 >>>> len=3D65536 >>>> unlocked for fnum 49966 file >>>> torrent_downloads_finished/Point.Break.1991.720p (1).mkv >>>> [2011/11/08 03:24:00.068021, 10] lib/recvfile.c:65(default_sys_recvfil= e) >>>> =A0default_sys_recvfile: from =3D 33, to =3D 39, offset=3D83431665, co= unt =3D >>>> 65536 >>>> [2011/11/08 03:24:00.068995, 10] smbd/fileio.c:143(real_write_file) >>>> =A0real_write_file (torrent_downloads_finished/Point.Break.1991.720p >>>> (1).mkv): pos =3D 83431665, size =3D 65536, returned 65536 >>>> [2011/11/08 03:24:00.069013, =A03] smbd/reply.c:4639(reply_write_and_X= ) >>>> =A0writeX fnum=3D49966 num=3D65536 wrote=3D65536 >>>> [2011/11/08 03:24:00.069038, 10] >>>> lib/util_sock.c:516(read_smb_length_return_keepalive) >>>> =A0got smb length of 65600 >>>> [2011/11/08 03:24:00.069052, 10] >>>> smbd/reply.c:4459(is_valid_writeX_buffer) >>>> =A0is_valid_writeX_buffer: true len =3D 65600, doff =3D 64, numtowrite= =3D 65536 >>>> [2011/11/08 03:24:00.069062, =A06] smbd/process.c:1659(process_smb) >>>> =A0got message type 0x0 of len 0x3f >>>> [2011/11/08 03:24:00.069072, =A03] smbd/process.c:1661(process_smb) >>>> =A0Transaction 15398 of length 67 (65536 toread) >>>> [2011/11/08 03:24:00.069081, =A05] lib/util.c:332(show_msg) >>>> [2011/11/08 03:24:00.069087, =A05] lib/util.c:342(show_msg) >>>> =A0size=3D63 >>>> =A0smb_com=3D0x2f >>>> =A0smb_rcls=3D0 >>>> =A0smb_reh=3D0 >>>> =A0smb_err=3D0 >>>> =A0smb_flg=3D24 >>>> =A0smb_flg2=3D51207 >>>> =A0smb_tid=3D1 >>>> =A0smb_pid=3D65279 >>>> =A0smb_uid=3D0 >>>> =A0smb_mid=3D36032 >>>> =A0smt_wct=3D14 >>>> =A0smb_vwv[ 0]=3D =A0255 (0xFF) >>>> =A0smb_vwv[ 1]=3D57054 (0xDEDE) >>>> =A0smb_vwv[ 2]=3D49966 (0xC32E) >>>> =A0smb_vwv[ 3]=3D 4337 (0x10F1) >>>> =A0smb_vwv[ 4]=3D 1274 (0x4FA) >>>> =A0smb_vwv[ 5]=3D65535 (0xFFFF) >>>> =A0smb_vwv[ 6]=3D65535 (0xFFFF) >>>> =A0smb_vwv[ 7]=3D =A0 =A00 (0x0) >>>> =A0smb_vwv[ 8]=3D =A0 =A00 (0x0) >>>> =A0smb_vwv[ 9]=3D =A0 =A01 (0x1) >>>> =A0smb_vwv[10]=3D =A0 =A00 (0x0) >>>> =A0smb_vwv[11]=3D =A0 64 (0x40) >>>> =A0smb_vwv[12]=3D =A0 =A00 (0x0) >>>> =A0smb_vwv[13]=3D =A0 =A00 (0x0) >>>> =A0smb_bcc=3D0 >>>> [2011/11/08 03:24:00.069163, 10] ../lib/util/util.c:415(dump_data) >>>> [2011/11/08 03:24:00.069170, =A03] smbd/process.c:1466(switch_message) >>>> =A0switch message SMBwriteX (pid 64308) conn 0x805008450 >>>> [2011/11/08 03:24:00.069179, =A04] smbd/uid.c:345(change_to_user) >>>> =A0Skipping user change - already user >>>> [2011/11/08 03:24:00.069188, 10] >>>> locking/locking.c:120(strict_lock_default) >>>> =A0is_locked: optimisation - exclusive oplock on file >>>> torrent_downloads_finished/Point.Break.1991.720p (1).mkv >>>> [2011/11/08 03:24:00.069197, 10] >>>> locking/locking.c:162(strict_lock_default) >>>> =A0strict_lock_default: flavour =3D WINDOWS_LOCK brl start=3D83497201 >>>> len=3D65536 >>>> unlocked for fnum 49966 file >>>> torrent_downloads_finished/Point.Break.1991.720p (1).mkv >>>> [2011/11/08 03:24:00.069221, 10] lib/recvfile.c:65(default_sys_recvfil= e) >>>> =A0default_sys_recvfile: from =3D 33, to =3D 39, offset=3D83497201, co= unt =3D >>>> 65536 >>>> [2011/11/08 03:24:00.069987, 10] smbd/fileio.c:143(real_write_file) >>>> =A0real_write_file (torrent_downloads_finished/Point.Break.1991.720p >>>> (1).mkv): pos =3D 83497201, size =3D 65536, returned 65536 >>>> [2011/11/08 03:24:00.070004, =A03] smbd/reply.c:4639(reply_write_and_X= ) >>>> =A0writeX fnum=3D49966 num=3D65536 wrote=3D65536 >>>> [2011/11/08 03:24:00.070030, 10] >>>> lib/util_sock.c:516(read_smb_length_return_keepalive) >>>> =A0got smb length of 65600 >>>> [2011/11/08 03:24:00.070044, 10] >>>> smbd/reply.c:4459(is_valid_writeX_buffer) >>>> =A0is_valid_writeX_buffer: true len =3D 65600, doff =3D 64, numtowrite= =3D 65536 >>>> [2011/11/08 03:24:00.070053, =A06] smbd/process.c:1659(process_smb) >>>> =A0got message type 0x0 of len 0x3f >>>> [2011/11/08 03:24:00.070063, =A03] smbd/process.c:1661(process_smb) >>>> =A0Transaction 15399 of length 67 (65536 toread) >>>> [2011/11/08 03:24:00.070072, =A05] lib/util.c:332(show_msg) >>>> [2011/11/08 03:24:00.070077, =A05] lib/util.c:342(show_msg) >>>> =A0size=3D63 >>>> =A0smb_com=3D0x2f >>>> =A0smb_rcls=3D0 >>>> =A0smb_reh=3D0 >>>> =A0smb_err=3D0 >>>> =A0smb_flg=3D24 >>>> =A0smb_flg2=3D51207 >>>> =A0smb_tid=3D1 >>>> =A0smb_pid=3D65279 >>>> =A0smb_uid=3D0 >>>> =A0smb_mid=3D36102 >>>> =A0smt_wct=3D14 >>>> =A0smb_vwv[ 0]=3D =A0255 (0xFF) >>>> =A0smb_vwv[ 1]=3D57054 (0xDEDE) >>>> =A0smb_vwv[ 2]=3D49966 (0xC32E) >>>> =A0smb_vwv[ 3]=3D 4337 (0x10F1) >>>> =A0smb_vwv[ 4]=3D 1275 (0x4FB) >>>> =A0smb_vwv[ 5]=3D65535 (0xFFFF) >>>> =A0smb_vwv[ 6]=3D65535 (0xFFFF) >>>> =A0smb_vwv[ 7]=3D =A0 =A00 (0x0) >>>> =A0smb_vwv[ 8]=3D =A0 =A00 (0x0) >>>> =A0smb_vwv[ 9]=3D =A0 =A01 (0x1) >>>> =A0smb_vwv[10]=3D =A0 =A00 (0x0) >>>> =A0smb_vwv[11]=3D =A0 64 (0x40) >>>> =A0smb_vwv[12]=3D =A0 =A00 (0x0) >>>> =A0smb_vwv[13]=3D =A0 =A00 (0x0) >>>> =A0smb_bcc=3D0 >>>> >>>> >>>> Hopefully maybe someone can shine some light on this.... >>>> >>>> >>>> Dan. >>>> >>>> >>>> -- >>>> Dan The Man >>>> CTO/ Senior System Administrator >>>> Websites, Domains and Everything else >>>> http://www.SunSaturn.com >>>> Email: Dan@SunSaturn.com >>>> >>>> On Fri, 28 Oct 2011, Garrett Cooper wrote: >>>> >>>>> On Thu, Oct 27, 2011 at 6:42 PM, Dan <dan@sunsaturn.com> wrote: >>>>>> >>>>>> >>>>>> Updated from 9.0 beta3 to RC1 and using mkvmerge over samba/zfs >>>>>> its taking over an hour to just mux in things like DTS english, wher= e >>>>>> it >>>>>> was >>>>>> 15 minutes on beta3. >>>>> >>>>> Hi Dan, >>>>> - Can you do more deterministic / scientific benchmarks? >>>>> - Did you upgrade Samba? >>>>> - What is your system's operating hardware profile? >>>>> Thanks! >>>>> -Garrett >>>>> >> >> I had noticed a similar problem with doing large writes from my >> windows workstation to my freebsd ZFS array. =A0Previously I had been >> able to write at around 30MB/s (limited by a slow SATA controller), >> and those speeds had dropped to 3-5MB/s with long pauses between >> writes to the array (monitoring with zpool iostat). =A0 However, I just >> updated to stable/9 r227357 and the issue seems to have gone away; I'm >> back up to 30MB/s writes. >> >> Hope you find the same sort of thing, >> -kurt >> > > Yep your right, on RC2 my samba ZFS write speeds are back. > Thanks for heads up Kurt. > > > Dan. > > > -- > Dan The Man > CTO/ Senior System Administrator > Websites, Domains and Everything else > http://www.SunSaturn.com > Email: Dan@SunSaturn.com > I may have spoken too soon. With the speeds back up to normal, I moved a bunch of content that I'd been storing temporarily on my workstation to the server. After a few hours and moving 40gb+ of content (not all at once), I noticed the same write problems cropping up again. I rebooted the server and speeds came back to normal. The oddity that I notice is that the array seems to be delaying writes to a single disc in my raidz1 vdev, and then writing out very slowly to it (1-2MB/s). The drive that it is doing it with has absolutely zero read/write/seek errors in SMART, and works just fine after the box is rebooted (so it shouldn't be a drive issue). The only thing interesting about this drive is that it replaced a failed drive a few weeks ago. Is anyone else seeing problems like this with samba/zfs ? Perhaps it's not exclusive to samba, either? -kurt
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CALM%2B6aJiM%2BNfikax9vNiYzBJqo3B8WLEXiZgrjUUQa9ngCQaKg>