Date: Fri, 06 Apr 2007 11:22:09 +0100 From: Tom Judge <tom@tomjudge.com> To: sven@dmv.com Cc: freebsd-stable@freebsd.org Subject: Re: ggate + gmirror write performance woes Message-ID: <46161F51.1070704@tomjudge.com> In-Reply-To: <1175804135.8280.21.camel@lanshark.dmv.com> References: <1175785136.8280.10.camel@lanshark.dmv.com> <20070405160313.GE1230@dkirhlarov.mow.oilspace.com> <46152615.1040602@tomjudge.com> <1175804135.8280.21.camel@lanshark.dmv.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Sven Willenberger wrote: > On Thu, 2007-04-05 at 17:38 +0100, Tom Judge wrote: >> Dmitriy Kirhlarov wrote: >>> On Thu, Apr 05, 2007 at 10:58:56AM -0400, Sven Willenberger wrote: >>>> I am trying to set up a HA type system involving two identical boxes and >>>> have gone through the following to set up the systems: >>>> >>>> Slave server: >>>> ggated -R 196608 -S 196608 >>>> (exporting /dev/amrd1 ) >>>> net.inet.tcp.sendspace: 65536 >>>> net.inet.tcp.recvspace: 131072 >>> Try >>> net.local.stream.recvspace=65535 >>> net.local.stream.sendspace=65535 >>> >>> Also, try increase this sysctls with >>> net.inet.tcp.rfc1323=1 >>> >>> I use it on FreeBSD 5.x with: >>> net.inet.tcp.sendspace=131072 >>> net.inet.tcp.recvspace=131072 >>> net.local.stream.recvspace=65535 >>> net.local.stream.sendspace=65535 >>> >>> ggated -R 1048576 -S 1048576 >>> ggatec -R 1048576 -S 1048576 >>> >>> WBR. >>> Dmitriy >>> _______________________________________________ >>> freebsd-stable@freebsd.org mailing list >>> http://lists.freebsd.org/mailman/listinfo/freebsd-stable >>> To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" >> >> I have seen sustained writes of 30Mb/s using the following configuration: >> >> cat /boot/loader.conf >> kern.ipc.nmbclusters="32768" >> >> cat /etc/sysctl.conf >> net.inet.tcp.sendspace=1048576 >> net.inet.tcp.recvspace=1048576 >> >> Server: >> /sbin/ggated -S 1310720 -R 1310720 -a 172.31.0.18 /etc/gg.exports >> >> Client: >> /sbin/ggatec create -q 2048 -t 5 -S 1310720 -R 1310720 172.31.0.18 >> /dev/amrd0s2 >> >> The raid array is a RAID 1 volume on a dell PERC4 (Dell PE1850) with >> adaptive read ahead and write back caching. >> >> Tom > > I have tried both the settings ideas suggested above but I cannot even > get out of the gate with those. Setting net.inet.tcp.{send,recv}space to > anything higher that 131072 results in ggated bailing with the error: > # ggated -v -a 10.10.0.19 > info: Reading exports file (/etc/gg.exports). > debug: Added 10.10.0.0/24 /dev/amrd1 RW to exports list. > debug: Added 10.10.0.0/24 /dev/amrd3 RW to exports list. > info: Exporting 2 object(s). > error: Cannot open stream socket: No buffer space available. > error: Exiting. > > setting net.inet.tcp.{send,recv}space to 131072 allows me to start > ggated with the default R and S values of 131072; anything higher > results in "no buffer space" errors. At 131072 ggated starts but then I > cannot even open a new connection (like ssh) to the server as the ssh > client bails with "no buffer space available". Did you also set kern.ipc.nmbclusters="32768" in /boot/loader.conf and reboot? It sounds like you did not as this is the exact same problem I came across before adjusting that value. <<SNIP>> > This is on a FreeBSD 6.2-RELENG box i386 SMP using the amr driver (SATA > Raid using LSiMegaRaid. Do you have the cache BBU fitted (Batery Backup Unit) and the array caching set to write back? Also have you tested writing to the array locally without ggate to test the write speed? > > The odd thing is that even after I set the send and recvspace down to > values like 65536, I continue to get the no buffer error when trying to > connect to it remotely again. > I found that the easyest way to fix this was to reboot the system with good values for net.inet.tcp.{send,recv}space.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?46161F51.1070704>