Date: Mon, 09 Apr 2007 12:55:43 -0400 From: Sven Willenberger <sven@dmv.com> To: Nikolay Pavlov <quetzal@zone3000.net> Cc: Tom Judge <tom@tomjudge.com>, freebsd-stable@freebsd.org Subject: Re: ggate + gmirror write performance woes Message-ID: <1176137743.21526.7.camel@lanshark.dmv.com> In-Reply-To: <20070406131817.GA1977@zone3000.net> References: <1175785136.8280.10.camel@lanshark.dmv.com> <20070405160313.GE1230@dkirhlarov.mow.oilspace.com> <46152615.1040602@tomjudge.com> <1175804135.8280.21.camel@lanshark.dmv.com> <20070406131817.GA1977@zone3000.net>
next in thread | previous in thread | raw e-mail | index | archive | help
On Fri, 2007-04-06 at 16:18 +0300, Nikolay Pavlov wrote: > On Thursday, 5 April 2007 at 16:15:35 -0400, Sven Willenberger wrote: > > On Thu, 2007-04-05 at 17:38 +0100, Tom Judge wrote: > > > Dmitriy Kirhlarov wrote: > > > > On Thu, Apr 05, 2007 at 10:58:56AM -0400, Sven Willenberger wrote: > > > >> I am trying to set up a HA type system involving two identical boxes and > > > >> have gone through the following to set up the systems: > > > >> > > > >> Slave server: > > > >> ggated -R 196608 -S 196608 > > > >> (exporting /dev/amrd1 ) > > > >> net.inet.tcp.sendspace: 65536 > > > >> net.inet.tcp.recvspace: 131072 > > > > > > > > Try > > > > net.local.stream.recvspace=65535 > > > > net.local.stream.sendspace=65535 > > > > > > > > Also, try increase this sysctls with > > > > net.inet.tcp.rfc1323=1 > > > > > > > > I use it on FreeBSD 5.x with: > > > > net.inet.tcp.sendspace=131072 > > > > net.inet.tcp.recvspace=131072 > > > > net.local.stream.recvspace=65535 > > > > net.local.stream.sendspace=65535 > > > > > > > > ggated -R 1048576 -S 1048576 > > > > ggatec -R 1048576 -S 1048576 > > > > > > > > WBR. > > > > Dmitriy > > > > _______________________________________________ > > > > freebsd-stable@freebsd.org mailing list > > > > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > > > > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" > > > > > > > > > I have seen sustained writes of 30Mb/s using the following configuration: > > > > > > cat /boot/loader.conf > > > kern.ipc.nmbclusters="32768" > > > > > > cat /etc/sysctl.conf > > > net.inet.tcp.sendspace=1048576 > > > net.inet.tcp.recvspace=1048576 > > > > > > Server: > > > /sbin/ggated -S 1310720 -R 1310720 -a 172.31.0.18 /etc/gg.exports > > > > > > Client: > > > /sbin/ggatec create -q 2048 -t 5 -S 1310720 -R 1310720 172.31.0.18 > > > /dev/amrd0s2 > > > > > > The raid array is a RAID 1 volume on a dell PERC4 (Dell PE1850) with > > > adaptive read ahead and write back caching. > > > > > > Tom > > > > I have tried both the settings ideas suggested above but I cannot even > > get out of the gate with those. Setting net.inet.tcp.{send,recv}space to > > anything higher that 131072 results in ggated bailing with the error: > > # ggated -v -a 10.10.0.19 > > info: Reading exports file (/etc/gg.exports). > > debug: Added 10.10.0.0/24 /dev/amrd1 RW to exports list. > > debug: Added 10.10.0.0/24 /dev/amrd3 RW to exports list. > > info: Exporting 2 object(s). > > error: Cannot open stream socket: No buffer space available. > > error: Exiting. > > For values of net.inet.tcp.{send,recv}space more than > 524288 you also need to adjust kern.ipc.maxsockbuf > > Try this configuration for example: > kern.ipc.maxsockbuf=2049152 > net.inet.tcp.recvspace=1024576 > net.inet.tcp.sendspace=1024576 > kern.ipc.maxsockbuf was the issue here; I increased its value and now I no longer get the buffer space error. Furthermore, the write speed issue was also tied to a hardware raid controller issue. After fixing that issue, and with just the following: kern.ipc.maxsockbuf=1048576 net.inet.tcp.sendspace=131072 net.inet.tcp.recvspace=131072 I can start ggated with -R 262144 -S 262144 as well as the ggatec and see write speeds of 60+MB/s. I may play around with the settings more (and see if any further speed improvements occur), but this is quite acceptable at this point. (For the record nmbclusters is set to 32768). The next part of the project will be writing the freevrrp failover scripts to deal with I/O locking issues that will happen if the ggated server fails, etc.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1176137743.21526.7.camel>