From owner-freebsd-stable@FreeBSD.ORG Mon Apr 9 16:55:27 2007 Return-Path: X-Original-To: freebsd-stable@freebsd.org Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 0F61C16A40E for ; Mon, 9 Apr 2007 16:55:27 +0000 (UTC) (envelope-from sven@dmv.com) Received: from smtp-gw-cl-d.dmv.com (smtp-gw-cl-d.dmv.com [216.240.97.42]) by mx1.freebsd.org (Postfix) with ESMTP id AD2FC13C4B8 for ; Mon, 9 Apr 2007 16:55:26 +0000 (UTC) (envelope-from sven@dmv.com) Received: from mail-gw-cl-a.dmv.com (mail-gw-cl-a.dmv.com [216.240.97.38]) by smtp-gw-cl-d.dmv.com (8.12.10/8.12.10) with ESMTP id l39GtGp1083105; Mon, 9 Apr 2007 12:55:16 -0400 (EDT) (envelope-from sven@dmv.com) Received: from [216.240.97.46] (lanshark.dmv.com [216.240.97.46]) by mail-gw-cl-a.dmv.com (8.12.9/8.12.9) with ESMTP id l39Gt7uk093246; Mon, 9 Apr 2007 12:55:11 -0400 (EDT) (envelope-from sven@dmv.com) From: Sven Willenberger To: Nikolay Pavlov In-Reply-To: <20070406131817.GA1977@zone3000.net> References: <1175785136.8280.10.camel@lanshark.dmv.com> <20070405160313.GE1230@dkirhlarov.mow.oilspace.com> <46152615.1040602@tomjudge.com> <1175804135.8280.21.camel@lanshark.dmv.com> <20070406131817.GA1977@zone3000.net> Content-Type: text/plain Date: Mon, 09 Apr 2007 12:55:43 -0400 Message-Id: <1176137743.21526.7.camel@lanshark.dmv.com> Mime-Version: 1.0 X-Mailer: Evolution 2.8.3 Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.48 on 216.240.97.42 X-Scanned-By: MIMEDefang 2.48 on 216.240.97.38 Cc: Tom Judge , freebsd-stable@freebsd.org Subject: Re: ggate + gmirror write performance woes X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 09 Apr 2007 16:55:27 -0000 On Fri, 2007-04-06 at 16:18 +0300, Nikolay Pavlov wrote: > On Thursday, 5 April 2007 at 16:15:35 -0400, Sven Willenberger wrote: > > On Thu, 2007-04-05 at 17:38 +0100, Tom Judge wrote: > > > Dmitriy Kirhlarov wrote: > > > > On Thu, Apr 05, 2007 at 10:58:56AM -0400, Sven Willenberger wrote: > > > >> I am trying to set up a HA type system involving two identical boxes and > > > >> have gone through the following to set up the systems: > > > >> > > > >> Slave server: > > > >> ggated -R 196608 -S 196608 > > > >> (exporting /dev/amrd1 ) > > > >> net.inet.tcp.sendspace: 65536 > > > >> net.inet.tcp.recvspace: 131072 > > > > > > > > Try > > > > net.local.stream.recvspace=65535 > > > > net.local.stream.sendspace=65535 > > > > > > > > Also, try increase this sysctls with > > > > net.inet.tcp.rfc1323=1 > > > > > > > > I use it on FreeBSD 5.x with: > > > > net.inet.tcp.sendspace=131072 > > > > net.inet.tcp.recvspace=131072 > > > > net.local.stream.recvspace=65535 > > > > net.local.stream.sendspace=65535 > > > > > > > > ggated -R 1048576 -S 1048576 > > > > ggatec -R 1048576 -S 1048576 > > > > > > > > WBR. > > > > Dmitriy > > > > _______________________________________________ > > > > freebsd-stable@freebsd.org mailing list > > > > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > > > > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" > > > > > > > > > I have seen sustained writes of 30Mb/s using the following configuration: > > > > > > cat /boot/loader.conf > > > kern.ipc.nmbclusters="32768" > > > > > > cat /etc/sysctl.conf > > > net.inet.tcp.sendspace=1048576 > > > net.inet.tcp.recvspace=1048576 > > > > > > Server: > > > /sbin/ggated -S 1310720 -R 1310720 -a 172.31.0.18 /etc/gg.exports > > > > > > Client: > > > /sbin/ggatec create -q 2048 -t 5 -S 1310720 -R 1310720 172.31.0.18 > > > /dev/amrd0s2 > > > > > > The raid array is a RAID 1 volume on a dell PERC4 (Dell PE1850) with > > > adaptive read ahead and write back caching. > > > > > > Tom > > > > I have tried both the settings ideas suggested above but I cannot even > > get out of the gate with those. Setting net.inet.tcp.{send,recv}space to > > anything higher that 131072 results in ggated bailing with the error: > > # ggated -v -a 10.10.0.19 > > info: Reading exports file (/etc/gg.exports). > > debug: Added 10.10.0.0/24 /dev/amrd1 RW to exports list. > > debug: Added 10.10.0.0/24 /dev/amrd3 RW to exports list. > > info: Exporting 2 object(s). > > error: Cannot open stream socket: No buffer space available. > > error: Exiting. > > For values of net.inet.tcp.{send,recv}space more than > 524288 you also need to adjust kern.ipc.maxsockbuf > > Try this configuration for example: > kern.ipc.maxsockbuf=2049152 > net.inet.tcp.recvspace=1024576 > net.inet.tcp.sendspace=1024576 > kern.ipc.maxsockbuf was the issue here; I increased its value and now I no longer get the buffer space error. Furthermore, the write speed issue was also tied to a hardware raid controller issue. After fixing that issue, and with just the following: kern.ipc.maxsockbuf=1048576 net.inet.tcp.sendspace=131072 net.inet.tcp.recvspace=131072 I can start ggated with -R 262144 -S 262144 as well as the ggatec and see write speeds of 60+MB/s. I may play around with the settings more (and see if any further speed improvements occur), but this is quite acceptable at this point. (For the record nmbclusters is set to 32768). The next part of the project will be writing the freevrrp failover scripts to deal with I/O locking issues that will happen if the ggated server fails, etc.