From owner-freebsd-stable@FreeBSD.ORG Thu Apr 5 20:15:35 2007 Return-Path: X-Original-To: freebsd-stable@freebsd.org Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 6051A16A403 for ; Thu, 5 Apr 2007 20:15:35 +0000 (UTC) (envelope-from sven@dmv.com) Received: from smtp-gw-cl-c.dmv.com (smtp-gw-cl-c.dmv.com [216.240.97.41]) by mx1.freebsd.org (Postfix) with ESMTP id 1B8F213C458 for ; Thu, 5 Apr 2007 20:15:34 +0000 (UTC) (envelope-from sven@dmv.com) Received: from mail-gw-cl-a.dmv.com (mail-gw-cl-a.dmv.com [216.240.97.38]) by smtp-gw-cl-c.dmv.com (8.12.10/8.12.10) with ESMTP id l35KFS47007891; Thu, 5 Apr 2007 16:15:28 -0400 (EDT) (envelope-from sven@dmv.com) Received: from [216.240.97.46] (lanshark.dmv.com [216.240.97.46]) by mail-gw-cl-a.dmv.com (8.12.9/8.12.9) with ESMTP id l35KFSuk017447; Thu, 5 Apr 2007 16:15:28 -0400 (EDT) (envelope-from sven@dmv.com) From: Sven Willenberger To: Tom Judge In-Reply-To: <46152615.1040602@tomjudge.com> References: <1175785136.8280.10.camel@lanshark.dmv.com> <20070405160313.GE1230@dkirhlarov.mow.oilspace.com> <46152615.1040602@tomjudge.com> Content-Type: text/plain Date: Thu, 05 Apr 2007 16:15:35 -0400 Message-Id: <1175804135.8280.21.camel@lanshark.dmv.com> Mime-Version: 1.0 X-Mailer: Evolution 2.8.3 Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.39 X-Scanned-By: MIMEDefang 2.48 on 216.240.97.38 Cc: freebsd-stable@freebsd.org Subject: Re: ggate + gmirror write performance woes X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 Apr 2007 20:15:35 -0000 On Thu, 2007-04-05 at 17:38 +0100, Tom Judge wrote: > Dmitriy Kirhlarov wrote: > > On Thu, Apr 05, 2007 at 10:58:56AM -0400, Sven Willenberger wrote: > >> I am trying to set up a HA type system involving two identical boxes and > >> have gone through the following to set up the systems: > >> > >> Slave server: > >> ggated -R 196608 -S 196608 > >> (exporting /dev/amrd1 ) > >> net.inet.tcp.sendspace: 65536 > >> net.inet.tcp.recvspace: 131072 > > > > Try > > net.local.stream.recvspace=65535 > > net.local.stream.sendspace=65535 > > > > Also, try increase this sysctls with > > net.inet.tcp.rfc1323=1 > > > > I use it on FreeBSD 5.x with: > > net.inet.tcp.sendspace=131072 > > net.inet.tcp.recvspace=131072 > > net.local.stream.recvspace=65535 > > net.local.stream.sendspace=65535 > > > > ggated -R 1048576 -S 1048576 > > ggatec -R 1048576 -S 1048576 > > > > WBR. > > Dmitriy > > _______________________________________________ > > freebsd-stable@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" > > > I have seen sustained writes of 30Mb/s using the following configuration: > > cat /boot/loader.conf > kern.ipc.nmbclusters="32768" > > cat /etc/sysctl.conf > net.inet.tcp.sendspace=1048576 > net.inet.tcp.recvspace=1048576 > > Server: > /sbin/ggated -S 1310720 -R 1310720 -a 172.31.0.18 /etc/gg.exports > > Client: > /sbin/ggatec create -q 2048 -t 5 -S 1310720 -R 1310720 172.31.0.18 > /dev/amrd0s2 > > The raid array is a RAID 1 volume on a dell PERC4 (Dell PE1850) with > adaptive read ahead and write back caching. > > Tom I have tried both the settings ideas suggested above but I cannot even get out of the gate with those. Setting net.inet.tcp.{send,recv}space to anything higher that 131072 results in ggated bailing with the error: # ggated -v -a 10.10.0.19 info: Reading exports file (/etc/gg.exports). debug: Added 10.10.0.0/24 /dev/amrd1 RW to exports list. debug: Added 10.10.0.0/24 /dev/amrd3 RW to exports list. info: Exporting 2 object(s). error: Cannot open stream socket: No buffer space available. error: Exiting. setting net.inet.tcp.{send,recv}space to 131072 allows me to start ggated with the default R and S values of 131072; anything higher results in "no buffer space" errors. At 131072 ggated starts but then I cannot even open a new connection (like ssh) to the server as the ssh client bails with "no buffer space available". more information: # netstat -m 514/641/1155 mbufs in use (current/cache/total) 512/284/796/32768 mbuf clusters in use (current/cache/total/max) 512/256 mbuf+clusters out of packet secondary zone in use (current/cache) 0/0/0/0 4k (page size) jumbo clusters in use (current/cache/total/max) 0/0/0/0 9k jumbo clusters in use (current/cache/total/max) 0/0/0/0 16k jumbo clusters in use (current/cache/total/max) 1152K/728K/1880K bytes allocated to network (current/cache/total) 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 0/4/6656 sfbufs in use (current/peak/max) 0 requests for sfbufs denied 0 requests for sfbufs delayed 0 requests for I/O initiated by sendfile 0 calls to protocol drain routines This is on a FreeBSD 6.2-RELENG box i386 SMP using the amr driver (SATA Raid using LSiMegaRaid. The odd thing is that even after I set the send and recvspace down to values like 65536, I continue to get the no buffer error when trying to connect to it remotely again. Sven