From owner-freebsd-stable@FreeBSD.ORG Thu Apr 5 15:26:12 2007 Return-Path: X-Original-To: freebsd-stable@freebsd.org Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id DDEC616A401 for ; Thu, 5 Apr 2007 15:26:12 +0000 (UTC) (envelope-from sven@dmv.com) Received: from smtp-gw-cl-c.dmv.com (smtp-gw-cl-c.dmv.com [216.240.97.41]) by mx1.freebsd.org (Postfix) with ESMTP id A364C13C45D for ; Thu, 5 Apr 2007 15:26:12 +0000 (UTC) (envelope-from sven@dmv.com) Received: from mail-gw-cl-b.dmv.com (mail-gw-cl-b.dmv.com [216.240.97.39]) by smtp-gw-cl-c.dmv.com (8.12.10/8.12.10) with ESMTP id l35Ewr47097641 for ; Thu, 5 Apr 2007 10:58:53 -0400 (EDT) (envelope-from sven@dmv.com) Received: from [216.240.97.46] (lanshark.dmv.com [216.240.97.46]) by mail-gw-cl-b.dmv.com (8.12.9/8.12.9) with ESMTP id l35EwpPC009261 for ; Thu, 5 Apr 2007 10:58:53 -0400 (EDT) (envelope-from sven@dmv.com) From: Sven Willenberger To: freebsd-stable@freebsd.org Content-Type: text/plain Date: Thu, 05 Apr 2007 10:58:56 -0400 Message-Id: <1175785136.8280.10.camel@lanshark.dmv.com> Mime-Version: 1.0 X-Mailer: Evolution 2.8.3 Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.39 X-Scanned-By: MIMEDefang 2.48 on 216.240.97.39 Subject: ggate + gmirror write performance woes X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 Apr 2007 15:26:13 -0000 I am trying to set up a HA type system involving two identical boxes and have gone through the following to set up the systems: Slave server: ggated -R 196608 -S 196608 (exporting /dev/amrd1 ) net.inet.tcp.sendspace: 65536 net.inet.tcp.recvspace: 131072 Master server: ggatec create -u 0 -R 196608 -S 196608 -o rw [slaveip] /dev/amrd1 net.inet.tcp.sendspace: 131072 net.inet.tcp.recvspace: 65536 #gmirror status Name Status Components mirror/gm0 COMPLETE amrd1s1 ggate0s1 the two servers are connected to each other via their 2ndary physical gigE interfaces using cat6 crossover cable. (Netperf shows 890 Mbps at 95% confidence). softupdates are enable on gm0 (though this does not affect the results). The results: /usr/bin/time -h cp testfile64M /data1 28.62s real 0.00s user 0.16s sys and this is very consistent ... about 3 MB/s over repeated runs dd if=/dev/zero of=/data1/testfile32M2 bs=32k count=1024 1024+0 records in 1024+0 records out 33554432 bytes transferred in 16.122641 secs (2081199 bytes/sec) What else can I tune here to make this functional? If I increase recvspace and sendspace much beyond those numbers, ggated will not start claiming to not have enough buffer space. Sven