From owner-freebsd-stable@FreeBSD.ORG Mon Mar 12 18:33:09 2012 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id B8D0E1065670 for ; Mon, 12 Mar 2012 18:33:09 +0000 (UTC) (envelope-from to.my.trociny@gmail.com) Received: from mail-bk0-f54.google.com (mail-bk0-f54.google.com [209.85.214.54]) by mx1.freebsd.org (Postfix) with ESMTP id 3C58B8FC0C for ; Mon, 12 Mar 2012 18:33:08 +0000 (UTC) Received: by bkcjc3 with SMTP id jc3so3987643bkc.13 for ; Mon, 12 Mar 2012 11:33:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:to:cc:subject:references:x-comment-to:date:in-reply-to :message-id:user-agent:mime-version:content-type; bh=IPpi4JsHhk1P50mZgEOHk7KXoE0uYTb88FEfGQlD2J8=; b=Y+2I6WLcivTFJac2Oe60lyUKfvLd8tJ/aRPyeU5VfLtCDXIhfa5SVdoUkKxWvgc2fV cGza8Uav7QZvRwaJqPLx98IQiTe+1RmqifphvRzNgj50pStrY8wlkDdiMuahxn9E5OzK HNBW76unIlF+6sdfvC09YL+E9/KwkJFzMhTJx0WY+QGTyebWaMOopSg8blG4ToUKVKId Sws82JQy9FcmqlBr+lfGvLy0pDSJPt5WZB5X3RaGMHRVP+f5zCckfidxbvcB3biFK+hp Ko6fNp09cQzNkYqieZTMqme+4QVFsT1i1yE1TfWZN9yOUiONRTfOIMAiAlzEHdqAefbU OFkQ== Received: by 10.204.136.220 with SMTP id s28mr5354119bkt.94.1331577187992; Mon, 12 Mar 2012 11:33:07 -0700 (PDT) Received: from localhost ([95.69.173.122]) by mx.google.com with ESMTPS id f5sm27111549bke.9.2012.03.12.11.33.05 (version=TLSv1/SSLv3 cipher=OTHER); Mon, 12 Mar 2012 11:33:06 -0700 (PDT) From: Mikolaj Golub To: Phil Regnauld References: <20120311185457.GB1684@macbook.bluepipe.net> <861uoyvpzh.fsf@kopusha.home.net> <20120311220911.GD1684@macbook.bluepipe.net> <20120312143127.GM12975@macbook.bluepipe.net> X-Comment-To: Phil Regnauld Date: Mon, 12 Mar 2012 20:33:04 +0200 In-Reply-To: <20120312143127.GM12975@macbook.bluepipe.net> (Phil Regnauld's message of "Mon, 12 Mar 2012 15:31:27 +0100") Message-ID: <86k42pu0tb.fsf@kopusha.home.net> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/23.3 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Cc: freebsd-stable@freebsd.org Subject: Re: Issue with hast replication X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Mar 2012 18:33:09 -0000 On Mon, 12 Mar 2012 15:31:27 +0100 Phil Regnauld wrote: PR> Phil Regnauld (regnauld) writes: >> >> 7) ktrace on the destination dd: >> >> fstat(0,{ mode=p--------- ,inode=5,size=16384,blksize=4096 }) = 0 (0x0) >> lseek(0,0x0,SEEK_CUR) ERR#29 'Illegal seek' PR> [...] >> Illegal seek, eh ? Any clues ? >> >> The boxes are identical (HP DL380 G6), though the RAM config is different. >> >> Summary: >> >> - ssh works fine >> - h1 zvol to h2 zvol over ssh fails >> - h1 zvol to h2 /tmp/x over ssh is fine >> - h2 /dev/zero locally to h2 zvol is fine >> - h2 /tmp/x locally to h2 zvol fails at first, but works afterwards... PR> A few more data points: dd from a local zvol to a local zvol on either PR> machine works fine. PR> Using nc instead of ssh, this time it's the sender nc dying: PR> ktrace on the sender: PR> 47704 nc CALL write(0x3,0x7fffffff5450,0x800) PR> 47704 nc RET write -1 errno 32 Broken pipe PR> 47704 nc PSIG SIGPIPE SIG_DFL code=0x10006 PR> truss on the sender: PR> poll({3/POLLIN 0/POLLIN},2,-1) = 2 (0x2) PR> read(3,0x7fffffff5450,2048) ERR#54 'Connection reset by peer' PR> close(3) = 0 (0x0) PR> On tcpdump, I do see the receiver send a FIN when using nc. PR> When using ssh, the sender is sending the FIN. PR> Anything else I can look for ? It looks like in the case of hastd this was send(2) who returned ENOMEM, but it would be good to check. Could you please start synchronization again, ktrace primary worker process when ENOMEM errors are observed and show output here? If it is send(2) who fails then monitoring netstat and network driver statistics might be helpful. Something like netstat -nax netstat -naT netstat -m netstat -nid sysctl -a dev. And may be vmstat -m vmstat -z -- Mikolaj Golub