From owner-freebsd-fs@FreeBSD.ORG Thu Oct 16 11:33:18 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3264D833; Thu, 16 Oct 2014 11:33:18 +0000 (UTC) Received: from mail.jrv.org (adsl-70-243-84-11.dsl.austtx.swbell.net [70.243.84.11]) by mx1.freebsd.org (Postfix) with ESMTP id F109D682; Thu, 16 Oct 2014 11:33:17 +0000 (UTC) Received: from localhost (localhost.localdomain [127.0.0.1]) by mail.jrv.org (Postfix) with ESMTP id B28D41A1D02; Thu, 16 Oct 2014 06:25:44 -0500 (CDT) Received: from mail.jrv.org ([127.0.0.1]) by localhost (zimbra64.housenet.jrv [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id QhW2_ibdt3fa; Thu, 16 Oct 2014 06:25:34 -0500 (CDT) Received: from localhost (localhost.localdomain [127.0.0.1]) by mail.jrv.org (Postfix) with ESMTP id D289A1A1CFA; Thu, 16 Oct 2014 06:25:34 -0500 (CDT) X-Virus-Scanned: amavisd-new at zimbra64.housenet.jrv Received: from mail.jrv.org ([127.0.0.1]) by localhost (zimbra64.housenet.jrv [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id Mt5NHfUP045b; Thu, 16 Oct 2014 06:25:34 -0500 (CDT) Received: from [192.168.138.128] (BMX.housenet.jrv [192.168.3.140]) by mail.jrv.org (Postfix) with ESMTPSA id AB7B61A1CF7; Thu, 16 Oct 2014 06:25:34 -0500 (CDT) Message-ID: <543FAB3C.4090503@jrv.org> Date: Thu, 16 Oct 2014 06:25:48 -0500 From: "James R. Van Artsdalen" User-Agent: Mozilla/5.0 (Windows NT 5.0; rv:12.0) Gecko/20120428 Thunderbird/12.0.1 MIME-Version: 1.0 Subject: Re: zfs recv hangs in kmem arena References: <54250AE9.6070609@jrv.org> In-Reply-To: <54250AE9.6070609@jrv.org> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org, current@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Oct 2014 11:33:18 -0000 The zfs recv / kmem arena hang happens with -CURRENT as well as 10-STABLE, on two different systems, with 16GB or 32GB of RAM, from memstick or normal multi-user environments, Hangs usually seem to hapeen 1TB to 3TB in, but last night one run hung after only 4.35MB. On 9/26/2014 1:42 AM, James R. Van Artsdalen wrote: > FreeBSD BLACKIE.housenet.jrv 10.1-BETA2 FreeBSD 10.1-BETA2 #2 r272070M: > Wed Sep 24 17:36:56 CDT 2014 > james@BLACKIE.housenet.jrv:/usr/obj/usr/src/sys/GENERIC amd64 > > With current STABLE10 I am unable to replicate a ZFS pool using zfs > send/recv without zfs hanging in state "kmem arena", within the first > 4TB or so (of a 23TB Pool). > > The most recent attempt used this command line > > SUPERTEX:/root# zfs send -R BIGTEX/UNIX@syssnap | ssh BLACKIE zfs recv > -duvF BIGTOX > > though local replications fail in kmem arena too. > > The two machines I've been attempting this on have 16BG and 32GB of RAM > each and are otherwise idle. > > Any suggestions on how to get around, or investigate, "kmem arena"? > > # top > last pid: 3272; load averages: 0.22, 0.22, 0.23 up > 0+08:25:02 01:32:07 > 34 processes: 1 running, 33 sleeping > CPU: 0.0% user, 0.0% nice, 0.1% system, 0.0% interrupt, 99.9% idle > Mem: 21M Active, 82M Inact, 15G Wired, 28M Cache, 450M Free > ARC: 12G Total, 24M MFU, 12G MRU, 23M Anon, 216M Header, 47M Other > Swap: 16G Total, 16G Free > > PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU > COMMAND > 1173 root 1 52 0 86476K 7780K select 0 124:33 0.00% sshd > 1176 root 1 46 0 87276K 47732K kmem a 3 48:36 0.00% zfs > 968 root 32 20 0 12344K 1888K rpcsvc 0 0:13 0.00% nfsd > 1009 root 1 20 0 25452K 2864K select 3 0:01 0.00% ntpd > ...