From owner-freebsd-fs@FreeBSD.ORG Sun Oct 14 00:16:18 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id C09F84F1 for ; Sun, 14 Oct 2012 00:16:18 +0000 (UTC) (envelope-from ivoras@gmail.com) Received: from mail-vc0-f182.google.com (mail-vc0-f182.google.com [209.85.220.182]) by mx1.freebsd.org (Postfix) with ESMTP id 6D6D88FC08 for ; Sun, 14 Oct 2012 00:16:18 +0000 (UTC) Received: by mail-vc0-f182.google.com with SMTP id fw7so6029720vcb.13 for ; Sat, 13 Oct 2012 17:16:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:from:date :x-google-sender-auth:message-id:subject:to:cc:content-type; bh=8ZPNwFWSu4oLIfM2pk34cMcz1aqJ9zBgeC8cV5tXv0M=; b=sD+dV5McU2+UlET0LJxxWD0PKyucorFFgHC+Ab5MOftC9qZE1+GHjom9EOV0dVkstL /Iiu0J9ylltLL/gEu4sZqt3b9Pbi/Mn2+DZNBo1rNCvRvTizqX2KZ9CtyDWj0BKfYjGI m6HzfuBfO79T+IFcJ/5k7oqgPb/YfNg4+PoVvQ1KWTxn3zpOWv/6MBrTkqBdGPPz8f6w Cu8OgKAPP47sA8LplVSBaaQ6ihHIDjs03SQvXb+1nfWB1/YESOr4kDl02aZDUYXm0+l0 rwWg4W+TmgFY65/ckR+Fc6TRKVBUAq/rhw2+217sTfTpPsoALidQMETrSYUdtjWwgBfp eqHA== Received: by 10.52.139.136 with SMTP id qy8mr3803800vdb.39.1350173777557; Sat, 13 Oct 2012 17:16:17 -0700 (PDT) MIME-Version: 1.0 Sender: ivoras@gmail.com Received: by 10.59.0.37 with HTTP; Sat, 13 Oct 2012 17:15:37 -0700 (PDT) In-Reply-To: <1979889899.2197790.1350164598183.JavaMail.root@erie.cs.uoguelph.ca> References: <1979889899.2197790.1350164598183.JavaMail.root@erie.cs.uoguelph.ca> From: Ivan Voras Date: Sun, 14 Oct 2012 02:15:37 +0200 X-Google-Sender-Auth: T7c0cVYz-7WGiAu0hTkhqJ-87vg Message-ID: Subject: Re: NFS server bottlenecks To: Rick Macklem Content-Type: text/plain; charset=UTF-8 Cc: FS List X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 14 Oct 2012 00:16:18 -0000 On 13 October 2012 23:43, Rick Macklem wrote: > If, as you proposed, use separate LRU lists for each hash bucket, then > how do you know if the least recently used for one hash backet isn't > much more recently used than the least recently used for another hash > bucket? (The hash code is using xid, which might be about the same for > different clients at the same time.) I'm not that familiar with the code to judge: would that be a problem, other than a (seemingly slight) loss of efficiency? Is there any other purpose to the LRU list except to help remove stale entries? I haven't done any real examination of how it works, but looking at the code in: http://fxr.watson.org/fxr/source/fs/nfsserver/nfs_nfsdcache.c#L780 ... I don't see how the LRU property of the list actually helps anything (I.e. - would the correctness of the code be damaged if this was an orfinary list without the LRU property?) > ps: I hope you didn't mind me adding the mailing list. I'd like others to > be able to comment/read the discussion. For the others to catch up, I was proposing this approach to Rick: http://people.freebsd.org/~ivoras/diffs/nfscache_lock.patch (this patch is far from being complete, it's just a sketch of an idea). Basically, I'd like to break the global hash lock into per-bucket locks and to break the global LRU list into per-bucket lists, protected by the same locks. From owner-freebsd-fs@FreeBSD.ORG Sun Oct 14 00:27:40 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id CA34173B; Sun, 14 Oct 2012 00:27:40 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: from mail-lb0-f182.google.com (mail-lb0-f182.google.com [209.85.217.182]) by mx1.freebsd.org (Postfix) with ESMTP id 3918E8FC18; Sun, 14 Oct 2012 00:27:39 +0000 (UTC) Received: by mail-lb0-f182.google.com with SMTP id b5so3399758lbd.13 for ; Sat, 13 Oct 2012 17:27:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:reply-to:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:content-type; bh=Rxc5g4wNO4y2ULjCG1cAmZqW7FuwEG5lV1TUllRIA5w=; b=wAq+knDkyPPgB6zk4zV6/YsLOZeeBYxHxIlJ3Wyd+MMc4t6B1UWHUT1VsBEsxwgG2O 0zSZmxN7tw2zB+lsubupYb9ak9atWbU+iIW0VXzoOq2vGaIX+Ccd5fnpwmveWt+UhsxL csCZzZk/omCtcCtkYcIpnxEYL60pbmze0CANcd25EXFK8jcjKZSvf3svp1UcJ7HpQtrM KKw/lf4JWCLK1CRW3ffAa6IfQyV6AmLe8jAud1WTXr9vHbfsWMbDLVo47Q+99bDMG/Sp y6wcV7veEWdK2rMDvoz7M4EQdRfsYDscaILXrFEAk7bxamml5NHUh6atsSrkkGB3jotj 28EQ== MIME-Version: 1.0 Received: by 10.152.105.103 with SMTP id gl7mr6858057lab.10.1350174457780; Sat, 13 Oct 2012 17:27:37 -0700 (PDT) Sender: asmrookie@gmail.com Received: by 10.112.101.234 with HTTP; Sat, 13 Oct 2012 17:27:37 -0700 (PDT) In-Reply-To: References: <20120829060158.GA38721@x2.osted.lan> <20120831052003.GA91340@x2.osted.lan> <20120905201531.GA54452@x2.osted.lan> <20120917140055.GA9037@x2.osted.lan> Date: Sun, 14 Oct 2012 01:27:37 +0100 X-Google-Sender-Auth: nrt0Lp6hoJ9ZS2YaXxitERXyJxo Message-ID: Subject: Re: MPSAFE VFS -- List of upcoming actions From: Attilio Rao To: FreeBSD FS , freebsd-current@freebsd.org, Peter Holm , =?UTF-8?Q?Gustau_P=C3=A9rez?= , George Neville-Neil , Florian Smeets , bdrewery@freebsd.org Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: attilio@FreeBSD.org List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 14 Oct 2012 00:27:40 -0000 On Fri, Sep 21, 2012 at 1:22 AM, Attilio Rao wrote: > On Wed, Sep 19, 2012 at 3:48 AM, Attilio Rao wrote: >> On Fri, Jul 13, 2012 at 12:18 AM, Attilio Rao wrote: >>> 2012/7/4 Attilio Rao : >>>> 2012/6/29 Attilio Rao : >>>>> As already published several times, according to the following plan: >>>>> http://wiki.freebsd.org/NONMPSAFE_DEORBIT_VFS >>>>> >>>> >>>> I still haven't heard from Vivien or Edward, anyway as NTFS is >>>> basically only used RO these days (also the mount_ntfs code just >>>> permits RO mounting) I stripped all the uncomplete/bogus write support >>>> with the following patch: >>>> http://www.freebsd.org/~attilio/ntfs_remove_write.patch >>>> >>>> This is an attempt to make the code smaller and possibly just focus on >>>> the locking that really matter (as read-only filesystem). >>>> On some points of the patch I'm a bit less sure as we could easily >>>> take into account also write for things like vaccess() arguments, and >>>> make easier to re-add correct write support at some point in the >>>> future, but still force RO, even if the approach used in the patch is >>>> more correct IMHO. >>>> As an added bonus this patch cleans some dirty code in the mount >>>> operation and fixes a bug as vfs_mountedfrom() is called before real >>>> mounting is completed and can still fail. >>> >>> A quick update on this. >>> It looks like NTFS won't be completed for this GSoC thus I seriously >>> need to find an alternative to not loose the NTFS support entirely. >>> >>> I tried to look into the NTFS implementation right now and it is >>> really a poor support. As Peter has also verified, it can deadlock in >>> no-time, it compeltely violates VFS rules, etc. IMHO it deserves a >>> complete rewrite if we would still support in-kernel NTFS. I also >>> tried to look at the NetBSD implementation. Their code is someway >>> similar to our, but they used very complicated (and very dirty) code >>> to do the locking. Even if I don't know well enough NetBSD VFS, I have >>> the impression not all the races are correctly handled. Definitively, >>> not something I would like to port. >>> >>> Considering all that the only viable option would be meaning an >>> userland filesystem implementation. My preferred choice would be to >>> import PUFFS and librefuse on top of it but honestly it requires a lot >>> of time to be completed, time which I don't currently have as in 2 >>> months Giant must be gone by the VFS. >>> >>> I then decided to switch to gnn's rewamp of FUSE patches. You can find >>> his initial e-mail here: >>> http://lists.freebsd.org/pipermail/freebsd-fs/2012-March/013876.html >>> >>> I've precisely got the second version of George's patch and created >>> this dolphin branch: >>> svn://svn.freebsd.org/base/projects/fuse >>> >>> I'm fixing low hanging fruit for the moment (see r238411 for example) >>> and I still have to make a throughful review. >>> However my idea is to commit the support once: >>> - ntfs-3g is well stress-tested and proves to be bug-free >>> - there is no major/big technical issue pending after the reviews >> >> In the last weeks Peter, Florian, Gustau and I have been working in >> stabilizing fuse support. In the specific, Peter has worked hard on >> producing several utilities to nit stress-test fuse and in particular >> ntfs, Florian has improved fuse related ports (as explained later) and >> Gustau has done sparse testing. I feel moderately satisfied by the >> level of stability of fuse now to propose to wider usage, in >> particular given the huge amount of complaints I'm hearing around >> about occasional fuse users. >> >> The final target of the project is to completely import into base the >> content of fusefs-kmod starting from earlier posted patches by George. >> So far, we took care only of importing in the fuse branch the kernel >> part, so that fusefs-kmod userland part is still needed to be >> installed from ports, but I was studying the mount_fusefs licensing >> before to process with the import for the userland bits of it. >> >> The fixing has been happening here: >> svn://svn.freebsd.org/base/projects/fuse/ >> >> which is essentially an HEAD branch + fuse kernel components. In order >> to get fuse, please compile a kernel from this branch with FUSE option >> or simply build and load fuse module. >> Alternatively, a kernel patch that should work with HEAD@240684 is here: >> http://www.freebsd.org/~attilio/fuse_import/fuse_240684.patch >> >> I guess the patch can easilly apply to all FreeBSD branches, really, >> but it is not tested to anything else different then -CURRENT. >> >> As said you still need currently to build fusefs-kmod port. However >> you need these further patches, to be put in the fusefs-kmod/files/ >> directory:: >> http://www.freebsd.org/~attilio/fuse_import/patch-Makefile >> http://www.freebsd.org/~attilio/fuse_import/patch-mount_fusefs__mount_fusefs2.c >> >> They both disable the old kernel building/linking and import new >> functionality to let the new kernel support work well in presence of >> many consumers. >> >> In addition to fusefs-kmod, Bryan and Florian have also updated >> fusefs-lib and fusefs-ntfs ports. For instance, please refer to this >> e-mail: >> http://lists.freebsd.org/pipermail/freebsd-ports/2012-August/077950.html >> >> Even if this work is someway independent by the fusefs-kmod import, I >> warmly suggest to all of you to use their patches (and this what we >> have been testing so far too. > > So, after Bryan and Florian ports update, I've also committed userland > part of fusefs-kmod and now the project branch fully mirrors > functionality of fusefs-kmod. The code in projects/fuse, infact, will > also install mount_fusefs as part of the fuse support. I've committed the FUSE support into base as r241519. Few things I would like to stress out at the present time: - The import of the fusefs in base, along with update of fusefs-ntfs and fusefs-libs ports (which happened by Florian and Brian), will bring us a fairly good ntfs (and possibly smbfs too, which is currently untested, as far as I know) support. In particular, in-kernel ntfs is so fragile that I don't understand how people cannot easily experience deadlocks, memory leaks and panics at the present time. - In -CURRENT, you don't need to install fusefs-kmod anymore. It would be good if a port committer (Florian, Brian?) will update the port in order to forbid install of fusefs-kmod in -CURRENT at the present time. I didn't bump the __FreeBSD_version because it is really unnecessary (port can just detect 10xxxx version and be ok with not installing the port) but once I will MFC it, I will bump the version in order to make the port not installable in STABLE branches - It would be good if someone could give a sweep to the style(9) of the fuse code. I tried to reduce the changes to the gnn's proposed patches at the minimum, thus I didn't fix style before to commit it, but there are several style violations that should be addressed. This could be a work for junior developers willing to take over. - There are still a couple of bugs, but they are infrequent and minor enough that they can be addressed after the committing of the infrastructure. gnn has agreed to take care of them in the long term and I will feed him as soon as possible with the details to reproduce and fix them. I will also help directly with fuse code when I will have available time. Next step for the VFS deorbit project now include the disconnect of non-MPSAFE filesystems, including NTFS and SMBFS that can be now used as FUSE modules instead. Thanks, Attilio -- Peace can only be achieved by understanding - A. Einstein From owner-freebsd-fs@FreeBSD.ORG Sun Oct 14 02:18:29 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 54C92C48; Sun, 14 Oct 2012 02:18:29 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id E4B078FC08; Sun, 14 Oct 2012 02:18:28 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Ap8EAN6MclCDaFvO/2dsb2JhbABFFoV7uhmCIAEBBAEjSwsbDgoCAg0ZAlkGE4d/BgumTJF3gSGKLhWEaYESA5VrgRWPGYMJgT88 X-IronPort-AV: E=Sophos;i="4.80,583,1344225600"; d="scan'208";a="183535966" Received: from erie.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.206]) by esa-jnhn-pri.mail.uoguelph.ca with ESMTP; 13 Oct 2012 22:18:22 -0400 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id 489F0B4037; Sat, 13 Oct 2012 22:18:22 -0400 (EDT) Date: Sat, 13 Oct 2012 22:18:22 -0400 (EDT) From: Rick Macklem To: Ivan Voras Message-ID: <1472578725.2201172.1350181102240.JavaMail.root@erie.cs.uoguelph.ca> In-Reply-To: Subject: Re: NFS server bottlenecks MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.91.201] X-Mailer: Zimbra 6.0.10_GA_2692 (ZimbraWebClient - IE7 (Win)/6.0.10_GA_2692) Cc: FS List X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 14 Oct 2012 02:18:29 -0000 Ivan Voras wrote: > On 13 October 2012 23:43, Rick Macklem wrote: > > > If, as you proposed, use separate LRU lists for each hash bucket, > > then > > how do you know if the least recently used for one hash backet isn't > > much more recently used than the least recently used for another > > hash > > bucket? (The hash code is using xid, which might be about the same > > for > > different clients at the same time.) > > I'm not that familiar with the code to judge: would that be a problem, > other than a (seemingly slight) loss of efficiency? > > Is there any other purpose to the LRU list except to help remove stale > entries? > I haven't done any real examination of how it works, but > looking at the code in: > > http://fxr.watson.org/fxr/source/fs/nfsserver/nfs_nfsdcache.c#L780 > > ... I don't see how the LRU property of the list actually helps > anything (I.e. - would the correctness of the code be damaged if this > was an orfinary list without the LRU property?) The concept behind the DRC is (published in Usenix long ago, the reference is in a comment in the code): - When NFS is run over UDP, the client will wait for a reply from the server with a timeout. When there is a timeout, the client will resend the RPC request. - If the timeout occurs because the server was slow to reply (due to heavy load or ???) or the reply was lost by the network, this retransmit of the RPC request would result in the RPC being re-done on the server. - for idempotent RPCs (like read), this increases load on the server - for non-idempotent RPCs, this can result in corrupted data - The DRC minimizes the likelyhood of this occurring, by caching replies for non-idempotent RPCs, so the server can reply from the cache instead of re-doing the RPC. As such, cached replies need to be cached long enough, so that it is unlikely that the server will be retrying the RPC. Unfortunately, there is no well defined time limit, since retry timeout and network delay varies for different clients. Therefore, the server wants to hold onto the cached reply as long as possible. This means that if you don't replace the least recently used cached reply, you make the DRC less effective. rick > > > ps: I hope you didn't mind me adding the mailing list. I'd like > > others to > > be able to comment/read the discussion. > > For the others to catch up, I was proposing this approach to Rick: > > http://people.freebsd.org/~ivoras/diffs/nfscache_lock.patch > > (this patch is far from being complete, it's just a sketch of an > idea). Basically, I'd like to break the global hash lock into > per-bucket locks and to break the global LRU list into per-bucket > lists, protected by the same locks. From owner-freebsd-fs@FreeBSD.ORG Sun Oct 14 03:43:44 2012 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id B71B7B03; Sun, 14 Oct 2012 03:43:44 +0000 (UTC) (envelope-from mckusick@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [8.8.178.135]) by mx1.freebsd.org (Postfix) with ESMTP id 8779A8FC08; Sun, 14 Oct 2012 03:43:44 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q9E3hilG023973; Sun, 14 Oct 2012 03:43:44 GMT (envelope-from mckusick@freefall.freebsd.org) Received: (from mckusick@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q9E3hi8w023969; Sun, 14 Oct 2012 03:43:44 GMT (envelope-from mckusick) Date: Sun, 14 Oct 2012 03:43:44 GMT Message-Id: <201210140343.q9E3hi8w023969@freefall.freebsd.org> To: mckusick@FreeBSD.org, freebsd-fs@FreeBSD.org, mckusick@FreeBSD.org From: mckusick@FreeBSD.org Subject: Re: kern/172631: [ufs] mksnap_ffs(8) deadlock with journaling and some daemons X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 14 Oct 2012 03:43:44 -0000 Synopsis: [ufs] mksnap_ffs(8) deadlock with journaling and some daemons Responsible-Changed-From-To: freebsd-fs->mckusick Responsible-Changed-By: mckusick Responsible-Changed-When: Sun Oct 14 03:42:41 UTC 2012 Responsible-Changed-Why: I will take responsibility for this one. http://www.freebsd.org/cgi/query-pr.cgi?pr=172631 From owner-freebsd-fs@FreeBSD.ORG Sun Oct 14 09:43:53 2012 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 54479744; Sun, 14 Oct 2012 09:43:53 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 6F18F8FC0A; Sun, 14 Oct 2012 09:43:51 +0000 (UTC) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id MAA28635; Sun, 14 Oct 2012 12:43:50 +0300 (EEST) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1TNKjd-000Cat-Sj; Sun, 14 Oct 2012 12:43:50 +0300 Message-ID: <507A8954.3000702@FreeBSD.org> Date: Sun, 14 Oct 2012 12:43:48 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:16.0) Gecko/20121013 Thunderbird/16.0.1 MIME-Version: 1.0 To: "freebsd-fs@freebsd.org" Subject: potential zfs/vfs trouble in force umount X-Enigmail-Version: 1.4.5 Content-Type: text/plain; charset=X-VIET-VPS Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 14 Oct 2012 09:43:53 -0000 I think that there is the following potentially troublesome scenario. One thread does zil_commit and obtains a znode pointer using zfs_zget. At this point the thread doesn't have any locks on either the znode or its vnode. the only thing that is supposed to keep them around is a reference on the vnode. If a force umount is going on in parallel, the one of the first things it does is calling vflush(FORCECLOSE) (this happens before closing down zil). vflush force-reclaims all vnodes in this case (even when v_usecount > 0). So the znode in question gets destroyed. Later, when the first thread tries to dereference the znode pointer it would crash. -- Andriy Gapon From owner-freebsd-fs@FreeBSD.ORG Sun Oct 14 11:25:12 2012 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 298FC4B2; Sun, 14 Oct 2012 11:25:12 +0000 (UTC) (envelope-from pawel@dawidek.net) Received: from mail.dawidek.net (garage.dawidek.net [91.121.88.72]) by mx1.freebsd.org (Postfix) with ESMTP id DF5B58FC18; Sun, 14 Oct 2012 11:25:11 +0000 (UTC) Received: from localhost (89-73-195-149.dynamic.chello.pl [89.73.195.149]) by mail.dawidek.net (Postfix) with ESMTPSA id 1E0E5118; Sun, 14 Oct 2012 13:23:57 +0200 (CEST) Date: Sun, 14 Oct 2012 13:25:47 +0200 From: Pawel Jakub Dawidek To: Andriy Gapon Subject: Re: potential zfs/vfs trouble in force umount Message-ID: <20121014112546.GH1383@garage.freebsd.pl> References: <507A8954.3000702@FreeBSD.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="FUFe+yI/t+r3nyH4" Content-Disposition: inline In-Reply-To: <507A8954.3000702@FreeBSD.org> X-OS: FreeBSD 10.0-CURRENT amd64 User-Agent: Mutt/1.5.21 (2010-09-15) Cc: "freebsd-fs@freebsd.org" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 14 Oct 2012 11:25:12 -0000 --FUFe+yI/t+r3nyH4 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sun, Oct 14, 2012 at 12:43:48PM +0300, Andriy Gapon wrote: >=20 > I think that there is the following potentially troublesome scenario. > One thread does zil_commit and obtains a znode pointer using zfs_zget. A= t this > point the thread doesn't have any locks on either the znode or its vnode.= the > only thing that is supposed to keep them around is a reference on the vno= de. > If a force umount is going on in parallel, the one of the first things it= does > is calling vflush(FORCECLOSE) (this happens before closing down zil). vf= lush > force-reclaims all vnodes in this case (even when v_usecount > 0). So th= e znode > in question gets destroyed. > Later, when the first thread tries to dereference the znode pointer it wo= uld crash. The z_teardown_lock lock is held for reading for every VOP and zfs_umount() obtains this lock for writing before calling vflush(FORCECLOSE) and sets z_unmounted to true. This in turn will make every new VOP to return with EIO. This ensures that no VOP is in-progress when vflush() is called. --=20 Pawel Jakub Dawidek http://www.wheelsystems.com FreeBSD committer http://www.FreeBSD.org Am I Evil? Yes, I Am! http://tupytaj.pl --FUFe+yI/t+r3nyH4 Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) iEYEARECAAYFAlB6oToACgkQForvXbEpPzTU9wCg1okzChpjWV6PtGUdtO2pBXNO drAAoPSbNK0Kf2OXuR78QeTmmzKAw2NS =ibzW -----END PGP SIGNATURE----- --FUFe+yI/t+r3nyH4-- From owner-freebsd-fs@FreeBSD.ORG Sun Oct 14 12:12:46 2012 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 4D8F4C8D; Sun, 14 Oct 2012 12:12:46 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 6A8CE8FC14; Sun, 14 Oct 2012 12:12:45 +0000 (UTC) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id PAA29371; Sun, 14 Oct 2012 15:12:43 +0300 (EEST) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1TNN3j-000Clz-9R; Sun, 14 Oct 2012 15:12:43 +0300 Message-ID: <507AAC38.3000709@FreeBSD.org> Date: Sun, 14 Oct 2012 15:12:40 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:16.0) Gecko/20121013 Thunderbird/16.0.1 MIME-Version: 1.0 To: Pawel Jakub Dawidek Subject: Re: potential zfs/vfs trouble in force umount References: <507A8954.3000702@FreeBSD.org> <20121014112546.GH1383@garage.freebsd.pl> In-Reply-To: <20121014112546.GH1383@garage.freebsd.pl> X-Enigmail-Version: 1.4.5 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: "freebsd-fs@freebsd.org" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 14 Oct 2012 12:12:46 -0000 on 14/10/2012 14:25 Pawel Jakub Dawidek said the following: > On Sun, Oct 14, 2012 at 12:43:48PM +0300, Andriy Gapon wrote: >> >> I think that there is the following potentially troublesome scenario. One >> thread does zil_commit and obtains a znode pointer using zfs_zget. At >> this point the thread doesn't have any locks on either the znode or its >> vnode. the only thing that is supposed to keep them around is a >> reference on the vnode. If a force umount is going on in parallel, the >> one of the first things it does is calling vflush(FORCECLOSE) (this >> happens before closing down zil). vflush force-reclaims all vnodes in >> this case (even when v_usecount > 0). So the znode in question gets >> destroyed. Later, when the first thread tries to dereference the znode >> pointer it would crash. > > The z_teardown_lock lock is held for reading for every VOP and zfs_umount() > obtains this lock for writing before calling vflush(FORCECLOSE) and sets > z_unmounted to true. This in turn will make every new VOP to return with > EIO. This ensures that no VOP is in-progress when vflush() is called. > What was/is not clear to me is whether zil operations are always called under z_teardown_lock (aka ZFS_ENTER)... -- Andriy Gapon From owner-freebsd-fs@FreeBSD.ORG Sun Oct 14 12:24:29 2012 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 5E958F43; Sun, 14 Oct 2012 12:24:29 +0000 (UTC) (envelope-from pawel@dawidek.net) Received: from mail.dawidek.net (garage.dawidek.net [91.121.88.72]) by mx1.freebsd.org (Postfix) with ESMTP id 1D41F8FC0C; Sun, 14 Oct 2012 12:24:29 +0000 (UTC) Received: from localhost (89-73-195-149.dynamic.chello.pl [89.73.195.149]) by mail.dawidek.net (Postfix) with ESMTPSA id 3D92513F; Sun, 14 Oct 2012 14:23:14 +0200 (CEST) Date: Sun, 14 Oct 2012 14:25:04 +0200 From: Pawel Jakub Dawidek To: Andriy Gapon Subject: Re: potential zfs/vfs trouble in force umount Message-ID: <20121014122503.GS1383@garage.freebsd.pl> References: <507A8954.3000702@FreeBSD.org> <20121014112546.GH1383@garage.freebsd.pl> <507AAC38.3000709@FreeBSD.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="D5tFrmRBv7YOLFOK" Content-Disposition: inline In-Reply-To: <507AAC38.3000709@FreeBSD.org> X-OS: FreeBSD 10.0-CURRENT amd64 User-Agent: Mutt/1.5.21 (2010-09-15) Cc: "freebsd-fs@freebsd.org" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 14 Oct 2012 12:24:29 -0000 --D5tFrmRBv7YOLFOK Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sun, Oct 14, 2012 at 03:12:40PM +0300, Andriy Gapon wrote: > on 14/10/2012 14:25 Pawel Jakub Dawidek said the following: > > On Sun, Oct 14, 2012 at 12:43:48PM +0300, Andriy Gapon wrote: > >>=20 > >> I think that there is the following potentially troublesome scenario. = One > >> thread does zil_commit and obtains a znode pointer using zfs_zget. At > >> this point the thread doesn't have any locks on either the znode or its > >> vnode. the only thing that is supposed to keep them around is a > >> reference on the vnode. If a force umount is going on in parallel, the > >> one of the first things it does is calling vflush(FORCECLOSE) (this > >> happens before closing down zil). vflush force-reclaims all vnodes in > >> this case (even when v_usecount > 0). So the znode in question gets > >> destroyed. Later, when the first thread tries to dereference the znode > >> pointer it would crash. > >=20 > > The z_teardown_lock lock is held for reading for every VOP and zfs_umou= nt() > > obtains this lock for writing before calling vflush(FORCECLOSE) and sets > > z_unmounted to true. This in turn will make every new VOP to return with > > EIO. This ensures that no VOP is in-progress when vflush() is called. > >=20 >=20 > What was/is not clear to me is whether zil operations are always called u= nder > z_teardown_lock (aka ZFS_ENTER)... All VOP start from acquiring this lock. If there is a zil_commit() you are talking about which is not part of a VOP, then it should be investigated separately. --=20 Pawel Jakub Dawidek http://www.wheelsystems.com FreeBSD committer http://www.FreeBSD.org Am I Evil? Yes, I Am! http://tupytaj.pl --D5tFrmRBv7YOLFOK Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) iEYEARECAAYFAlB6rx8ACgkQForvXbEpPzSorACg9C7x+wDVuAlLNqTt/DzmS3rF +6IAni4i8L4eD+cEmSuw0X8FA7MIDmnB =52b2 -----END PGP SIGNATURE----- --D5tFrmRBv7YOLFOK-- From owner-freebsd-fs@FreeBSD.ORG Sun Oct 14 13:37:26 2012 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 40242D72; Sun, 14 Oct 2012 13:37:26 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 57EE48FC14; Sun, 14 Oct 2012 13:37:24 +0000 (UTC) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id QAA29690; Sun, 14 Oct 2012 16:37:23 +0300 (EEST) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1TNONe-000CrP-Cq; Sun, 14 Oct 2012 16:37:23 +0300 Message-ID: <507AC00E.7060407@FreeBSD.org> Date: Sun, 14 Oct 2012 16:37:18 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:16.0) Gecko/20121013 Thunderbird/16.0.1 MIME-Version: 1.0 To: Pawel Jakub Dawidek Subject: Re: potential zfs/vfs trouble in force umount References: <507A8954.3000702@FreeBSD.org> <20121014112546.GH1383@garage.freebsd.pl> <507AAC38.3000709@FreeBSD.org> In-Reply-To: <507AAC38.3000709@FreeBSD.org> X-Enigmail-Version: 1.4.5 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: "freebsd-fs@freebsd.org" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 14 Oct 2012 13:37:26 -0000 on 14/10/2012 15:12 Andriy Gapon said the following: > on 14/10/2012 14:25 Pawel Jakub Dawidek said the following: >> On Sun, Oct 14, 2012 at 12:43:48PM +0300, Andriy Gapon wrote: >>> >>> I think that there is the following potentially troublesome scenario. One >>> thread does zil_commit and obtains a znode pointer using zfs_zget. At >>> this point the thread doesn't have any locks on either the znode or its >>> vnode. the only thing that is supposed to keep them around is a >>> reference on the vnode. If a force umount is going on in parallel, the >>> one of the first things it does is calling vflush(FORCECLOSE) (this >>> happens before closing down zil). vflush force-reclaims all vnodes in >>> this case (even when v_usecount > 0). So the znode in question gets >>> destroyed. Later, when the first thread tries to dereference the znode >>> pointer it would crash. >> >> The z_teardown_lock lock is held for reading for every VOP and zfs_umount() >> obtains this lock for writing before calling vflush(FORCECLOSE) and sets >> z_unmounted to true. This in turn will make every new VOP to return with >> EIO. This ensures that no VOP is in-progress when vflush() is called. >> > > What was/is not clear to me is whether zil operations are always called under > z_teardown_lock (aka ZFS_ENTER)... > OK, this appears to be true. The only special case is zil_commit being called from zil_close in umount. Because zil_close is called after vflush all the znodes would already be properly disposed of. If there is anything in zil at that point, then I am not sure what zfs_zget would do... Return an error and some the records would remain uncommitted/lost? Or create znodes (and vnodes) despite the unmount going on? Something else?.. Umm, looks like there would be an error in insmntque and so zfs_zget would return an error. Not sure if losing some zil entries (non-sync operations) is something to worry about for the forceful unmounting, though. -- Andriy Gapon From owner-freebsd-fs@FreeBSD.ORG Sun Oct 14 23:33:17 2012 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 54C46115; Sun, 14 Oct 2012 23:33:17 +0000 (UTC) (envelope-from linimon@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [8.8.178.135]) by mx1.freebsd.org (Postfix) with ESMTP id 24BAE8FC0A; Sun, 14 Oct 2012 23:33:17 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q9ENXHSl037263; Sun, 14 Oct 2012 23:33:17 GMT (envelope-from linimon@freefall.freebsd.org) Received: (from linimon@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q9ENXHpI037259; Sun, 14 Oct 2012 23:33:17 GMT (envelope-from linimon) Date: Sun, 14 Oct 2012 23:33:17 GMT Message-Id: <201210142333.q9ENXHpI037259@freefall.freebsd.org> To: linimon@FreeBSD.org, freebsd-bugs@FreeBSD.org, freebsd-fs@FreeBSD.org From: linimon@FreeBSD.org Subject: Re: kern/171761: [zfs] [patch] Small patch to (temporarily) remove -r from zfs send usage and zfs(8) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 14 Oct 2012 23:33:17 -0000 Old Synopsis: Small patch to (temporarily) remove -r from zfs send usage and zfs(8) New Synopsis: [zfs] [patch] Small patch to (temporarily) remove -r from zfs send usage and zfs(8) Responsible-Changed-From-To: freebsd-bugs->freebsd-fs Responsible-Changed-By: linimon Responsible-Changed-When: Sun Oct 14 23:32:41 UTC 2012 Responsible-Changed-Why: Over to maintainer(s). http://www.freebsd.org/cgi/query-pr.cgi?pr=171761 From owner-freebsd-fs@FreeBSD.ORG Mon Oct 15 07:28:50 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 68FFF918 for ; Mon, 15 Oct 2012 07:28:50 +0000 (UTC) (envelope-from ndenev@gmail.com) Received: from mail-we0-f182.google.com (mail-we0-f182.google.com [74.125.82.182]) by mx1.freebsd.org (Postfix) with ESMTP id DEF448FC12 for ; Mon, 15 Oct 2012 07:28:49 +0000 (UTC) Received: by mail-we0-f182.google.com with SMTP id x43so3629348wey.13 for ; Mon, 15 Oct 2012 00:28:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=content-type:mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to:x-mailer; bh=Hegv+s9zQXN//HYxNQXnScT0ZBLZPkDnF3DkiwUSrtc=; b=NEjVV9ZRc+lj8gvb1YuQB6yw1Ujkq2eYqVUEGGPSkjHmwXhzntihgV5Xaz+EORMToV KWKBdKSiU6TFlveNx1+q6xOps5MiHS+UzriKZSPV8NJuaEfFy6f/q3MmU4kdseJobZxY M+wk0clQ628iX6zb/jLlcy4U+5K4jfAZZ9U4COsvAVBB/hbFDXK92pXsvxyiuSklVFgA wSTqpoeDaFCOJhPnUviXGJ4p47dX2HAbVxZzjQV4B/2NIJDTfmvuuQzOL8diS2424yUj KBuet6WxgkdbxmG10VEoTV5LHaXEWieGoQrBocCIwyaWfxLFPPBmACDh5B3CC+HHqaUi E6gg== Received: by 10.180.8.134 with SMTP id r6mr21596253wia.18.1350286128803; Mon, 15 Oct 2012 00:28:48 -0700 (PDT) Received: from [10.0.0.86] ([93.152.184.10]) by mx.google.com with ESMTPS id ei1sm14399360wid.7.2012.10.15.00.28.45 (version=TLSv1/SSLv3 cipher=OTHER); Mon, 15 Oct 2012 00:28:47 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 6.1 \(1498\)) Subject: Bad ZFS - NFS interaction? [ was: NFS server bottlenecks ] From: Nikolay Denev In-Reply-To: <302BF685-4B9D-49C8-8000-8D0F6540C8F7@gmail.com> Date: Mon, 15 Oct 2012 10:28:44 +0300 Content-Transfer-Encoding: quoted-printable Message-Id: <65F06188-F333-4961-B3E9-CB8EB8696945@gmail.com> References: <937460294.2185822.1350093954059.JavaMail.root@erie.cs.uoguelph.ca> <302BF685-4B9D-49C8-8000-8D0F6540C8F7@gmail.com> To: "freebsd-fs@freebsd.org" X-Mailer: Apple Mail (2.1498) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Oct 2012 07:28:50 -0000 On Oct 13, 2012, at 6:22 PM, Nikolay Denev wrote: >=20 > On Oct 13, 2012, at 5:05 AM, Rick Macklem = wrote: >=20 >> I wrote: >>> Oops, I didn't get the "readahead" option description >>> quite right in the last post. The default read ahead >>> is 1, which does result in "rsize * 2", since there is >>> the read + 1 readahead. >>>=20 >>> "rsize * 16" would actually be for the option "readahead=3D15" >>> and for "readahead=3D16" the calculation would be "rsize * 17". >>>=20 >>> However, the example was otherwise ok, I think? rick >>=20 >> I've attached the patch drc3.patch (it assumes drc2.patch has already = been >> applied) that replaces the single mutex with one for each hash list >> for tcp. It also increases the size of NFSRVCACHE_HASHSIZE to 200. >>=20 >> These patches are also at: >> http://people.freebsd.org/~rmacklem/drc2.patch >> http://people.freebsd.org/~rmacklem/drc3.patch >> in case the attachments don't get through. >>=20 >> rick >> ps: I haven't tested drc3.patch a lot, but I think it's ok? >=20 > drc3.patch applied and build cleanly and shows nice improvement! >=20 > I've done a quick benchmark using iozone over the NFS mount from the = Linux host. >=20 > drc2.pach (but with NFSRVCACHE_HASHSIZE=3D500) >=20 > TEST WITH 8K > = --------------------------------------------------------------------------= ----------------------- > Auto Mode > Using Minimum Record Size 8 KB > Using Maximum Record Size 8 KB > Using minimum file size of 2097152 kilobytes. > Using maximum file size of 2097152 kilobytes. > O_DIRECT feature enabled > SYNC Mode.=20 > OPS Mode. Output is in operations per second. > Command line used: iozone -a -y 8k -q 8k -n 2g -g 2g -C -I -o = -O -i 0 -i 1 -i 2 > Time Resolution =3D 0.000001 seconds. > Processor cache size set to 1024 Kbytes. > Processor cache line size set to 32 bytes. > File stride size set to 17 * record size. > random = random bkwd record stride =20 > KB reclen write rewrite read reread read = write read rewrite read fwrite frewrite fread freread > 2097152 8 1919 1914 2356 2321 2335 = 1706 =20 >=20 > TEST WITH 1M > = --------------------------------------------------------------------------= ----------------------- > Auto Mode > Using Minimum Record Size 1024 KB > Using Maximum Record Size 1024 KB > Using minimum file size of 2097152 kilobytes. > Using maximum file size of 2097152 kilobytes. > O_DIRECT feature enabled > SYNC Mode.=20 > OPS Mode. Output is in operations per second. > Command line used: iozone -a -y 1m -q 1m -n 2g -g 2g -C -I -o = -O -i 0 -i 1 -i 2 > Time Resolution =3D 0.000001 seconds. > Processor cache size set to 1024 Kbytes. > Processor cache line size set to 32 bytes. > File stride size set to 17 * record size. > random = random bkwd record stride =20 > KB reclen write rewrite read reread read = write read rewrite read fwrite frewrite fread freread > 2097152 1024 73 64 477 486 496 = 61 =20 >=20 >=20 > drc3.patch >=20 > TEST WITH 8K > = --------------------------------------------------------------------------= ----------------------- > Auto Mode > Using Minimum Record Size 8 KB > Using Maximum Record Size 8 KB > Using minimum file size of 2097152 kilobytes. > Using maximum file size of 2097152 kilobytes. > O_DIRECT feature enabled > SYNC Mode.=20 > OPS Mode. Output is in operations per second. > Command line used: iozone -a -y 8k -q 8k -n 2g -g 2g -C -I -o = -O -i 0 -i 1 -i 2 > Time Resolution =3D 0.000001 seconds. > Processor cache size set to 1024 Kbytes. > Processor cache line size set to 32 bytes. > File stride size set to 17 * record size. > random = random bkwd record stride =20 > KB reclen write rewrite read reread read = write read rewrite read fwrite frewrite fread freread > 2097152 8 2108 2397 3001 3013 3010 = 2389 =20 >=20 >=20 > TEST WITH 1M > = --------------------------------------------------------------------------= ----------------------- > Auto Mode > Using Minimum Record Size 1024 KB > Using Maximum Record Size 1024 KB > Using minimum file size of 2097152 kilobytes. > Using maximum file size of 2097152 kilobytes. > O_DIRECT feature enabled > SYNC Mode.=20 > OPS Mode. Output is in operations per second. > Command line used: iozone -a -y 1m -q 1m -n 2g -g 2g -C -I -o = -O -i 0 -i 1 -i 2 > Time Resolution =3D 0.000001 seconds. > Processor cache size set to 1024 Kbytes. > Processor cache line size set to 32 bytes. > File stride size set to 17 * record size. > random = random bkwd record stride =20 > KB reclen write rewrite read reread read = write read rewrite read fwrite frewrite fread freread > 2097152 1024 80 79 521 536 528 = 75 =20 >=20 >=20 > Also with drc3 the CPU usage on the server is noticeably lower. Most = of the time I could see only the geom{g_up}/{g_down} threads, > and a few nfsd threads, before that nfsd's were much more prominent. >=20 > I guess under bigger load the performance improvement can be bigger. >=20 > I'll run some more tests with heavier loads this week. >=20 > Thanks, > Nikolay >=20 >=20 If anyone is interested here's a FlameGraph generated using DTrace and Brendan Gregg's tools from https://github.com/brendangregg/FlameGraph : https://home.totalterror.net/freebsd/goliath-kernel.svg It was sampled during Oracle database restore from Linux host over the = nfs mount. Currently all IO on the dataset that the linux machine writes is stuck, = simple ls in the directory hangs for maybe 10-15 minutes and then eventually completes. Looks like some weird locking issue. [*] http://dtrace.org/blogs/brendan/2011/12/16/flame-graphs/ P.S.: The machine runs with drc3.patch for the NFS server. P.S.2: The nfsd server is configured for vfs.nfsd.maxthreads=3D200, = maybe that's too much? From owner-freebsd-fs@FreeBSD.ORG Mon Oct 15 08:13:11 2012 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 6EF203EB for ; Mon, 15 Oct 2012 08:13:11 +0000 (UTC) (envelope-from mm@FreeBSD.org) Received: from mail.vx.sk (mail.vx.sk [176.9.45.25]) by mx1.freebsd.org (Postfix) with ESMTP id 23FF68FC0C for ; Mon, 15 Oct 2012 08:13:10 +0000 (UTC) Received: from core.vx.sk (localhost [127.0.0.2]) by mail.vx.sk (Postfix) with ESMTP id A50593F3B2; Mon, 15 Oct 2012 10:13:04 +0200 (CEST) X-Virus-Scanned: amavisd-new at mail.vx.sk Received: from mail.vx.sk by core.vx.sk (amavisd-new, unix socket) with LMTP id b1VJqiMSsaic; Mon, 15 Oct 2012 10:12:59 +0200 (CEST) Received: from [10.9.8.1] (188-167-78-15.dynamic.chello.sk [188.167.78.15]) by mail.vx.sk (Postfix) with ESMTPSA id 4FDB13F36E; Mon, 15 Oct 2012 10:12:59 +0200 (CEST) Message-ID: <507BC58A.40803@FreeBSD.org> Date: Mon, 15 Oct 2012 10:12:58 +0200 From: Martin Matuska User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:16.0) Gecko/20121010 Thunderbird/16.0.1 MIME-Version: 1.0 To: Volodymyr Kostyrko Subject: Re: [CFT] ZFS feature flag support for 9-STABLE References: <506B4508.3030406@FreeBSD.org> <5077F3DF.2030108@gmail.com> In-Reply-To: <5077F3DF.2030108@gmail.com> X-Enigmail-Version: 1.4.5 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Cc: freebsd-fs@FreeBSD.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Oct 2012 08:13:11 -0000 On 12.10.2012 12:41, Volodymyr Kostyrko wrote: > 02.10.2012 22:48, Martin Matuska wrote: >> ZFS feature flag support is ready to be merged to 9-STABLE. >> The scheduled merge date is short after 9.1-RELEASE. >> >> Early adopters can test new features by applying the following patch >> (stable/9 r241135): >> http://people.freebsd.org/~mm/patches/zfs/9-stable-zfs-features.patch.gz >> >> Steps to apply to a clean checked-out source: >> cd /path/to/src >> patch -p0 < /path/to/9-stable-zfs-features.patch > > Does this suppose to work on i386? On my test machine this gives an > instant panic on trying to mount root system. I've already rebuilt my > kernel WITH INVARIANTS, WITNESS and DIAGNOSTIC but it still panics > even before I can dump core anywhere. I now proceeding to build a tiny > virtual machine to capture the backtrace. > Hi, I am unable to reproduce this under i386 VirtualBox. Does your system have some special settings? Was patched stable/9 built correctly? Cheers, mm -- Martin Matuska FreeBSD committer http://blog.vx.sk From owner-freebsd-fs@FreeBSD.ORG Mon Oct 15 11:06:12 2012 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 2BC4F507 for ; Mon, 15 Oct 2012 11:06:12 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [8.8.178.135]) by mx1.freebsd.org (Postfix) with ESMTP id EE79E8FC28 for ; Mon, 15 Oct 2012 11:06:11 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q9FB6BbW011470 for ; Mon, 15 Oct 2012 11:06:11 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q9FB6B67011469 for freebsd-fs@FreeBSD.org; Mon, 15 Oct 2012 11:06:11 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 15 Oct 2012 11:06:11 GMT Message-Id: <201210151106.q9FB6B67011469@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-fs@FreeBSD.org Subject: Current problem reports assigned to freebsd-fs@FreeBSD.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Oct 2012 11:06:12 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). From owner-freebsd-fs@FreeBSD.ORG Mon Oct 15 14:06:20 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 14F1E863 for ; Mon, 15 Oct 2012 14:06:20 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id 97BE88FC08 for ; Mon, 15 Oct 2012 14:06:19 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AqAEAN6MclCDaFvO/2dsb2JhbABFFoV7uhmCIAEBAQQBAQEgKyALGw4KAgINGQIjBgEJJgYIBwQBHASHUgMPC6ZMiBYNiVSBIYlIZhqEZIESA5M+WIFVgRWKDoULgwmBRzQ X-IronPort-AV: E=Sophos;i="4.80,588,1344225600"; d="scan'208";a="183660337" Received: from erie.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.206]) by esa-jnhn-pri.mail.uoguelph.ca with ESMTP; 15 Oct 2012 10:06:18 -0400 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id 4A4B3B402B; Mon, 15 Oct 2012 10:06:18 -0400 (EDT) Date: Mon, 15 Oct 2012 10:06:18 -0400 (EDT) From: Rick Macklem To: Nikolay Denev Message-ID: <831941180.2238334.1350309978281.JavaMail.root@erie.cs.uoguelph.ca> In-Reply-To: <65F06188-F333-4961-B3E9-CB8EB8696945@gmail.com> Subject: Re: Bad ZFS - NFS interaction? [ was: NFS server bottlenecks ] MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.91.202] X-Mailer: Zimbra 6.0.10_GA_2692 (ZimbraWebClient - FF3.0 (Win)/6.0.10_GA_2692) Cc: "freebsd-fs@freebsd.org" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Oct 2012 14:06:20 -0000 Nikolay Denev wrote: > On Oct 13, 2012, at 6:22 PM, Nikolay Denev wrote: > > > > > On Oct 13, 2012, at 5:05 AM, Rick Macklem > > wrote: > > > >> I wrote: > >>> Oops, I didn't get the "readahead" option description > >>> quite right in the last post. The default read ahead > >>> is 1, which does result in "rsize * 2", since there is > >>> the read + 1 readahead. > >>> > >>> "rsize * 16" would actually be for the option "readahead=15" > >>> and for "readahead=16" the calculation would be "rsize * 17". > >>> > >>> However, the example was otherwise ok, I think? rick > >> > >> I've attached the patch drc3.patch (it assumes drc2.patch has > >> already been > >> applied) that replaces the single mutex with one for each hash list > >> for tcp. It also increases the size of NFSRVCACHE_HASHSIZE to 200. > >> > >> These patches are also at: > >> http://people.freebsd.org/~rmacklem/drc2.patch > >> http://people.freebsd.org/~rmacklem/drc3.patch > >> in case the attachments don't get through. > >> > >> rick > >> ps: I haven't tested drc3.patch a lot, but I think it's ok? > > > > drc3.patch applied and build cleanly and shows nice improvement! > > > > I've done a quick benchmark using iozone over the NFS mount from the > > Linux host. > > > > drc2.pach (but with NFSRVCACHE_HASHSIZE=500) > > > > TEST WITH 8K > > ------------------------------------------------------------------------------------------------- > > Auto Mode > > Using Minimum Record Size 8 KB > > Using Maximum Record Size 8 KB > > Using minimum file size of 2097152 kilobytes. > > Using maximum file size of 2097152 kilobytes. > > O_DIRECT feature enabled > > SYNC Mode. > > OPS Mode. Output is in operations per second. > > Command line used: iozone -a -y 8k -q 8k -n 2g -g 2g -C -I -o > > -O -i 0 -i 1 -i 2 > > Time Resolution = 0.000001 seconds. > > Processor cache size set to 1024 Kbytes. > > Processor cache line size set to 32 bytes. > > File stride size set to 17 * record size. > > random > > random > > bkwd > > record > > stride > > KB reclen write rewrite read reread read write read > > rewrite read fwrite frewrite fread freread > > 2097152 8 1919 1914 2356 2321 2335 1706 > > > > TEST WITH 1M > > ------------------------------------------------------------------------------------------------- > > Auto Mode > > Using Minimum Record Size 1024 KB > > Using Maximum Record Size 1024 KB > > Using minimum file size of 2097152 kilobytes. > > Using maximum file size of 2097152 kilobytes. > > O_DIRECT feature enabled > > SYNC Mode. > > OPS Mode. Output is in operations per second. > > Command line used: iozone -a -y 1m -q 1m -n 2g -g 2g -C -I -o > > -O -i 0 -i 1 -i 2 > > Time Resolution = 0.000001 seconds. > > Processor cache size set to 1024 Kbytes. > > Processor cache line size set to 32 bytes. > > File stride size set to 17 * record size. > > random > > random > > bkwd > > record > > stride > > KB reclen write rewrite read reread read write read > > rewrite read fwrite frewrite fread freread > > 2097152 1024 73 64 477 486 496 61 > > > > > > drc3.patch > > > > TEST WITH 8K > > ------------------------------------------------------------------------------------------------- > > Auto Mode > > Using Minimum Record Size 8 KB > > Using Maximum Record Size 8 KB > > Using minimum file size of 2097152 kilobytes. > > Using maximum file size of 2097152 kilobytes. > > O_DIRECT feature enabled > > SYNC Mode. > > OPS Mode. Output is in operations per second. > > Command line used: iozone -a -y 8k -q 8k -n 2g -g 2g -C -I -o > > -O -i 0 -i 1 -i 2 > > Time Resolution = 0.000001 seconds. > > Processor cache size set to 1024 Kbytes. > > Processor cache line size set to 32 bytes. > > File stride size set to 17 * record size. > > random > > random > > bkwd > > record > > stride > > KB reclen write rewrite read reread read write read > > rewrite read fwrite frewrite fread freread > > 2097152 8 2108 2397 3001 3013 3010 2389 > > > > > > TEST WITH 1M > > ------------------------------------------------------------------------------------------------- > > Auto Mode > > Using Minimum Record Size 1024 KB > > Using Maximum Record Size 1024 KB > > Using minimum file size of 2097152 kilobytes. > > Using maximum file size of 2097152 kilobytes. > > O_DIRECT feature enabled > > SYNC Mode. > > OPS Mode. Output is in operations per second. > > Command line used: iozone -a -y 1m -q 1m -n 2g -g 2g -C -I -o > > -O -i 0 -i 1 -i 2 > > Time Resolution = 0.000001 seconds. > > Processor cache size set to 1024 Kbytes. > > Processor cache line size set to 32 bytes. > > File stride size set to 17 * record size. > > random > > random > > bkwd > > record > > stride > > KB reclen write rewrite read reread read write read > > rewrite read fwrite frewrite fread freread > > 2097152 1024 80 79 521 536 528 75 > > > > > > Also with drc3 the CPU usage on the server is noticeably lower. Most > > of the time I could see only the geom{g_up}/{g_down} threads, > > and a few nfsd threads, before that nfsd's were much more prominent. > > > > I guess under bigger load the performance improvement can be bigger. > > > > I'll run some more tests with heavier loads this week. > > > > Thanks, > > Nikolay > > > > > > If anyone is interested here's a FlameGraph generated using DTrace and > Brendan Gregg's tools from https://github.com/brendangregg/FlameGraph > : > > https://home.totalterror.net/freebsd/goliath-kernel.svg > > It was sampled during Oracle database restore from Linux host over the > nfs mount. > Currently all IO on the dataset that the linux machine writes is > stuck, simple ls in the directory > hangs for maybe 10-15 minutes and then eventually completes. > > Looks like some weird locking issue. > > [*] http://dtrace.org/blogs/brendan/2011/12/16/flame-graphs/ > > P.S.: The machine runs with drc3.patch for the NFS server. > P.S.2: The nfsd server is configured for vfs.nfsd.maxthreads=200, > maybe that's too much? > You could try trimming the size of vfs.nfsd.tcphighwater down. Remember that, with this patch, when you increase this tunable, you are trading space for CPU overhead. If it's still "running", you could do "vmstat -m" and "vmstat -z" to see where the memory is allocated. ("nfsstat -e -s" will tell you the size of the cache.) rick > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@FreeBSD.ORG Mon Oct 15 15:08:24 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 97DD33F5 for ; Mon, 15 Oct 2012 15:08:24 +0000 (UTC) (envelope-from ndenev@gmail.com) Received: from mail-wg0-f42.google.com (mail-wg0-f42.google.com [74.125.82.42]) by mx1.freebsd.org (Postfix) with ESMTP id 239098FC0A for ; Mon, 15 Oct 2012 15:08:23 +0000 (UTC) Received: by mail-wg0-f42.google.com with SMTP id fm10so249677wgb.1 for ; Mon, 15 Oct 2012 08:08:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer; bh=GJsGagae2L2KWrIg2Smh1jt6/3QLowyZWAtDYKkZChw=; b=Xs2dxhxunfMuyl+1GmERZcbDdGxxDN25vI5KDV2DjLB2eZSUy+g1r14MWLqz+CdvKb qerM8bj11gdQbVm7W9WTQf01agYTy5WwKmJjd0ORB8c7mDv6P3JH38XOZIJeH2hCSMHr l+z8vf3zqr/o2EFfPC7whbOls6zd6xNEb2XVvPyW3SAN7VYem/Drty3wm2Kya49hAchm HiPhL4W0xX6M9gjFUZ+VmcESyB6EilHx/pU/xZYqBG8EhOsLbKTMrSmxYzrpRJD3kplo IoOdPmKtHUMaItTmtaXhFbUK9eRDAEHsFNR5UAiH6MJKBhYRbRitlRlorBgYzl0dSY7n 2dsQ== Received: by 10.216.27.84 with SMTP id d62mr7108644wea.3.1350313703137; Mon, 15 Oct 2012 08:08:23 -0700 (PDT) Received: from ndenevsa.sf.moneybookers.net (g1.moneybookers.com. [217.18.249.148]) by mx.google.com with ESMTPS id cl8sm14356877wib.10.2012.10.15.08.08.20 (version=TLSv1/SSLv3 cipher=OTHER); Mon, 15 Oct 2012 08:08:22 -0700 (PDT) Subject: Re: Bad ZFS - NFS interaction? [ was: NFS server bottlenecks ] Mime-Version: 1.0 (Mac OS X Mail 6.1 \(1498\)) Content-Type: text/plain; charset=us-ascii From: Nikolay Denev In-Reply-To: <831941180.2238334.1350309978281.JavaMail.root@erie.cs.uoguelph.ca> Date: Mon, 15 Oct 2012 18:08:19 +0300 Content-Transfer-Encoding: quoted-printable Message-Id: <9BE97E36-8995-4968-B8ED-1B17D308ED19@gmail.com> References: <831941180.2238334.1350309978281.JavaMail.root@erie.cs.uoguelph.ca> To: Rick Macklem X-Mailer: Apple Mail (2.1498) Cc: "freebsd-fs@freebsd.org" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Oct 2012 15:08:24 -0000 On Oct 15, 2012, at 5:06 PM, Rick Macklem wrote: > Nikolay Denev wrote: >> On Oct 13, 2012, at 6:22 PM, Nikolay Denev wrote: >>=20 >>>=20 >>> On Oct 13, 2012, at 5:05 AM, Rick Macklem >>> wrote: >>>=20 >>>> I wrote: >>>>> Oops, I didn't get the "readahead" option description >>>>> quite right in the last post. The default read ahead >>>>> is 1, which does result in "rsize * 2", since there is >>>>> the read + 1 readahead. >>>>>=20 >>>>> "rsize * 16" would actually be for the option "readahead=3D15" >>>>> and for "readahead=3D16" the calculation would be "rsize * 17". >>>>>=20 >>>>> However, the example was otherwise ok, I think? rick >>>>=20 >>>> I've attached the patch drc3.patch (it assumes drc2.patch has >>>> already been >>>> applied) that replaces the single mutex with one for each hash list >>>> for tcp. It also increases the size of NFSRVCACHE_HASHSIZE to 200. >>>>=20 >>>> These patches are also at: >>>> http://people.freebsd.org/~rmacklem/drc2.patch >>>> http://people.freebsd.org/~rmacklem/drc3.patch >>>> in case the attachments don't get through. >>>>=20 >>>> rick >>>> ps: I haven't tested drc3.patch a lot, but I think it's ok? >>>=20 >>> drc3.patch applied and build cleanly and shows nice improvement! >>>=20 >>> I've done a quick benchmark using iozone over the NFS mount from the >>> Linux host. >>>=20 >>> drc2.pach (but with NFSRVCACHE_HASHSIZE=3D500) >>>=20 >>> TEST WITH 8K >>> = --------------------------------------------------------------------------= ----------------------- >>> Auto Mode >>> Using Minimum Record Size 8 KB >>> Using Maximum Record Size 8 KB >>> Using minimum file size of 2097152 kilobytes. >>> Using maximum file size of 2097152 kilobytes. >>> O_DIRECT feature enabled >>> SYNC Mode. >>> OPS Mode. Output is in operations per second. >>> Command line used: iozone -a -y 8k -q 8k -n 2g -g 2g -C -I -o >>> -O -i 0 -i 1 -i 2 >>> Time Resolution =3D 0.000001 seconds. >>> Processor cache size set to 1024 Kbytes. >>> Processor cache line size set to 32 bytes. >>> File stride size set to 17 * record size. >>> random >>> random >>> bkwd >>> record >>> stride >>> KB reclen write rewrite read reread read write read >>> rewrite read fwrite frewrite fread freread >>> 2097152 8 1919 1914 2356 2321 2335 1706 >>>=20 >>> TEST WITH 1M >>> = --------------------------------------------------------------------------= ----------------------- >>> Auto Mode >>> Using Minimum Record Size 1024 KB >>> Using Maximum Record Size 1024 KB >>> Using minimum file size of 2097152 kilobytes. >>> Using maximum file size of 2097152 kilobytes. >>> O_DIRECT feature enabled >>> SYNC Mode. >>> OPS Mode. Output is in operations per second. >>> Command line used: iozone -a -y 1m -q 1m -n 2g -g 2g -C -I -o >>> -O -i 0 -i 1 -i 2 >>> Time Resolution =3D 0.000001 seconds. >>> Processor cache size set to 1024 Kbytes. >>> Processor cache line size set to 32 bytes. >>> File stride size set to 17 * record size. >>> random >>> random >>> bkwd >>> record >>> stride >>> KB reclen write rewrite read reread read write read >>> rewrite read fwrite frewrite fread freread >>> 2097152 1024 73 64 477 486 496 61 >>>=20 >>>=20 >>> drc3.patch >>>=20 >>> TEST WITH 8K >>> = --------------------------------------------------------------------------= ----------------------- >>> Auto Mode >>> Using Minimum Record Size 8 KB >>> Using Maximum Record Size 8 KB >>> Using minimum file size of 2097152 kilobytes. >>> Using maximum file size of 2097152 kilobytes. >>> O_DIRECT feature enabled >>> SYNC Mode. >>> OPS Mode. Output is in operations per second. >>> Command line used: iozone -a -y 8k -q 8k -n 2g -g 2g -C -I -o >>> -O -i 0 -i 1 -i 2 >>> Time Resolution =3D 0.000001 seconds. >>> Processor cache size set to 1024 Kbytes. >>> Processor cache line size set to 32 bytes. >>> File stride size set to 17 * record size. >>> random >>> random >>> bkwd >>> record >>> stride >>> KB reclen write rewrite read reread read write read >>> rewrite read fwrite frewrite fread freread >>> 2097152 8 2108 2397 3001 3013 3010 2389 >>>=20 >>>=20 >>> TEST WITH 1M >>> = --------------------------------------------------------------------------= ----------------------- >>> Auto Mode >>> Using Minimum Record Size 1024 KB >>> Using Maximum Record Size 1024 KB >>> Using minimum file size of 2097152 kilobytes. >>> Using maximum file size of 2097152 kilobytes. >>> O_DIRECT feature enabled >>> SYNC Mode. >>> OPS Mode. Output is in operations per second. >>> Command line used: iozone -a -y 1m -q 1m -n 2g -g 2g -C -I -o >>> -O -i 0 -i 1 -i 2 >>> Time Resolution =3D 0.000001 seconds. >>> Processor cache size set to 1024 Kbytes. >>> Processor cache line size set to 32 bytes. >>> File stride size set to 17 * record size. >>> random >>> random >>> bkwd >>> record >>> stride >>> KB reclen write rewrite read reread read write read >>> rewrite read fwrite frewrite fread freread >>> 2097152 1024 80 79 521 536 528 75 >>>=20 >>>=20 >>> Also with drc3 the CPU usage on the server is noticeably lower. Most >>> of the time I could see only the geom{g_up}/{g_down} threads, >>> and a few nfsd threads, before that nfsd's were much more prominent. >>>=20 >>> I guess under bigger load the performance improvement can be bigger. >>>=20 >>> I'll run some more tests with heavier loads this week. >>>=20 >>> Thanks, >>> Nikolay >>>=20 >>>=20 >>=20 >> If anyone is interested here's a FlameGraph generated using DTrace = and >> Brendan Gregg's tools from https://github.com/brendangregg/FlameGraph >> : >>=20 >> https://home.totalterror.net/freebsd/goliath-kernel.svg >>=20 >> It was sampled during Oracle database restore from Linux host over = the >> nfs mount. >> Currently all IO on the dataset that the linux machine writes is >> stuck, simple ls in the directory >> hangs for maybe 10-15 minutes and then eventually completes. >>=20 >> Looks like some weird locking issue. >>=20 >> [*] http://dtrace.org/blogs/brendan/2011/12/16/flame-graphs/ >>=20 >> P.S.: The machine runs with drc3.patch for the NFS server. >> P.S.2: The nfsd server is configured for vfs.nfsd.maxthreads=3D200, >> maybe that's too much? >>=20 > You could try trimming the size of vfs.nfsd.tcphighwater down. = Remember that, > with this patch, when you increase this tunable, you are trading space > for CPU overhead. >=20 > If it's still "running", you could do "vmstat -m" and "vmstat -z" to > see where the memory is allocated. ("nfsstat -e -s" will tell you the > size of the cache.) >=20 > rick >> _______________________________________________ >> freebsd-fs@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-fs >> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" Are you saying that the time spent in _mtx_spin_lock can be because of = this? To me it looks like that there was some heavy contention in ZFS, maybe = specific to the way it's accessed by the NFS server? Probably due to high maxthreads = value ? Here's the nfsstat -s -e, seems like it's wrong as it's negative number, = maybe overflowed? Server: Retfailed Faults Clients 0 0 0 OpenOwner Opens LockOwner Locks Delegs=20 0 0 0 0 0=20 Server Cache Stats: Inprog Idem Non-idem Misses CacheSize TCPPeak 0 0 0 83500632 -24072 16385 Also here are the following sysctls : vfs.nfsd.request_space_used: 0 vfs.nfsd.request_space_used_highest: 13121808 vfs.nfsd.request_space_high: 13107200 vfs.nfsd.request_space_low: 8738133 vfs.nfsd.request_space_throttled: 0 vfs.nfsd.request_space_throttle_count: 0 Are they related to the same request cache? I have stats that show at some point nfsd has allocated all 200 threads = and=20 vfs.nfsd.request_space_used hits the ceiling too. From owner-freebsd-fs@FreeBSD.ORG Mon Oct 15 16:04:08 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 925E699 for ; Mon, 15 Oct 2012 16:04:08 +0000 (UTC) (envelope-from freebsd@pki2.com) Received: from btw.pki2.com (btw.pki2.com [IPv6:2001:470:a:6fd::2]) by mx1.freebsd.org (Postfix) with ESMTP id 077CA8FC14 for ; Mon, 15 Oct 2012 16:04:02 +0000 (UTC) Received: from [127.0.0.1] (localhost [127.0.0.1]) by btw.pki2.com (8.14.5/8.14.5) with ESMTP id q9FG3d7j074277 for ; Mon, 15 Oct 2012 09:03:39 -0700 (PDT) (envelope-from freebsd@pki2.com) Subject: I have a DDB session open to a crashed ZFS server From: Dennis Glatting To: freebsd-fs@freebsd.org Date: Mon, 15 Oct 2012 09:03:39 -0700 Message-ID: <1350317019.71982.50.camel@btw.pki2.com> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port X-yoursite-MailScanner-Information: Dennis Glatting X-yoursite-MailScanner-ID: q9FG3d7j074277 X-yoursite-MailScanner: Found to be clean X-MailScanner-From: freebsd@pki2.com X-Mailman-Approved-At: Mon, 15 Oct 2012 17:11:41 +0000 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.14 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Oct 2012 16:04:08 -0000 What do you want to see? I Enclosed is: * Boot output, * "top" from another window, and * a DDB ps output. The boot sequence: iirc# tip com1 ______ ____ _____ _____ | ____| | _ \ / ____| __ \ | |___ _ __ ___ ___ | |_) | (___ | | | | | ___| '__/ _ \/ _ \| _ < \___ \| | | | | | | | | __/ __/| |_) |____) | |__| | | | | | | | || | | | |_| |_| \___|\___||____/|_____/|_____/ ``` ` s` `.....---.......--.``` -/ �������������Welcome to FreeBSD������������ +o .--` /y:` +. � � yo`:. :o ` +- � 1. Boot [ENTER] � y/ -/` -o/ � 2. [Esc]ape to loader prompt � .- ::/sy +:. � 3. Reboot � / `-- / � � `: :` � Options: � `: :` � 4. [A]CPI Support: Enabled � / / � 5. Boot Safe [M]ode: NO � .- -. � 6. Boot [S]ingle User: NO � -- -. � 7. Boot [V]erbose: NO � `:` `:` � � .-- `--. � � .---.....----. ������������������������������������������� KDB: debugger backends: ddb KDB: current backend: ddb Copyright (c) 1992-2012 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 9.1-PRERELEASE #0 r241306M: Sat Oct 6 20:41:22 PDT 2012 root@mc:/usr/obj/disk-1/src/sys/DTRACE amd64 CPU: AMD Opteron(TM) Processor 6274 (2200.07-MHz K8-class CPU) Origin = "AuthenticAMD" Id = 0x600f12 Family = 0x15 Model = 0x1 Stepping = 2 Features=0x178bfbff Features2=0x1e98220b AMD Features=0x2e500800 AMD Features2=0x1c9bfff,> TSC: P-state invariant, performance statistics real memory = 137438953472 (131072 MB) avail memory = 132423512064 (126288 MB) Event timer "LAPIC" quality 400 ACPI APIC Table: <120911 APIC1027> FreeBSD/SMP: Multiprocessor System Detected: 64 CPUs FreeBSD/SMP: 4 package(s) x 16 core(s) cpu0 (BSP): APIC ID: 32 cpu1 (AP): APIC ID: 33 cpu2 (AP): APIC ID: 34 cpu3 (AP): APIC ID: 35 cpu4 (AP): APIC ID: 36 cpu5 (AP): APIC ID: 37 cpu6 (AP): APIC ID: 38 cpu7 (AP): APIC ID: 39 cpu8 (AP): APIC ID: 40 cpu9 (AP): APIC ID: 41 cpu10 (AP): APIC ID: 42 cpu11 (AP): APIC ID: 43 cpu12 (AP): APIC ID: 44 cpu13 (AP): APIC ID: 45 cpu14 (AP): APIC ID: 46 cpu15 (AP): APIC ID: 47 cpu16 (AP): APIC ID: 64 cpu17 (AP): APIC ID: 65 cpu18 (AP): APIC ID: 66 cpu19 (AP): APIC ID: 67 cpu20 (AP): APIC ID: 68 cpu21 (AP): APIC ID: 69 cpu22 (AP): APIC ID: 70 cpu23 (AP): APIC ID: 71 cpu24 (AP): APIC ID: 72 cpu25 (AP): APIC ID: 73 cpu26 (AP): APIC ID: 74 cpu27 (AP): APIC ID: 75 cpu28 (AP): APIC ID: 76 cpu29 (AP): APIC ID: 77 cpu30 (AP): APIC ID: 78 cpu31 (AP): APIC ID: 79 cpu32 (AP): APIC ID: 96 cpu33 (AP): APIC ID: 97 cpu34 (AP): APIC ID: 98 cpu35 (AP): APIC ID: 99 cpu36 (AP): APIC ID: 100 cpu37 (AP): APIC ID: 101 cpu38 (AP): APIC ID: 102 cpu39 (AP): APIC ID: 103 cpu40 (AP): APIC ID: 104 cpu41 (AP): APIC ID: 105 cpu42 (AP): APIC ID: 106 cpu43 (AP): APIC ID: 107 cpu44 (AP): APIC ID: 108 cpu45 (AP): APIC ID: 109 cpu46 (AP): APIC ID: 110 cpu47 (AP): APIC ID: 111 cpu48 (AP): APIC ID: 128 cpu49 (AP): APIC ID: 129 cpu50 (AP): APIC ID: 130 cpu51 (AP): APIC ID: 131 cpu52 (AP): APIC ID: 132 cpu53 (AP): APIC ID: 133 cpu54 (AP): APIC ID: 134 cpu55 (AP): APIC ID: 135 cpu56 (AP): APIC ID: 136 cpu57 (AP): APIC ID: 137 cpu58 (AP): APIC ID: 138 cpu59 (AP): APIC ID: 139 cpu60 (AP): APIC ID: 140 cpu61 (AP): APIC ID: 141 cpu62 (AP): APIC ID: 142 cpu63 (AP): APIC ID: 143 ACPI Warning: Optional field Pm2ControlBlock has zero address or length: 0x0000000000000000/0x1 (20110527/tbfadt-583) ioapic0 irqs 0-23 on motherboard ioapic1 irqs 24-55 on motherboard kbd1 at kbdmux0 module_register_init: MOD_LOAD (vesa, 0xffffffff80bcb7a0, 0) error 19 ctl: CAM Target Layer loaded cryptosoft0: on motherboard aesni0: on motherboard acpi0: <120911 XSDT1027> on motherboard acpi0: Power Button (fixed) acpi0: reservation of fec00000, 1000 (3) failed acpi0: reservation of fee00000, 1000 (3) failed acpi0: reservation of ffb80000, 80000 (3) failed acpi0: reservation of fec10000, 20 (3) failed acpi0: reservation of 0, a0000 (3) failed acpi0: reservation of 100000, dff00000 (3) failed cpu0: on acpi0 cpu1: on acpi0 cpu2: on acpi0 cpu3: on acpi0 cpu4: on acpi0 cpu5: on acpi0 cpu6: on acpi0 cpu7: on acpi0 cpu8: on acpi0 cpu9: on acpi0 cpu10: on acpi0 cpu11: on acpi0 cpu12: on acpi0 cpu13: on acpi0 cpu14: on acpi0 cpu15: on acpi0 cpu16: on acpi0 cpu17: on acpi0 cpu18: on acpi0 cpu19: on acpi0 cpu20: on acpi0 cpu21: on acpi0 cpu22: on acpi0 cpu23: on acpi0 cpu24: on acpi0 cpu25: on acpi0 cpu26: on acpi0 cpu27: on acpi0 cpu28: on acpi0 cpu29: on acpi0 cpu30: on acpi0 cpu31: on acpi0 cpu32: on acpi0 cpu33: on acpi0 cpu34: on acpi0 cpu35: on acpi0 cpu36: on acpi0 cpu37: on acpi0 cpu38: on acpi0 cpu39: on acpi0 cpu40: on acpi0 cpu41: on acpi0 cpu42: on acpi0 cpu43: on acpi0 cpu44: on acpi0 cpu45: on acpi0 cpu46: on acpi0 cpu47: on acpi0 cpu48: on acpi0 cpu49: on acpi0 cpu50: on acpi0 cpu51: on acpi0 cpu52: on acpi0 cpu53: on acpi0 cpu54: on acpi0 cpu55: on acpi0 cpu56: on acpi0 cpu57: on acpi0 cpu58: on acpi0 cpu59: on acpi0 cpu60: on acpi0 cpu61: on acpi0 cpu62: on acpi0 cpu63: on acpi0 attimer0: port 0x40-0x43 irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 atrtc0: port 0x70-0x71 irq 8 on acpi0 Event timer "RTC" frequency 32768 Hz quality 0 hpet0: iomem 0xfed00000-0xfed003ff on acpi0 Timecounter "HPET" frequency 14318180 Hz quality 950 Timecounter "ACPI-safe" frequency 3579545 Hz quality 850 acpi_timer0: <32-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pci0: at device 0.2 (no driver attached) pcib1: irq 52 at device 4.0 on pci0 pci5: on pcib1 igb0: port 0xe400-0xe41f mem 0xfeb60000-0xfeb7ffff,0xfeb40000-0xfeb5ffff,0xfeb1c000-0xfeb1ffff irq 44 at device 0.0 on pci5 igb0: Using MSIX interrupts with 9 vectors igb0: Ethernet address: 00:e0:81:c8:ee:8a igb0: Bound queue 0 to cpu 0 igb0: Bound queue 1 to cpu 1 igb0: Bound queue 2 to cpu 2 igb0: Bound queue 3 to cpu 3 igb0: Bound queue 4 to cpu 4 igb0: Bound queue 5 to cpu 5 igb0: Bound queue 6 to cpu 6 igb0: Bound queue 7 to cpu 7 igb1: port 0xe800-0xe81f mem 0xfebe0000-0xfebfffff,0xfebc0000-0xfebdffff,0xfeb9c000-0xfeb9ffff irq 45 at device 0.1 on pci5 igb1: Using MSIX interrupts with 9 vectors igb1: Ethernet address: 00:e0:81:c8:ee:8b igb1: Bound queue 0 to cpu 8 igb1: Bound queue 1 to cpu 9 igb1: Bound queue 2 to cpu 10 igb1: Bound queue 3 to cpu 11 igb1: Bound queue 4 to cpu 12 igb1: Bound queue 5 to cpu 13 igb1: Bound queue 6 to cpu 14 igb1: Bound queue 7 to cpu 15 pcib2: irq 53 at device 9.0 on pci0 pci4: on pcib2 em0: port 0xd800-0xd81f mem 0xfeae0000-0xfeafffff,0xfeadc000-0xfeadffff irq 48 at device 0.0 on pci4 em0: Using MSIX interrupts with 3 vectors em0: Ethernet address: 00:e0:81:c8:ee:ff pcib3: irq 54 at device 11.0 on pci0 pci3: on pcib3 mps0: port 0xc000-0xc0ff mem 0xfe93c000-0xfe93ffff,0xfe940000-0xfe97ffff irq 32 at device 0.0 on pci3 mps0: Firmware: 14.00.00.00, Driver: 14.00.00.01-fbsd mps0: IOCCapabilities: 1285c pcib4: irq 54 at device 13.0 on pci0 pci2: on pcib4 mps1: port 0xb000-0xb0ff mem 0xfe83c000-0xfe83ffff,0xfe840000-0xfe87ffff irq 40 at device 0.0 on pci2 mps1: Firmware: 14.00.00.00, Driver: 14.00.00.01-fbsd mps1: IOCCapabilities: 185c ahci0: port 0x9000-0x9007,0x8000-0x8003,0x7000-0x7007,0x6000-0x6003,0x5000-0x500f mem 0xfdefe400-0xfdefe7ff irq 22 at device 17.0 on pci0 ahci0: AHCI v1.10 with 4 3Gbps ports, Port Multiplier supported ahcich0: at channel 0 on ahci0 ahcich1: at channel 1 on ahci0 ahcich2: at channel 2 on ahci0 ahcich3: at channel 3 on ahci0 ohci0: mem 0xfdefa000-0xfdefafff irq 16 at device 18.0 on pci0 usbus0 on ohci0 ohci1: mem 0xfdefb000-0xfdefbfff irq 16 at device 18.1 on pci0 usbus1 on ohci1 ehci0: mem 0xfdefe800-0xfdefe8ff irq 17 at device 18.2 on pci0 usbus2: EHCI version 1.0 usbus2 on ehci0 ohci2: mem 0xfdefc000-0xfdefcfff irq 18 at device 19.0 on pci0 usbus3 on ohci2 ohci3: mem 0xfdefd000-0xfdefdfff irq 18 at device 19.1 on pci0 usbus4 on ohci3 ehci1: mem 0xfdefec00-0xfdefecff irq 19 at device 19.2 on pci0 usbus5: EHCI version 1.0 usbus5 on ehci1 pci0: at device 20.0 (no driver attached) atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xff00-0xff0f at device 20.1 on pci0 ata0: at channel 0 on atapci0 ata1: at channel 1 on atapci0 isab0: at device 20.3 on pci0 isa0: on isab0 pcib5: at device 20.4 on pci0 pci1: on pcib5 vgapci0: port 0xa800-0xa87f mem 0xfe000000-0xfe7fffff,0xfdfe0000-0xfdffffff irq 23 at device 9.0 on pci1 ohci4: mem 0xfdeff000-0xfdefffff irq 18 at device 20.5 on pci0 usbus6 on ohci4 acpi_button0: on acpi0 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart0: console (9600,n,8,1) uart1: port 0x2f8-0x2ff irq 3 on acpi0 ppc0: port 0x378-0x37f irq 7 on acpi0 ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode ppbus0: on ppc0 plip0: on ppbus0 lpt0: on ppbus0 lpt0: Interrupt-driven port ppi0: on ppbus0 orm0: at iomem 0xc0000-0xc7fff on isa0 sc0: at flags 0x100 on isa0 sc0: CGA <16 virtual consoles, flags=0x100> vga0: at port 0x3d0-0x3db iomem 0xb8000-0xbffff on isa0 atkbdc0: at port 0x60,0x64 on isa0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] acpi_throttle0: on cpu0 Timecounters tick every 1.000 msec usbus0: 12Mbps Full Speed USB v1.0 usbus1: 12Mbps Full Speed USB v1.0 ugen0.1: at usbus0 uhub0: on usbus0 ugen1.1: at usbus1 uhub1: on usbus1 usbus2: 480Mbps High Speed USB v2.0 usbus3: 12Mbps Full Speed USB v1.0 ugen2.1: at usbus2 uhub2: on usbus2 ugen3.1: at usbus3 uhub3: on usbus3 usbus4: 12Mbps Full Speed USB v1.0 usbus5: 480Mbps High Speed USB v2.0 ugen4.1: at usbus4 uhub4: on usbus4 uhub0: 3 ports with 3 removable, self powered ugen5.1: at usbus5 uhub5: on usbus5 uhub1: 3 ports with 3 removable, self powered usbus6: 12Mbps Full Speed USB v1.0 ugen6.1: at usbus6 uhub6: on usbus6 uhub3: 3 ports with 3 removable, self powered uhub4: 3 ports with 3 removable, self powered uhub6: 2 ports with 2 removable, self powered uhub2: 6 ports with 6 removable, self powered uhub5: 6 ports with 6 removable, self powered ugen5.2: at usbus5 uhub7: on usbus5 uhub7: 3 ports with 3 removable, self powered (probe258:mps1:0:3:0): INQUIRY. CDB: 12 0 0 0 24 0 (probe258:mps1:0:3:0): CAM status: Invalid Target ID (probe258:mps1:0:3:0): Error 22, Unretryable error (probe0:mps1:0:3:1): INQUIRY. CDB: 12 20 0 0 24 0 (probe0:mps1:0:3:1): CAM status: Invalid Target ID (probe0:mps1:0:3:1): Error 22, Unretryable error (probe0:mps1:0:3:2): INQUIRY. CDB: 12 40 0 0 24 0 (probe0:mps1:0:3:2): CAM status: Invalid Target ID (probe0:mps1:0:3:2): Error 22, Unretryable error (probe0:mps1:0:3:3): INQUIRY. CDB: 12 60 0 0 24 0 (probe0:mps1:0:3:3): CAM status: Invalid Target ID (probe0:mps1:0:3:3): Error 22, Unretryable error (probe0:mps1:0:3:4): INQUIRY. CDB: 12 80 0 0 24 0 (probe0:mps1:0:3:4): CAM status: Invalid Target ID (probe0:mps1:0:3:4): Error 22, Unretryable error (probe0:mps1:0:3:5): INQUIRY. CDB: 12 a0 0 0 24 0 (probe0:mps1:0:3:5): CAM status: Invalid Target ID (probe0:mps1:0:3:5): Error 22, Unretryable error (probe0:mps1:0:3:6): INQUIRY. CDB: 12 c0 0 0 24 0 (probe0:mps1:0:3:6): CAM status: Invalid Target ID (probe0:mps1:0:3:6): Error 22, Unretryable error (probe0:mps1:0:3:7): INQUIRY. CDB: 12 e0 0 0 24 0 (probe0:mps1:0:3:7): CAM status: Invalid Target ID (probe0:mps1:0:3:7): Error 22, Unretryable error (probe255:mps1:0:0:0): REPORT LUNS. CDB: a0 0 0 0 0 0 0 0 0 10 0 0 (probe255:mps1:0:0:0): CAM status: SCSI Status Error (probe255:mps1:0:0:0): SCSI status: Check Condition (probe255:mps1:0:0:0): SCSI sense: ILLEGAL REQUEST asc:20,0 (Invalid command operation code) (probe255:mps1:0:0:0): Command Specific Info: 0x22222222 (probe255:mps1:0:0:0): Error 22, Unretryable error (probe1:mps1:0:3:0): INQUIRY. CDB: 12 0 0 0 24 0 (probe1:mps1:0:3:0): CAM status: Invalid Target ID (probe1:mps1:0:3:0): Error 22, Unretryable error (probe1:mps1:0:3:0): INQUIRY. CDB: 12 0 0 0 24 0 (probe1:mps1:0:3:0): CAM status: Invalid Target ID (probe1:mps1:0:3:0): Error 22, Unretryable error (probe0:mps1:0:3:1): INQUIRY. CDB: 12 20 0 0 24 0 (probe0:mps1:0:3:1): CAM status: Invalid Target ID (probe0:mps1:0:3:1): Error 22, Unretryable error (probe0:mps1:0:3:1): INQUIRY. CDB: 12 20 0 0 24 0 (probe0:mps1:0:3:1): CAM status: Invalid Target ID (probe0:mps1:0:3:1): Error 22, Unretryable error (probe4:mps1:0:3:2): INQUIRY. CDB: 12 40 0 0 24 0 (probe4:mps1:0:3:2): CAM status: Invalid Target ID (probe4:mps1:0:3:2): Error 22, Unretryable error (probe0:mps1:0:3:2): INQUIRY. CDB: 12 40 0 0 24 0 (probe0:mps1:0:3:2): CAM status: Invalid Target ID (probe0:mps1:0:3:2): Error 22, Unretryable error (probe4:mps1:0:3:3): INQUIRY. CDB: 12 60 0 0 24 0 (probe4:mps1:0:3:3): CAM status: Invalid Target ID (probe4:mps1:0:3:3): Error 22, Unretryable error (probe0:mps1:0:3:3): INQUIRY. CDB: 12 60 0 0 24 0 (probe0:mps1:0:3:3): CAM status: Invalid Target ID (probe0:mps1:0:3:3): Error 22, Unretryable error (probe4:mps1:0:3:4): INQUIRY. CDB: 12 80 0 0 24 0 (probe4:mps1:0:3:4): CAM status: Invalid Target ID (probe4:mps1:0:3:4): Error 22, Unretryable error (probe0:mps1:0:3:4): INQUIRY. CDB: 12 80 0 0 24 0 (probe0:mps1:0:3:4): CAM status: Invalid Target ID (probe0:mps1:0:3:4): Error 22, Unretryable error (probe4:mps1:0:3:5): INQUIRY. CDB: 12 a0 0 0 24 0 (probe4:mps1:0:3:5): CAM status: Invalid Target ID (probe4:mps1:0:3:5): Error 22, Unretryable error (probe0:mps1:0:3:5): INQUIRY. CDB: 12 a0 0 0 24 0 (probe0:mps1:0:3:5): CAM status: Invalid Target ID (probe0:mps1:0:3:5): Error 22, Unretryable error (probe4:mps1:0:3:6): INQUIRY. CDB: 12 c0 0 0 24 0 (probe4:mps1:0:3:6): CAM status: Invalid Target ID (probe4:mps1:0:3:6): Error 22, Unretryable error (probe0:mps1:0:3:6): INQUIRY. CDB: 12 c0 0 0 24 0 (probe0:mps1:0:3:6): CAM status: Invalid Target ID (probe0:mps1:0:3:6): Error 22, Unretryable error (probe4:mps1:0:3:7): INQUIRY. CDB: 12 e0 0 0 24 0 (probe4:mps1:0:3:7): CAM status: Invalid Target ID (probe4:mps1:0:3:7): Error 22, Unretryable error (probe0:mps1:0:3:7): INQUIRY. CDB: 12 e0 0 0 24 0 (probe0:mps1:0:3:7): CAM status: Invalid Target ID (probe0:mps1:0:3:7): Error 22, Unretryable error (probe2:mps1:0:0:0): REPORT LUNS. CDB: a0 0 0 0 0 0 0 0 0 10 0 0 (probe2:mps1:0:0:0): CAM status: SCSI Status Error (probe2:mps1:0:0:0): SCSI status: Check Condition (probe2:mps1:0:0:0): SCSI sense: ILLEGAL REQUEST asc:20,0 (Invalid command operation code) (probe2:mps1:0:0:0): Field Replaceable Unit: 56 (probe2:mps1:0:0:0): Command Specific Info: 0x33353839 (probe2:mps1:0:0:0): Error 22, Unretryable error da0 at mps0 bus 0 scbus0 target 3 lun 0 da0: Fixed Direct Access SCSI-6 device da0: 600.000MB/s transfers da0: Command Queueing enabled da0: 244198MB (500118192 512 byte sectors: 255H 63S/T 31130C) da8 at mps1 bus 0 scbus1 target 0 lun 0 da8: Fixed Direct Access SCSI-6 device da8: 150.000MB/s transfers da8: Command Queueing enabled da8: 68664MB (140623872 512 byte sectors: 255H 63S/T 8753C) da4 at mps0 bus 0 scbus0 target 8 lun 0 da4: Fixed Direct Access SCSI-6 device da4: 600.000MB/s transfers da4: Command Queueing enabled da4: 2861588MB (5860533168 512 byte sectors: 255H 63S/T 364801C) da2 at mps0 bus 0 scbus0 target 6 lun 0 da2: Fixed Direct Access SCSI-6 device da2: 600.000MB/s transfers da2: Command Queueing enabled da2: 2861588MB (5860533168 512 byte sectors: 255H 63S/T 364801C) da3 at mps0 bus 0 scbus0 target 7 lun 0 da3: Fixed Direct Access SCSI-6 device da3: 600.000MB/s transfers da3: Command Queueing enabled da3: 2861588MB (5860533168 512 byte sectors: 255H 63S/T 364801C) da1 at mps0 bus 0 scbus0 target 5 lun 0 da1: Fixed Direct Access SCSI-6 device da1: 600.000MB/s transfers da1: Command Queueing enabled da1: 238475MB (488397168 512 byte sectors: 255H 63S/T 30401C) da5 at mps0 bus 0 scbus0 target 9 lun 0 da5: Fixed Direct Access SCSI-6 device da5: 600.000MB/s transfers da5: Command Queueing enabled da5: 2861588MB (5860533168 512 byte sectors: 255H 63S/T 364801C) da7 at mps0 bus 0 scbus0 target 11 lun 0 da7: Fixed Direct Access SCSI-6 device da7: 600.000MB/s transfers da7: Command Queueing enabled da7: 2861588MB (5860533168 512 byte sectors: 255H 63S/T 364801C) da6 at mps0 bus 0 scbus0 target 10 lun 0 da6: Fixed Direct Access SCSI-6 device da6: 600.000MB/s transfers da6: Command Queueing enabled da6: 2861588MB (5860533168 512 byte sectors: 255H 63S/T 364801C) SMP: AP CPU #1 Launched! cd0 at ahcich0 bus 0 scbus2 target 0 lun 0 SMP: AP CPU #41 Launched! cd0: SMP: AP CPU #34 Launched! Removable CD-ROM SCSI-0 device SMP: AP CPU #35 Launched! cd0: 150.000MB/s transfersSMP: AP CPU #2 Launched! (SATA 1.x, SMP: AP CPU #3 Launched! UDMA6, SMP: AP CPU #39 Launched! ATAPI 12bytes, PIO 8192bytesSMP: AP CPU #5 Launched! ) SMP: AP CPU #12 Launched! cd0: Attempt to query device size failed: NOT READY, Medium not present - tray closed SMP: AP CPU #44 Launched! SMP: AP CPU #20 Launched! SMP: AP CPU #19 Launched! SMP: AP CPU #38 Launched! SMP: AP CPU #37 Launched! SMP: AP CPU #25 Launched! SMP: AP CPU #40 Launched! SMP: AP CPU #18 Launched! SMP: AP CPU #4 Launched! SMP: AP CPU #59 Launched! SMP: AP CPU #8 Launched! SMP: AP CPU #63 Launched! SMP: AP CPU #50 Launched! SMP: AP CPU #30 Launched! SMP: AP CPU #26 Launched! SMP: AP CPU #45 Launched! SMP: AP CPU #29 Launched! SMP: AP CPU #32 Launched! SMP: AP CPU #22 Launched! SMP: AP CPU #55 Launched! SMP: AP CPU #27 Launched! SMP: AP CPU #6 Launched! SMP: AP CPU #31 Launched! SMP: AP CPU #14 Launched! SMP: AP CPU #15 Launched! SMP: AP CPU #42 Launched! SMP: AP CPU #28 Launched! SMP: AP CPU #48 Launched! SMP: AP CPU #21 Launched! SMP: AP CPU #54 Launched! SMP: AP CPU #58 Launched! SMP: AP CPU #23 Launched! SMP: AP CPU #10 Launched! SMP: AP CPU #13 Launched! SMP: AP CPU #57 Launched! SMP: AP CPU #62 Launched! SMP: AP CPU #7 Launched! SMP: AP CPU #61 Launched! SMP: AP CPU #53 Launched! SMP: AP CPU #16 Launched! SMP: AP CPU #46 Launched! SMP: AP CPU #52 Launched! SMP: AP CPU #9 Launched! SMP: AP CPU #11 Launched! SMP: AP CPU #24 Launched! SMP: AP CPU #47 Launched! SMP: AP CPU #56 Launched! SMP: AP CPU #49 Launched! SMP: AP CPU #17 Launched! SMP: AP CPU #43 Launched! SMP: AP CPU #36 Launched! SMP: AP CPU #33 Launched! SMP: AP CPU #51 Launched! SMP: AP CPU #60 Launched! da9 at mps1 bus 0 scbus1 target 9 lun 0 da9: Fixed Direct Access SCSI-6 device da9: 600.000MB/s transfers da9: Command Queueing enabled da9: 3815447MB (7814037168 512 byte sectors: 255H 63S/T 486401C) da10 at mps1 bus 0 scbus1 target 11 lun 0 da10: Fixed Direct Access SCSI-6 device da10: 600.000MB/s transfers da10: Command Queueing enabled da10: 3815447MB (7814037168 512 byte sectors: 255H 63S/T 486401C) Timecounter "TSC-low" frequency 8594040 Hz quality 800 ugen5.3: at usbus5 ukbd0: on usbus5 kbd2 at ukbd0 ums0: on usbus5 ums0: 3 buttons and [Z] coordinates ID=0 Trying to mount root from ufs:/dev/gpt/disk0 [rw]... Setting hostuuid: 03000200-0400-0500-0006-000700080009. Setting hostid: 0xfe4ac89c. ZFS filesystem version 5 ZFS storage pool version 28 Entropy harvesting: interrupts ethernet point_to_point kickstart. Starting file system checks: /dev/gpt/disk0: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/gpt/disk0: clean, 14195352 free (12176 frags, 1772897 blocks, 0.1% fragmentation) Mounting local file systems:. Setting hostname: mc. Starting Network: lo0 igb0 igb1 em0 plip0. lo0: flags=8049 metric 0 mtu 16384 options=600003 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0xc inet 127.0.0.1 netmask 0xff000000 nd6 options=21 em0: link state changed to UP igb0: flags=8c02igb0: link state changed to UP metric 0 mtu 1500 options=401bb ether 00:e0:81:c8:ee:8a nd6 options=29 media: Ethernet autoselect (1000baseT ) status: active igb1: flags=8c02 metric 0 mtu 1500 options=401bb ether 00:e0:81:c8:ee:8b nd6 options=29 media: Ethernet autoselect status: no carrier em0: flags=8843 metric 0 mtu 1500 options=4219b ether 00:e0:81:c8:ee:ff inet 192.168.23.10 netmask 0xffffff00 broadcast 192.168.23.255 inet6 fe80::2e0:81ff:fec8:eeff%em0 prefixlen 64 scopeid 0x3 nd6 options=29 media: Ethernet autoselect (1000baseT ) status: active plip0: flags=8810 metric 0 mtu 1500 nd6 options=29 Starting devd. Starting Network: igb0. igb0: flags=8c02 metric 0 mtu 1500 options=401bb ether 00:e0:81:c8:ee:8a nd6 options=29 media: Ethernet autoselect (1000baseT ) status: active Starting Network: igb1. igb1: flags=8c02 metric 0 mtu 1500 options=401bb ether 00:e0:81:c8:ee:8b nd6 options=29 media: Ethernet autoselect status: no carrier Starting Network: usbus0. Starting Network: usbus1. Starting Network: usbus2. Starting Network: usbus3. Starting Network: usbus4. Starting Network: usbus5. Starting Network: usbus6. Starting Network: plip0. plip0: flags=8810 metric 0 mtu 1500 nd6 options=29 Starting ums0 moused. add net default: gateway 192.168.23.1 add net ::ffff:0.0.0.0: gateway ::1 add net ::0.0.0.0: gateway ::1 add net fe80::: gateway ::1 add net ff02::: gateway ::1 ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/kde4/lib /usr/local/lib/compat /usr/local/lib/gcc47 /usr/local/lib/qt4 32-bit compatibility ldconfig path: /usr/lib32 /usr/local/lib32/compat Creating and/or trimming log files. Starting syslogd. No core dumps found. Additional ABI support: linux. Setting date via ntp. 14 Oct 17:42:47 ntpdate[1793]: step time server 192.168.23.1 offset -0.758215 sec Clearing /tmp. Updating motd:. Starting ntpd. Starting smartd. (pass11:ahcich0:0:0:0): ATA_IDENTIFY. ACB: ec 00 00 00 00 40 00 00 00 00 01 00 (pass11:ahcich0:0:0:0): CAM status: ATA Status Error (pass11:ahcich0:0:0:0): ATA status: 51 (DRDY SERV ERR), error: 04 (ABRT ) (pass11:ahcich0:0:0:0): RES: 51 04 00 14 eb 40 00 00 00 03 00 Starting rsyncd. Starting munin_node. Starting dumper. Configuring syscons: blanktime screensaver. Starting sshd. Starting cron. Sun Oct 14 17:43:00 PDT 2012 FreeBSD/amd64 (mc) (ttyu0) login: NMI ... going to debugger [ thread pid 11 tid 100003 ] Stopped at acpi_cpu_c1+0x2: ret Top displayed on another server: last pid: 48210; load averages: 0.05, 0.03, 0.01 up 0+15:00:19 08:41:44 85 processes: 1 running, 84 sleeping CPU: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle Mem: 119M Active, 2593M Inact, 34G Wired, 287M Buf, 87G Free ARC: 32G Total, 4110M MRU, 27G MFU, 43M Anon, 399M Header, 143M Other Swap: 233G Total, 233G Free PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 2817 root 11 32 4 180M 110M tx->tx 51 21.3H 0.00% pbzip2 2826 root 11 31 4 180M 112M tx->tx 21 21.3H 0.00% pbzip2 2775 root 11 27 4 176M 107M tx->tx 26 21.1H 0.00% pbzip2 2766 root 11 32 4 180M 110M tx->tx 55 20.9H 0.00% pbzip2 2790 root 11 26 4 180M 112M tx->tx 0 20.7H 0.00% pbzip2 2760 root 11 31 4 180M 107M tx->tx 40 20.7H 0.00% pbzip2 2814 root 11 31 4 180M 113M tx->tx 56 20.6H 0.00% pbzip2 2802 root 11 32 4 180M 112M tx->tx 8 20.6H 0.00% pbzip2 2778 root 11 29 4 180M 113M tx->tx 61 20.5H 0.00% pbzip2 2757 root 11 31 4 180M 110M tx->tx 56 20.5H 0.00% pbzip2 2799 root 11 30 4 180M 113M tx->tx 26 20.4H 0.00% pbzip2 2811 root 11 31 4 180M 112M tx->tx 32 20.4H 0.00% pbzip2 2805 root 11 33 4 180M 109M tx->tx 37 20.4H 0.00% pbzip2 2808 root 11 39 4 180M 112M tx->tx 20 20.3H 0.00% pbzip2 2820 root 11 30 4 180M 115M tx->tx 48 20.3H 0.00% pbzip2 2793 root 11 31 4 180M 111M tx->tx 36 20.3H 0.00% pbzip2 2763 root 11 31 4 180M 113M tx->tx 34 20.3H 0.00% pbzip2 2796 root 11 32 4 180M 114M tx->tx 7 20.2H 0.00% pbzip2 2823 root 11 33 4 180M 115M tx->tx 0 20.1H 0.00% pbzip2 2781 root 11 33 4 180M 113M tx->tx 47 20.1H 0.00% pbzip2 2787 root 11 32 4 180M 116M tx->tx 60 19.8H 0.00% pbzip2 2772 root 11 34 4 180M 112M tx->tx 39 19.6H 0.00% pbzip2 2769 root 11 29 4 180M 117M tx->tx 21 19.6H 0.00% pbzip2 2784 root 11 32 4 180M 114M tx->tx 55 19.6H 0.00% pbzip2 2819 root 1 20 0 28648K 5500K pipewr 29 43:28 0.00% john 2768 root 1 20 0 28648K 5392K pipewr 63 43:07 0.00% john 2783 root 1 20 0 28648K 5424K pipewr 53 41:52 0.00% john 2732 root 1 20 0 16508K 2676K CPU36 18 40:09 0.00% top 2780 root 1 20 0 28648K 5424K pipewr 55 40:04 0.00% john 2777 root 1 20 0 28648K 5424K pipewr 45 39:11 0.00% john 2810 root 1 20 0 28648K 5464K pipewr 21 39:02 0.00% john 2816 root 1 20 0 28648K 5500K pipewr 17 38:44 0.00% john 2822 root 1 20 0 28648K 5496K pipewr 37 38:29 0.00% john 2786 root 1 20 0 28648K 5440K pipewr 29 38:25 0.00% john 2825 root 1 20 0 28648K 5500K pipewr 31 37:09 0.00% john 2807 root 1 20 0 28648K 5464K pipewr 46 36:35 0.00% john 2813 root 1 20 0 28648K 5500K pipewr 58 35:56 0.00% john 2771 root 1 20 0 28648K 5392K pipewr 41 34:52 0.00% john 2798 root 1 20 0 28648K 5440K pipewr 36 34:49 0.00% john 2762 root 1 22 0 28648K 5388K zio->i 50 34:47 0.00% john 2795 root 1 20 0 28648K 5440K pipewr 45 34:21 0.00% john And from DDB a ps output: db> ps pid ppid pgrp uid state wmesg wchan cmd 70648 70645 70632 0 S piperd 0xfffffe006796e2d8 wc 70647 70645 70632 0 S piperd 0xfffffe006e46e000 tee 70646 70645 70632 0 D tx->tx_q 0xfffffe0059b60240 find 70645 70641 70632 0 S wait 0xfffffe006e4974a0 sh 70641 70639 70632 0 S piperd 0xfffffe006ea3ab60 sh 70640 70632 70632 0 S piperd 0xfffffe0a765cc2d8 mail 70639 70632 70632 0 S wait 0xfffffe0067716940 sh 70632 70630 70632 0 Ss wait 0xfffffe00674dd940 sh 70630 2164 2164 0 S piperd 0xfffffe0067fc0b60 cron 2826 2736 2825 0 D+ (threaded) pbzip2 103329 D tx->tx_q 0xfffffe0059b60240 pbzip2 103328 S uwait 0xfffffe003fce9b00 pbzip2 103327 S uwait 0xfffffe003fce9a00 pbzip2 103326 S uwait 0xfffffe003fce9900 pbzip2 103325 S uwait 0xfffffe003fce9800 pbzip2 103324 S uwait 0xfffffe003fce9700 pbzip2 103323 S uwait 0xfffffe003fce9600 pbzip2 103322 S uwait 0xfffffe003fce9500 pbzip2 103321 S uwait 0xfffffe003fce9400 pbzip2 103320 S sigwait 0xfffffe005c3ed000 pbzip2 103189 S uwait 0xfffffe00679db100 pbzip2 2825 2736 2825 0 S+ pipewr 0xfffffe003ff795b0 john 2823 2736 2822 0 D+ (threaded) pbzip2 103385 D tx->tx_q 0xfffffe0059b60240 pbzip2 103384 S uwait 0xfffffe006701db80 pbzip2 103383 S uwait 0xfffffe006701da80 pbzip2 103382 S uwait 0xfffffe006701d980 pbzip2 103381 S uwait 0xfffffe006701d880 pbzip2 103380 S uwait 0xfffffe006701d780 pbzip2 103379 S uwait 0xfffffe006701d680 pbzip2 103378 S uwait 0xfffffe006701d580 pbzip2 103376 S uwait 0xfffffe005a7d2c80 pbzip2 103375 S sigwait 0xfffffe005c3f1000 pbzip2 103186 S uwait 0xfffffe00679dac80 pbzip2 2822 2736 2822 0 S+ pipewr 0xfffffe00678eb5b0 john 2820 2736 2819 0 D+ (threaded) pbzip2 103359 D tx->tx_q 0xfffffe0059b60240 pbzip2 103358 S uwait 0xfffffe003f93a280 pbzip2 103357 S uwait 0xfffffe003f93a180 pbzip2 103356 S uwait 0xfffffe003f93a080 pbzip2 103355 S uwait 0xfffffe003f939e00 pbzip2 103354 S uwait 0xfffffe003f939d00 pbzip2 103353 S uwait 0xfffffe003f939c00 pbzip2 103352 S uwait 0xfffffe003f939b00 pbzip2 103351 S uwait 0xfffffe003f939a00 pbzip2 103350 S sigwait 0xfffffe005c3f4000 pbzip2 103183 S uwait 0xfffffe00679da980 pbzip2 2819 2736 2819 0 S+ pipewr 0xfffffe00678eb000 john 2817 2736 2816 0 D+ (threaded) pbzip2 103409 D tx->tx_q 0xfffffe0059b60240 pbzip2 103408 S uwait 0xfffffe00595ea480 pbzip2 103407 S uwait 0xfffffe00595ea280 pbzip2 103406 S uwait 0xfffffe0059bd2c80 pbzip2 103377 S uwait 0xfffffe006748f800 pbzip2 103374 S uwait 0xfffffe003f8a1500 pbzip2 103373 S uwait 0xfffffe003f0c1900 pbzip2 103372 S uwait 0xfffffe003f8a1880 pbzip2 103371 S uwait 0xfffffe003f8a1b00 pbzip2 103370 S sigwait 0xfffffe005c3f8000 pbzip2 103180 S uwait 0xfffffe00679da680 pbzip2 2816 2736 2816 0 S+ pipewr 0xfffffe00678ea888 john 2814 2736 2813 0 D+ (threaded) pbzip2 103405 D tx->tx_q 0xfffffe0059b60240 pbzip2 103404 S uwait 0xfffffe005a7d3700 pbzip2 103403 S uwait 0xfffffe005a7d3600 pbzip2 103402 S uwait 0xfffffe005a7d3500 pbzip2 103401 S uwait 0xfffffe005a7d3400 pbzip2 103400 S uwait 0xfffffe005a7d3300 pbzip2 103399 S uwait 0xfffffe005a7d3200 pbzip2 103398 S uwait 0xfffffe005a7d3100 pbzip2 103397 S uwait 0xfffffe005a7d3000 pbzip2 103396 S sigwait 0xfffffe005c3fb000 pbzip2 103177 S uwait 0xfffffe00679da380 pbzip2 2813 2736 2813 0 S+ pipewr 0xfffffe00678ea2d8 john 2811 2736 2810 0 D+ (threaded) pbzip2 103239 D tx->tx_q 0xfffffe0059b60240 pbzip2 103238 S uwait 0xfffffe003f835280 pbzip2 103237 S uwait 0xfffffe003f835180 pbzip2 103236 S uwait 0xfffffe003f835080 pbzip2 103235 S uwait 0xfffffe003f834e00 pbzip2 103234 S uwait 0xfffffe003f834d00 pbzip2 103233 S uwait 0xfffffe003f834c00 pbzip2 103232 S uwait 0xfffffe003f834b00 pbzip2 103231 S uwait 0xfffffe003f834a00 pbzip2 103230 S sigwait 0xfffffe005a637000 pbzip2 101845 S uwait 0xfffffe00590b1680 pbzip2 2810 2736 2810 0 S+ pipewr 0xfffffe005cc515b0 john 2808 2736 2807 0 D+ (threaded) pbzip2 103299 D tx->tx_q 0xfffffe0059b60240 pbzip2 103298 S uwait 0xfffffe006780d380 pbzip2 103297 S uwait 0xfffffe006780d280 pbzip2 103296 S uwait 0xfffffe006780d180 pbzip2 103295 S uwait 0xfffffe006780d080 pbzip2 103294 S uwait 0xfffffe006780ce00 pbzip2 103293 S uwait 0xfffffe006780cd00 pbzip2 103292 S uwait 0xfffffe006780cc00 pbzip2 103287 S uwait 0xfffffe006780cb00 pbzip2 103286 S sigwait 0xfffffe005ab95000 pbzip2 102389 S uwait 0xfffffe00590d0000 pbzip2 2807 2736 2807 0 S+ pipewr 0xfffffe005cc7c000 john 2805 2736 2804 0 D+ (threaded) pbzip2 103249 D tx->tx_q 0xfffffe0059b60240 pbzip2 103248 S uwait 0xfffffe005905d300 pbzip2 103247 S uwait 0xfffffe005905d200 pbzip2 103246 S uwait 0xfffffe005905d100 pbzip2 103245 S uwait 0xfffffe005905d000 pbzip2 103244 S uwait 0xfffffe0059051d80 pbzip2 103243 S uwait 0xfffffe0059051c80 pbzip2 103242 S uwait 0xfffffe0059051b80 pbzip2 103241 S uwait 0xfffffe0059051a80 pbzip2 103240 S sigwait 0xfffffe005a269000 pbzip2 102391 S uwait 0xfffffe00590d0200 pbzip2 2804 2736 2804 0 S+ pipewr 0xfffffe006e8a0b60 john 2802 2736 2801 0 D+ (threaded) pbzip2 103369 D tx->tx_q 0xfffffe0059b60240 pbzip2 103368 S uwait 0xfffffe005a917480 pbzip2 103367 S uwait 0xfffffe005a917380 pbzip2 103366 S uwait 0xfffffe005a917280 pbzip2 103365 S uwait 0xfffffe005a917180 pbzip2 103364 S uwait 0xfffffe005a917080 pbzip2 103363 S uwait 0xfffffe005a916e00 pbzip2 103362 S uwait 0xfffffe005a916d00 pbzip2 103361 S uwait 0xfffffe005a916c00 pbzip2 103360 S sigwait 0xfffffe0059a00000 pbzip2 102388 S uwait 0xfffffe00590b1d80 pbzip2 2801 2736 2801 0 S+ pipewr 0xfffffe0067a115b0 john 2799 2736 2798 0 D+ (threaded) pbzip2 103419 D tx->tx_q 0xfffffe0059b60240 pbzip2 103418 S uwait 0xfffffe006780d780 pbzip2 103417 S uwait 0xfffffe006780d680 pbzip2 103416 S uwait 0xfffffe006780d580 pbzip2 103415 S uwait 0xfffffe003f6d1b80 pbzip2 103414 S uwait 0xfffffe003f6d1a80 pbzip2 103413 S uwait 0xfffffe003f6d1980 pbzip2 103412 S uwait 0xfffffe005c0b3800 pbzip2 103411 S uwait 0xfffffe005c634300 pbzip2 103410 S sigwait 0xfffffe00599ff000 pbzip2 101688 S uwait 0xfffffe003fee1380 pbzip2 2798 2736 2798 0 S+ pipewr 0xfffffe0067dc8888 john 2796 2736 2795 0 D+ (threaded) pbzip2 103339 D tx->tx_q 0xfffffe0059b60240 pbzip2 103338 S uwait 0xfffffe003f8a2400 pbzip2 103337 S uwait 0xfffffe003f8a2600 pbzip2 103336 S uwait 0xfffffe003f8a2880 pbzip2 103335 S uwait 0xfffffe003f8a2a80 pbzip2 103334 S uwait 0xfffffe003f8a2c80 pbzip2 103333 S uwait 0xfffffe003f8a3000 pbzip2 103332 S uwait 0xfffffe003f8a3100 pbzip2 103331 S uwait 0xfffffe006eb61c00 pbzip2 103330 S sigwait 0xfffffe0059a23000 pbzip2 101411 S uwait 0xfffffe005c630380 pbzip2 2795 2736 2795 0 S+ pipewr 0xfffffe0067dc8000 john 2793 2736 2792 0 D+ (threaded) pbzip2 103199 D tx->tx_q 0xfffffe0059b60240 pbzip2 103198 S uwait 0xfffffe0059679800 pbzip2 103197 S uwait 0xfffffe0059679700 pbzip2 103196 S uwait 0xfffffe0059679600 pbzip2 103195 S uwait 0xfffffe0059679500 pbzip2 103194 S uwait 0xfffffe0059679400 pbzip2 103193 S uwait 0xfffffe0059679300 pbzip2 103192 S uwait 0xfffffe0059679200 pbzip2 103191 S uwait 0xfffffe0059679100 pbzip2 103190 S sigwait 0xfffffe0059a20000 pbzip2 101689 S uwait 0xfffffe003fee1480 pbzip2 2792 2736 2792 0 S+ pipewr 0xfffffe0067dc85b0 john 2790 2736 2789 0 D+ (threaded) pbzip2 103269 D tx->tx_q 0xfffffe0059b60240 pbzip2 103268 S uwait 0xfffffe006733b780 pbzip2 103267 S uwait 0xfffffe006733b680 pbzip2 103266 S uwait 0xfffffe006733b580 pbzip2 103265 S uwait 0xfffffe006733b480 pbzip2 103264 S uwait 0xfffffe006733b380 pbzip2 103263 S uwait 0xfffffe006733b280 pbzip2 103262 S uwait 0xfffffe006733b180 pbzip2 103261 S uwait 0xfffffe006733b080 pbzip2 103260 S sigwait 0xfffffe003fea3000 pbzip2 101746 S uwait 0xfffffe003fee1780 pbzip2 2789 2736 2789 0 S+ pipewr 0xfffffe0067dc8b60 john 2787 2736 2786 0 D+ (threaded) pbzip2 103319 D tx->tx_q 0xfffffe0059b60240 pbzip2 103318 S uwait 0xfffffe005c443880 pbzip2 103315 S uwait 0xfffffe005c443980 pbzip2 103314 S uwait 0xfffffe006ec7a180 pbzip2 103311 S uwait 0xfffffe006ec7a080 pbzip2 103307 S uwait 0xfffffe006ec79e00 pbzip2 103306 S uwait 0xfffffe006ec79d00 pbzip2 103305 S uwait 0xfffffe006ec79c00 pbzip2 103303 S uwait 0xfffffe006ec79b00 pbzip2 103302 S sigwait 0xfffffe005caf3000 pbzip2 101711 S uwait 0xfffffe005c9f8d80 pbzip2 2786 2736 2786 0 S+ pipewr 0xfffffe0067a11000 john 2784 2736 2783 0 D+ (threaded) pbzip2 103229 D tx->tx_q 0xfffffe0059b60240 pbzip2 103228 S uwait 0xfffffe0059fdbc80 pbzip2 103227 S uwait 0xfffffe0059fdbb80 pbzip2 103226 S uwait 0xfffffe0059fdba80 pbzip2 103225 S uwait 0xfffffe0059fdb980 pbzip2 103224 S uwait 0xfffffe0059fdb880 pbzip2 103223 S uwait 0xfffffe0059fdb780 pbzip2 103222 S uwait 0xfffffe0059fdb680 pbzip2 103221 S uwait 0xfffffe0059fdb580 pbzip2 103220 S sigwait 0xfffffe005caf6000 pbzip2 102401 S uwait 0xfffffe005cabd800 pbzip2 2783 2736 2783 0 S+ pipewr 0xfffffe0067a10888 john 2781 2736 2780 0 D+ (threaded) pbzip2 103317 D tx->tx_q 0xfffffe0059b60240 pbzip2 103316 S uwait 0xfffffe005904f680 pbzip2 103313 S uwait 0xfffffe005904f580 pbzip2 103312 S uwait 0xfffffe005904f480 pbzip2 103310 S uwait 0xfffffe005904f380 pbzip2 103309 S uwait 0xfffffe005904f280 pbzip2 103308 S uwait 0xfffffe005904f180 pbzip2 103304 S uwait 0xfffffe005904f080 pbzip2 103301 S uwait 0xfffffe005904ee00 pbzip2 103300 S sigwait 0xfffffe005cafa000 pbzip2 102583 S uwait 0xfffffe005cabdd00 pbzip2 2780 2736 2780 0 S+ pipewr 0xfffffe0067a105b0 john 2778 2736 2777 0 D+ (threaded) pbzip2 103395 D tx->tx_q 0xfffffe0059b60240 pbzip2 103394 S uwait 0xfffffe006e9db680 pbzip2 103393 S uwait 0xfffffe006e9db580 pbzip2 103392 S uwait 0xfffffe006e9db480 pbzip2 103391 S uwait 0xfffffe006e9db380 pbzip2 103390 S uwait 0xfffffe006e9db280 pbzip2 103389 S uwait 0xfffffe006e9db180 pbzip2 103388 S uwait 0xfffffe006e9db080 pbzip2 103387 S uwait 0xfffffe006e9dae00 pbzip2 103386 S sigwait 0xfffffe005cafd000 pbzip2 102582 S uwait 0xfffffe005cabdc00 pbzip2 2777 2736 2777 0 S+ pipewr 0xfffffe00678e85b0 john 2775 2736 2774 0 D+ (threaded) pbzip2 103279 D tx->tx_q 0xfffffe0059b60240 pbzip2 103278 S uwait 0xfffffe005c9ac200 pbzip2 103277 S uwait 0xfffffe005c9ac100 pbzip2 103276 S uwait 0xfffffe005c9ac000 pbzip2 103275 S uwait 0xfffffe005c9abd80 pbzip2 103274 S uwait 0xfffffe005c9abc80 pbzip2 103273 S uwait 0xfffffe005c9abb80 pbzip2 103272 S uwait 0xfffffe005c9aba80 pbzip2 103271 S uwait 0xfffffe005c9ab980 pbzip2 103270 S sigwait 0xfffffe005cb01000 pbzip2 101674 S uwait 0xfffffe003f23b600 pbzip2 2774 2736 2774 0 S+ pipewr 0xfffffe005ca5a2d8 john 2772 2736 2771 0 D+ (threaded) pbzip2 103219 D tx->tx_q 0xfffffe0059b60240 pbzip2 103218 S uwait 0xfffffe006701b200 pbzip2 103217 S uwait 0xfffffe006701b100 pbzip2 103216 S uwait 0xfffffe006701b000 pbzip2 103215 S uwait 0xfffffe006701ad80 pbzip2 103214 S uwait 0xfffffe006701ac80 pbzip2 103213 S uwait 0xfffffe006701ab80 pbzip2 103212 S uwait 0xfffffe006701aa80 pbzip2 103211 S uwait 0xfffffe006701a980 pbzip2 103210 S sigwait 0xfffffe005cb04000 pbzip2 102560 S uwait 0xfffffe005cabda00 pbzip2 2771 2736 2771 0 S+ pipewr 0xfffffe005ca5bb60 john 2769 2736 2768 0 D+ (threaded) pbzip2 103291 D tx->tx_q 0xfffffe0059b60240 pbzip2 103290 S uwait 0xfffffe005a97f300 pbzip2 103289 S uwait 0xfffffe005a97f200 pbzip2 103288 S uwait 0xfffffe005a97f100 pbzip2 103285 S uwait 0xfffffe005a97f000 pbzip2 103284 S uwait 0xfffffe005a97ed80 pbzip2 103283 S uwait 0xfffffe005a97ec80 pbzip2 103282 S uwait 0xfffffe005a97eb80 pbzip2 103281 S uwait 0xfffffe005a97ea80 pbzip2 103280 S sigwait 0xfffffe005cb08000 pbzip2 102400 S uwait 0xfffffe005cabd700 pbzip2 2768 2736 2768 0 S+ pipewr 0xfffffe006e6602d8 john 2766 2736 2765 0 D+ (threaded) pbzip2 103259 D tx->tx_q 0xfffffe0059b60240 pbzip2 103258 S uwait 0xfffffe0067926180 pbzip2 103257 S uwait 0xfffffe0067926080 pbzip2 103256 S uwait 0xfffffe0067925e00 pbzip2 103255 S uwait 0xfffffe0067925d00 pbzip2 103254 S uwait 0xfffffe0067925c00 pbzip2 103253 S uwait 0xfffffe0067925b00 pbzip2 103252 S uwait 0xfffffe0067925a00 pbzip2 103251 S uwait 0xfffffe0067925900 pbzip2 103250 S sigwait 0xfffffe005cb0b000 pbzip2 103007 S uwait 0xfffffe006e6eb100 pbzip2 2765 2736 2765 0 S+ pipewr 0xfffffe005ca7a5b0 john 2763 2736 2762 0 D+ (threaded) pbzip2 103429 D tx->tx_q 0xfffffe0059b60240 pbzip2 103428 S uwait 0xfffffe0067023500 pbzip2 103427 S uwait 0xfffffe0067023400 pbzip2 103426 S uwait 0xfffffe0067023300 pbzip2 103425 S uwait 0xfffffe0067023200 pbzip2 103424 S uwait 0xfffffe0067023100 pbzip2 103423 S uwait 0xfffffe0067023000 pbzip2 103422 S uwait 0xfffffe006701dd80 pbzip2 103421 S uwait 0xfffffe0067e2e800 pbzip2 103420 S sigwait 0xfffffe005cb0e000 pbzip2 102181 S piperd 0xfffffe005ca7a000 pbzip2 2762 2736 2762 0 D+ zio->io_ 0xfffffe00593f3690 john 2760 2736 2759 0 D+ (threaded) pbzip2 103209 D tx->tx_q 0xfffffe0059b60240 pbzip2 103208 S uwait 0xfffffe005c632480 pbzip2 103207 S uwait 0xfffffe005c632380 pbzip2 103206 S uwait 0xfffffe005c632280 pbzip2 103205 S uwait 0xfffffe005c632180 pbzip2 103204 S uwait 0xfffffe005c632080 pbzip2 103203 S uwait 0xfffffe005c631e00 pbzip2 103202 S uwait 0xfffffe005c631d00 pbzip2 103201 S uwait 0xfffffe005c631c00 pbzip2 103200 S sigwait 0xfffffe005cb12000 pbzip2 101628 S uwait 0xfffffe005c9f8b80 pbzip2 2759 2736 2759 0 S+ pipewr 0xfffffe005ca5a5b0 john 2757 2736 2756 0 D+ (threaded) pbzip2 103349 D tx->tx_q 0xfffffe0059b60240 pbzip2 103348 S uwait 0xfffffe005cabe880 pbzip2 103347 S uwait 0xfffffe005cabe780 pbzip2 103346 S uwait 0xfffffe005cabe680 pbzip2 103345 S uwait 0xfffffe005cabe580 pbzip2 103344 S uwait 0xfffffe005cabe480 pbzip2 103343 S uwait 0xfffffe005cabe380 pbzip2 103342 S uwait 0xfffffe005cabe280 pbzip2 103341 S uwait 0xfffffe005cabe180 pbzip2 103340 S sigwait 0xfffffe005cb15000 pbzip2 101441 S uwait 0xfffffe005c9f8680 pbzip2 2756 2736 2756 0 S+ pipewr 0xfffffe005ca5a000 john 2736 2733 2736 0 Ss+ ttyin 0xfffffe003f70a4a8 csh 2733 2008 2733 0 Ss select 0xfffffe0067e2c8c0 sshd 2732 2728 2732 0 S+ select 0xfffffe0067146a40 top 2728 2725 2728 0 Ss+ pause 0xfffffe0067fd09e0 csh 2725 2008 2725 0 Ss select 0xfffffe00590e5b40 sshd 2262 1 1 0 S ttydcd 0xfffffe00394a70e8 getty 2261 1 2261 0 Ss+ ttyin 0xfffffe00394a44a8 getty 2259 1 2259 0 Ss+ ttyin 0xfffffe00394ca4a8 getty 2258 1 2258 0 Ss+ ttyin 0xfffffe00394ca8a8 getty 2257 1 2257 0 Ss+ ttyin 0xfffffe00394caca8 getty 2256 1 2256 0 Ss+ ttyin 0xfffffe00394cb0a8 getty 2255 1 2255 0 Ss+ ttyin 0xfffffe00394cb4a8 getty 2254 1 2254 0 Ss+ ttyin 0xfffffe00394cb8a8 getty 2253 1 2253 0 Ss+ ttyin 0xfffffe00394cbca8 getty 2252 1 2252 0 Ss+ ttyin 0xfffffe00394cc0a8 getty 2164 1 2164 0 Ss nanslp 0xffffffff81225ea0 cron 2160 1 2160 25 Ss pause 0xfffffe005c2109e0 sendmail 2157 1 2157 0 Ss select 0xfffffe00676f83c0 sendmail 2008 1 2008 0 Ss select 0xfffffe005ccbe7c0 sshd 1961 1 1961 0 Ss select 0xfffffe006ec772c0 sshd 1892 1 1892 0 Ss select 0xfffffe00590e54c0 perl5.16.0 1886 1 1886 0 Ss select 0xfffffe00c53ae940 rsync 1882 1 1881 0 D cbwait 0xfffffe003f33b048 smartd 1864 1 1864 0 Ss select 0xfffffe006e62e240 ntpd 1778 1 1778 0 Ss select 0xfffffe003f938bc0 syslogd 1572 1 1572 0 Ss select 0xfffffe006748e040 devd 1539 1 1539 0 Ss select 0xfffffe00590e51c0 moused 120 1 120 0 Ss pause 0xfffffe005cee09e0 adjkerntz 41 0 0 0 DL (threaded) [zfskern] 101382 D tx->tx_s 0xfffffe003f4b9210 [txg_thread_enter] 101381 D tx->tx_q 0xfffffe003f4b9230 [txg_thread_enter] 100856 D zio->io_ 0xfffffe005a023d70 [txg_thread_enter] 100855 D tx->tx_q 0xfffffe0059b60230 [txg_thread_enter] 100328 D zio->io_ 0xfffffe00593ee690 [l2arc_feed_thread] 100327 D arc_recl 0xffffffff81934910 [arc_reclaim_thread] 21 0 0 0 DL sdflush 0xffffffff812777bc [softdepflush] 20 0 0 0 DL zio->io_ 0xfffffe01187c5320 [syncer] 19 0 0 0 DL vlruwt 0xfffffe003f4c0000 [vnlru] 18 0 0 0 DL psleep 0xffffffff812709cc [bufdaemon] 17 0 0 0 DL pgzero 0xffffffff8127906c [pagezero] 16 0 0 0 DL psleep 0xffffffff812782b8 [vmdaemon] 9 0 0 0 DL psleep 0xffffffff812df304 [pagedaemon] 8 0 0 0 DL ccb_scan 0xffffffff811e9a30 [xpt_thrd] 7 0 0 0 DL waiting_ 0xffffffff812d3298 [sctp_iterator] 15 0 0 0 DL (threaded) [usb] 100230 D - 0xffffff8005bc7460 [usbus6] 100229 D - 0xffffff8005bc7408 [usbus6] 100228 D - 0xffffff8005bc73b0 [usbus6] 100227 D - 0xffffff8005bc7358 [usbus6] 100224 D - 0xffffff8005bc3e18 [usbus5] 100223 D - 0xffffff8005bc3dc0 [usbus5] 100222 D - 0xffffff8005bc3d68 [usbus5] 100221 D - 0xffffff8005bc3d10 [usbus5] 100219 D - 0xffffff8005bbb460 [usbus4] 100218 D - 0xffffff8005bbb408 [usbus4] 100217 D - 0xffffff8005bbb3b0 [usbus4] 100216 D - 0xffffff8005bbb358 [usbus4] 100215 D - 0xffffff8005bb8460 [usbus3] 100214 D - 0xffffff8005bb8408 [usbus3] 100213 D - 0xffffff8005bb83b0 [usbus3] 100212 D - 0xffffff8005bb8358 [usbus3] 100210 D - 0xffffff8005bb4e18 [usbus2] 100209 D - 0xffffff8005bb4dc0 [usbus2] 100208 D - 0xffffff8005bb4d68 [usbus2] 100207 D - 0xffffff8005bb4d10 [usbus2] 100205 D - 0xffffff8005bac460 [usbus1] 100204 D - 0xffffff8005bac408 [usbus1] 100203 D - 0xffffff8005bac3b0 [usbus1] 100202 D - 0xffffff8005bac358 [usbus1] 100201 D - 0xffffff8005ba9460 [usbus0] 100200 D - 0xffffff8005ba9408 [usbus0] 100199 D - 0xffffff8005ba93b0 [usbus0] 100198 D - 0xffffff8005ba9358 [usbus0] 6 0 0 0 DL mps_scan 0xfffffe0030e9dca8 [mps_scan1] 5 0 0 0 DL mps_scan 0xfffffe0030e9d2a8 [mps_scan0] 4 0 0 0 DL ctl_work 0xffffff8004601000 [ctl_thrd] 3 0 0 0 DL crypto_r 0xffffffff8160d0d8 [crypto returns] 2 0 0 0 DL crypto_w 0xffffffff8160cf80 [crypto] 14 0 0 0 DL - 0xffffffff81224d60 [yarrow] 13 0 0 0 DL (threaded) [geom] 100135 D - 0xffffffff812cbd50 [g_down] 100134 D - 0xffffffff812cbd48 [g_up] 100133 D - 0xffffffff812cbd40 [g_event] 12 0 0 0 WL (threaded) [intr] 100233 I [irq1: atkbd0] 100232 I [irq7: ppc0] 100231 I [swi0: uart uart] 100226 I [irq15: ata1] 100225 I [irq14: ata0] 100220 I [irq19: ehci1] 100211 I [irq18: ohci2 ohci3+] 100206 I [irq17: ehci0] 100197 I [irq16: ohci0 ohci1] 100196 I [irq22: ahci0] 100195 I [irq278: mps1] 100192 I [irq277: mps0] 100189 I [irq276: em0:link] 100187 I [irq275: em0:tx 0] 100185 I [irq274: em0:rx 0] 100184 I [irq273: igb1:link] 100182 I [irq272: igb1:que 7] 100180 I [irq271: igb1:que 6] 100178 I [irq270: igb1:que 5] 100176 I [irq269: igb1:que 4] 100174 I [irq268: igb1:que 3] 100172 I [irq267: igb1:que 2] 100170 I [irq266: igb1:que 1] 100168 I [irq265: igb1:que 0] 100167 I [irq264: igb0:link] 100165 I [irq263: igb0:que 7] 100163 I [irq262: igb0:que 6] 100161 I [irq261: igb0:que 5] 100159 I [irq260: igb0:que 4] 100157 I [irq259: igb0:que 3] 100155 I [irq258: igb0:que 2] 100153 I [irq257: igb0:que 1] 100151 I [irq256: igb0:que 0] 100149 I [swi6: task queue] 100148 I [swi2: cambio] 100143 I [swi5: fast taskq] 100140 I [swi6: Giant taskq] 100132 I [swi1: netisr 0] 100131 I [swi4: clock] 100130 I [swi4: clock] 100129 I [swi4: clock] 100128 I [swi4: clock] 100127 I [swi4: clock] 100126 I [swi4: clock] 100125 I [swi4: clock] 100124 I [swi4: clock] 100123 I [swi4: clock] 100122 I [swi4: clock] 100121 I [swi4: clock] 100120 I [swi4: clock] 100119 I [swi4: clock] 100118 I [swi4: clock] 100117 I [swi4: clock] 100116 I [swi4: clock] 100115 I [swi4: clock] 100114 I [swi4: clock] 100113 I [swi4: clock] 100112 I [swi4: clock] 100111 I [swi4: clock] 100110 I [swi4: clock] 100109 I [swi4: clock] 100108 I [swi4: clock] 100107 I [swi4: clock] 100106 I [swi4: clock] 100105 I [swi4: clock] 100104 I [swi4: clock] 100103 I [swi4: clock] 100102 I [swi4: clock] 100101 I [swi4: clock] 100100 I [swi4: clock] 100099 I [swi4: clock] 100098 I [swi4: clock] 100097 I [swi4: clock] 100096 I [swi4: clock] 100095 I [swi4: clock] 100094 I [swi4: clock] 100093 I [swi4: clock] 100092 I [swi4: clock] 100091 I [swi4: clock] 100090 I [swi4: clock] 100089 I [swi4: clock] 100088 I [swi4: clock] 100087 I [swi4: clock] 100086 I [swi4: clock] 100085 I [swi4: clock] 100084 I [swi4: clock] 100083 I [swi4: clock] 100082 I [swi4: clock] 100081 I [swi4: clock] 100080 I [swi4: clock] 100079 I [swi4: clock] 100078 I [swi4: clock] 100077 I [swi4: clock] 100076 I [swi4: clock] 100075 I [swi4: clock] 100074 I [swi4: clock] 100073 I [swi4: clock] 100072 I [swi4: clock] 100071 I [swi4: clock] 100070 I [swi4: clock] 100069 I [swi4: clock] 100068 I [swi4: clock] 100067 I [swi3: vm] 11 0 0 0 RL (threaded) [idle] 100066 Run CPU 63 [idle: cpu63] 100065 Run CPU 62 [idle: cpu62] 100064 Run CPU 61 [idle: cpu61] 100063 Run CPU 60 [idle: cpu60] 100062 Run CPU 59 [idle: cpu59] 100061 Run CPU 58 [idle: cpu58] 100060 Run CPU 57 [idle: cpu57] 100059 Run CPU 56 [idle: cpu56] 100058 Run CPU 55 [idle: cpu55] 100057 Run CPU 54 [idle: cpu54] 100056 Run CPU 53 [idle: cpu53] 100055 Run CPU 52 [idle: cpu52] 100054 Run CPU 51 [idle: cpu51] 100053 Run CPU 50 [idle: cpu50] 100052 Run CPU 49 [idle: cpu49] 100051 Run CPU 48 [idle: cpu48] 100050 Run CPU 47 [idle: cpu47] 100049 Run CPU 46 [idle: cpu46] 100048 Run CPU 45 [idle: cpu45] 100047 Run CPU 44 [idle: cpu44] 100046 Run CPU 43 [idle: cpu43] 100045 Run CPU 42 [idle: cpu42] 100044 Run CPU 41 [idle: cpu41] 100043 Run CPU 40 [idle: cpu40] 100042 Run CPU 39 [idle: cpu39] 100041 Run CPU 38 [idle: cpu38] 100040 Run CPU 37 [idle: cpu37] 100039 Run CPU 36 [idle: cpu36] 100038 Run CPU 35 [idle: cpu35] 100037 Run CPU 34 [idle: cpu34] 100036 Run CPU 33 [idle: cpu33] 100035 Run CPU 32 [idle: cpu32] 100034 Run CPU 31 [idle: cpu31] 100033 Run CPU 30 [idle: cpu30] 100032 Run CPU 29 [idle: cpu29] 100031 Run CPU 28 [idle: cpu28] 100030 Run CPU 27 [idle: cpu27] 100029 Run CPU 26 [idle: cpu26] 100028 Run CPU 25 [idle: cpu25] 100027 Run CPU 24 [idle: cpu24] 100026 Run CPU 23 [idle: cpu23] 100025 Run CPU 22 [idle: cpu22] 100024 Run CPU 21 [idle: cpu21] 100023 Run CPU 20 [idle: cpu20] 100022 Run CPU 19 [idle: cpu19] 100021 Run CPU 18 [idle: cpu18] 100020 Run CPU 17 [idle: cpu17] 100019 Run CPU 16 [idle: cpu16] 100018 Run CPU 15 [idle: cpu15] 100017 Run CPU 14 [idle: cpu14] 100016 Run CPU 13 [idle: cpu13] 100015 Run CPU 12 [idle: cpu12] 100014 Run CPU 11 [idle: cpu11] 100013 Run CPU 10 [idle: cpu10] 100012 Run CPU 9 [idle: cpu9] 100011 Run CPU 8 [idle: cpu8] 100010 Run CPU 7 [idle: cpu7] 100009 Run CPU 6 [idle: cpu6] 100008 Run CPU 5 [idle: cpu5] 100007 Run CPU 4 [idle: cpu4] 100006 Run CPU 3 [idle: cpu3] 100005 Run CPU 2 [idle: cpu2] 100004 Run CPU 1 [idle: cpu1] 100003 Run CPU 0 [idle: cpu0] 1 0 1 0 SLs wait 0xfffffe00308d1940 [init] 10 0 0 0 DL audit_wo 0xffffffff812d6a18 [audit] 0 0 0 0 DLs (threaded) [kernel] 101428 D - 0xfffffe005c9f7b80 [zil_clean] 101427 D - 0xfffffe005c9ac880 [zil_clean] 101426 D - 0xfffffe003f1bb780 [zil_clean] 101425 D - 0xfffffe003f0bf580 [zil_clean] 101424 D - 0xfffffe005a919500 [zil_clean] 101423 D - 0xfffffe003f836880 [zil_clean] 101380 D - 0xfffffe003f0c1180 [zfs_vn_rele_taskq] 101379 D - 0xfffffe00590d3400 [zio_ioctl_intr] 101378 D - 0xfffffe00590d3480 [zio_ioctl_issue] 101377 D - 0xfffffe00590d3500 [zio_claim_intr] 101376 D - 0xfffffe00590d3580 [zio_claim_issue] 101375 D - 0xfffffe00590d3600 [zio_free_intr] 101374 D - 0xfffffe00590d3680 [zio_free_issue_99] 101373 D - 0xfffffe00590d3680 [zio_free_issue_98] 101372 D - 0xfffffe00590d3680 [zio_free_issue_97] 101371 D - 0xfffffe00590d3680 [zio_free_issue_96] 101370 D - 0xfffffe00590d3680 [zio_free_issue_95] 101369 D - 0xfffffe00590d3680 [zio_free_issue_94] 101368 D - 0xfffffe00590d3680 [zio_free_issue_93] 101367 D - 0xfffffe00590d3680 [zio_free_issue_92] 101366 D - 0xfffffe00590d3680 [zio_free_issue_91] 101365 D - 0xfffffe00590d3680 [zio_free_issue_90] 101364 D - 0xfffffe00590d3680 [zio_free_issue_89] 101363 D - 0xfffffe00590d3680 [zio_free_issue_88] 101362 D - 0xfffffe00590d3680 [zio_free_issue_87] 101361 D - 0xfffffe00590d3680 [zio_free_issue_86] 101360 D - 0xfffffe00590d3680 [zio_free_issue_85] 101359 D - 0xfffffe00590d3680 [zio_free_issue_84] 101358 D - 0xfffffe00590d3680 [zio_free_issue_83] 101357 D - 0xfffffe00590d3680 [zio_free_issue_82] 101356 D - 0xfffffe00590d3680 [zio_free_issue_81] 101355 D - 0xfffffe00590d3680 [zio_free_issue_80] 101354 D - 0xfffffe00590d3680 [zio_free_issue_79] 101353 D - 0xfffffe00590d3680 [zio_free_issue_78] 101352 D - 0xfffffe00590d3680 [zio_free_issue_77] 101351 D - 0xfffffe00590d3680 [zio_free_issue_76] 101350 D - 0xfffffe00590d3680 [zio_free_issue_75] 101349 D - 0xfffffe00590d3680 [zio_free_issue_74] 101348 D - 0xfffffe00590d3680 [zio_free_issue_73] 101347 D - 0xfffffe00590d3680 [zio_free_issue_72] 101346 D - 0xfffffe00590d3680 [zio_free_issue_71] 101345 D - 0xfffffe00590d3680 [zio_free_issue_70] 101344 D - 0xfffffe00590d3680 [zio_free_issue_69] 101343 D - 0xfffffe00590d3680 [zio_free_issue_68] 101342 D - 0xfffffe00590d3680 [zio_free_issue_67] 101341 D - 0xfffffe00590d3680 [zio_free_issue_66] 101340 D - 0xfffffe00590d3680 [zio_free_issue_65] 101339 D - 0xfffffe00590d3680 [zio_free_issue_64] 101338 D - 0xfffffe00590d3680 [zio_free_issue_63] 101337 D - 0xfffffe00590d3680 [zio_free_issue_62] 101336 D - 0xfffffe00590d3680 [zio_free_issue_61] 101335 D - 0xfffffe00590d3680 [zio_free_issue_60] 101334 D - 0xfffffe00590d3680 [zio_free_issue_59] 101333 D - 0xfffffe00590d3680 [zio_free_issue_58] 101332 D - 0xfffffe00590d3680 [zio_free_issue_57] 101331 D - 0xfffffe00590d3680 [zio_free_issue_56] 101330 D - 0xfffffe00590d3680 [zio_free_issue_55] 101329 D - 0xfffffe00590d3680 [zio_free_issue_54] 101328 D - 0xfffffe00590d3680 [zio_free_issue_53] 101327 D - 0xfffffe00590d3680 [zio_free_issue_52] 101326 D - 0xfffffe00590d3680 [zio_free_issue_51] 101325 D - 0xfffffe00590d3680 [zio_free_issue_50] 101324 D - 0xfffffe00590d3680 [zio_free_issue_49] 101323 D - 0xfffffe00590d3680 [zio_free_issue_48] 101322 D - 0xfffffe00590d3680 [zio_free_issue_47] 101321 D - 0xfffffe00590d3680 [zio_free_issue_46] 101320 D - 0xfffffe00590d3680 [zio_free_issue_45] 101319 D - 0xfffffe00590d3680 [zio_free_issue_44] 101318 D - 0xfffffe00590d3680 [zio_free_issue_43] 101317 D - 0xfffffe00590d3680 [zio_free_issue_42] 101316 D - 0xfffffe00590d3680 [zio_free_issue_41] 101315 D - 0xfffffe00590d3680 [zio_free_issue_40] 101314 D - 0xfffffe00590d3680 [zio_free_issue_39] 101313 D - 0xfffffe00590d3680 [zio_free_issue_38] 101312 D - 0xfffffe00590d3680 [zio_free_issue_37] 101311 D - 0xfffffe00590d3680 [zio_free_issue_36] 101310 D - 0xfffffe00590d3680 [zio_free_issue_35] 101309 D - 0xfffffe00590d3680 [zio_free_issue_34] 101308 D - 0xfffffe00590d3680 [zio_free_issue_33] 101307 D - 0xfffffe00590d3680 [zio_free_issue_32] 101306 D - 0xfffffe00590d3680 [zio_free_issue_31] 101305 D - 0xfffffe00590d3680 [zio_free_issue_30] 101304 D - 0xfffffe00590d3680 [zio_free_issue_29] 101303 D - 0xfffffe00590d3680 [zio_free_issue_28] 101302 D - 0xfffffe00590d3680 [zio_free_issue_27] 101301 D - 0xfffffe00590d3680 [zio_free_issue_26] 101300 D - 0xfffffe00590d3680 [zio_free_issue_25] 101299 D - 0xfffffe00590d3680 [zio_free_issue_24] 101298 D - 0xfffffe00590d3680 [zio_free_issue_23] 101297 D - 0xfffffe00590d3680 [zio_free_issue_22] 101296 D - 0xfffffe00590d3680 [zio_free_issue_21] 101295 D - 0xfffffe00590d3680 [zio_free_issue_20] 101294 D - 0xfffffe00590d3680 [zio_free_issue_19] 101293 D - 0xfffffe00590d3680 [zio_free_issue_18] 101292 D - 0xfffffe00590d3680 [zio_free_issue_17] 101291 D - 0xfffffe00590d3680 [zio_free_issue_16] 101290 D - 0xfffffe00590d3680 [zio_free_issue_15] 101289 D - 0xfffffe00590d3680 [zio_free_issue_14] 101288 D - 0xfffffe00590d3680 [zio_free_issue_13] 101287 D - 0xfffffe00590d3680 [zio_free_issue_12] 101286 D - 0xfffffe00590d3680 [zio_free_issue_11] 101285 D - 0xfffffe00590d3680 [zio_free_issue_10] 101284 D - 0xfffffe00590d3680 [zio_free_issue_9] 101283 D - 0xfffffe00590d3680 [zio_free_issue_8] 101282 D - 0xfffffe00590d3680 [zio_free_issue_7] 101281 D - 0xfffffe00590d3680 [zio_free_issue_6] 101280 D - 0xfffffe00590d3680 [zio_free_issue_5] 101279 D - 0xfffffe00590d3680 [zio_free_issue_4] 101278 D - 0xfffffe00590d3680 [zio_free_issue_3] 101277 D - 0xfffffe00590d3680 [zio_free_issue_2] 101276 D - 0xfffffe00590d3680 [zio_free_issue_1] 101275 D - 0xfffffe00590d3680 [zio_free_issue_0] 101274 D - 0xfffffe00590d3700 [zio_write_intr_high] 101273 D - 0xfffffe00590d3700 [zio_write_intr_high] 101272 D - 0xfffffe00590d3700 [zio_write_intr_high] 101271 D - 0xfffffe00590d3700 [zio_write_intr_high] 101270 D - 0xfffffe00590d3700 [zio_write_intr_high] 101269 D - 0xfffffe00590d3780 [zio_write_intr_7] 101268 D - 0xfffffe00590d3780 [zio_write_intr_6] 101267 D - 0xfffffe00590d3780 [zio_write_intr_5] 101266 D - 0xfffffe00590d3780 [zio_write_intr_4] 101265 D - 0xfffffe00590d3780 [zio_write_intr_3] 101264 D - 0xfffffe00590d3780 [zio_write_intr_2] 101263 D - 0xfffffe00590d3780 [zio_write_intr_1] 101262 D - 0xfffffe00590d3780 [zio_write_intr_0] 101261 D - 0xfffffe00590d3800 [zio_write_issue_hig] 101260 D - 0xfffffe00590d3800 [zio_write_issue_hig] 101259 D - 0xfffffe00590d3800 [zio_write_issue_hig] 101258 D - 0xfffffe00590d3800 [zio_write_issue_hig] 101257 D - 0xfffffe00590d3800 [zio_write_issue_hig] 101256 D - 0xfffffe00590d3880 [zio_write_issue_63] 101255 D - 0xfffffe00590d3880 [zio_write_issue_62] 101254 D - 0xfffffe00590d3880 [zio_write_issue_61] 101253 D - 0xfffffe00590d3880 [zio_write_issue_60] 101252 D - 0xfffffe00590d3880 [zio_write_issue_59] 101251 D - 0xfffffe00590d3880 [zio_write_issue_58] 101250 D - 0xfffffe00590d3880 [zio_write_issue_57] 101249 D - 0xfffffe00590d3880 [zio_write_issue_56] 101248 D - 0xfffffe00590d3880 [zio_write_issue_55] 101247 D - 0xfffffe00590d3880 [zio_write_issue_54] 101246 D - 0xfffffe00590d3880 [zio_write_issue_53] 101245 D - 0xfffffe00590d3880 [zio_write_issue_52] 101244 D - 0xfffffe00590d3880 [zio_write_issue_51] 101243 D - 0xfffffe00590d3880 [zio_write_issue_50] 101242 D - 0xfffffe00590d3880 [zio_write_issue_49] 101241 D - 0xfffffe00590d3880 [zio_write_issue_48] 101240 D - 0xfffffe00590d3880 [zio_write_issue_47] 101239 D - 0xfffffe00590d3880 [zio_write_issue_46] 101238 D - 0xfffffe00590d3880 [zio_write_issue_45] 101237 D - 0xfffffe00590d3880 [zio_write_issue_44] 101236 D - 0xfffffe00590d3880 [zio_write_issue_43] 101235 D - 0xfffffe00590d3880 [zio_write_issue_42] 101234 D - 0xfffffe00590d3880 [zio_write_issue_41] 101233 D - 0xfffffe00590d3880 [zio_write_issue_40] 101232 D - 0xfffffe00590d3880 [zio_write_issue_39] 101231 D - 0xfffffe00590d3880 [zio_write_issue_38] 101230 D - 0xfffffe00590d3880 [zio_write_issue_37] 101229 D - 0xfffffe00590d3880 [zio_write_issue_36] 101228 D - 0xfffffe00590d3880 [zio_write_issue_35] 101227 D - 0xfffffe00590d3880 [zio_write_issue_34] 101226 D - 0xfffffe00590d3880 [zio_write_issue_33] 101225 D - 0xfffffe00590d3880 [zio_write_issue_32] 101224 D - 0xfffffe00590d3880 [zio_write_issue_31] 101223 D - 0xfffffe00590d3880 [zio_write_issue_30] 101222 D - 0xfffffe00590d3880 [zio_write_issue_29] 101221 D - 0xfffffe00590d3880 [zio_write_issue_28] 101220 D - 0xfffffe00590d3880 [zio_write_issue_27] 101219 D - 0xfffffe00590d3880 [zio_write_issue_26] 101218 D - 0xfffffe00590d3880 [zio_write_issue_25] 101217 D - 0xfffffe00590d3880 [zio_write_issue_24] 101216 D - 0xfffffe00590d3880 [zio_write_issue_23] 101215 D - 0xfffffe00590d3880 [zio_write_issue_22] 101214 D - 0xfffffe00590d3880 [zio_write_issue_21] 101213 D - 0xfffffe00590d3880 [zio_write_issue_20] 101212 D - 0xfffffe00590d3880 [zio_write_issue_19] 101211 D - 0xfffffe00590d3880 [zio_write_issue_18] 101210 D - 0xfffffe00590d3880 [zio_write_issue_17] 101209 D - 0xfffffe00590d3880 [zio_write_issue_16] 101208 D - 0xfffffe00590d3880 [zio_write_issue_15] 101207 D - 0xfffffe00590d3880 [zio_write_issue_14] 101206 D - 0xfffffe00590d3880 [zio_write_issue_13] 101205 D - 0xfffffe00590d3880 [zio_write_issue_12] 101204 D - 0xfffffe00590d3880 [zio_write_issue_11] 101203 D - 0xfffffe00590d3880 [zio_write_issue_10] 101202 D - 0xfffffe00590d3880 [zio_write_issue_9] 101201 D - 0xfffffe00590d3880 [zio_write_issue_8] 101200 D - 0xfffffe00590d3880 [zio_write_issue_7] 101199 D - 0xfffffe00590d3880 [zio_write_issue_6] 101198 D - 0xfffffe00590d3880 [zio_write_issue_5] 101197 D - 0xfffffe00590d3880 [zio_write_issue_4] 101196 D - 0xfffffe00590d3880 [zio_write_issue_3] 101195 D - 0xfffffe00590d3880 [zio_write_issue_2] 101194 D - 0xfffffe00590d3880 [zio_write_issue_1] 101193 D - 0xfffffe00590d3880 [zio_write_issue_0] 101192 D - 0xfffffe00590d3900 [zio_read_intr_63] 101191 D - 0xfffffe00590d3900 [zio_read_intr_62] 101190 D - 0xfffffe00590d3900 [zio_read_intr_61] 101189 D - 0xfffffe00590d3900 [zio_read_intr_60] 101188 D - 0xfffffe00590d3900 [zio_read_intr_59] 101187 D - 0xfffffe00590d3900 [zio_read_intr_58] 101186 D - 0xfffffe00590d3900 [zio_read_intr_57] 101185 D - 0xfffffe00590d3900 [zio_read_intr_56] 101184 D - 0xfffffe00590d3900 [zio_read_intr_55] 101183 D - 0xfffffe00590d3900 [zio_read_intr_54] 101182 D - 0xfffffe00590d3900 [zio_read_intr_53] 101181 D - 0xfffffe00590d3900 [zio_read_intr_52] 101180 D - 0xfffffe00590d3900 [zio_read_intr_51] 101179 D - 0xfffffe00590d3900 [zio_read_intr_50] 101178 D - 0xfffffe00590d3900 [zio_read_intr_49] 101177 D - 0xfffffe00590d3900 [zio_read_intr_48] 101176 D - 0xfffffe00590d3900 [zio_read_intr_47] 101175 D - 0xfffffe00590d3900 [zio_read_intr_46] 101174 D - 0xfffffe00590d3900 [zio_read_intr_45] 101173 D - 0xfffffe00590d3900 [zio_read_intr_44] 101172 D - 0xfffffe00590d3900 [zio_read_intr_43] 101171 D - 0xfffffe00590d3900 [zio_read_intr_42] 101170 D - 0xfffffe00590d3900 [zio_read_intr_41] 101169 D - 0xfffffe00590d3900 [zio_read_intr_40] 101168 D - 0xfffffe00590d3900 [zio_read_intr_39] 101167 D - 0xfffffe00590d3900 [zio_read_intr_38] 101166 D - 0xfffffe00590d3900 [zio_read_intr_37] 101165 D - 0xfffffe00590d3900 [zio_read_intr_36] 101164 D - 0xfffffe00590d3900 [zio_read_intr_35] 101163 D - 0xfffffe00590d3900 [zio_read_intr_34] 101162 D - 0xfffffe00590d3900 [zio_read_intr_33] 101161 D - 0xfffffe00590d3900 [zio_read_intr_32] 101160 D - 0xfffffe00590d3900 [zio_read_intr_31] 101159 D - 0xfffffe00590d3900 [zio_read_intr_30] 101158 D - 0xfffffe00590d3900 [zio_read_intr_29] 101157 D - 0xfffffe00590d3900 [zio_read_intr_28] 101156 D - 0xfffffe00590d3900 [zio_read_intr_27] 101155 D - 0xfffffe00590d3900 [zio_read_intr_26] 101154 D - 0xfffffe00590d3900 [zio_read_intr_25] 101153 D - 0xfffffe00590d3900 [zio_read_intr_24] 101152 D - 0xfffffe00590d3900 [zio_read_intr_23] 101151 D - 0xfffffe00590d3900 [zio_read_intr_22] 101150 D - 0xfffffe00590d3900 [zio_read_intr_21] 101149 D - 0xfffffe00590d3900 [zio_read_intr_20] 101148 D - 0xfffffe00590d3900 [zio_read_intr_19] 101147 D - 0xfffffe00590d3900 [zio_read_intr_18] 101146 D - 0xfffffe00590d3900 [zio_read_intr_17] 101145 D - 0xfffffe00590d3900 [zio_read_intr_16] 101144 D - 0xfffffe00590d3900 [zio_read_intr_15] 101143 D - 0xfffffe00590d3900 [zio_read_intr_14] 101142 D - 0xfffffe00590d3900 [zio_read_intr_13] 101141 D - 0xfffffe00590d3900 [zio_read_intr_12] 101140 D - 0xfffffe00590d3900 [zio_read_intr_11] 101139 D - 0xfffffe00590d3900 [zio_read_intr_10] 101138 D - 0xfffffe00590d3900 [zio_read_intr_9] 101137 D - 0xfffffe00590d3900 [zio_read_intr_8] 101136 D - 0xfffffe00590d3900 [zio_read_intr_7] 101135 D - 0xfffffe00590d3900 [zio_read_intr_6] 101134 D - 0xfffffe00590d3900 [zio_read_intr_5] 101133 D - 0xfffffe00590d3900 [zio_read_intr_4] 101132 D - 0xfffffe00590d3900 [zio_read_intr_3] 101131 D - 0xfffffe00590d3900 [zio_read_intr_2] 101130 D - 0xfffffe00590d3900 [zio_read_intr_1] 101129 D - 0xfffffe00590d3900 [zio_read_intr_0] 101128 D - 0xfffffe00590d3980 [zio_read_issue_7] 101127 D - 0xfffffe00590d3980 [zio_read_issue_6] 101126 D - 0xfffffe00590d3980 [zio_read_issue_5] 101125 D - 0xfffffe00590d3980 [zio_read_issue_4] 101124 D - 0xfffffe00590d3980 [zio_read_issue_3] 101123 D - 0xfffffe00590d3980 [zio_read_issue_2] 101122 D - 0xfffffe00590d3980 [zio_read_issue_1] 101121 D - 0xfffffe00590d3980 [zio_read_issue_0] 101120 D - 0xfffffe00590d3a00 [zio_null_intr] 101119 D - 0xfffffe00590d3a80 [zio_null_issue] 100854 D - 0xfffffe003f6b8480 [zfs_vn_rele_taskq] 100853 D - 0xfffffe005967b880 [zio_ioctl_intr] 100852 D - 0xfffffe005967b900 [zio_ioctl_issue] 100851 D - 0xfffffe005967b980 [zio_claim_intr] 100850 D - 0xfffffe005967ba00 [zio_claim_issue] 100849 D - 0xfffffe005967ba80 [zio_free_intr] 100848 D - 0xfffffe005967bb00 [zio_free_issue_99] 100847 D - 0xfffffe005967bb00 [zio_free_issue_98] 100846 D - 0xfffffe005967bb00 [zio_free_issue_97] 100845 D - 0xfffffe005967bb00 [zio_free_issue_96] 100844 D - 0xfffffe005967bb00 [zio_free_issue_95] 100843 D - 0xfffffe005967bb00 [zio_free_issue_94] 100842 D - 0xfffffe005967bb00 [zio_free_issue_93] 100841 D - 0xfffffe005967bb00 [zio_free_issue_92] 100840 D - 0xfffffe005967bb00 [zio_free_issue_91] 100839 D - 0xfffffe005967bb00 [zio_free_issue_90] 100838 D - 0xfffffe005967bb00 [zio_free_issue_89] 100837 D - 0xfffffe005967bb00 [zio_free_issue_88] 100836 D - 0xfffffe005967bb00 [zio_free_issue_87] 100835 D - 0xfffffe005967bb00 [zio_free_issue_86] 100834 D - 0xfffffe005967bb00 [zio_free_issue_85] 100833 D - 0xfffffe005967bb00 [zio_free_issue_84] 100832 D - 0xfffffe005967bb00 [zio_free_issue_83] 100831 D - 0xfffffe005967bb00 [zio_free_issue_82] 100830 D - 0xfffffe005967bb00 [zio_free_issue_81] 100829 D - 0xfffffe005967bb00 [zio_free_issue_80] 100828 D - 0xfffffe005967bb00 [zio_free_issue_79] 100827 D - 0xfffffe005967bb00 [zio_free_issue_78] 100826 D - 0xfffffe005967bb00 [zio_free_issue_77] 100825 D - 0xfffffe005967bb00 [zio_free_issue_76] 100824 D - 0xfffffe005967bb00 [zio_free_issue_75] 100823 D - 0xfffffe005967bb00 [zio_free_issue_74] 100822 D - 0xfffffe005967bb00 [zio_free_issue_73] 100821 D - 0xfffffe005967bb00 [zio_free_issue_72] 100820 D - 0xfffffe005967bb00 [zio_free_issue_71] 100819 D - 0xfffffe005967bb00 [zio_free_issue_70] 100818 D - 0xfffffe005967bb00 [zio_free_issue_69] 100817 D - 0xfffffe005967bb00 [zio_free_issue_68] 100816 D - 0xfffffe005967bb00 [zio_free_issue_67] 100815 D - 0xfffffe005967bb00 [zio_free_issue_66] 100814 D - 0xfffffe005967bb00 [zio_free_issue_65] 100813 D - 0xfffffe005967bb00 [zio_free_issue_64] 100812 D - 0xfffffe005967bb00 [zio_free_issue_63] 100811 D - 0xfffffe005967bb00 [zio_free_issue_62] 100810 D - 0xfffffe005967bb00 [zio_free_issue_61] 100809 D - 0xfffffe005967bb00 [zio_free_issue_60] 100808 D - 0xfffffe005967bb00 [zio_free_issue_59] 100807 D - 0xfffffe005967bb00 [zio_free_issue_58] 100806 D - 0xfffffe005967bb00 [zio_free_issue_57] 100805 D - 0xfffffe005967bb00 [zio_free_issue_56] 100804 D - 0xfffffe005967bb00 [zio_free_issue_55] 100803 D - 0xfffffe005967bb00 [zio_free_issue_54] 100802 D - 0xfffffe005967bb00 [zio_free_issue_53] 100801 D - 0xfffffe005967bb00 [zio_free_issue_52] 100800 D - 0xfffffe005967bb00 [zio_free_issue_51] 100799 D - 0xfffffe005967bb00 [zio_free_issue_50] 100798 D - 0xfffffe005967bb00 [zio_free_issue_49] 100797 D - 0xfffffe005967bb00 [zio_free_issue_48] 100796 D - 0xfffffe005967bb00 [zio_free_issue_47] 100795 D - 0xfffffe005967bb00 [zio_free_issue_46] 100794 D - 0xfffffe005967bb00 [zio_free_issue_45] 100793 D - 0xfffffe005967bb00 [zio_free_issue_44] 100792 D - 0xfffffe005967bb00 [zio_free_issue_43] 100791 D - 0xfffffe005967bb00 [zio_free_issue_42] 100790 D - 0xfffffe005967bb00 [zio_free_issue_41] 100789 D - 0xfffffe005967bb00 [zio_free_issue_40] 100788 D - 0xfffffe005967bb00 [zio_free_issue_39] 100787 D - 0xfffffe005967bb00 [zio_free_issue_38] 100786 D - 0xfffffe005967bb00 [zio_free_issue_37] 100785 D - 0xfffffe005967bb00 [zio_free_issue_36] 100784 D - 0xfffffe005967bb00 [zio_free_issue_35] 100783 D - 0xfffffe005967bb00 [zio_free_issue_34] 100782 D - 0xfffffe005967bb00 [zio_free_issue_33] 100781 D - 0xfffffe005967bb00 [zio_free_issue_32] 100780 D - 0xfffffe005967bb00 [zio_free_issue_31] 100779 D - 0xfffffe005967bb00 [zio_free_issue_30] 100778 D - 0xfffffe005967bb00 [zio_free_issue_29] 100777 D - 0xfffffe005967bb00 [zio_free_issue_28] 100776 D - 0xfffffe005967bb00 [zio_free_issue_27] 100775 D - 0xfffffe005967bb00 [zio_free_issue_26] 100774 D - 0xfffffe005967bb00 [zio_free_issue_25] 100773 D - 0xfffffe005967bb00 [zio_free_issue_24] 100772 D - 0xfffffe005967bb00 [zio_free_issue_23] 100771 D - 0xfffffe005967bb00 [zio_free_issue_22] 100770 D - 0xfffffe005967bb00 [zio_free_issue_21] 100769 D - 0xfffffe005967bb00 [zio_free_issue_20] 100768 D - 0xfffffe005967bb00 [zio_free_issue_19] 100767 D - 0xfffffe005967bb00 [zio_free_issue_18] 100766 D - 0xfffffe005967bb00 [zio_free_issue_17] 100765 D - 0xfffffe005967bb00 [zio_free_issue_16] 100764 D - 0xfffffe005967bb00 [zio_free_issue_15] 100763 D - 0xfffffe005967bb00 [zio_free_issue_14] 100762 D - 0xfffffe005967bb00 [zio_free_issue_13] 100761 D - 0xfffffe005967bb00 [zio_free_issue_12] 100760 D - 0xfffffe005967bb00 [zio_free_issue_11] 100759 D - 0xfffffe005967bb00 [zio_free_issue_10] 100758 D - 0xfffffe005967bb00 [zio_free_issue_9] 100757 D - 0xfffffe005967bb00 [zio_free_issue_8] 100756 D - 0xfffffe005967bb00 [zio_free_issue_7] 100755 D - 0xfffffe005967bb00 [zio_free_issue_6] 100754 D - 0xfffffe005967bb00 [zio_free_issue_5] 100753 D - 0xfffffe005967bb00 [zio_free_issue_4] 100752 D - 0xfffffe005967bb00 [zio_free_issue_3] 100751 D - 0xfffffe005967bb00 [zio_free_issue_2] 100750 D - 0xfffffe005967bb00 [zio_free_issue_1] 100749 D - 0xfffffe005967bb00 [zio_free_issue_0] 100748 D - 0xfffffe005967bb80 [zio_write_intr_high] 100747 D - 0xfffffe005967bb80 [zio_write_intr_high] 100746 D - 0xfffffe005967bb80 [zio_write_intr_high] 100745 D - 0xfffffe005967bb80 [zio_write_intr_high] 100744 D - 0xfffffe005967bb80 [zio_write_intr_high] 100743 D - 0xfffffe005967bc00 [zio_write_intr_7] 100742 D - 0xfffffe005967bc00 [zio_write_intr_6] 100741 D - 0xfffffe005967bc00 [zio_write_intr_5] 100740 D - 0xfffffe005967bc00 [zio_write_intr_4] 100739 D - 0xfffffe005967bc00 [zio_write_intr_3] 100738 D - 0xfffffe005967bc00 [zio_write_intr_2] 100737 D - 0xfffffe005967bc00 [zio_write_intr_1] 100736 D - 0xfffffe005967bc00 [zio_write_intr_0] 100735 D - 0xfffffe005967bc80 [zio_write_issue_hig] 100734 D - 0xfffffe005967bc80 [zio_write_issue_hig] 100733 D - 0xfffffe005967bc80 [zio_write_issue_hig] 100732 D - 0xfffffe005967bc80 [zio_write_issue_hig] 100731 D - 0xfffffe005967bc80 [zio_write_issue_hig] 100730 D - 0xfffffe005967bd00 [zio_write_issue_63] 100729 D - 0xfffffe005967bd00 [zio_write_issue_62] 100728 D - 0xfffffe005967bd00 [zio_write_issue_61] 100727 D - 0xfffffe005967bd00 [zio_write_issue_60] 100726 D - 0xfffffe005967bd00 [zio_write_issue_59] 100725 D - 0xfffffe005967bd00 [zio_write_issue_58] 100724 D - 0xfffffe005967bd00 [zio_write_issue_57] 100723 D - 0xfffffe005967bd00 [zio_write_issue_56] 100722 D - 0xfffffe005967bd00 [zio_write_issue_55] 100721 D - 0xfffffe005967bd00 [zio_write_issue_54] 100720 D - 0xfffffe005967bd00 [zio_write_issue_53] 100719 D - 0xfffffe005967bd00 [zio_write_issue_52] 100718 D - 0xfffffe005967bd00 [zio_write_issue_51] 100717 D - 0xfffffe005967bd00 [zio_write_issue_50] 100716 D - 0xfffffe005967bd00 [zio_write_issue_49] 100715 D - 0xfffffe005967bd00 [zio_write_issue_48] 100714 D - 0xfffffe005967bd00 [zio_write_issue_47] 100713 D - 0xfffffe005967bd00 [zio_write_issue_46] 100712 D - 0xfffffe005967bd00 [zio_write_issue_45] 100711 D - 0xfffffe005967bd00 [zio_write_issue_44] 100710 D - 0xfffffe005967bd00 [zio_write_issue_43] 100709 D - 0xfffffe005967bd00 [zio_write_issue_42] 100708 D - 0xfffffe005967bd00 [zio_write_issue_41] 100707 D - 0xfffffe005967bd00 [zio_write_issue_40] 100706 D - 0xfffffe005967bd00 [zio_write_issue_39] 100705 D - 0xfffffe005967bd00 [zio_write_issue_38] 100704 D - 0xfffffe005967bd00 [zio_write_issue_37] 100703 D - 0xfffffe005967bd00 [zio_write_issue_36] 100702 D - 0xfffffe005967bd00 [zio_write_issue_35] 100701 D - 0xfffffe005967bd00 [zio_write_issue_34] 100700 D - 0xfffffe005967bd00 [zio_write_issue_33] 100699 D - 0xfffffe005967bd00 [zio_write_issue_32] 100698 D - 0xfffffe005967bd00 [zio_write_issue_31] 100697 D - 0xfffffe005967bd00 [zio_write_issue_30] 100696 D - 0xfffffe005967bd00 [zio_write_issue_29] 100695 D - 0xfffffe005967bd00 [zio_write_issue_28] 100694 D - 0xfffffe005967bd00 [zio_write_issue_27] 100693 D - 0xfffffe005967bd00 [zio_write_issue_26] 100692 D - 0xfffffe005967bd00 [zio_write_issue_25] 100691 D - 0xfffffe005967bd00 [zio_write_issue_24] 100690 D - 0xfffffe005967bd00 [zio_write_issue_23] 100689 D - 0xfffffe005967bd00 [zio_write_issue_22] 100688 D - 0xfffffe005967bd00 [zio_write_issue_21] 100687 D - 0xfffffe005967bd00 [zio_write_issue_20] 100686 D - 0xfffffe005967bd00 [zio_write_issue_19] 100685 D - 0xfffffe005967bd00 [zio_write_issue_18] 100684 D - 0xfffffe005967bd00 [zio_write_issue_17] 100683 D - 0xfffffe005967bd00 [zio_write_issue_16] 100682 D - 0xfffffe005967bd00 [zio_write_issue_15] 100681 D - 0xfffffe005967bd00 [zio_write_issue_14] 100680 D - 0xfffffe005967bd00 [zio_write_issue_13] 100679 D - 0xfffffe005967bd00 [zio_write_issue_12] 100678 D - 0xfffffe005967bd00 [zio_write_issue_11] 100677 D - 0xfffffe005967bd00 [zio_write_issue_10] 100676 D - 0xfffffe005967bd00 [zio_write_issue_9] 100675 D - 0xfffffe005967bd00 [zio_write_issue_8] 100674 D - 0xfffffe005967bd00 [zio_write_issue_7] 100673 D - 0xfffffe005967bd00 [zio_write_issue_6] 100672 D - 0xfffffe005967bd00 [zio_write_issue_5] 100671 D - 0xfffffe005967bd00 [zio_write_issue_4] 100670 D - 0xfffffe005967bd00 [zio_write_issue_3] 100669 D - 0xfffffe005967bd00 [zio_write_issue_2] 100668 D - 0xfffffe005967bd00 [zio_write_issue_1] 100667 D - 0xfffffe005967bd00 [zio_write_issue_0] 100666 D - 0xfffffe005967bd80 [zio_read_intr_63] 100665 D - 0xfffffe005967bd80 [zio_read_intr_62] 100664 D - 0xfffffe005967bd80 [zio_read_intr_61] 100663 D - 0xfffffe005967bd80 [zio_read_intr_60] 100662 D - 0xfffffe005967bd80 [zio_read_intr_59] 100661 D - 0xfffffe005967bd80 [zio_read_intr_58] 100660 D - 0xfffffe005967bd80 [zio_read_intr_57] 100659 D - 0xfffffe005967bd80 [zio_read_intr_56] 100658 D - 0xfffffe005967bd80 [zio_read_intr_55] 100657 D - 0xfffffe005967bd80 [zio_read_intr_54] 100656 D - 0xfffffe005967bd80 [zio_read_intr_53] 100655 D - 0xfffffe005967bd80 [zio_read_intr_52] 100654 D - 0xfffffe005967bd80 [zio_read_intr_51] 100653 D - 0xfffffe005967bd80 [zio_read_intr_50] 100652 D - 0xfffffe005967bd80 [zio_read_intr_49] 100651 D - 0xfffffe005967bd80 [zio_read_intr_48] 100650 D - 0xfffffe005967bd80 [zio_read_intr_47] 100649 D - 0xfffffe005967bd80 [zio_read_intr_46] 100648 D - 0xfffffe005967bd80 [zio_read_intr_45] 100647 D - 0xfffffe005967bd80 [zio_read_intr_44] 100646 D - 0xfffffe005967bd80 [zio_read_intr_43] 100645 D - 0xfffffe005967bd80 [zio_read_intr_42] 100644 D - 0xfffffe005967bd80 [zio_read_intr_41] 100643 D - 0xfffffe005967bd80 [zio_read_intr_40] 100642 D - 0xfffffe005967bd80 [zio_read_intr_39] 100641 D - 0xfffffe005967bd80 [zio_read_intr_38] 100640 D - 0xfffffe005967bd80 [zio_read_intr_37] 100639 D - 0xfffffe005967bd80 [zio_read_intr_36] 100638 D - 0xfffffe005967bd80 [zio_read_intr_35] 100637 D - 0xfffffe005967bd80 [zio_read_intr_34] 100636 D - 0xfffffe005967bd80 [zio_read_intr_33] 100635 D - 0xfffffe005967bd80 [zio_read_intr_32] 100634 D - 0xfffffe005967bd80 [zio_read_intr_31] 100633 D - 0xfffffe005967bd80 [zio_read_intr_30] 100632 D - 0xfffffe005967bd80 [zio_read_intr_29] 100631 D - 0xfffffe005967bd80 [zio_read_intr_28] 100630 D - 0xfffffe005967bd80 [zio_read_intr_27] 100629 D - 0xfffffe005967bd80 [zio_read_intr_26] 100628 D - 0xfffffe005967bd80 [zio_read_intr_25] 100627 D - 0xfffffe005967bd80 [zio_read_intr_24] 100626 D - 0xfffffe005967bd80 [zio_read_intr_23] 100625 D - 0xfffffe005967bd80 [zio_read_intr_22] 100624 D - 0xfffffe005967bd80 [zio_read_intr_21] 100623 D - 0xfffffe005967bd80 [zio_read_intr_20] 100622 D - 0xfffffe005967bd80 [zio_read_intr_19] 100621 D - 0xfffffe005967bd80 [zio_read_intr_18] 100620 D - 0xfffffe005967bd80 [zio_read_intr_17] 100619 D - 0xfffffe005967bd80 [zio_read_intr_16] 100618 D - 0xfffffe005967bd80 [zio_read_intr_15] 100617 D - 0xfffffe005967bd80 [zio_read_intr_14] 100616 D - 0xfffffe005967bd80 [zio_read_intr_13] 100615 D - 0xfffffe005967bd80 [zio_read_intr_12] 100614 D - 0xfffffe005967bd80 [zio_read_intr_11] 100613 D - 0xfffffe005967bd80 [zio_read_intr_10] 100612 D - 0xfffffe005967bd80 [zio_read_intr_9] 100611 D - 0xfffffe005967bd80 [zio_read_intr_8] 100610 D - 0xfffffe005967bd80 [zio_read_intr_7] 100609 D - 0xfffffe005967bd80 [zio_read_intr_6] 100608 D - 0xfffffe005967bd80 [zio_read_intr_5] 100607 D - 0xfffffe005967bd80 [zio_read_intr_4] 100606 D - 0xfffffe005967bd80 [zio_read_intr_3] 100605 D - 0xfffffe005967bd80 [zio_read_intr_2] 100604 D - 0xfffffe005967bd80 [zio_read_intr_1] 100603 D - 0xfffffe005967bd80 [zio_read_intr_0] 100602 D - 0xfffffe005967be00 [zio_read_issue_7] 100601 D - 0xfffffe005967be00 [zio_read_issue_6] 100600 D - 0xfffffe005967be00 [zio_read_issue_5] 100599 D - 0xfffffe005967be00 [zio_read_issue_4] 100598 D - 0xfffffe005967be00 [zio_read_issue_3] 100597 D - 0xfffffe005967be00 [zio_read_issue_2] 100596 D - 0xfffffe005967be00 [zio_read_issue_1] 100595 D - 0xfffffe005967be00 [zio_read_issue_0] 100594 D - 0xfffffe005967c000 [zio_null_intr] 100593 D - 0xfffffe005967c080 [zio_null_issue] 100326 D - 0xfffffe003fe7b880 [system_taskq_63] 100325 D - 0xfffffe003fe7b880 [system_taskq_62] 100324 D - 0xfffffe003fe7b880 [system_taskq_61] 100323 D - 0xfffffe003fe7b880 [system_taskq_60] 100322 D - 0xfffffe003fe7b880 [system_taskq_59] 100321 D - 0xfffffe003fe7b880 [system_taskq_58] 100320 D - 0xfffffe003fe7b880 [system_taskq_57] 100319 D - 0xfffffe003fe7b880 [system_taskq_56] 100318 D - 0xfffffe003fe7b880 [system_taskq_55] 100317 D - 0xfffffe003fe7b880 [system_taskq_54] 100316 D - 0xfffffe003fe7b880 [system_taskq_53] 100315 D - 0xfffffe003fe7b880 [system_taskq_52] 100314 D - 0xfffffe003fe7b880 [system_taskq_51] 100313 D - 0xfffffe003fe7b880 [system_taskq_50] 100312 D - 0xfffffe003fe7b880 [system_taskq_49] 100311 D - 0xfffffe003fe7b880 [system_taskq_48] 100310 D - 0xfffffe003fe7b880 [system_taskq_47] 100309 D - 0xfffffe003fe7b880 [system_taskq_46] 100308 D - 0xfffffe003fe7b880 [system_taskq_45] 100307 D - 0xfffffe003fe7b880 [system_taskq_44] 100306 D - 0xfffffe003fe7b880 [system_taskq_43] 100305 D - 0xfffffe003fe7b880 [system_taskq_42] 100304 D - 0xfffffe003fe7b880 [system_taskq_41] 100303 D - 0xfffffe003fe7b880 [system_taskq_40] 100302 D - 0xfffffe003fe7b880 [system_taskq_39] 100301 D - 0xfffffe003fe7b880 [system_taskq_38] 100300 D - 0xfffffe003fe7b880 [system_taskq_37] 100299 D - 0xfffffe003fe7b880 [system_taskq_36] 100298 D - 0xfffffe003fe7b880 [system_taskq_35] 100297 D - 0xfffffe003fe7b880 [system_taskq_34] 100296 D - 0xfffffe003fe7b880 [system_taskq_33] 100295 D - 0xfffffe003fe7b880 [system_taskq_32] 100294 D - 0xfffffe003fe7b880 [system_taskq_31] 100293 D - 0xfffffe003fe7b880 [system_taskq_30] 100292 D - 0xfffffe003fe7b880 [system_taskq_29] 100291 D - 0xfffffe003fe7b880 [system_taskq_28] 100290 D - 0xfffffe003fe7b880 [system_taskq_27] 100289 D - 0xfffffe003fe7b880 [system_taskq_26] 100288 D - 0xfffffe003fe7b880 [system_taskq_25] 100287 D - 0xfffffe003fe7b880 [system_taskq_24] 100286 D - 0xfffffe003fe7b880 [system_taskq_23] 100285 D - 0xfffffe003fe7b880 [system_taskq_22] 100284 D - 0xfffffe003fe7b880 [system_taskq_21] 100283 D - 0xfffffe003fe7b880 [system_taskq_20] 100282 D - 0xfffffe003fe7b880 [system_taskq_19] 100281 D - 0xfffffe003fe7b880 [system_taskq_18] 100280 D - 0xfffffe003fe7b880 [system_taskq_17] 100279 D - 0xfffffe003fe7b880 [system_taskq_16] 100278 D - 0xfffffe003fe7b880 [system_taskq_15] 100277 D - 0xfffffe003fe7b880 [system_taskq_14] 100276 D - 0xfffffe003fe7b880 [system_taskq_13] 100275 D - 0xfffffe003fe7b880 [system_taskq_12] 100274 D - 0xfffffe003fe7b880 [system_taskq_11] 100273 D - 0xfffffe003fe7b880 [system_taskq_10] 100272 D - 0xfffffe003fe7b880 [system_taskq_9] 100271 D - 0xfffffe003fe7b880 [system_taskq_8] 100270 D - 0xfffffe003fe7b880 [system_taskq_7] 100269 D - 0xfffffe003fe7b880 [system_taskq_6] 100268 D - 0xfffffe003fe7b880 [system_taskq_5] 100267 D - 0xfffffe003fe7b880 [system_taskq_4] 100266 D - 0xfffffe003fe7b880 [system_taskq_3] 100265 D - 0xfffffe003fe7b880 [system_taskq_2] 100264 D - 0xfffffe003fe7b880 [system_taskq_1] 100263 D - 0xfffffe003fe7b880 [system_taskq_0] 100234 D - 0xfffffe00394d9580 [mca taskq] 100193 D - 0xfffffe0030f00080 [mps1 taskq] 100190 D - 0xfffffe0030f02200 [mps0 taskq] 100188 D - 0xfffffe0030ea8680 [em0 txq] 100186 D - 0xfffffe0030ea8800 [em0 rxq] 100183 D - 0xfffffe0030ea4400 [igb1 que] 100181 D - 0xfffffe0030ea4580 [igb1 que] 100179 D - 0xfffffe0030e9c480 [igb1 que] 100177 D - 0xfffffe0030e9c600 [igb1 que] 100175 D - 0xfffffe0030e95480 [igb1 que] 100173 D - 0xfffffe0030e95600 [igb1 que] 100171 D - 0xfffffe0030e95780 [igb1 que] 100169 D - 0xfffffe0030e95900 [igb1 que] 100166 D - 0xfffffe0030e11400 [igb0 que] 100164 D - 0xfffffe0030e11580 [igb0 que] 100162 D - 0xfffffe0030e0b780 [igb0 que] 100160 D - 0xfffffe0030e0b900 [igb0 que] 100158 D - 0xfffffe0030e0ba80 [igb0 que] 100156 D - 0xfffffe0030e0bc00 [igb0 que] 100154 D - 0xfffffe0030e07a80 [igb0 que] 100152 D - 0xfffffe0030e07c00 [igb0 que] 100147 D - 0xfffffe0030ae6680 [acpi_task_2] 100146 D - 0xfffffe0030ae6680 [acpi_task_1] 100145 D - 0xfffffe0030ae6680 [acpi_task_0] 100144 D - 0xfffffe0030ae6700 [kqueue taskq] 100142 D - 0xfffffe0030ae6880 [ffs_trim taskq] 100141 D - 0xfffffe0030ae6900 [thread taskq] 100136 D - 0xfffffe003093f200 [firmware taskq] 100000 D sched 0xffffffff812cbdd8 [swapper] db> From owner-freebsd-fs@FreeBSD.ORG Mon Oct 15 20:34:59 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 2EF888F0 for ; Mon, 15 Oct 2012 20:34:59 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id 8AB2D8FC0A for ; Mon, 15 Oct 2012 20:34:58 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AqAEACFyfFCDaFvO/2dsb2JhbABFFoV8unyCIAEBAQMBAQEBICsgCwUWDgoCAg0ZAiMGAQkmBggHBAEcBIdRAwkGC6oOiSQNiVSBIYlSZhUFhRGBEgOTP1iBVYEVig6FDYMJgT8INA X-IronPort-AV: E=Sophos;i="4.80,590,1344225600"; d="scan'208";a="183734513" Received: from erie.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.206]) by esa-jnhn-pri.mail.uoguelph.ca with ESMTP; 15 Oct 2012 16:34:56 -0400 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id 01B95B4022; Mon, 15 Oct 2012 16:34:57 -0400 (EDT) Date: Mon, 15 Oct 2012 16:34:56 -0400 (EDT) From: Rick Macklem To: Nikolay Denev Message-ID: <1632051502.2285525.1350333296994.JavaMail.root@erie.cs.uoguelph.ca> In-Reply-To: <9BE97E36-8995-4968-B8ED-1B17D308ED19@gmail.com> Subject: Re: Bad ZFS - NFS interaction? [ was: NFS server bottlenecks ] MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.91.203] X-Mailer: Zimbra 6.0.10_GA_2692 (ZimbraWebClient - IE7 (Win)/6.0.10_GA_2692) Cc: "freebsd-fs@freebsd.org" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Oct 2012 20:34:59 -0000 Nikolay Denev wrote: > On Oct 15, 2012, at 5:06 PM, Rick Macklem > wrote: > > > Nikolay Denev wrote: > >> On Oct 13, 2012, at 6:22 PM, Nikolay Denev > >> wrote: > >> > >>> > >>> On Oct 13, 2012, at 5:05 AM, Rick Macklem > >>> wrote: > >>> > >>>> I wrote: > >>>>> Oops, I didn't get the "readahead" option description > >>>>> quite right in the last post. The default read ahead > >>>>> is 1, which does result in "rsize * 2", since there is > >>>>> the read + 1 readahead. > >>>>> > >>>>> "rsize * 16" would actually be for the option "readahead=15" > >>>>> and for "readahead=16" the calculation would be "rsize * 17". > >>>>> > >>>>> However, the example was otherwise ok, I think? rick > >>>> > >>>> I've attached the patch drc3.patch (it assumes drc2.patch has > >>>> already been > >>>> applied) that replaces the single mutex with one for each hash > >>>> list > >>>> for tcp. It also increases the size of NFSRVCACHE_HASHSIZE to > >>>> 200. > >>>> > >>>> These patches are also at: > >>>> http://people.freebsd.org/~rmacklem/drc2.patch > >>>> http://people.freebsd.org/~rmacklem/drc3.patch > >>>> in case the attachments don't get through. > >>>> > >>>> rick > >>>> ps: I haven't tested drc3.patch a lot, but I think it's ok? > >>> > >>> drc3.patch applied and build cleanly and shows nice improvement! > >>> > >>> I've done a quick benchmark using iozone over the NFS mount from > >>> the > >>> Linux host. > >>> > >>> drc2.pach (but with NFSRVCACHE_HASHSIZE=500) > >>> > >>> TEST WITH 8K > >>> ------------------------------------------------------------------------------------------------- > >>> Auto Mode > >>> Using Minimum Record Size 8 KB > >>> Using Maximum Record Size 8 KB > >>> Using minimum file size of 2097152 kilobytes. > >>> Using maximum file size of 2097152 kilobytes. > >>> O_DIRECT feature enabled > >>> SYNC Mode. > >>> OPS Mode. Output is in operations per second. > >>> Command line used: iozone -a -y 8k -q 8k -n 2g -g 2g -C -I > >>> -o > >>> -O -i 0 -i 1 -i 2 > >>> Time Resolution = 0.000001 seconds. > >>> Processor cache size set to 1024 Kbytes. > >>> Processor cache line size set to 32 bytes. > >>> File stride size set to 17 * record size. > >>> random > >>> random > >>> bkwd > >>> record > >>> stride > >>> KB reclen write rewrite read reread read write read > >>> rewrite read fwrite frewrite fread freread > >>> 2097152 8 1919 1914 2356 2321 2335 1706 > >>> > >>> TEST WITH 1M > >>> ------------------------------------------------------------------------------------------------- > >>> Auto Mode > >>> Using Minimum Record Size 1024 KB > >>> Using Maximum Record Size 1024 KB > >>> Using minimum file size of 2097152 kilobytes. > >>> Using maximum file size of 2097152 kilobytes. > >>> O_DIRECT feature enabled > >>> SYNC Mode. > >>> OPS Mode. Output is in operations per second. > >>> Command line used: iozone -a -y 1m -q 1m -n 2g -g 2g -C -I > >>> -o > >>> -O -i 0 -i 1 -i 2 > >>> Time Resolution = 0.000001 seconds. > >>> Processor cache size set to 1024 Kbytes. > >>> Processor cache line size set to 32 bytes. > >>> File stride size set to 17 * record size. > >>> random > >>> random > >>> bkwd > >>> record > >>> stride > >>> KB reclen write rewrite read reread read write read > >>> rewrite read fwrite frewrite fread freread > >>> 2097152 1024 73 64 477 486 496 61 > >>> > >>> > >>> drc3.patch > >>> > >>> TEST WITH 8K > >>> ------------------------------------------------------------------------------------------------- > >>> Auto Mode > >>> Using Minimum Record Size 8 KB > >>> Using Maximum Record Size 8 KB > >>> Using minimum file size of 2097152 kilobytes. > >>> Using maximum file size of 2097152 kilobytes. > >>> O_DIRECT feature enabled > >>> SYNC Mode. > >>> OPS Mode. Output is in operations per second. > >>> Command line used: iozone -a -y 8k -q 8k -n 2g -g 2g -C -I > >>> -o > >>> -O -i 0 -i 1 -i 2 > >>> Time Resolution = 0.000001 seconds. > >>> Processor cache size set to 1024 Kbytes. > >>> Processor cache line size set to 32 bytes. > >>> File stride size set to 17 * record size. > >>> random > >>> random > >>> bkwd > >>> record > >>> stride > >>> KB reclen write rewrite read reread read write read > >>> rewrite read fwrite frewrite fread freread > >>> 2097152 8 2108 2397 3001 3013 3010 2389 > >>> > >>> > >>> TEST WITH 1M > >>> ------------------------------------------------------------------------------------------------- > >>> Auto Mode > >>> Using Minimum Record Size 1024 KB > >>> Using Maximum Record Size 1024 KB > >>> Using minimum file size of 2097152 kilobytes. > >>> Using maximum file size of 2097152 kilobytes. > >>> O_DIRECT feature enabled > >>> SYNC Mode. > >>> OPS Mode. Output is in operations per second. > >>> Command line used: iozone -a -y 1m -q 1m -n 2g -g 2g -C -I > >>> -o > >>> -O -i 0 -i 1 -i 2 > >>> Time Resolution = 0.000001 seconds. > >>> Processor cache size set to 1024 Kbytes. > >>> Processor cache line size set to 32 bytes. > >>> File stride size set to 17 * record size. > >>> random > >>> random > >>> bkwd > >>> record > >>> stride > >>> KB reclen write rewrite read reread read write read > >>> rewrite read fwrite frewrite fread freread > >>> 2097152 1024 80 79 521 536 528 75 > >>> > >>> > >>> Also with drc3 the CPU usage on the server is noticeably lower. > >>> Most > >>> of the time I could see only the geom{g_up}/{g_down} threads, > >>> and a few nfsd threads, before that nfsd's were much more > >>> prominent. > >>> > >>> I guess under bigger load the performance improvement can be > >>> bigger. > >>> > >>> I'll run some more tests with heavier loads this week. > >>> > >>> Thanks, > >>> Nikolay > >>> > >>> > >> > >> If anyone is interested here's a FlameGraph generated using DTrace > >> and > >> Brendan Gregg's tools from > >> https://github.com/brendangregg/FlameGraph > >> : > >> > >> https://home.totalterror.net/freebsd/goliath-kernel.svg > >> > >> It was sampled during Oracle database restore from Linux host over > >> the > >> nfs mount. > >> Currently all IO on the dataset that the linux machine writes is > >> stuck, simple ls in the directory > >> hangs for maybe 10-15 minutes and then eventually completes. > >> > >> Looks like some weird locking issue. > >> > >> [*] http://dtrace.org/blogs/brendan/2011/12/16/flame-graphs/ > >> > >> P.S.: The machine runs with drc3.patch for the NFS server. > >> P.S.2: The nfsd server is configured for vfs.nfsd.maxthreads=200, > >> maybe that's too much? > >> > > You could try trimming the size of vfs.nfsd.tcphighwater down. > > Remember that, > > with this patch, when you increase this tunable, you are trading > > space > > for CPU overhead. > > > > If it's still "running", you could do "vmstat -m" and "vmstat -z" to > > see where the memory is allocated. ("nfsstat -e -s" will tell you > > the > > size of the cache.) > > > > rick > >> _______________________________________________ > >> freebsd-fs@freebsd.org mailing list > >> http://lists.freebsd.org/mailman/listinfo/freebsd-fs > >> To unsubscribe, send any mail to > >> "freebsd-fs-unsubscribe@freebsd.org" > > > Are you saying that the time spent in _mtx_spin_lock can be because of > this? No. I was thinking that memory used by the DRC cache isn't available to ZFS and that ZFS might be getting contrained because of this. AS I've said before, I'm not a ZFS guy, but you don't have to look very hard to find problems related to ZFS running low on what I think they call the ARC cache. (I believe it is usually a lack of kernel virtual address space, but I'm not the guy to know if that's correct or how to tell.) > To me it looks like that there was some heavy contention in ZFS, maybe > specific to the > way it's accessed by the NFS server? Probably due to high maxthreads > value ? > Using fewer nfsd threads would set a lower upper limit on load for ZFS, since that sets the upper limit on the # of concurrent VOP_xxx() calls. > > Here's the nfsstat -s -e, seems like it's wrong as it's negative > number, maybe overflowed? > There was a bug fixed a while ago, where "nfsstat -e -z" would zero the count out, and then it would go negative when it decreased. It will also wrap around when it hits 2B, since it's a signed 32bit. (jwd@ suggested changing the printf() to at least show unsigned, but I don't think we ever got around to a patch.) > Server: > Retfailed Faults Clients > 0 0 0 > OpenOwner Opens LockOwner Locks Delegs > 0 0 0 0 0 > Server Cache Stats: > Inprog Idem Non-idem Misses CacheSize TCPPeak > 0 0 0 83500632 -24072 16385 > > > > Also here are the following sysctls : > > vfs.nfsd.request_space_used: 0 > vfs.nfsd.request_space_used_highest: 13121808 > vfs.nfsd.request_space_high: 13107200 > vfs.nfsd.request_space_low: 8738133 > vfs.nfsd.request_space_throttled: 0 > vfs.nfsd.request_space_throttle_count: 0 > > Are they related to the same request cache? > Nope. They are in the krpc (sys/rpc/svc.c) and control/limit the space used by requests (mbuf clusters). Again, a bigger DRC will mean less mbuf/mbuf cluster space available for the rest of the system. Reduce vfs.nfsd.tcphighwater and you reduce the mbuf/mbuf cluster usage for the DRC. (It caches the reply by m_copy()ing the mbuf list.) > I have stats that show at some point nfsd has allocated all 200 > threads and > vfs.nfsd.request_space_used hits the ceiling too. When all the threads are busy, new requests will be queued in the receive side of the krpc code, which means more request_space_used. As I mentioned, use "vmstat -z" to see what the mbuf/mbuf cluster use is, among others, rick. From owner-freebsd-fs@FreeBSD.ORG Tue Oct 16 03:42:06 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id D4D9388B for ; Tue, 16 Oct 2012 03:42:06 +0000 (UTC) (envelope-from vcfdser@yahoo.com) Received: from s7.send-out.co.cc (s7.send-out.co.cc [41.32.158.117]) by mx1.freebsd.org (Postfix) with ESMTP id 38B078FC08 for ; Tue, 16 Oct 2012 03:41:57 +0000 (UTC) Received: from PC2 ([41.32.158.117]) by s7.send-out.co.cc with Microsoft SMTPSVC(6.0.2600.2096); Mon, 15 Oct 2012 21:51:43 +0200 From: "vcfdser@yahoo.com" To: freebsd-fs@freebsd.org Subject: Message to whole mankind X-Mailer: SendBlaster.1.6.0 Date: Mon, 15 Oct 2012 21:51:43 +0200 Message-ID: <295263127008117298796@PC2> X-OriginalArrivalTime: 15 Oct 2012 19:51:43.0781 (UTC) FILETIME=[80869D50:01CDAB0E] MIME-Version: 1.0 Content-Type: text/plain X-Content-Filtered-By: Mailman/MimeDel 2.1.14 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: vcfdser@yahoo.com List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Oct 2012 03:42:07 -0000 The submission of man to His Creator is the essence of Islam. The name "Islam" is chosen by God (Allah) and not by man. It is the same unifying Message revealed to all the Prophets and Messengers by Allah and which they spread amongst their respective nations. In its Final form it was revealed to Muhammad (Peace & Mercy of Allah be upon him) as a complete Message to whole mankind. The Lord, Allah, is the True and Only Creator that deserves to be worshipped. No worship is worthy of being given to a stone, statue, a cross, a triangle, Khomeini, Farakhan, Eliajahs, Malcom's X or Y, Ghandi, Krishna, Guru, Buddha, Mahatma, Emperor, Joseph Smith, Sun, Moon (not to that from Korea too!), Light, Fire, rivers, cows, Rama, Temples, Prophets, Messengers (Yes! Muslims do not worship Muhammad-peace be upon him), Saints, Priests, Monks, Movie Stars, Sheiks, etc.!!! All are created beings or things. ALLAH, is the Name of the One True God. His Name is not chosen by man and does not have a number or gender. It is known that Allah is the Name of God in Aramaic, the language of our beloved Prophet Jesus and a sister language of Arabic. The Name "Allah" has been used by all previous Prophets starting with Adam and by the last and final Prophet, Muhammad (Peace be upon them all). The Innate Nature in man recognizes what is good and bad, what is true and false. It recognizes that the Attributes of Allah must be True, Unique, and All-Perfect. It does not feel comfortable towards any kind of degradation of His Attributes not does it qualities to the Creator. Many who became "discontent with God" did so because of the practices of the Church in medieval Europe and because of the claims of "god dwelling in a son" and the concept of the "original sin". However, they "escaped" into worshipping a new theory called "mother nature" as well as the "material" World. With the advancement of materialistic technology others from different religions adopted the concept of "forgetting about God" and "let us live this life and enjoy it!", not realizing that they have chosen the worship of the "original god" of Rome: Desire!. NOW we can see that all of this materialistic and secular progress produced a spiritual vacuum that led to complex social, economical, political, and psychological problems. Many of those who "fled" their "religions" are in search again. Some try to "escape" the complexity of their daily lives via various means. Those who had the chance to examine the Qur'an and Islam, proceed with a complete way of life that relates man to establish a purpose for his presence on earth. This is well recognized in the Attributes of Allah and what does He require from man. He does not want man to be enslaved to any false deity: nature, drugs, lust, money, other man, desire, or sex. He provides man with the proofs that He is the One who can redeem so that man can free himself from the slavery to any form of creation and to turn to his Creator Alone. THIS Creator Has Perfect Attributes. He is the First, nothing is before Him, the Ever Living. To Him is the Final Return where everyone will be dealt with in the Most Perfect and Just way. He does not begot nor He is begotten. Those who attribute Divinity to Jesus forget or ignore the fact that Jesus was in a mother's womb. He needed nutrition; he was born and grew up to be a man. He was trusted with the Gospel as a Message to the Children of Israel: "For there is One God, and one mediator (i.e. a messenger) between God and men (the Children of Israel), the man Christ Jesus) (I Timothy 2:5). A man-messenger calling his nation not to worship him: "But in vain they do worship me!" (Mathew 15:9). A man who needs to eat, walk, sleed, rest, etc.. cannot have Divine Attributes because he is in need and God (Allah) is Self-Sufficient. AS far as Buddhism, Hinduism, Zoroastrianism, Marxism, and Capitalism, there is the devotion of worshipping created being/things in one form or another. Jews had attributed a "Nationalistic" belonging to Allah. They labeled Him "The Tribal God" for the Children of Israel. Men and women following these "religions" were born with the natural inclination of submission to their Creator, Allah. It is their parents who had driven them into their respective traditions. However, once people are exposed to the Signs of Allah around them, or in the Qur'an or to someone who triggers thei Fitra (natural inclination to worship Allah Alone), the reverting process begins and that is why we see a universal spreading of Islam. In the West and despite tha many distortions of Islam in the Media, many admit that Islam may be the fastest growing Faith. No sense of fairness can be achieved without a genuine attempt to know the Word of Allah in the Qur'an and not on the 30-min-Evening News. This is the real challenge for those who seek the Truth. Man is created for a purpose: to live a life in accordance with Allah's way. Why Not? Do we posses the air we breath? Did we create ourselves or others? Or were we ourselves the Creators? We are limited and weak. So is our right to ignore our Creator where we all need Him? ISLAM is the submission in worship to Allah Alone and it is the essence of all the Messages sent to all nations before us. Allah is All-Just and All-Wise. He does not intend confusion for His Creation. The religion accepted to Him is the one chosen by Him. Its essence must be One, because He is One. It is free from geographical, racist, and status oriented concepts. It is Perfect and it is the complete way of life. All these qualities are chosen by Allah in His Only Religion: Islam. Its details are in in the Qur'an, read it and come with an open heart because none can expose better than the World of Allah. The Qur'an was revealed to Prophet Muhammad. He did not author it. He was unlettered. Its translation is available in many languages in bookstores or in an Islamic Center close to you. Take the time to read it and come/call the Islamic Center, or speak to someone who re-verted and submitted to Allah Alone. Prepared by Dr. Saleh As-Saleh From owner-freebsd-fs@FreeBSD.ORG Tue Oct 16 09:25:30 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 4A47E185; Tue, 16 Oct 2012 09:25:30 +0000 (UTC) (envelope-from c.kworr@gmail.com) Received: from mail-ea0-f182.google.com (mail-ea0-f182.google.com [209.85.215.182]) by mx1.freebsd.org (Postfix) with ESMTP id 99B258FC17; Tue, 16 Oct 2012 09:25:29 +0000 (UTC) Received: by mail-ea0-f182.google.com with SMTP id c10so1493969eaa.13 for ; Tue, 16 Oct 2012 02:25:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=pUqurpjTOLc5trEtxm7LJsPBgcXcrZOacrxUFQKe3I0=; b=YqmZjU+Sx4pmagftgu9CRWYcR/yN9TIoBwpyXSvnBGUSHcJZcnW5ykKi3tHbhbIf0f spXMB3Oq52P1F39HXohswdqkO5cjmzbUmT5brY07b5cTmMADRc+KamLLaxyRAV7cY9ZS oQ6HOjZd0MbEqPx17AlF+4FtB+c+QlmiI8bZAR5p1S5A3Svo8ZBKGd+hoWGaD+oGJaRh B8Qs/GgCn2OC80J5oM0pzH/mqE4clz+lEKwk2xQpzXtHRHE6BadK+dTWRCGf/7liGFjo Lu9kpyBlnBRPiGY1xsd+iov4WA4xItFq0mWNvItagpIHeAENR3l/mN5/BvcCfl+DPO5C 4bFQ== Received: by 10.14.214.2 with SMTP id b2mr11509690eep.32.1350379522213; Tue, 16 Oct 2012 02:25:22 -0700 (PDT) Received: from limbo.xim.bz ([46.150.100.6]) by mx.google.com with ESMTPS id 42sm28978618eee.0.2012.10.16.02.25.20 (version=SSLv3 cipher=OTHER); Tue, 16 Oct 2012 02:25:21 -0700 (PDT) Message-ID: <507D27FD.2090207@gmail.com> Date: Tue, 16 Oct 2012 12:25:17 +0300 From: Volodymyr Kostyrko User-Agent: Mozilla/5.0 (X11; FreeBSD i386; rv:16.0) Gecko/20121013 Thunderbird/16.0.1 MIME-Version: 1.0 To: Martin Matuska Subject: Re: [CFT] ZFS feature flag support for 9-STABLE References: <506B4508.3030406@FreeBSD.org> <5077F3DF.2030108@gmail.com> <507BC58A.40803@FreeBSD.org> In-Reply-To: <507BC58A.40803@FreeBSD.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@FreeBSD.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Oct 2012 09:25:30 -0000 15.10.2012 11:12, Martin Matuska wrote: >>> ZFS feature flag support is ready to be merged to 9-STABLE. >>> The scheduled merge date is short after 9.1-RELEASE. >>> >>> Early adopters can test new features by applying the following patch >>> (stable/9 r241135): >>> http://people.freebsd.org/~mm/patches/zfs/9-stable-zfs-features.patch.gz >>> >>> Steps to apply to a clean checked-out source: >>> cd /path/to/src >>> patch -p0 < /path/to/9-stable-zfs-features.patch >> >> Does this suppose to work on i386? On my test machine this gives an >> instant panic on trying to mount root system. I've already rebuilt my >> kernel WITH INVARIANTS, WITNESS and DIAGNOSTIC but it still panics >> even before I can dump core anywhere. I now proceeding to build a tiny >> virtual machine to capture the backtrace. > > I am unable to reproduce this under i386 VirtualBox. Does your system > have some special settings? Was patched stable/9 built correctly? I can't reproduce this either with the same kernel on clean pool. It just doesn't breaks. Looks like rather I've got some problem with my pool. > uname -a FreeBSD limbo.xim.bz 9.1-PRERELEASE FreeBSD 9.1-PRERELEASE #1 r241326M: Mon Oct 8 05:00:21 EEST 2012 arcade@limbo.xim.bz:/usr/obj/usr/src/sys/MINIMALx32 i386 World is built WITH_CLANG_IS_CC, tmpfs nrbtree patch, CPUTYPE=native. Everything works perfectly in qemu. -- Sphinx of black quartz judge my vow. From owner-freebsd-fs@FreeBSD.ORG Tue Oct 16 11:25:59 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id E0A5A9AD for ; Tue, 16 Oct 2012 11:25:59 +0000 (UTC) (envelope-from olivier@gid0.org) Received: from mail-lb0-f182.google.com (mail-lb0-f182.google.com [209.85.217.182]) by mx1.freebsd.org (Postfix) with ESMTP id 5D1AB8FC1B for ; Tue, 16 Oct 2012 11:25:58 +0000 (UTC) Received: by mail-lb0-f182.google.com with SMTP id b5so5131043lbd.13 for ; Tue, 16 Oct 2012 04:25:57 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding:x-gm-message-state; bh=JWS/1ekXrQFiJnhTiHCvBmdmKhEoOjmikFgVYfQtHgw=; b=EVt1SGcubsOIe+8uCk6oEqAlhqOdZ/qSs8VPUDrXX6nF93RvKZlR9fyrZRwoe7G1OS GcOjLpm2aFkq2MQLib/RFkrg0+6od1kpUvcPkEWe3MqnzDTKH0k0HuVLuurRSjhGYFDe ZOr8PuW26kr/Z2MpnsX/Kg8GYAHEGM90X1fWWA6vOCnF64fYU6mm8UPkv9UC3EGSGHWo m3O3bj3fLAxM8GLhY/FHSPcIdHWRcPPqrhQQZG/vhm14uxJu1tk3CaFmL+XzIPtGMVqM rfPGoHSOaFeBDZrpUcaAZcr/jWe1FyynxVD7UtSF9Dqs7+adXvBJmi5OFe9U262SLq9A A8PQ== MIME-Version: 1.0 Received: by 10.112.45.231 with SMTP id q7mr5380448lbm.133.1350386757796; Tue, 16 Oct 2012 04:25:57 -0700 (PDT) Received: by 10.112.87.202 with HTTP; Tue, 16 Oct 2012 04:25:57 -0700 (PDT) In-Reply-To: <6116A56E-4565-4485-887E-46E3ED231606@ebureau.com> References: <6116A56E-4565-4485-887E-46E3ED231606@ebureau.com> Date: Tue, 16 Oct 2012 13:25:57 +0200 Message-ID: Subject: Re: Imposing ZFS latency limits From: Olivier Smedts To: Dustin Wenz Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Gm-Message-State: ALoCoQl0YmwBT3v/aWwKEcjlgApltKCn7FHI2iWdE4F1Zq5BFw5B/BqIPPVNswO8BshZGSKjBr4s Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Oct 2012 11:25:59 -0000 2012/10/8 Dustin Wenz : > I'm trying to find a way to get lower read latency in ZFS when there is a= failing disk in a mirror. The specific failure mode that I'm trying to han= dle is when a disk takes significantly longer than usual to respond to read= requests, but is not so far gone that the OS drops it from the bus. In tha= t case, the whole pool can become much less responsive than I think it need= s to be. > > Since ZFS has access to the data I'm reading through other members in a m= irror, I would like ZFS to defer to the redundant disks when individual dis= k latency exceeds some threshold that I would define. Is there any provisio= n for this in the FreeBSD ZFS implementation? That would be great - no need for TLER drives. But if you want to "drop" the drive from the bus, that would be a GEOM thing. Don't know if that's possible to implement. --=20 Olivier Smedts _ ASCII ribbon campaign ( ) e-mail: olivier@gid0.org - against HTML email & vCards X www: http://www.gid0.org - against proprietary attachments / \ "Il y a seulement 10 sortes de gens dans le monde : ceux qui comprennent le binaire, et ceux qui ne le comprennent pas." From owner-freebsd-fs@FreeBSD.ORG Tue Oct 16 12:49:20 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 08817120 for ; Tue, 16 Oct 2012 12:49:20 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from bigwig.baldwin.cx (bigknife-pt.tunnel.tserv9.chi1.ipv6.he.net [IPv6:2001:470:1f10:75::2]) by mx1.freebsd.org (Postfix) with ESMTP id D16418FC16 for ; Tue, 16 Oct 2012 12:49:19 +0000 (UTC) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 2F8E5B983; Tue, 16 Oct 2012 08:49:19 -0400 (EDT) From: John Baldwin To: freebsd-fs@freebsd.org Subject: Re: I have a DDB session open to a crashed ZFS server Date: Tue, 16 Oct 2012 08:44:41 -0400 User-Agent: KMail/1.13.5 (FreeBSD/8.2-CBSD-20110714-p20; KDE/4.5.5; amd64; ; ) References: <1350317019.71982.50.camel@btw.pki2.com> In-Reply-To: <1350317019.71982.50.camel@btw.pki2.com> MIME-Version: 1.0 Content-Type: Text/Plain; charset="utf-8" Content-Transfer-Encoding: 7bit Message-Id: <201210160844.41042.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Tue, 16 Oct 2012 08:49:19 -0400 (EDT) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Oct 2012 12:49:20 -0000 On Monday, October 15, 2012 12:03:39 pm Dennis Glatting wrote: > FreeBSD/amd64 (mc) (ttyu0) > > login: NMI ... going to debugger > [ thread pid 11 tid 100003 ] You got an NMI, not a crash. What happens if you just continue ('c' command) from DDB? I have heard of machines sending spurious NMIs in the past. If that is what you are seeing, there is a sysctl to disable dropping into DDB due to an NMI: machdep.kdb_on_nmi: 1 If you keep getting NMIs, try setting that to 0. -- John Baldwin From owner-freebsd-fs@FreeBSD.ORG Tue Oct 16 12:56:27 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id EE354572 for ; Tue, 16 Oct 2012 12:56:27 +0000 (UTC) (envelope-from feld@feld.me) Received: from feld.me (unknown [IPv6:2607:f4e0:100:300::2]) by mx1.freebsd.org (Postfix) with ESMTP id AF2A58FC17 for ; Tue, 16 Oct 2012 12:56:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=feld.me; s=blargle; h=In-Reply-To:Message-Id:From:Mime-Version:Date:References:Subject:Cc:To:Content-Type; bh=aZZWUfmHvMauy/QogoK+/6PCmpxF0GeQ4l9P294VnSU=; b=M4hgQxzkg6l2fgOzjyR45LozdPPmbKUUn21cjeH2dJikdbXDExQ0IjjthkbXkqjELYSTxocn6uinw3ibRMeQuSopSCpaX5iniRf/zU7Rq1zyAOTTZRAmBGPs2Od5mgTq; Received: from localhost ([127.0.0.1] helo=mwi1.coffeenet.org) by feld.me with esmtp (Exim 4.80 (FreeBSD)) (envelope-from ) id 1TO6h2-000FHE-5v; Tue, 16 Oct 2012 07:56:25 -0500 Received: from feld@feld.me by mwi1.coffeenet.org (Archiveopteryx 3.1.4) with esmtpa id 1350392174-53137-53136/5/7; Tue, 16 Oct 2012 12:56:14 +0000 Content-Type: text/plain; charset=utf-8; format=flowed; delsp=yes To: Dustin Wenz , Olivier Smedts Subject: Re: Imposing ZFS latency limits References: <6116A56E-4565-4485-887E-46E3ED231606@ebureau.com> Date: Tue, 16 Oct 2012 07:56:14 -0500 Mime-Version: 1.0 From: Mark Felder Message-Id: In-Reply-To: User-Agent: Opera Mail/12.10 (FreeBSD) X-SA-Report: ALL_TRUSTED=-1, KHOP_THREADED=-0.5 X-SA-Score: -1.5 Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Oct 2012 12:56:28 -0000 On Tue, 16 Oct 2012 06:25:57 -0500, Olivier Smedts wrote: > > That would be great - no need for TLER drives. But if you want to > "drop" the drive from the bus, that would be a GEOM thing. Don't know > if that's possible to implement. This would be GREATLY appreciated. I've seen this happen on my own ZFS boxes as well as on a custom made SAN. It's painful but easy to detect when you notice the symptoms... From owner-freebsd-fs@FreeBSD.ORG Tue Oct 16 15:16:46 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 55860A1C; Tue, 16 Oct 2012 15:16:46 +0000 (UTC) (envelope-from dg17@penx.com) Received: from btw.pki2.com (btw.pki2.com [IPv6:2001:470:a:6fd::2]) by mx1.freebsd.org (Postfix) with ESMTP id F11078FC0A; Tue, 16 Oct 2012 15:16:44 +0000 (UTC) Received: from [127.0.0.1] (localhost [127.0.0.1]) by btw.pki2.com (8.14.5/8.14.5) with ESMTP id q9GFGbt3041969; Tue, 16 Oct 2012 08:16:37 -0700 (PDT) (envelope-from dg17@penx.com) Subject: Re: I have a DDB session open to a crashed ZFS server From: Dennis Glatting To: John Baldwin In-Reply-To: <201210160844.41042.jhb@freebsd.org> References: <1350317019.71982.50.camel@btw.pki2.com> <201210160844.41042.jhb@freebsd.org> Content-Type: text/plain; charset="us-ascii" Date: Tue, 16 Oct 2012 08:16:37 -0700 Message-ID: <1350400597.72003.32.camel@btw.pki2.com> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port Content-Transfer-Encoding: 7bit X-yoursite-MailScanner-Information: Dennis Glatting X-yoursite-MailScanner-ID: q9GFGbt3041969 X-yoursite-MailScanner: Found to be clean X-MailScanner-From: dg17@penx.com Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: dg17@penx.com List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Oct 2012 15:16:46 -0000 On Tue, 2012-10-16 at 08:44 -0400, John Baldwin wrote: > On Monday, October 15, 2012 12:03:39 pm Dennis Glatting wrote: > > FreeBSD/amd64 (mc) (ttyu0) > > > > login: NMI ... going to debugger > > [ thread pid 11 tid 100003 ] > > You got an NMI, not a crash. What happens if you just continue ('c' command) > from DDB? > I hit the NMI button because of the "crash," which is a misword, to get into DDB. The problem I am having with ZFS where the file systems go dorment under load within 24 hours. Specifically, the processes are still alive, stuck in disk wait, and there is no disk I/O. I have this problem across four machines for months. The network and console still work but if I enter a command requiring a pull from disk, nothing comes back. > I have heard of machines sending spurious NMIs in the past. If that is what > you are seeing, there is a sysctl to disable dropping into DDB due to an NMI: > > machdep.kdb_on_nmi: 1 > > If you keep getting NMIs, try setting that to 0. > The DDB session is still open but I don't see why this system is stuck. I have been looking at locked processes (two below) but I'm not familiar with the code. Maybe a deadlock? Maybe a missed interrupt? Maybe an unsupported controller? I dunno. 0xfffffe0b989803f0: 0xfffffe0b989803f0: tag zfs, type VDIR tag zfs, type VDIR usecount 0, writecount 0, refcount 2 mountedhere 0 usecount 0, writecount 0, refcount 2 mountedhere 0 flags (VI_DOINGINACT|VI(0x200)) flags (VI_DOINGINACT|VI(0x200)) v_object 0xfffffe0b87af7488 ref 0 pages 0 v_object 0xfffffe0b87af7488 ref 0 pages 0 lock type zfs: EXCL by thread 0xfffffe09b48fe900 (pid 70646) lock type zfs: EXCL by thread 0xfffffe09b48fe900 (pid 70646) db> show thread 0xfffffe09b48fe900 Thread 104609 at 0xfffffe09b48fe900: proc (pid 70646): 0xfffffe0af7e79940 name: find stack: 0xffffffa3329b2000-0xffffffa3329b5fff flags: 0x4 pflags: 0 state: INHIBITED: {SLEEPING} wmesg: tx->tx_quiesce_done_cv) wchan: 0xfffffe0059b60240 priority: 120 container lock: sleepq chain (0xffffffff8126d498) db> sh proc 70646 Process 70646 (find) at 0xfffffe0af7e79940: state: NORMAL uid: 0 gids: 0, 5 parent: pid 70645 at 0xfffffe006e4974a0 ABI: FreeBSD ELF64 arguments: find threads: 1 104609 D tx->tx_q 0xfffffe0059b60240 find db> tr 104609 Tracing pid 70646 tid 104609 td 0xfffffe09b48fe900 sched_switch() at sched_switch+0x28b mi_switch() at mi_switch+0xdf sleepq_wait() at sleepq_wait+0x3a _cv_wait() at _cv_wait+0x164 txg_wait_open() at txg_wait_open+0x85 dmu_tx_assign() at dmu_tx_assign+0x38 zfs_inactive() at zfs_inactive+0x8e zfs_freebsd_inactive() at zfs_freebsd_inactive+0xd VOP_INACTIVE_APV() at VOP_INACTIVE_APV+0x5d vinactive() at vinactive+0xef vputx() at vputx+0x244 sys_fchdir() at sys_fchdir+0x3f0 amd64_syscall() at amd64_syscall+0x334 Xfast_syscall() at Xfast_syscall+0xfb --- syscall (13, FreeBSD ELF64, sys_fchdir), rip = 0x80088396c, rsp = 0x7fffffffd998, rbp = 0x7fffffffda40 --- 0xfffffe005c8855e8: 0xfffffe005c8855e8: tag syncer, type VNON tag syncer, type VNON usecount 1, writecount 0, refcount 2 mountedhere 0 usecount 1, writecount 0, refcount 2 mountedhere 0 flags (VI(0x200)) flags (VI(0x200)) lock type syncer: EXCL by thread 0xfffffe0039147480 (pid 20) lock type syncer: EXCL by thread 0xfffffe0039147480 (pid 20) db> sh thread 0xfffffe0039147480 Thread 100242 at 0xfffffe0039147480: proc (pid 20): 0xfffffe003f4bc940 name: syncer stack: 0xffffffa2fc73b000-0xffffffa2fc73efff flags: 0x4 pflags: 0x240800 state: INHIBITED: {SLEEPING} wmesg: zio->io_cv) wchan: 0xfffffe01187c5320 priority: 116 container lock: sleepq chain (0xffffffff8126e140) db> sh proc 20 Process 20 (syncer) at 0xfffffe003f4bc940: state: NORMAL uid: 0 gids: 0 parent: pid 0 at 0xffffffff812cbdd8 ABI: null threads: 1 100242 D zio->io_ 0xfffffe01187c5320 [syncer] db> tr 100242 Tracing pid 20 tid 100242 td 0xfffffe0039147480 sched_switch() at sched_switch+0x28b mi_switch() at mi_switch+0xdf sleepq_wait() at sleepq_wait+0x3a _cv_wait() at _cv_wait+0x164 zio_wait() at zio_wait+0x5b zil_commit() at zil_commit+0x833 zfs_sync() at zfs_sync+0xaa sync_fsync() at sync_fsync+0x168 VOP_FSYNC_APV() at VOP_FSYNC_APV+0x5d sync_vnode() at sync_vnode+0x1b0 sched_sync() at sched_sync+0x29f fork_exit() at fork_exit+0x9a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffa2fc73eb30, rbp = 0 --- From owner-freebsd-fs@FreeBSD.ORG Tue Oct 16 15:25:31 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id E93A7D77; Tue, 16 Oct 2012 15:25:31 +0000 (UTC) (envelope-from freebsd@penx.com) Received: from btw.pki2.com (btw.pki2.com [IPv6:2001:470:a:6fd::2]) by mx1.freebsd.org (Postfix) with ESMTP id 7B21E8FC19; Tue, 16 Oct 2012 15:25:30 +0000 (UTC) Received: from [127.0.0.1] (localhost [127.0.0.1]) by btw.pki2.com (8.14.5/8.14.5) with ESMTP id q9GFPOb5046028; Tue, 16 Oct 2012 08:25:24 -0700 (PDT) (envelope-from freebsd@penx.com) Subject: Re: I have a DDB session open to a crashed ZFS server From: Dennis Glatting To: John Baldwin In-Reply-To: <1350400597.72003.32.camel@btw.pki2.com> References: <1350317019.71982.50.camel@btw.pki2.com> <201210160844.41042.jhb@freebsd.org> <1350400597.72003.32.camel@btw.pki2.com> Content-Type: text/plain; charset="us-ascii" Date: Tue, 16 Oct 2012 08:25:24 -0700 Message-ID: <1350401124.72003.38.camel@btw.pki2.com> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port Content-Transfer-Encoding: 7bit X-yoursite-MailScanner-Information: Dennis Glatting X-yoursite-MailScanner-ID: q9GFPOb5046028 X-yoursite-MailScanner: Found to be clean X-MailScanner-From: freebsd@penx.com Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Oct 2012 15:25:32 -0000 On Tue, 2012-10-16 at 08:44 -0400, John Baldwin wrote: > On Monday, October 15, 2012 12:03:39 pm Dennis Glatting wrote: > > FreeBSD/amd64 (mc) (ttyu0) > > > > login: NMI ... going to debugger > > [ thread pid 11 tid 100003 ] > > You got an NMI, not a crash. What happens if you just continue ('c' command) > from DDB? > I hit the NMI button because of the "crash," which is a misword, to get into DDB. The problem I am having with ZFS where the file systems go dorment under load within 24 hours. Specifically, the processes are still alive, stuck in disk wait, and there is no disk I/O. I have this problem across four machines for months. The network and console still work but if I enter a command requiring a pull from disk, nothing comes back. > I have heard of machines sending spurious NMIs in the past. If that is what > you are seeing, there is a sysctl to disable dropping into DDB due to an NMI: > > machdep.kdb_on_nmi: 1 > > If you keep getting NMIs, try setting that to 0. > The DDB session is still open but I don't see why this system is stuck. I have been looking at locked processes (two below) but I'm not familiar with the code. Maybe a deadlock? Maybe a missed interrupt? Maybe an unsupported controller? I dunno. 0xfffffe0b989803f0: 0xfffffe0b989803f0: tag zfs, type VDIR tag zfs, type VDIR usecount 0, writecount 0, refcount 2 mountedhere 0 usecount 0, writecount 0, refcount 2 mountedhere 0 flags (VI_DOINGINACT|VI(0x200)) flags (VI_DOINGINACT|VI(0x200)) v_object 0xfffffe0b87af7488 ref 0 pages 0 v_object 0xfffffe0b87af7488 ref 0 pages 0 lock type zfs: EXCL by thread 0xfffffe09b48fe900 (pid 70646) lock type zfs: EXCL by thread 0xfffffe09b48fe900 (pid 70646) db> show thread 0xfffffe09b48fe900 Thread 104609 at 0xfffffe09b48fe900: proc (pid 70646): 0xfffffe0af7e79940 name: find stack: 0xffffffa3329b2000-0xffffffa3329b5fff flags: 0x4 pflags: 0 state: INHIBITED: {SLEEPING} wmesg: tx->tx_quiesce_done_cv) wchan: 0xfffffe0059b60240 priority: 120 container lock: sleepq chain (0xffffffff8126d498) db> sh proc 70646 Process 70646 (find) at 0xfffffe0af7e79940: state: NORMAL uid: 0 gids: 0, 5 parent: pid 70645 at 0xfffffe006e4974a0 ABI: FreeBSD ELF64 arguments: find threads: 1 104609 D tx->tx_q 0xfffffe0059b60240 find db> tr 104609 Tracing pid 70646 tid 104609 td 0xfffffe09b48fe900 sched_switch() at sched_switch+0x28b mi_switch() at mi_switch+0xdf sleepq_wait() at sleepq_wait+0x3a _cv_wait() at _cv_wait+0x164 txg_wait_open() at txg_wait_open+0x85 dmu_tx_assign() at dmu_tx_assign+0x38 zfs_inactive() at zfs_inactive+0x8e zfs_freebsd_inactive() at zfs_freebsd_inactive+0xd VOP_INACTIVE_APV() at VOP_INACTIVE_APV+0x5d vinactive() at vinactive+0xef vputx() at vputx+0x244 sys_fchdir() at sys_fchdir+0x3f0 amd64_syscall() at amd64_syscall+0x334 Xfast_syscall() at Xfast_syscall+0xfb --- syscall (13, FreeBSD ELF64, sys_fchdir), rip = 0x80088396c, rsp = 0x7fffffffd998, rbp = 0x7fffffffda40 --- 0xfffffe005c8855e8: 0xfffffe005c8855e8: tag syncer, type VNON tag syncer, type VNON usecount 1, writecount 0, refcount 2 mountedhere 0 usecount 1, writecount 0, refcount 2 mountedhere 0 flags (VI(0x200)) flags (VI(0x200)) lock type syncer: EXCL by thread 0xfffffe0039147480 (pid 20) lock type syncer: EXCL by thread 0xfffffe0039147480 (pid 20) db> sh thread 0xfffffe0039147480 Thread 100242 at 0xfffffe0039147480: proc (pid 20): 0xfffffe003f4bc940 name: syncer stack: 0xffffffa2fc73b000-0xffffffa2fc73efff flags: 0x4 pflags: 0x240800 state: INHIBITED: {SLEEPING} wmesg: zio->io_cv) wchan: 0xfffffe01187c5320 priority: 116 container lock: sleepq chain (0xffffffff8126e140) db> sh proc 20 Process 20 (syncer) at 0xfffffe003f4bc940: state: NORMAL uid: 0 gids: 0 parent: pid 0 at 0xffffffff812cbdd8 ABI: null threads: 1 100242 D zio->io_ 0xfffffe01187c5320 [syncer] db> tr 100242 Tracing pid 20 tid 100242 td 0xfffffe0039147480 sched_switch() at sched_switch+0x28b mi_switch() at mi_switch+0xdf sleepq_wait() at sleepq_wait+0x3a _cv_wait() at _cv_wait+0x164 zio_wait() at zio_wait+0x5b zil_commit() at zil_commit+0x833 zfs_sync() at zfs_sync+0xaa sync_fsync() at sync_fsync+0x168 VOP_FSYNC_APV() at VOP_FSYNC_APV+0x5d sync_vnode() at sync_vnode+0x1b0 sched_sync() at sched_sync+0x29f fork_exit() at fork_exit+0x9a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffa2fc73eb30, rbp = 0 --- From owner-freebsd-fs@FreeBSD.ORG Tue Oct 16 16:17:32 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 3F31DD05 for ; Tue, 16 Oct 2012 16:17:32 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from bigwig.baldwin.cx (bigknife-pt.tunnel.tserv9.chi1.ipv6.he.net [IPv6:2001:470:1f10:75::2]) by mx1.freebsd.org (Postfix) with ESMTP id 127A38FC18 for ; Tue, 16 Oct 2012 16:17:32 +0000 (UTC) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 6A670B924; Tue, 16 Oct 2012 12:17:31 -0400 (EDT) From: John Baldwin To: dg17@penx.com Subject: Re: I have a DDB session open to a crashed ZFS server Date: Tue, 16 Oct 2012 12:15:33 -0400 User-Agent: KMail/1.13.5 (FreeBSD/8.2-CBSD-20110714-p20; KDE/4.5.5; amd64; ; ) References: <1350317019.71982.50.camel@btw.pki2.com> <201210160844.41042.jhb@freebsd.org> <1350400597.72003.32.camel@btw.pki2.com> In-Reply-To: <1350400597.72003.32.camel@btw.pki2.com> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <201210161215.33369.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Tue, 16 Oct 2012 12:17:31 -0400 (EDT) Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Oct 2012 16:17:32 -0000 On Tuesday, October 16, 2012 11:16:37 am Dennis Glatting wrote: > On Tue, 2012-10-16 at 08:44 -0400, John Baldwin wrote: > > On Monday, October 15, 2012 12:03:39 pm Dennis Glatting wrote: > > > FreeBSD/amd64 (mc) (ttyu0) > > > > > > login: NMI ... going to debugger > > > [ thread pid 11 tid 100003 ] > > > > You got an NMI, not a crash. What happens if you just continue ('c' command) > > from DDB? > > > > I hit the NMI button because of the "crash," which is a misword, to get > into DDB. Ah, I would suggest "hung" or "deadlocked" next time. It certainly seems like a deadlock since all CPUs are idle. Some helpful commands here might be 'show sleepchain' and 'show lockchain'. Pick a "stuck" process (like find) and run: 'show sleepchain ' In your case though it seems both 'find' and the various 'pbzip2' threads are stuck on a condition variable, so there isn't an easy way to identify an "owner" that is supposed to awaken these threads. It could be a case of a missed wakeup perhaps, but you'll need to get someone more familiar with ZFS to identify where these codes should be awakened normally. -- John Baldwin From owner-freebsd-fs@FreeBSD.ORG Tue Oct 16 16:29:42 2012 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 76C93115; Tue, 16 Oct 2012 16:29:42 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 8DC1F8FC14; Tue, 16 Oct 2012 16:29:40 +0000 (UTC) Received: from odyssey.starpoint.kiev.ua (alpha-e.starpoint.kiev.ua [212.40.38.101]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id TAA25582; Tue, 16 Oct 2012 19:29:29 +0300 (EEST) (envelope-from avg@FreeBSD.org) Message-ID: <507D8B69.3090903@FreeBSD.org> Date: Tue, 16 Oct 2012 19:29:29 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:16.0) Gecko/20121012 Thunderbird/16.0.1 MIME-Version: 1.0 To: dg17@penx.com Subject: Re: I have a DDB session open to a crashed ZFS server References: <1350317019.71982.50.camel@btw.pki2.com> <201210160844.41042.jhb@freebsd.org> <1350400597.72003.32.camel@btw.pki2.com> <201210161215.33369.jhb@freebsd.org> In-Reply-To: <201210161215.33369.jhb@freebsd.org> X-Enigmail-Version: 1.4.5 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Cc: freebsd-fs@FreeBSD.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Oct 2012 16:29:42 -0000 on 16/10/2012 19:15 John Baldwin said the following: > On Tuesday, October 16, 2012 11:16:37 am Dennis Glatting wrote: >> On Tue, 2012-10-16 at 08:44 -0400, John Baldwin wrote: >>> On Monday, October 15, 2012 12:03:39 pm Dennis Glatting wrote: >>>> FreeBSD/amd64 (mc) (ttyu0) >>>> >>>> login: NMI ... going to debugger >>>> [ thread pid 11 tid 100003 ] >>> >>> You got an NMI, not a crash. What happens if you just continue ('c' command) >>> from DDB? >>> >> >> I hit the NMI button because of the "crash," which is a misword, to get >> into DDB. > > Ah, I would suggest "hung" or "deadlocked" next time. It certainly seems like > a deadlock since all CPUs are idle. Some helpful commands here might be > 'show sleepchain' and 'show lockchain'. > > Pick a "stuck" process (like find) and run: > > 'show sleepchain ' > > In your case though it seems both 'find' and the various 'pbzip2' threads > are stuck on a condition variable, so there isn't an easy way to identify > an "owner" that is supposed to awaken these threads. It could be a case > of a missed wakeup perhaps, but you'll need to get someone more familiar > with ZFS to identify where these codes should be awakened normally. > I would also re-iterate a suggestion that I made to Nikolay ealrier: http://article.gmane.org/gmane.os.freebsd.devel.file-systems/15981 BTW, in that case it turned out to be a genuine deadlock in ZFS ARC handling of lowmem. procstat -kk -a is a great help for analyzing such situations. -- Andriy Gapon From owner-freebsd-fs@FreeBSD.ORG Tue Oct 16 18:48:22 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id DABA7DAC; Tue, 16 Oct 2012 18:48:22 +0000 (UTC) (envelope-from dg@pki2.com) Received: from btw.pki2.com (btw.pki2.com [IPv6:2001:470:a:6fd::2]) by mx1.freebsd.org (Postfix) with ESMTP id 7FC398FC16; Tue, 16 Oct 2012 18:48:22 +0000 (UTC) Received: from btw.pki2.com (btw.pki2.com [192.168.23.1]) by btw.pki2.com (8.14.5/8.14.5) with ESMTP id q9GImE1d029056 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT); Tue, 16 Oct 2012 11:48:16 -0700 (PDT) (envelope-from dg@pki2.com) Date: Tue, 16 Oct 2012 11:48:14 -0700 (PDT) From: Dennis Glatting X-X-Sender: dennisg@btw.pki2.com To: Andriy Gapon Subject: Re: I have a DDB session open to a crashed ZFS server In-Reply-To: <507D8B69.3090903@FreeBSD.org> Message-ID: References: <1350317019.71982.50.camel@btw.pki2.com> <201210160844.41042.jhb@freebsd.org> <1350400597.72003.32.camel@btw.pki2.com> <201210161215.33369.jhb@freebsd.org> <507D8B69.3090903@FreeBSD.org> User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-yoursite-MailScanner-Information: Dennis Glatting X-yoursite-MailScanner-ID: q9GImE1d029056 X-yoursite-MailScanner: Found to be clean X-MailScanner-From: dg@pki2.com Cc: freebsd-fs@freebsd.org, dg17@penx.com X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Oct 2012 18:48:23 -0000 On Tue, 16 Oct 2012, Andriy Gapon wrote: > on 16/10/2012 19:15 John Baldwin said the following: >> On Tuesday, October 16, 2012 11:16:37 am Dennis Glatting wrote: >>> On Tue, 2012-10-16 at 08:44 -0400, John Baldwin wrote: >>>> On Monday, October 15, 2012 12:03:39 pm Dennis Glatting wrote: >>>>> FreeBSD/amd64 (mc) (ttyu0) >>>>> >>>>> login: NMI ... going to debugger >>>>> [ thread pid 11 tid 100003 ] >>>> >>>> You got an NMI, not a crash. What happens if you just continue ('c' command) >>>> from DDB? >>>> >>> >>> I hit the NMI button because of the "crash," which is a misword, to get >>> into DDB. >> >> Ah, I would suggest "hung" or "deadlocked" next time. It certainly seems like >> a deadlock since all CPUs are idle. Some helpful commands here might be >> 'show sleepchain' and 'show lockchain'. >> >> Pick a "stuck" process (like find) and run: >> >> 'show sleepchain ' >> >> In your case though it seems both 'find' and the various 'pbzip2' threads >> are stuck on a condition variable, so there isn't an easy way to identify >> an "owner" that is supposed to awaken these threads. It could be a case >> of a missed wakeup perhaps, but you'll need to get someone more familiar >> with ZFS to identify where these codes should be awakened normally. >> > > I would also re-iterate a suggestion that I made to Nikolay ealrier: > http://article.gmane.org/gmane.os.freebsd.devel.file-systems/15981 > > BTW, in that case it turned out to be a genuine deadlock in ZFS ARC handling of > lowmem. > procstat -kk -a is a great help for analyzing such situations. > Without restarting the server and from memory, I believe the ARC on this server is 32GB. The L2ARC is a 50-60GB SSD. The ZIL is a 16GB partitioned SSD but my non-ZIL systems have the same problem. Main memory is 128GB. I can run procstat to a serial console and scarf the output. What interval would be helpful? Five seconds? Remember when the system hangs, no commands will run so the data will be pre-hang. BTW, it takes 4-24 hours to hang under load. Also, are you suggesting I apply the patch in the URL and run again? I have been following your other posts but the patches you posted did not cleanly apply, so I removed them from my rev. From owner-freebsd-fs@FreeBSD.ORG Tue Oct 16 20:00:02 2012 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 0201AE14 for ; Tue, 16 Oct 2012 20:00:02 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [8.8.178.135]) by mx1.freebsd.org (Postfix) with ESMTP id C4C118FC0C for ; Tue, 16 Oct 2012 20:00:01 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q9GK01Ru095721 for ; Tue, 16 Oct 2012 20:00:01 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q9GK015o095720; Tue, 16 Oct 2012 20:00:01 GMT (envelope-from gnats) Date: Tue, 16 Oct 2012 20:00:01 GMT Message-Id: <201210162000.q9GK015o095720@freefall.freebsd.org> To: freebsd-fs@FreeBSD.org Cc: From: Martin Matuska Subject: Re: kern/171761: [zfs] [patch] Small patch to (temporarily) remove -r from zfs send usage and zfs(8) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: Martin Matuska List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Oct 2012 20:00:02 -0000 The following reply was made to PR kern/171761; it has been noted by GNATS. From: Martin Matuska To: bug-followup@FreeBSD.org, thomas@gibfest.dk Cc: Subject: Re: kern/171761: [zfs] [patch] Small patch to (temporarily) remove -r from zfs send usage and zfs(8) Date: Tue, 16 Oct 2012 21:51:30 +0200 This has already been fixed in r240955 and will be committed to 9-STABLE together with feature flags after 9.1-RELEASE is out. http://svn.freebsd.org/changeset/base/240955 -- Martin Matuska FreeBSD committer http://blog.vx.sk From owner-freebsd-fs@FreeBSD.ORG Tue Oct 16 20:01:03 2012 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id EDF3DEA8; Tue, 16 Oct 2012 20:01:03 +0000 (UTC) (envelope-from mm@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [8.8.178.135]) by mx1.freebsd.org (Postfix) with ESMTP id BF2E18FC17; Tue, 16 Oct 2012 20:01:03 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q9GK13LV095996; Tue, 16 Oct 2012 20:01:03 GMT (envelope-from mm@freefall.freebsd.org) Received: (from mm@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q9GK13TS095992; Tue, 16 Oct 2012 20:01:03 GMT (envelope-from mm) Date: Tue, 16 Oct 2012 20:01:03 GMT Message-Id: <201210162001.q9GK13TS095992@freefall.freebsd.org> To: mm@FreeBSD.org, freebsd-fs@FreeBSD.org, mm@FreeBSD.org From: mm@FreeBSD.org Subject: Re: kern/171761: [zfs] [patch] Small patch to (temporarily) remove -r from zfs send usage and zfs(8) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Oct 2012 20:01:04 -0000 Synopsis: [zfs] [patch] Small patch to (temporarily) remove -r from zfs send usage and zfs(8) Responsible-Changed-From-To: freebsd-fs->mm Responsible-Changed-By: mm Responsible-Changed-When: Tue Oct 16 20:01:03 UTC 2012 Responsible-Changed-Why: I'll take it. http://www.freebsd.org/cgi/query-pr.cgi?pr=171761 From owner-freebsd-fs@FreeBSD.ORG Tue Oct 16 20:10:22 2012 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 7AE2C22D for ; Tue, 16 Oct 2012 20:10:22 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id A5E8F8FC14 for ; Tue, 16 Oct 2012 20:10:21 +0000 (UTC) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id XAA26724; Tue, 16 Oct 2012 23:10:14 +0300 (EEST) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1TODSv-000LzB-Um; Tue, 16 Oct 2012 23:10:13 +0300 Message-ID: <507DBF23.4050303@FreeBSD.org> Date: Tue, 16 Oct 2012 23:10:11 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:16.0) Gecko/20121013 Thunderbird/16.0.1 MIME-Version: 1.0 To: Dennis Glatting Subject: Re: I have a DDB session open to a crashed ZFS server References: <1350317019.71982.50.camel@btw.pki2.com> <201210160844.41042.jhb@freebsd.org> <1350400597.72003.32.camel@btw.pki2.com> <201210161215.33369.jhb@freebsd.org> <507D8B69.3090903@FreeBSD.org> In-Reply-To: X-Enigmail-Version: 1.4.5 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-fs@FreeBSD.org, dg17@penx.com X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Oct 2012 20:10:22 -0000 on 16/10/2012 21:48 Dennis Glatting said the following: > > > On Tue, 16 Oct 2012, Andriy Gapon wrote: > >> on 16/10/2012 19:15 John Baldwin said the following: >>> On Tuesday, October 16, 2012 11:16:37 am Dennis Glatting wrote: >>>> On Tue, 2012-10-16 at 08:44 -0400, John Baldwin wrote: >>>>> On Monday, October 15, 2012 12:03:39 pm Dennis Glatting wrote: >>>>>> FreeBSD/amd64 (mc) (ttyu0) >>>>>> >>>>>> login: NMI ... going to debugger >>>>>> [ thread pid 11 tid 100003 ] >>>>> >>>>> You got an NMI, not a crash. What happens if you just continue ('c' command) >>>>> from DDB? >>>>> >>>> >>>> I hit the NMI button because of the "crash," which is a misword, to get >>>> into DDB. >>> >>> Ah, I would suggest "hung" or "deadlocked" next time. It certainly seems like >>> a deadlock since all CPUs are idle. Some helpful commands here might be >>> 'show sleepchain' and 'show lockchain'. >>> >>> Pick a "stuck" process (like find) and run: >>> >>> 'show sleepchain ' >>> >>> In your case though it seems both 'find' and the various 'pbzip2' threads >>> are stuck on a condition variable, so there isn't an easy way to identify >>> an "owner" that is supposed to awaken these threads. It could be a case >>> of a missed wakeup perhaps, but you'll need to get someone more familiar >>> with ZFS to identify where these codes should be awakened normally. >>> >> >> I would also re-iterate a suggestion that I made to Nikolay ealrier: >> http://article.gmane.org/gmane.os.freebsd.devel.file-systems/15981 >> >> BTW, in that case it turned out to be a genuine deadlock in ZFS ARC handling of >> lowmem. >> procstat -kk -a is a great help for analyzing such situations. >> > > Without restarting the server and from memory, I believe the ARC on this server > is 32GB. The L2ARC is a 50-60GB SSD. The ZIL is a 16GB partitioned SSD but my > non-ZIL systems have the same problem. Main memory is 128GB. This information doesn't help with the debugging unfortunately... > I can run procstat to a serial console and scarf the output. What interval would > be helpful? Five seconds? Remember when the system hangs, no commands will run > so the data will be pre-hang. Hmm... No, I need its output when the system hangs. If you can arrange to have a memory disk (mdmfs, mdconfig) with UFS on it, then I believe that you should be able to run procstat (and the shared libraries that it uses) from it. > BTW, it takes 4-24 hours to hang under load. > > Also, are you suggesting I apply the patch in the URL and run again? I have been > following your other posts but the patches you posted did not cleanly apply, so > I removed them from my rev. > The patches are intended for head and recent stable. The patch that you are referring to may help with debugging. -- Andriy Gapon From owner-freebsd-fs@FreeBSD.ORG Wed Oct 17 07:04:49 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 91EC08E0; Wed, 17 Oct 2012 07:04:49 +0000 (UTC) (envelope-from ndenev@gmail.com) Received: from mail-we0-f182.google.com (mail-we0-f182.google.com [74.125.82.182]) by mx1.freebsd.org (Postfix) with ESMTP id E81988FC0C; Wed, 17 Oct 2012 07:04:48 +0000 (UTC) Received: by mail-we0-f182.google.com with SMTP id x43so5276803wey.13 for ; Wed, 17 Oct 2012 00:04:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer; bh=rEt/+eA91eEpqN3jTjiYKzdDI5qyyUcq/FvbyVbqzzY=; b=VDDkhUJZsQEURODc9Y0WVKMQHijxsfS4Bbtky2Ixs9pjNAqAhf9oDkC7NW8FVT3Juh StoVAJkEK/dJoat/r1H7rMj3yKTr3oqC0a7YTfANoB2AOSuGuOT+ip+yKsrMAF3mNKgU rYFo9y4k1d9kQK8FL3YqFvc3na1Wk3t+hVjaF91yMU6ocRYUcjk1cJ88ItQZjw0jWM1d y3E62m3aGtR2+CDP/hyWHOG4LD6ilKvQorR+mA7YO1WIpavPkn7nuHfHbNnLOzbGJO+H gzks026+6l/xs7M6dl+6yBQeEUeKnqiv+UG4Rc+zLaavmm9wkwf0oYcB7Dr7DWTEhnUN Yo2w== Received: by 10.180.83.101 with SMTP id p5mr1943415wiy.2.1350457481904; Wed, 17 Oct 2012 00:04:41 -0700 (PDT) Received: from [10.0.0.86] ([93.152.184.10]) by mx.google.com with ESMTPS id w8sm22912550wif.4.2012.10.17.00.04.39 (version=TLSv1/SSLv3 cipher=OTHER); Wed, 17 Oct 2012 00:04:41 -0700 (PDT) Subject: Re: I have a DDB session open to a crashed ZFS server Mime-Version: 1.0 (Mac OS X Mail 6.1 \(1498\)) Content-Type: text/plain; charset=us-ascii From: Nikolay Denev In-Reply-To: Date: Wed, 17 Oct 2012 10:04:38 +0300 Content-Transfer-Encoding: quoted-printable Message-Id: <0B0CA833-79FA-4C8E-86AC-828E7947FF67@gmail.com> References: <1350317019.71982.50.camel@btw.pki2.com> <201210160844.41042.jhb@freebsd.org> <1350400597.72003.32.camel@btw.pki2.com> <201210161215.33369.jhb@freebsd.org> <507D8B69.3090903@FreeBSD.org> To: Dennis Glatting X-Mailer: Apple Mail (2.1498) Cc: freebsd-fs@freebsd.org, dg17@penx.com, Andriy Gapon X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Oct 2012 07:04:49 -0000 On Oct 16, 2012, at 9:48 PM, Dennis Glatting wrote: >=20 >=20 > On Tue, 16 Oct 2012, Andriy Gapon wrote: >=20 >> on 16/10/2012 19:15 John Baldwin said the following: >>> On Tuesday, October 16, 2012 11:16:37 am Dennis Glatting wrote: >>>> On Tue, 2012-10-16 at 08:44 -0400, John Baldwin wrote: >>>>> On Monday, October 15, 2012 12:03:39 pm Dennis Glatting wrote: >>>>>> FreeBSD/amd64 (mc) (ttyu0) >>>>>>=20 >>>>>> login: NMI ... going to debugger >>>>>> [ thread pid 11 tid 100003 ] >>>>>=20 >>>>> You got an NMI, not a crash. What happens if you just continue = ('c' command) >>>>> from DDB? >>>>>=20 >>>>=20 >>>> I hit the NMI button because of the "crash," which is a misword, to = get >>>> into DDB. >>>=20 >>> Ah, I would suggest "hung" or "deadlocked" next time. It certainly = seems like >>> a deadlock since all CPUs are idle. Some helpful commands here = might be >>> 'show sleepchain' and 'show lockchain'. >>>=20 >>> Pick a "stuck" process (like find) and run: >>>=20 >>> 'show sleepchain ' >>>=20 >>> In your case though it seems both 'find' and the various 'pbzip2' = threads >>> are stuck on a condition variable, so there isn't an easy way to = identify >>> an "owner" that is supposed to awaken these threads. It could be a = case >>> of a missed wakeup perhaps, but you'll need to get someone more = familiar >>> with ZFS to identify where these codes should be awakened normally. >>>=20 >>=20 >> I would also re-iterate a suggestion that I made to Nikolay ealrier: >> http://article.gmane.org/gmane.os.freebsd.devel.file-systems/15981 >>=20 >> BTW, in that case it turned out to be a genuine deadlock in ZFS ARC = handling of >> lowmem. >> procstat -kk -a is a great help for analyzing such situations. >>=20 >=20 > Without restarting the server and from memory, I believe the ARC on = this server is 32GB. The L2ARC is a 50-60GB SSD. The ZIL is a 16GB = partitioned SSD but my non-ZIL systems have the same problem. Main = memory is 128GB. >=20 > I can run procstat to a serial console and scarf the output. What = interval would be helpful? Five seconds? Remember when the system hangs, = no commands will run so the data will be pre-hang. >=20 > BTW, it takes 4-24 hours to hang under load. >=20 > Also, are you suggesting I apply the patch in the URL and run again? I = have been following your other posts but the patches you posted did not = cleanly apply, so I removed them from my rev. >=20 > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" Hi, I'm running with the patch from here : = http://thread.gmane.org/gmane.os.freebsd.devel.file-systems/16000/focus=3D= 16017 And there were no deadlocks since it's applied. If you're hitting the same issue as I was, this should help. Regards, Nikolay= From owner-freebsd-fs@FreeBSD.ORG Wed Oct 17 11:29:59 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id E790D4BB for ; Wed, 17 Oct 2012 11:29:59 +0000 (UTC) (envelope-from prvs=1636d8f419=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id 6C0668FC0C for ; Wed, 17 Oct 2012 11:29:58 +0000 (UTC) Received: from r2d2 ([188.220.16.49]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50000716770.msg for ; Tue, 16 Oct 2012 16:46:11 +0100 X-Spam-Processed: mail1.multiplay.co.uk, Tue, 16 Oct 2012 16:46:11 +0100 (not processed: message from valid local sender) X-MDRemoteIP: 188.220.16.49 X-Return-Path: prvs=1636d8f419=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-fs@freebsd.org Message-ID: <089898A4493042448C934643FD5C3887@multiplay.co.uk> From: "Steven Hartland" To: "Dustin Wenz" , "Olivier Smedts" , "Mark Felder" References: <6116A56E-4565-4485-887E-46E3ED231606@ebureau.com> Subject: Re: Imposing ZFS latency limits Date: Tue, 16 Oct 2012 16:46:00 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=response Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Oct 2012 11:30:00 -0000 ----- Original Message ----- From: "Mark Felder" > On Tue, 16 Oct 2012 06:25:57 -0500, Olivier Smedts > wrote: > >> >> That would be great - no need for TLER drives. But if you want to >> "drop" the drive from the bus, that would be a GEOM thing. Don't know >> if that's possible to implement. > > This would be GREATLY appreciated. I've seen this happen on my own ZFS > boxes as well as on a custom made SAN. It's painful but easy to detect > when you notice the symptoms... Interesting, what metrics where you using which made it easy to detect, work be nice to know your process there Mark? Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-fs@FreeBSD.ORG Wed Oct 17 13:38:59 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 09575CC6 for ; Wed, 17 Oct 2012 13:38:59 +0000 (UTC) (envelope-from prvs=16374f1e9b=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id 904EF8FC19 for ; Wed, 17 Oct 2012 13:38:57 +0000 (UTC) Received: from r2d2 ([188.220.16.49]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50000727532.msg for ; Wed, 17 Oct 2012 14:38:55 +0100 X-Spam-Processed: mail1.multiplay.co.uk, Wed, 17 Oct 2012 14:38:55 +0100 (not processed: message from valid local sender) X-MDRemoteIP: 188.220.16.49 X-Return-Path: prvs=16374f1e9b=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-fs@freebsd.org Message-ID: <8C42CB56F50F4D8AB949BDD91970D388@multiplay.co.uk> From: "Steven Hartland" To: "Dustin Wenz" , "Olivier Smedts" , "Mark Felder" Subject: Re: Imposing ZFS latency limits Date: Wed, 17 Oct 2012 14:38:46 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=response Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Oct 2012 13:38:59 -0000 ----- Original Message ----- From: "Mark Felder" > On Tue, 16 Oct 2012 06:25:57 -0500, Olivier Smedts > wrote: > >> >> That would be great - no need for TLER drives. But if you want to >> "drop" the drive from the bus, that would be a GEOM thing. Don't know >> if that's possible to implement. > > This would be GREATLY appreciated. I've seen this happen on my own ZFS > boxes as well as on a custom made SAN. It's painful but easy to detect > when you notice the symptoms... Interesting, what metrics where you using which made it easy to detect, work be nice to know your process there Mark? Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-fs@FreeBSD.ORG Wed Oct 17 14:35:53 2012 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 83EB0CB9 for ; Wed, 17 Oct 2012 14:35:53 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id C8A4B8FC16 for ; Wed, 17 Oct 2012 14:35:52 +0000 (UTC) Received: from odyssey.starpoint.kiev.ua (alpha-e.starpoint.kiev.ua [212.40.38.101]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id RAA06786; Wed, 17 Oct 2012 17:35:47 +0300 (EEST) (envelope-from avg@FreeBSD.org) Message-ID: <507EC242.1070809@FreeBSD.org> Date: Wed, 17 Oct 2012 17:35:46 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:16.0) Gecko/20121012 Thunderbird/16.0.1 MIME-Version: 1.0 To: Nikolay Denev Subject: Re: I have a DDB session open to a crashed ZFS server References: <1350317019.71982.50.camel@btw.pki2.com> <201210160844.41042.jhb@freebsd.org> <1350400597.72003.32.camel@btw.pki2.com> <201210161215.33369.jhb@freebsd.org> <507D8B69.3090903@FreeBSD.org> <0B0CA833-79FA-4C8E-86AC-828E7947FF67@gmail.com> In-Reply-To: <0B0CA833-79FA-4C8E-86AC-828E7947FF67@gmail.com> X-Enigmail-Version: 1.4.5 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Cc: freebsd-fs@FreeBSD.org, dg17@penx.com, Dennis Glatting X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Oct 2012 14:35:53 -0000 on 17/10/2012 10:04 Nikolay Denev said the following: > I'm running with the patch from here : http://thread.gmane.org/gmane.os.freebsd.devel.file-systems/16000/focus=16017 > And there were no deadlocks since it's applied. > If you're hitting the same issue as I was, this should help. I am going to commit that change soon-ish, if nobody expressly objects to it. BTW, Nikolay, is your system (that used to deadlock) amd64? Could you please capture 'sysctl vm | fgrep kmem' output when it is under load? -- Andriy Gapon From owner-freebsd-fs@FreeBSD.ORG Wed Oct 17 14:41:52 2012 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 8A085D8B; Wed, 17 Oct 2012 14:41:52 +0000 (UTC) (envelope-from prvs=16374f1e9b=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id E10DD8FC12; Wed, 17 Oct 2012 14:41:51 +0000 (UTC) Received: from r2d2 ([188.220.16.49]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50000728115.msg; Wed, 17 Oct 2012 15:41:48 +0100 X-Spam-Processed: mail1.multiplay.co.uk, Wed, 17 Oct 2012 15:41:48 +0100 (not processed: message from valid local sender) X-MDRemoteIP: 188.220.16.49 X-Return-Path: prvs=16374f1e9b=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk Message-ID: <9826CE7F20544176825D7C71CD4A36B4@multiplay.co.uk> From: "Steven Hartland" To: "Andriy Gapon" , "Nikolay Denev" References: <1350317019.71982.50.camel@btw.pki2.com> <201210160844.41042.jhb@freebsd.org> <1350400597.72003.32.camel@btw.pki2.com> <201210161215.33369.jhb@freebsd.org> <507D8B69.3090903@FreeBSD.org> <0B0CA833-79FA-4C8E-86AC-828E7947FF67@gmail.com> <507EC242.1070809@FreeBSD.org> Subject: Re: I have a DDB session open to a crashed ZFS server Date: Wed, 17 Oct 2012 15:41:39 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 Cc: freebsd-fs@FreeBSD.org, dg17@penx.com, Dennis Glatting X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Oct 2012 14:41:52 -0000 ----- Original Message ----- From: "Andriy Gapon" >> And there were no deadlocks since it's applied. >> If you're hitting the same issue as I was, this should help. > > I am going to commit that change soon-ish, if nobody expressly objects to it. > > BTW, Nikolay, is your system (that used to deadlock) amd64? > Could you please capture 'sysctl vm | fgrep kmem' output when it is under load? We have a machine here which is locking up under high NFS / ZFS load but unfortunately when it locks nothing is responsive. I've patched out 8.3-RELEASE install with this and intend to give it a test but as it only happens every week or so won't be able to give you an answer for a few weeks I'm afraid. Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-fs@FreeBSD.ORG Wed Oct 17 14:57:11 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 6783824F; Wed, 17 Oct 2012 14:57:11 +0000 (UTC) (envelope-from ndenev@gmail.com) Received: from mail-wg0-f50.google.com (mail-wg0-f50.google.com [74.125.82.50]) by mx1.freebsd.org (Postfix) with ESMTP id BDFC48FC12; Wed, 17 Oct 2012 14:57:10 +0000 (UTC) Received: by mail-wg0-f50.google.com with SMTP id 16so6062537wgi.31 for ; Wed, 17 Oct 2012 07:57:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer; bh=/lTHutyJ32j7MElTe/qFMtIv9h02akERnRUgYfYUDpw=; b=p1N8yS/TmWPivfGxO9jeR4O07O/trYd9WoR15NjHr06FE3ftBNO/ZlSUNAVQb6ok43 YapeqEehJRUy72ITbDp8eh50qhLQ9r81U8HVqQSPZIWvJIgJNfvpvgdVQUM5QW2c3s/H ymzJFjhvPuFTLor8HDmSUTK9SqSlyCFmJVc0Z7pKQZLXyywjoaHDyvUGj8U+oaZoPKoC 2J2reodqZw2++8u0fn0cxCKiFDUtW+oW7Hk1ogk+JNl6mzoQYNWSOWaR+GzqKVeylF2V qDJf16QD8F7AYWbaP3PDWJ80n48Tw/YIU292ow5D2DlRe2sVjlnjE4XGlfpsxvuSPRuL YzqQ== Received: by 10.180.19.71 with SMTP id c7mr4742886wie.2.1350485829164; Wed, 17 Oct 2012 07:57:09 -0700 (PDT) Received: from ndenevsa.sf.moneybookers.net (g1.moneybookers.com. [217.18.249.148]) by mx.google.com with ESMTPS id gg4sm28024206wib.6.2012.10.17.07.57.07 (version=TLSv1/SSLv3 cipher=OTHER); Wed, 17 Oct 2012 07:57:08 -0700 (PDT) Subject: Re: I have a DDB session open to a crashed ZFS server Mime-Version: 1.0 (Mac OS X Mail 6.1 \(1498\)) Content-Type: text/plain; charset=us-ascii From: Nikolay Denev In-Reply-To: <507EC242.1070809@FreeBSD.org> Date: Wed, 17 Oct 2012 17:57:05 +0300 Content-Transfer-Encoding: quoted-printable Message-Id: <20F56CFC-3B97-4921-A2BF-2CCFA20669FD@gmail.com> References: <1350317019.71982.50.camel@btw.pki2.com> <201210160844.41042.jhb@freebsd.org> <1350400597.72003.32.camel@btw.pki2.com> <201210161215.33369.jhb@freebsd.org> <507D8B69.3090903@FreeBSD.org> <0B0CA833-79FA-4C8E-86AC-828E7947FF67@gmail.com> <507EC242.1070809@FreeBSD.org> To: Andriy Gapon X-Mailer: Apple Mail (2.1498) Cc: freebsd-fs@FreeBSD.org, dg17@penx.com, Dennis Glatting X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Oct 2012 14:57:11 -0000 On Oct 17, 2012, at 5:35 PM, Andriy Gapon wrote: > on 17/10/2012 10:04 Nikolay Denev said the following: >> I'm running with the patch from here : = http://thread.gmane.org/gmane.os.freebsd.devel.file-systems/16000/focus=3D= 16017 >> And there were no deadlocks since it's applied. >> If you're hitting the same issue as I was, this should help. >=20 > I am going to commit that change soon-ish, if nobody expressly objects = to it. >=20 > BTW, Nikolay, is your system (that used to deadlock) amd64? > Could you please capture 'sysctl vm | fgrep kmem' output when it is = under load? >=20 > --=20 > Andriy Gapon Yep, It's a amd64, currently I'm doing some iozone throughput testing = over NFS and I can say it's loaded : last pid: 9818; load averages: 13.26, 13.98, 15.14 = up = 4+00:09:55 16:56:10 44 processes: 1 running, 43 sleeping CPU: 0.2% user, 0.1% nice, 63.5% system, 4.0% interrupt, 32.1% idle Mem: 193M Active, 1689M Inact, 163G Wired, 70M Cache, 22G Free =20 ARC: 147G Total, 6884M MRU, 128G MFU, 2824K Anon, 8674M Header, 3896M = Other vm.kmem_map_free: 46525120512 =20 vm.kmem_map_size: 149978828800 =20 vm.kmem_size_scale: 1 vm.kmem_size_max: 329853485875 vm.kmem_size_min: 0 vm.kmem_size: 199974834176 From owner-freebsd-fs@FreeBSD.ORG Wed Oct 17 16:06:17 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id A770974D; Wed, 17 Oct 2012 16:06:17 +0000 (UTC) (envelope-from amvandemore@gmail.com) Received: from mail-oa0-f54.google.com (mail-oa0-f54.google.com [209.85.219.54]) by mx1.freebsd.org (Postfix) with ESMTP id 47A418FC12; Wed, 17 Oct 2012 16:06:17 +0000 (UTC) Received: by mail-oa0-f54.google.com with SMTP id n9so10118346oag.13 for ; Wed, 17 Oct 2012 09:06:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=BeQ3lUaaSBKisxRttlANAsoNH5oK5FbqR8ovujayeVo=; b=V2hx7SjXw5EIVpkqG4HrbwQzKnBCf4ArpzngdRR8hVl3ut4xdgeVGWzT4FzBNkp4r6 94XZRAYFJ3M7aaR5t4FynglnxoS/LoUN2Y9ihcoXn4211Oj3DsTbk86bYrWTxmGf1wXo n4ilvRMuqxHGwBcqcTb0ONz/WzfLdEyqx0DxvW/V+zrgWH6vRxtl+HH0l0Y1uIEjyj7N dWf7FO4ApbdUpavZ+R8tuY4uMUyGdQ4Bz3Ibt+3kY+KyACNRWqSFXXIQTqSK2KQZgIHw A+Su98Ab/J2QgX8Er5iZt7hkUTCqVuDpnJVxuqmsW+qW31hGcTZW/QPrJfSRzwoZ154I q3qg== MIME-Version: 1.0 Received: by 10.60.32.136 with SMTP id j8mr16327073oei.0.1350489975721; Wed, 17 Oct 2012 09:06:15 -0700 (PDT) Received: by 10.76.80.104 with HTTP; Wed, 17 Oct 2012 09:06:15 -0700 (PDT) In-Reply-To: <507EC242.1070809@FreeBSD.org> References: <1350317019.71982.50.camel@btw.pki2.com> <201210160844.41042.jhb@freebsd.org> <1350400597.72003.32.camel@btw.pki2.com> <201210161215.33369.jhb@freebsd.org> <507D8B69.3090903@FreeBSD.org> <0B0CA833-79FA-4C8E-86AC-828E7947FF67@gmail.com> <507EC242.1070809@FreeBSD.org> Date: Wed, 17 Oct 2012 11:06:15 -0500 Message-ID: Subject: Re: I have a DDB session open to a crashed ZFS server From: Adam Vande More To: Andriy Gapon Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: freebsd-fs@freebsd.org, dg17@penx.com, Dennis Glatting X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Oct 2012 16:06:18 -0000 On Wed, Oct 17, 2012 at 9:35 AM, Andriy Gapon wrote: > on 17/10/2012 10:04 Nikolay Denev said the following: > > I'm running with the patch from here : > http://thread.gmane.org/gmane.os.freebsd.devel.file-systems/16000/focus=16017 > > And there were no deadlocks since it's applied. > > If you're hitting the same issue as I was, this should help. > > I am going to commit that change soon-ish, if nobody expressly objects to > it. I imagine you don't have much to do with the release process, but this seems to be a rather major bug warranting serious consideration about inserting into the 9_1 branch prior to final release. -- Adam Vande More From owner-freebsd-fs@FreeBSD.ORG Wed Oct 17 16:39:57 2012 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 50015356 for ; Wed, 17 Oct 2012 16:39:57 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 906E88FC18 for ; Wed, 17 Oct 2012 16:39:56 +0000 (UTC) Received: from odyssey.starpoint.kiev.ua (alpha-e.starpoint.kiev.ua [212.40.38.101]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id TAA08041; Wed, 17 Oct 2012 19:39:47 +0300 (EEST) (envelope-from avg@FreeBSD.org) Message-ID: <507EDF52.4050705@FreeBSD.org> Date: Wed, 17 Oct 2012 19:39:46 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:16.0) Gecko/20121012 Thunderbird/16.0.1 MIME-Version: 1.0 To: Adam Vande More Subject: Re: I have a DDB session open to a crashed ZFS server References: <1350317019.71982.50.camel@btw.pki2.com> <201210160844.41042.jhb@freebsd.org> <1350400597.72003.32.camel@btw.pki2.com> <201210161215.33369.jhb@freebsd.org> <507D8B69.3090903@FreeBSD.org> <0B0CA833-79FA-4C8E-86AC-828E7947FF67@gmail.com> <507EC242.1070809@FreeBSD.org> In-Reply-To: X-Enigmail-Version: 1.4.5 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Cc: freebsd-fs@FreeBSD.org, dg17@penx.com, Dennis Glatting X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Oct 2012 16:39:57 -0000 on 17/10/2012 19:06 Adam Vande More said the following: > On Wed, Oct 17, 2012 at 9:35 AM, Andriy Gapon > wrote: > > on 17/10/2012 10:04 Nikolay Denev said the following: > > I'm running with the patch from here : > http://thread.gmane.org/gmane.os.freebsd.devel.file-systems/16000/focus=16017 > > And there were no deadlocks since it's applied. > > If you're hitting the same issue as I was, this should help. > > I am going to commit that change soon-ish, if nobody expressly objects to it. > > > I imagine you don't have much to do with the release process, but this seems to be > a rather major bug warranting serious consideration about inserting into the 9_1 > branch prior to final release. No, I don't think so. The bug can only be triggered by running out of KVA space (vm.kmem_map_free getting too low), and that can be worked around by increasing vm.kmem_size via a loader tunable. I hear that a value twice the physical memory size is pretty safe. -- Andriy Gapon From owner-freebsd-fs@FreeBSD.ORG Wed Oct 17 17:39:52 2012 Return-Path: Delivered-To: FS@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 891006AB for ; Wed, 17 Oct 2012 17:39:52 +0000 (UTC) (envelope-from heikki@suonsivu.net) Received: from mail.bbnetworks.net (mail.bbnetworks.net [IPv6:2a00:1c30:1:100::10]) by mx1.freebsd.org (Postfix) with ESMTP id 7CC258FC1B for ; Wed, 17 Oct 2012 17:39:51 +0000 (UTC) Received: from toimisto.suonsivu.net (dyn73.olari.suonsivu.net [212.16.96.73]) by mail.bbnetworks.net (8.14.5/8.14.5) with ESMTP id q9HHdaTi004768; Wed, 17 Oct 2012 20:39:37 +0300 (EEST) (envelope-from heikki@suonsivu.net) Message-ID: <507EED58.80409@suonsivu.net> Date: Wed, 17 Oct 2012 20:39:36 +0300 From: Heikki Suonsivu User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:15.0) Gecko/20120831 Thunderbird/15.0 MIME-Version: 1.0 To: FS@freebsd.org Subject: ZFS raidz2, errors in file? Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Oct 2012 17:39:52 -0000 Summary: This could indicate that in some situations ZFS or FreeBSD ignore disk errors in some situations, or disks do not pass them to FreeBSD. There are three SCSI timeouts all on different disks reported, though it seems unlikely they hit all three redundancies of a single block of single file, so the more likely scenario would be that there are other errors causing this, which are not reported or are ignored. Also, timeouts should trigger retries. SMART data indicates problems on two other disks, but no indication of those are seen in logs (the disks work, but SMART information indicates problems). The fs is 8*3T pool on raidz2. Further data below: ------------------------------------------------------------------------ zpool status -v pool: raid6 state: ONLINE status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://www.sun.com/msg/ZFS-8000-8A scan: none requested config: NAME STATE READ WRITE CKSUM raid6 ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 da0 ONLINE 0 0 0 da1 ONLINE 0 0 0 da2 ONLINE 0 0 0 da3 ONLINE 0 0 0 da4 ONLINE 0 0 0 da5 ONLINE 0 0 0 da6 ONLINE 0 0 0 da7 ONLINE 0 0 0 errors: Permanent errors have been detected in the following files: /raid6/backup/rsnapshot/daily.1/foobar.net/users/mail/foo ------------------------------------------------------------------------ All disks online, so this is kind of odd. One disk indeed has pending sector, not unusual and should be survivable: ------------------------------------------------------------------------ ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0 3 Spin_Up_Time 0x0027 162 154 021 Pre-fail Always - 8891 4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 46 5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0 7 Seek_Error_Rate 0x002e 100 253 000 Old_age Always - 0 9 Power_On_Hours 0x0032 094 094 000 Old_age Always - 4776 10 Spin_Retry_Count 0x0032 100 253 000 Old_age Always - 0 11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 43 192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 40 193 Load_Cycle_Count 0x0032 196 196 000 Old_age Always - 13595 194 Temperature_Celsius 0x0022 119 106 000 Old_age Always - 33 196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0 197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 1 198 Offline_Uncorrectable 0x0030 200 200 000 Old_age Offline - 1 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0 200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 1 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 No self-tests have been logged. [To run self-tests, use: smartctl -t] ------------------------------------------------------------------------ But there is no corresponding read error in /var/log/messages, which would indicate that FreeBSD never saw the error, or ignored it. In addition, there seems to be ICRC DMA errors on da0. Looks nasty, but only show up in SMART log, not in /var/log/messages. ------------------------------------------------------------------------ smartctl 5.43 2012-06-30 r3573 [FreeBSD 9.0-RELEASE-p3 amd64] (local build) Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net === START OF INFORMATION SECTION === Model Family: Hitachi Deskstar 7K3000 Device Model: Hitachi HDS723030ALA640 Serial Number: MK0331YHGVDK6A LU WWN Device Id: 5 000cca 225cc00c7 Firmware Version: MKAOA5C0 User Capacity: 3,000,592,982,016 bytes [3.00 TB] Sector Size: 512 bytes logical/physical Device is: In smartctl database [for details use: -P show] ATA Version is: 8 ATA Standard is: ATA-8-ACS revision 4 Local Time is: Wed Oct 17 20:03:05 2012 EEST SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x84) Offline data collection activity was suspended by an interrupting command from host. Auto Offline Data Collection: Enabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: (29509) seconds. Offline data collection capabilities: (0x5b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. No Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 1) minutes. Extended self-test routine recommended polling time: ( 492) minutes. SCT capabilities: (0x003d) SCT Status supported. SCT Error Recovery Control supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000b 100 100 016 Pre-fail Always - 0 2 Throughput_Performance 0x0005 136 136 054 Pre-fail Offline - 83 3 Spin_Up_Time 0x0007 125 125 024 Pre-fail Always - 617 (Average 617) 4 Start_Stop_Count 0x0012 100 100 000 Old_age Always - 62 5 Reallocated_Sector_Ct 0x0033 100 100 005 Pre-fail Always - 0 7 Seek_Error_Rate 0x000b 100 100 067 Pre-fail Always - 0 8 Seek_Time_Performance 0x0005 135 135 020 Pre-fail Offline - 26 9 Power_On_Hours 0x0012 100 100 000 Old_age Always - 4570 10 Spin_Retry_Count 0x0013 100 100 060 Pre-fail Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 30 192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 129 193 Load_Cycle_Count 0x0012 100 100 000 Old_age Always - 129 194 Temperature_Celsius 0x0002 166 166 000 Old_age Always - 36 (Min/Max 25/50) 196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 0 197 Current_Pending_Sector 0x0022 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0008 100 100 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x000a 200 200 000 Old_age Always - 112 SMART Error Log Version: 1 ATA Error Count: 112 (device log contains only the most recent five errors) CR = Command Register [HEX] FR = Features Register [HEX] SC = Sector Count Register [HEX] SN = Sector Number Register [HEX] CL = Cylinder Low Register [HEX] CH = Cylinder High Register [HEX] DH = Device/Head Register [HEX] DC = Device Command Register [HEX] ER = Error register [HEX] ST = Status register [HEX] Powered_Up_Time is measured from power on, and printed as DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes, SS=sec, and sss=millisec. It "wraps" after 49.710 days. Error 112 occurred at disk power-on lifetime: 52 hours (2 days + 4 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 84 51 20 28 c5 ae 06 Error: ICRC, ABRT 32 sectors at LBA = 0x06aec528 = 112117032 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- 35 03 f0 58 c4 ae 40 00 2d+04:00:39.416 WRITE DMA EXT 35 03 20 38 c4 ae 40 00 2d+04:00:39.409 WRITE DMA EXT 35 03 f0 48 c3 ae 40 00 2d+04:00:39.402 WRITE DMA EXT 35 03 f0 58 c2 ae 40 00 2d+04:00:39.394 WRITE DMA EXT 35 03 60 f8 c1 ae 40 00 2d+04:00:39.388 WRITE DMA EXT Error 111 occurred at disk power-on lifetime: 52 hours (2 days + 4 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 84 51 30 08 c4 90 06 Error: ICRC, ABRT 48 sectors at LBA = 0x0690c408 = 110150664 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- 35 03 f0 48 c3 90 40 00 2d+03:59:29.925 WRITE DMA EXT 35 03 f0 58 c2 90 40 00 2d+03:59:29.917 WRITE DMA EXT 35 03 60 f8 c1 90 40 00 2d+03:59:29.910 WRITE DMA EXT 35 03 b0 48 c1 90 40 00 2d+03:59:29.907 WRITE DMA EXT 35 03 f0 58 c0 90 40 00 2d+03:59:29.899 WRITE DMA EXT Error 110 occurred at disk power-on lifetime: 52 hours (2 days + 4 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 84 51 20 28 47 90 05 Error: ICRC, ABRT 32 sectors at LBA = 0x05904728 = 93341480 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- 35 da f0 58 46 90 40 00 2d+03:49:28.919 WRITE DMA EXT 35 da 20 38 46 90 40 00 2d+03:49:28.917 WRITE DMA EXT 35 da f0 48 45 90 40 00 2d+03:49:28.909 WRITE DMA EXT 35 da f0 58 44 90 40 00 2d+03:49:28.901 WRITE DMA EXT 35 da 20 38 44 90 40 00 2d+03:49:28.899 WRITE DMA EXT Error 109 occurred at disk power-on lifetime: 52 hours (2 days + 4 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 84 51 10 38 e7 41 05 Error: ICRC, ABRT 16 sectors at LBA = 0x0541e738 = 88205112 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- 35 03 f0 58 e6 41 40 00 2d+03:46:20.939 WRITE DMA EXT 35 03 20 38 e6 41 40 00 2d+03:46:20.933 WRITE DMA EXT 35 03 f0 48 e5 41 40 00 2d+03:46:20.926 WRITE DMA EXT 35 03 f0 58 e4 41 40 00 2d+03:46:20.921 WRITE DMA EXT 35 03 20 38 e4 41 40 00 2d+03:46:20.916 WRITE DMA EXT Error 108 occurred at disk power-on lifetime: 52 hours (2 days + 4 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 84 51 10 28 0a 2f 05 Error: ICRC, ABRT 16 sectors at LBA = 0x052f0a28 = 86968872 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- 35 03 f0 48 09 2f 40 00 2d+03:45:06.250 WRITE DMA EXT 35 03 f0 58 08 2f 40 00 2d+03:45:06.242 WRITE DMA EXT 35 03 20 38 08 2f 40 00 2d+03:45:06.238 WRITE DMA EXT 35 03 f0 48 07 2f 40 00 2d+03:45:06.231 WRITE DMA EXT 35 03 08 40 bc 25 40 00 2d+03:45:06.225 WRITE DMA EXT SMART Self-test log structure revision number 1 No self-tests have been logged. [To run self-tests, use: smartctl -t] SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay. ------------------------------------------------------------------------ These only show up in disk's SMART log, not as errors in other logs (logs exists for all the history of the system). These do not seem to have passed to the FreeBSD, or FreeBSD has ignored the errors. From logs, I can find da3, da4, da6 all have reported scsi timeout once (different times, within one month in July. They all look like this, one case for each of the disks: ------------------------------------------------------------------------ Jul 24 12:16:55 tila kernel: (da6:mps0:0:6:0): SCSI command timeout on device handle 0x000d SMID 699 Jul 24 12:16:55 tila kernel: (da6:mps0:0:6:0): SCSI command timeout on device handle 0x000d SMID 419 Jul 24 12:16:55 tila kernel: (da6:mps0:0:6:0): SCSI command timeout on device handle 0x000d SMID 359 Jul 24 12:16:55 tila kernel: (da6:mps0:0:6:0): SCSI command timeout on device handle 0x000d SMID 472 Jul 24 12:16:55 tila kernel: (da6:mps0:0:6:0): SCSI command timeout on device handle 0x000d SMID 124 Jul 24 12:16:55 tila kernel: (da6:mps0:0:6:0): SCSI command timeout on device handle 0x000d SMID 762 Jul 24 12:16:55 tila kernel: (da6:mps0:0:6:0): SCSI command timeout on device handle 0x000d SMID 243 Jul 24 12:16:55 tila kernel: (da6:mps0:0:6:0): SCSI command timeout on device handle 0x000d SMID 890 Jul 24 12:16:55 tila kernel: (da6:mps0:0:6:0): SCSI command timeout on device handle 0x000d SMID 204 Jul 24 12:16:55 tila kernel: (da6:mps0:0:6:0): SCSI command timeout on device handle 0x000d SMID 2 Jul 24 12:16:55 tila kernel: (da6:mps0:0:6:0): SCSI command timeout on device handle 0x000d SMID 291 Jul 24 12:16:55 tila kernel: mps0: (0:6:0) terminated ioc 804b scsi 0 state c xfer 0 Jul 24 12:16:55 tila kernel: mps0: mpssas_abort_complete: abort request on handle 0x0d SMID 699 complete Jul 24 12:16:55 tila kernel: mps0: mpssas_complete_tm_request: sending deferred task management request for handle 0x0d SMID 419 Jul 24 12:16:55 tila kernel: mps0: mpssas_abort_complete: abort request on handle 0x0d SMID 419 complete Jul 24 12:16:55 tila kernel: mps0: mpssas_complete_tm_request: sending deferred task management request for handle 0x0d SMID 359 Jul 24 12:16:55 tila kernel: mps0: mpssas_abort_complete: abort request on handle 0x0d SMID 359 complete Jul 24 12:16:55 tila kernel: mps0: mpssas_complete_tm_request: sending deferred task management request for handle 0x0d SMID 472 Jul 24 12:16:55 tila kernel: mps0: mpssas_abort_complete: abort request on handle 0x0d SMID 472 complete Jul 24 12:16:55 tila kernel: mps0: mpssas_complete_tm_request: sending deferred task management request for handle 0x0d SMID 124 Jul 24 12:16:55 tila kernel: mps0: mpssas_abort_complete: abort request on handle 0x0d SMID 124 complete Jul 24 12:16:55 tila kernel: mps0: mpssas_complete_tm_request: sending deferred task management request for handle 0x0d SMID 762 Jul 24 12:16:55 tila kernel: mps0: mpssas_abort_complete: abort request on handle 0x0d SMID 762 complete Jul 24 12:16:55 tila kernel: mps0: mpssas_complete_tm_request: sending deferred task management request for handle 0x0d SMID 243 Jul 24 12:16:55 tila kernel: mps0: mpssas_abort_complete: abort request on handle 0x0d SMID 243 complete Jul 24 12:16:55 tila kernel: mps0: mpssas_complete_tm_request: sending deferred task management request for handle 0x0d SMID 890 Jul 24 12:16:55 tila kernel: mps0: mpssas_abort_complete: abort request on handle 0x0d SMID 890 complete Jul 24 12:16:55 tila kernel: mps0: mpssas_complete_tm_request: sending deferred task management request for handle 0x0d SMID 204 Jul 24 12:16:55 tila kernel: mps0: mpssas_abort_complete: abort request on handle 0x0d SMID 204 complete Jul 24 12:16:55 tila kernel: mps0: mpssas_complete_tm_request: sending deferred task management request for handle 0x0d SMID 2 Jul 24 12:16:55 tila kernel: mps0: mpssas_abort_complete: abort request on handle 0x0d SMID 2 complete Jul 24 12:16:55 tila kernel: mps0: mpssas_complete_tm_request: sending deferred task management request for handle 0x0d SMID 291 Jul 24 12:16:55 tila kernel: mps0: mpssas_abort_complete: abort request on handle 0x0d SMID 291 complete Jul 24 12:16:55 tila kernel: (da6:mps0:0:6:0): WRITE(16). CDB: 8a 0 0 0 0 1 4a 17 dc 5e 0 0 0 57 0 0 Jul 24 12:16:55 tila kernel: (da6:mps0:0:6:0): CAM status: SCSI Status Error Jul 24 12:16:55 tila kernel: (da6:mps0:0:6:0): SCSI status: Check Condition Jul 24 12:16:55 tila kernel: (da6:mps0:0:6:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred) Jul 24 12:16:56 tila kernel: (da6:mps0:0:6:0): WRITE(10). CDB: 2a 0 28 0 85 fb 0 0 1 0 Jul 24 12:16:56 tila kernel: (da6:mps0:0:6:0): CAM status: SCSI Status Error Jul 24 12:16:56 tila kernel: (da6:mps0:0:6:0): SCSI status: Check Condition Jul 24 12:16:56 tila kernel: (da6:mps0:0:6:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred) ------------------------------------------------------------------------ If the da0 ICRC errors would have been seen by ZFS, it should have made a) note of that in some log? b) retried write? c) Something else? If we assume that the disk firmware is broken and does not report these to OS, so da0 might be corrupt. But that should still be ok with raidz2. We do have 3 random SCSI timeouts, which were seen by FreeBSD, and thus should have prompted ZFS do handle the errors, and one read error on data, which is not reported as read error in any log, other than disk's SMART info says so. Is it possible to get more information, which block of the file is corrupt, on which disks the offending blocks reside, and similar information? From owner-freebsd-fs@FreeBSD.ORG Wed Oct 17 19:17:42 2012 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 95B64527 for ; Wed, 17 Oct 2012 19:17:42 +0000 (UTC) (envelope-from Martin.Birgmeier@aon.at) Received: from email.aon.at (nat-warsl417-01.aon.at [195.3.96.119]) by mx1.freebsd.org (Postfix) with ESMTP id CF0828FC12 for ; Wed, 17 Oct 2012 19:17:40 +0000 (UTC) Received: (qmail 4451 invoked from network); 17 Oct 2012 19:17:33 -0000 Received: from smarthub82.res.a1.net (HELO email.aon.at) ([172.18.1.202]) (envelope-sender ) by fallback43.highway.telekom.at (qmail-ldap-1.03) with SMTP for ; 17 Oct 2012 19:17:33 -0000 Received: (qmail 10318 invoked from network); 17 Oct 2012 19:17:25 -0000 X-Spam-Checker-Version: SpamAssassin 3.2.0 (2007-05-01) on WARSBL507.highway.telekom.at X-Spam-Level: Received: from 194-166-159-44.adsl.highway.telekom.at (HELO gandalf.xyzzy) ([194.166.159.44]) (envelope-sender ) by smarthub82.res.a1.net (qmail-ldap-1.03) with AES256-SHA encrypted SMTP for ; 17 Oct 2012 19:17:25 -0000 Received: from mizar-v1.xyzzy (mizar-v1.xyzzy [192.168.1.51]) by gandalf.xyzzy (8.14.5/8.14.5) with ESMTP id q9HJHO1X015851; Wed, 17 Oct 2012 21:17:24 +0200 (CEST) (envelope-from Martin.Birgmeier@aon.at) Message-ID: <507F0446.2060805@aon.at> Date: Wed, 17 Oct 2012 21:17:26 +0200 From: Martin Birgmeier User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:16.0) Gecko/20121012 Thunderbird/16.0.1 MIME-Version: 1.0 To: bug-followup@FreeBSD.org, Martin.Birgmeier@aon.at, freebsd-fs@FreeBSD.org Subject: Re: kern/171415: [zfs] zfs recv fails with " cannot receive incremental stream: invalid backup stream" X-Enigmail-Version: 1.4.5 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Oct 2012 19:17:42 -0000 Good news... I upgraded the FreeBSD head running in the virtual machine v903 to r241507, with the goal to see whether the problem described has been fixed in the meantime. (The server, hal, is still at 8.2 release.) In addition, instead of using iSCSI to make disk{31..36}p4 from hal appear in v903, I attached them as devices directly using VirtualBox (all to one and the same virtual SATA controller). Now the test worked, and I could successfully receive the backup stream. I even made an md5 comparison of all the files in the received file system (> 53000), which went through without a single difference. Very good! What was left was testing with the original test setup again, i.e., using iSCSI. Unfortunately, this still fails in the same way, so it seems that something with iscontrol or istgt is broken. On a hunch, I'd say it is iscontrol, simply because I assume that on the server side (ports/net/istgt) there is not that much that can break, it being a userland application, whereas iscontrol is a kernel module hooking into the SCSI subsystem. Therefore, this defect should possibly be reclassified and instead of [zfs] have some other sticker applied. Regards, Martin p.s. I am using istgt also with disk images of MS Windows XP installations, exporting them to remotely running VirtualBox instances without any issues - to me a hint that istgt is quite o.k. Of course, the zfs scenario uses 6 iSCSI targets at the same time, whereas the XP emulation uses only one, so maybe either istgt or iscontrol have problems with multiple connections/instances. From owner-freebsd-fs@FreeBSD.ORG Wed Oct 17 19:20:01 2012 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id B238D690 for ; Wed, 17 Oct 2012 19:20:01 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [8.8.178.135]) by mx1.freebsd.org (Postfix) with ESMTP id 990DD8FC17 for ; Wed, 17 Oct 2012 19:20:01 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q9HJK13p014486 for ; Wed, 17 Oct 2012 19:20:01 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q9HJK1b4014485; Wed, 17 Oct 2012 19:20:01 GMT (envelope-from gnats) Date: Wed, 17 Oct 2012 19:20:01 GMT Message-Id: <201210171920.q9HJK1b4014485@freefall.freebsd.org> To: freebsd-fs@FreeBSD.org Cc: From: Martin Birgmeier Subject: Re: kern/171415: [zfs] zfs recv fails with " cannot receive incremental stream: invalid backup stream" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: Martin Birgmeier List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Oct 2012 19:20:01 -0000 The following reply was made to PR kern/171415; it has been noted by GNATS. From: Martin Birgmeier To: bug-followup@FreeBSD.org, Martin.Birgmeier@aon.at, freebsd-fs@FreeBSD.org Cc: Subject: Re: kern/171415: [zfs] zfs recv fails with "cannot receive incremental stream: invalid backup stream" Date: Wed, 17 Oct 2012 21:17:26 +0200 Good news... I upgraded the FreeBSD head running in the virtual machine v903 to r241507, with the goal to see whether the problem described has been fixed in the meantime. (The server, hal, is still at 8.2 release.) In addition, instead of using iSCSI to make disk{31..36}p4 from hal appear in v903, I attached them as devices directly using VirtualBox (all to one and the same virtual SATA controller). Now the test worked, and I could successfully receive the backup stream. I even made an md5 comparison of all the files in the received file system (> 53000), which went through without a single difference. Very good! What was left was testing with the original test setup again, i.e., using iSCSI. Unfortunately, this still fails in the same way, so it seems that something with iscontrol or istgt is broken. On a hunch, I'd say it is iscontrol, simply because I assume that on the server side (ports/net/istgt) there is not that much that can break, it being a userland application, whereas iscontrol is a kernel module hooking into the SCSI subsystem. Therefore, this defect should possibly be reclassified and instead of [zfs] have some other sticker applied. Regards, Martin p.s. I am using istgt also with disk images of MS Windows XP installations, exporting them to remotely running VirtualBox instances without any issues - to me a hint that istgt is quite o.k. Of course, the zfs scenario uses 6 iSCSI targets at the same time, whereas the XP emulation uses only one, so maybe either istgt or iscontrol have problems with multiple connections/instances. From owner-freebsd-fs@FreeBSD.ORG Wed Oct 17 21:51:01 2012 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id E7ED92E5 for ; Wed, 17 Oct 2012 21:51:01 +0000 (UTC) (envelope-from Martin.Birgmeier@aon.at) Received: from email.aon.at (nat-warsl417-01.aon.at [195.3.96.119]) by mx1.freebsd.org (Postfix) with ESMTP id 3055F8FC12 for ; Wed, 17 Oct 2012 21:51:00 +0000 (UTC) Received: (qmail 30197 invoked from network); 17 Oct 2012 21:50:59 -0000 Received: from smarthub81.res.a1.net (HELO email.aon.at) ([172.18.1.201]) (envelope-sender ) by fallback44.highway.telekom.at (qmail-ldap-1.03) with SMTP for ; 17 Oct 2012 21:50:59 -0000 Received: (qmail 30680 invoked from network); 17 Oct 2012 21:50:53 -0000 X-Spam-Checker-Version: SpamAssassin 3.2.0 (2007-05-01) on WARSBL606.highway.telekom.at X-Spam-Level: Received: from 194-166-159-44.adsl.highway.telekom.at (HELO gandalf.xyzzy) ([194.166.159.44]) (envelope-sender ) by smarthub81.res.a1.net (qmail-ldap-1.03) with AES256-SHA encrypted SMTP for ; 17 Oct 2012 21:50:52 -0000 Received: from mizar-v1.xyzzy (mizar-v1.xyzzy [192.168.1.51]) by gandalf.xyzzy (8.14.5/8.14.5) with ESMTP id q9HLoqWj024427; Wed, 17 Oct 2012 23:50:52 +0200 (CEST) (envelope-from Martin.Birgmeier@aon.at) Message-ID: <507F283D.5060003@aon.at> Date: Wed, 17 Oct 2012 23:50:53 +0200 From: Martin Birgmeier User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:16.0) Gecko/20121012 Thunderbird/16.0.1 MIME-Version: 1.0 To: bug-followup@FreeBSD.org, freebsd-fs@FreeBSD.org Subject: Re: kern/171415: [zfs] zfs recv fails with " cannot receive incremental stream: invalid backup stream" X-Enigmail-Version: 1.4.5 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Oct 2012 21:51:02 -0000 One more test with the goal of determining whether iscontrol or istgt is the culprit... The test setup is altered as follows: In addition to the 6 partitions which are being used for the zpool on v903, also v903's OS partition (GPT, holding /, /usr, and swap) is being exported from hal via istgt (i.e., as iSCSI targets). Then v903 is started as a VirtualBox client on a Windows 7 host, where the (now 7) partitions are included via VirtualBox's iSCSI facility (http://www.virtualbox.org/manual/ch05.html#idp17868816) as ada0 and ada1..ada6. This setup results in a flawless zfs receive of the sent stream, again checked via md5 of all files. In addition the number (but not md5) of files in all snapshot directories (.zfs/snapshot/*) were checked. This is a strong indication that indeed iscontrol is problematic. Regards, Martin p.s. Some observations using this setup: It is a lot faster than running v903 in a VirtualBox client on hal. I had to do a disk check of v903 because it had crashed, and checking the UFS file systems / and /usr was in fact blazingly fast, much faster than when its containing partition was directly connected in hal as VirtualBox host. Also, the subsequent zfs import test ran with a sustained rate of approximately 23 MB/s, whereas with v903 on hal as host it ran with only 15 MB/s (see my previous test), a speed increase of 50%. One should note that in the setup of this test, there is a 1 Gbps Ethernet between hal as the server and the Windows host, whereas otherwise v903 runs directly on hal, which at least in theory should result in much quicker I/O. So it seems that not only iscontrol needs fixing, but VirtualBox on FreeBSD hosts could use some optimization. But then, FreeBSD is the work of volunteers, and at this point of my tests is certainly the time to thank them for all the great possibilities this system offers! From owner-freebsd-fs@FreeBSD.ORG Wed Oct 17 22:00:02 2012 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 24AF46EF for ; Wed, 17 Oct 2012 22:00:02 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [8.8.178.135]) by mx1.freebsd.org (Postfix) with ESMTP id E43108FC08 for ; Wed, 17 Oct 2012 22:00:01 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q9HM01Dr026554 for ; Wed, 17 Oct 2012 22:00:01 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q9HM01Zn026553; Wed, 17 Oct 2012 22:00:01 GMT (envelope-from gnats) Date: Wed, 17 Oct 2012 22:00:01 GMT Message-Id: <201210172200.q9HM01Zn026553@freefall.freebsd.org> To: freebsd-fs@FreeBSD.org Cc: From: Martin Birgmeier Subject: Re: kern/171415: [zfs] zfs recv fails with " cannot receive incremental stream: invalid backup stream" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: Martin Birgmeier List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Oct 2012 22:00:02 -0000 The following reply was made to PR kern/171415; it has been noted by GNATS. From: Martin Birgmeier To: bug-followup@FreeBSD.org, freebsd-fs@FreeBSD.org Cc: Subject: Re: kern/171415: [zfs] zfs recv fails with "cannot receive incremental stream: invalid backup stream" Date: Wed, 17 Oct 2012 23:50:53 +0200 One more test with the goal of determining whether iscontrol or istgt is the culprit... The test setup is altered as follows: In addition to the 6 partitions which are being used for the zpool on v903, also v903's OS partition (GPT, holding /, /usr, and swap) is being exported from hal via istgt (i.e., as iSCSI targets). Then v903 is started as a VirtualBox client on a Windows 7 host, where the (now 7) partitions are included via VirtualBox's iSCSI facility (http://www.virtualbox.org/manual/ch05.html#idp17868816) as ada0 and ada1..ada6. This setup results in a flawless zfs receive of the sent stream, again checked via md5 of all files. In addition the number (but not md5) of files in all snapshot directories (.zfs/snapshot/*) were checked. This is a strong indication that indeed iscontrol is problematic. Regards, Martin p.s. Some observations using this setup: It is a lot faster than running v903 in a VirtualBox client on hal. I had to do a disk check of v903 because it had crashed, and checking the UFS file systems / and /usr was in fact blazingly fast, much faster than when its containing partition was directly connected in hal as VirtualBox host. Also, the subsequent zfs import test ran with a sustained rate of approximately 23 MB/s, whereas with v903 on hal as host it ran with only 15 MB/s (see my previous test), a speed increase of 50%. One should note that in the setup of this test, there is a 1 Gbps Ethernet between hal as the server and the Windows host, whereas otherwise v903 runs directly on hal, which at least in theory should result in much quicker I/O. So it seems that not only iscontrol needs fixing, but VirtualBox on FreeBSD hosts could use some optimization. But then, FreeBSD is the work of volunteers, and at this point of my tests is certainly the time to thank them for all the great possibilities this system offers! From owner-freebsd-fs@FreeBSD.ORG Thu Oct 18 01:21:16 2012 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 858F5971; Thu, 18 Oct 2012 01:21:16 +0000 (UTC) (envelope-from freebsd@pki2.com) Received: from btw.pki2.com (btw.pki2.com [IPv6:2001:470:a:6fd::2]) by mx1.freebsd.org (Postfix) with ESMTP id 4CC988FC14; Thu, 18 Oct 2012 01:21:16 +0000 (UTC) Received: from [127.0.0.1] (localhost [127.0.0.1]) by btw.pki2.com (8.14.5/8.14.5) with ESMTP id q9I1LAdJ017772; Wed, 17 Oct 2012 18:21:10 -0700 (PDT) (envelope-from freebsd@pki2.com) Subject: Re: I have a DDB session open to a crashed ZFS server From: Dennis Glatting To: Andriy Gapon In-Reply-To: <507EC242.1070809@FreeBSD.org> References: <1350317019.71982.50.camel@btw.pki2.com> <201210160844.41042.jhb@freebsd.org> <1350400597.72003.32.camel@btw.pki2.com> <201210161215.33369.jhb@freebsd.org> <507D8B69.3090903@FreeBSD.org> <0B0CA833-79FA-4C8E-86AC-828E7947FF67@gmail.com> <507EC242.1070809@FreeBSD.org> Content-Type: text/plain; charset="ISO-8859-1" Date: Wed, 17 Oct 2012 18:21:10 -0700 Message-ID: <1350523270.3968.1.camel@btw.pki2.com> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port Content-Transfer-Encoding: 7bit X-yoursite-MailScanner-Information: Dennis Glatting X-yoursite-MailScanner-ID: q9I1LAdJ017772 X-yoursite-MailScanner: Found to be clean X-MailScanner-From: freebsd@pki2.com Cc: freebsd-fs@FreeBSD.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Oct 2012 01:21:16 -0000 That system for which I sent the procstat output was hung but I am unable to confirm if the hang was identical as the past because I lost the window. However, it was behaving identically as prior hangs. From owner-freebsd-fs@FreeBSD.ORG Thu Oct 18 01:28:10 2012 Return-Path: Delivered-To: FS@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 7F2ABB50 for ; Thu, 18 Oct 2012 01:28:10 +0000 (UTC) (envelope-from freebsd@pki2.com) Received: from btw.pki2.com (btw.pki2.com [IPv6:2001:470:a:6fd::2]) by mx1.freebsd.org (Postfix) with ESMTP id 15FC38FC14 for ; Thu, 18 Oct 2012 01:28:10 +0000 (UTC) Received: from [127.0.0.1] (localhost [127.0.0.1]) by btw.pki2.com (8.14.5/8.14.5) with ESMTP id q9I1S2B2019779; Wed, 17 Oct 2012 18:28:02 -0700 (PDT) (envelope-from freebsd@pki2.com) Subject: Re: ZFS raidz2, errors in file? From: Dennis Glatting To: Heikki Suonsivu In-Reply-To: <507EED58.80409@suonsivu.net> References: <507EED58.80409@suonsivu.net> Content-Type: text/plain; charset="ISO-8859-1" Date: Wed, 17 Oct 2012 18:28:02 -0700 Message-ID: <1350523682.3968.6.camel@btw.pki2.com> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port Content-Transfer-Encoding: 7bit X-yoursite-MailScanner-Information: Dennis Glatting X-yoursite-MailScanner-ID: q9I1S2B2019779 X-yoursite-MailScanner: Found to be clean X-yoursite-MailScanner-SpamScore: 2 X-MailScanner-From: freebsd@pki2.com Cc: FS@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Oct 2012 01:28:10 -0000 On Wed, 2012-10-17 at 20:39 +0300, Heikki Suonsivu wrote: > Summary: This could indicate that in some situations ZFS or FreeBSD > ignore disk errors in some situations, or disks do not pass them to > FreeBSD. > > There are three SCSI timeouts all on different disks reported, though it > seems unlikely they hit all three redundancies of a single block of > single file, so the more likely scenario would be that there are other > errors causing this, which are not reported or are ignored. Also, > timeouts should trigger retries. > > SMART data indicates problems on two other disks, but no indication of > those are seen in logs (the disks work, but SMART information indicates > problems). The fs is 8*3T pool on raidz2. > I can't say about your disks but I had nine 4TB Hitachi disks suffering a 30% failure rate in their first year and consistently, and randomly, being marked off-line by ZFS in a RAIDz2 pool. I eventually stripped those disks from that role, replacing them with 3TB Seagates (the Hitachies appear to work find in a ZFS mirror). I haven't experienced the same failure since I replaced the Hitachies with Seagate. > Further data below: > > ------------------------------------------------------------------------ > zpool status -v > pool: raid6 > state: ONLINE > status: One or more devices has experienced an error resulting in data > corruption. Applications may be affected. > action: Restore the file in question if possible. Otherwise restore the > entire pool from backup. > see: http://www.sun.com/msg/ZFS-8000-8A > scan: none requested > config: > > NAME STATE READ WRITE CKSUM > raid6 ONLINE 0 0 0 > raidz2-0 ONLINE 0 0 0 > da0 ONLINE 0 0 0 > da1 ONLINE 0 0 0 > da2 ONLINE 0 0 0 > da3 ONLINE 0 0 0 > da4 ONLINE 0 0 0 > da5 ONLINE 0 0 0 > da6 ONLINE 0 0 0 > da7 ONLINE 0 0 0 > > errors: Permanent errors have been detected in the following files: > > /raid6/backup/rsnapshot/daily.1/foobar.net/users/mail/foo > ------------------------------------------------------------------------ > > All disks online, so this is kind of odd. > > One disk indeed has pending sector, not unusual and should be survivable: > > ------------------------------------------------------------------------ > ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE > UPDATED WHEN_FAILED RAW_VALUE > 1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail > Always - 0 > 3 Spin_Up_Time 0x0027 162 154 021 Pre-fail > Always - 8891 > 4 Start_Stop_Count 0x0032 100 100 000 Old_age > Always - 46 > 5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail > Always - 0 > 7 Seek_Error_Rate 0x002e 100 253 000 Old_age > Always - 0 > 9 Power_On_Hours 0x0032 094 094 000 Old_age > Always - 4776 > 10 Spin_Retry_Count 0x0032 100 253 000 Old_age > Always - 0 > 11 Calibration_Retry_Count 0x0032 100 253 000 Old_age > Always - 0 > 12 Power_Cycle_Count 0x0032 100 100 000 Old_age > Always - 43 > 192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always > - 40 > 193 Load_Cycle_Count 0x0032 196 196 000 Old_age Always > - 13595 > 194 Temperature_Celsius 0x0022 119 106 000 Old_age Always > - 33 > 196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always > - 0 > 197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always > - 1 > 198 Offline_Uncorrectable 0x0030 200 200 000 Old_age > Offline - 1 > 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always > - 0 > 200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age > Offline - 1 > > SMART Error Log Version: 1 > No Errors Logged > > SMART Self-test log structure revision number 1 > No self-tests have been logged. [To run self-tests, use: smartctl -t] > ------------------------------------------------------------------------ > > But there is no corresponding read error in /var/log/messages, which > would indicate that FreeBSD never saw the error, or ignored it. > > In addition, there seems to be ICRC DMA errors on da0. Looks nasty, but > only show up in SMART log, not in /var/log/messages. > > ------------------------------------------------------------------------ > smartctl 5.43 2012-06-30 r3573 [FreeBSD 9.0-RELEASE-p3 amd64] (local build) > Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net > > === START OF INFORMATION SECTION === > Model Family: Hitachi Deskstar 7K3000 > Device Model: Hitachi HDS723030ALA640 > Serial Number: MK0331YHGVDK6A > LU WWN Device Id: 5 000cca 225cc00c7 > Firmware Version: MKAOA5C0 > User Capacity: 3,000,592,982,016 bytes [3.00 TB] > Sector Size: 512 bytes logical/physical > Device is: In smartctl database [for details use: -P show] > ATA Version is: 8 > ATA Standard is: ATA-8-ACS revision 4 > Local Time is: Wed Oct 17 20:03:05 2012 EEST > SMART support is: Available - device has SMART capability. > SMART support is: Enabled > > === START OF READ SMART DATA SECTION === > SMART overall-health self-assessment test result: PASSED > > General SMART Values: > Offline data collection status: (0x84) Offline data collection activity > was suspended by an > interrupting command from host. > Auto Offline Data Collection: > Enabled. > Self-test execution status: ( 0) The previous self-test routine > completed > without error or no self-test > has ever > been run. > Total time to complete Offline > data collection: (29509) seconds. > Offline data collection > capabilities: (0x5b) SMART execute Offline immediate. > Auto Offline data collection > on/off support. > Suspend Offline collection upon new > command. > Offline surface scan supported. > Self-test supported. > No Conveyance Self-test supported. > Selective Self-test supported. > SMART capabilities: (0x0003) Saves SMART data before entering > power-saving mode. > Supports SMART auto save timer. > Error logging capability: (0x01) Error logging supported. > General Purpose Logging supported. > Short self-test routine > recommended polling time: ( 1) minutes. > Extended self-test routine > recommended polling time: ( 492) minutes. > SCT capabilities: (0x003d) SCT Status supported. > SCT Error Recovery Control > supported. > SCT Feature Control supported. > SCT Data Table supported. > > SMART Attributes Data Structure revision number: 16 > Vendor Specific SMART Attributes with Thresholds: > ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE > UPDATED WHEN_FAILED RAW_VALUE > 1 Raw_Read_Error_Rate 0x000b 100 100 016 Pre-fail > Always - 0 > 2 Throughput_Performance 0x0005 136 136 054 Pre-fail > Offline - 83 > 3 Spin_Up_Time 0x0007 125 125 024 Pre-fail > Always - 617 (Average 617) > 4 Start_Stop_Count 0x0012 100 100 000 Old_age > Always - 62 > 5 Reallocated_Sector_Ct 0x0033 100 100 005 Pre-fail > Always - 0 > 7 Seek_Error_Rate 0x000b 100 100 067 Pre-fail > Always - 0 > 8 Seek_Time_Performance 0x0005 135 135 020 Pre-fail > Offline - 26 > 9 Power_On_Hours 0x0012 100 100 000 Old_age > Always - 4570 > 10 Spin_Retry_Count 0x0013 100 100 060 Pre-fail > Always - 0 > 12 Power_Cycle_Count 0x0032 100 100 000 Old_age > Always - 30 > 192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always > - 129 > 193 Load_Cycle_Count 0x0012 100 100 000 Old_age Always > - 129 > 194 Temperature_Celsius 0x0002 166 166 000 Old_age Always > - 36 (Min/Max 25/50) > 196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always > - 0 > 197 Current_Pending_Sector 0x0022 100 100 000 Old_age Always > - 0 > 198 Offline_Uncorrectable 0x0008 100 100 000 Old_age > Offline - 0 > 199 UDMA_CRC_Error_Count 0x000a 200 200 000 Old_age Always > - 112 > > SMART Error Log Version: 1 > ATA Error Count: 112 (device log contains only the most recent five errors) > CR = Command Register [HEX] > FR = Features Register [HEX] > SC = Sector Count Register [HEX] > SN = Sector Number Register [HEX] > CL = Cylinder Low Register [HEX] > CH = Cylinder High Register [HEX] > DH = Device/Head Register [HEX] > DC = Device Command Register [HEX] > ER = Error register [HEX] > ST = Status register [HEX] > Powered_Up_Time is measured from power on, and printed as > DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes, > SS=sec, and sss=millisec. It "wraps" after 49.710 days. > > Error 112 occurred at disk power-on lifetime: 52 hours (2 days + 4 hours) > When the command that caused the error occurred, the device was > active or idle. > > After command completion occurred, registers were: > ER ST SC SN CL CH DH > -- -- -- -- -- -- -- > 84 51 20 28 c5 ae 06 Error: ICRC, ABRT 32 sectors at LBA = > 0x06aec528 = 112117032 > > Commands leading to the command that caused the error were: > CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name > -- -- -- -- -- -- -- -- ---------------- -------------------- > 35 03 f0 58 c4 ae 40 00 2d+04:00:39.416 WRITE DMA EXT > 35 03 20 38 c4 ae 40 00 2d+04:00:39.409 WRITE DMA EXT > 35 03 f0 48 c3 ae 40 00 2d+04:00:39.402 WRITE DMA EXT > 35 03 f0 58 c2 ae 40 00 2d+04:00:39.394 WRITE DMA EXT > 35 03 60 f8 c1 ae 40 00 2d+04:00:39.388 WRITE DMA EXT > > Error 111 occurred at disk power-on lifetime: 52 hours (2 days + 4 hours) > When the command that caused the error occurred, the device was > active or idle. > > After command completion occurred, registers were: > ER ST SC SN CL CH DH > -- -- -- -- -- -- -- > 84 51 30 08 c4 90 06 Error: ICRC, ABRT 48 sectors at LBA = > 0x0690c408 = 110150664 > > Commands leading to the command that caused the error were: > CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name > -- -- -- -- -- -- -- -- ---------------- -------------------- > 35 03 f0 48 c3 90 40 00 2d+03:59:29.925 WRITE DMA EXT > 35 03 f0 58 c2 90 40 00 2d+03:59:29.917 WRITE DMA EXT > 35 03 60 f8 c1 90 40 00 2d+03:59:29.910 WRITE DMA EXT > 35 03 b0 48 c1 90 40 00 2d+03:59:29.907 WRITE DMA EXT > 35 03 f0 58 c0 90 40 00 2d+03:59:29.899 WRITE DMA EXT > > Error 110 occurred at disk power-on lifetime: 52 hours (2 days + 4 hours) > When the command that caused the error occurred, the device was > active or idle. > > After command completion occurred, registers were: > ER ST SC SN CL CH DH > -- -- -- -- -- -- -- > 84 51 20 28 47 90 05 Error: ICRC, ABRT 32 sectors at LBA = > 0x05904728 = 93341480 > > Commands leading to the command that caused the error were: > CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name > -- -- -- -- -- -- -- -- ---------------- -------------------- > 35 da f0 58 46 90 40 00 2d+03:49:28.919 WRITE DMA EXT > 35 da 20 38 46 90 40 00 2d+03:49:28.917 WRITE DMA EXT > 35 da f0 48 45 90 40 00 2d+03:49:28.909 WRITE DMA EXT > 35 da f0 58 44 90 40 00 2d+03:49:28.901 WRITE DMA EXT > 35 da 20 38 44 90 40 00 2d+03:49:28.899 WRITE DMA EXT > > Error 109 occurred at disk power-on lifetime: 52 hours (2 days + 4 hours) > When the command that caused the error occurred, the device was > active or idle. > > After command completion occurred, registers were: > ER ST SC SN CL CH DH > -- -- -- -- -- -- -- > 84 51 10 38 e7 41 05 Error: ICRC, ABRT 16 sectors at LBA = > 0x0541e738 = 88205112 > > Commands leading to the command that caused the error were: > CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name > -- -- -- -- -- -- -- -- ---------------- -------------------- > 35 03 f0 58 e6 41 40 00 2d+03:46:20.939 WRITE DMA EXT > 35 03 20 38 e6 41 40 00 2d+03:46:20.933 WRITE DMA EXT > 35 03 f0 48 e5 41 40 00 2d+03:46:20.926 WRITE DMA EXT > 35 03 f0 58 e4 41 40 00 2d+03:46:20.921 WRITE DMA EXT > 35 03 20 38 e4 41 40 00 2d+03:46:20.916 WRITE DMA EXT > > Error 108 occurred at disk power-on lifetime: 52 hours (2 days + 4 hours) > When the command that caused the error occurred, the device was > active or idle. > > After command completion occurred, registers were: > ER ST SC SN CL CH DH > -- -- -- -- -- -- -- > 84 51 10 28 0a 2f 05 Error: ICRC, ABRT 16 sectors at LBA = > 0x052f0a28 = 86968872 > > Commands leading to the command that caused the error were: > CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name > -- -- -- -- -- -- -- -- ---------------- -------------------- > 35 03 f0 48 09 2f 40 00 2d+03:45:06.250 WRITE DMA EXT > 35 03 f0 58 08 2f 40 00 2d+03:45:06.242 WRITE DMA EXT > 35 03 20 38 08 2f 40 00 2d+03:45:06.238 WRITE DMA EXT > 35 03 f0 48 07 2f 40 00 2d+03:45:06.231 WRITE DMA EXT > 35 03 08 40 bc 25 40 00 2d+03:45:06.225 WRITE DMA EXT > > SMART Self-test log structure revision number 1 > No self-tests have been logged. [To run self-tests, use: smartctl -t] > > > SMART Selective self-test log data structure revision number 1 > SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS > 1 0 0 Not_testing > 2 0 0 Not_testing > 3 0 0 Not_testing > 4 0 0 Not_testing > 5 0 0 Not_testing > Selective self-test flags (0x0): > After scanning selected spans, do NOT read-scan remainder of disk. > If Selective self-test is pending on power-up, resume after 0 minute delay. > ------------------------------------------------------------------------ > > These only show up in disk's SMART log, not as errors in other logs > (logs exists for all the history of the system). These do not seem to > have passed to the FreeBSD, or FreeBSD has ignored the errors. > > From logs, I can find da3, da4, da6 all have reported scsi timeout once > (different times, within one month in July. > > They all look like this, one case for each of the disks: > > ------------------------------------------------------------------------ > Jul 24 12:16:55 tila kernel: (da6:mps0:0:6:0): SCSI command timeout on > device handle 0x000d SMID 699 > Jul 24 12:16:55 tila kernel: (da6:mps0:0:6:0): SCSI command timeout on > device handle 0x000d SMID 419 > Jul 24 12:16:55 tila kernel: (da6:mps0:0:6:0): SCSI command timeout on > device handle 0x000d SMID 359 > Jul 24 12:16:55 tila kernel: (da6:mps0:0:6:0): SCSI command timeout on > device handle 0x000d SMID 472 > Jul 24 12:16:55 tila kernel: (da6:mps0:0:6:0): SCSI command timeout on > device handle 0x000d SMID 124 > Jul 24 12:16:55 tila kernel: (da6:mps0:0:6:0): SCSI command timeout on > device handle 0x000d SMID 762 > Jul 24 12:16:55 tila kernel: (da6:mps0:0:6:0): SCSI command timeout on > device handle 0x000d SMID 243 > Jul 24 12:16:55 tila kernel: (da6:mps0:0:6:0): SCSI command timeout on > device handle 0x000d SMID 890 > Jul 24 12:16:55 tila kernel: (da6:mps0:0:6:0): SCSI command timeout on > device handle 0x000d SMID 204 > Jul 24 12:16:55 tila kernel: (da6:mps0:0:6:0): SCSI command timeout on > device handle 0x000d SMID 2 > Jul 24 12:16:55 tila kernel: (da6:mps0:0:6:0): SCSI command timeout on > device handle 0x000d SMID 291 > Jul 24 12:16:55 tila kernel: mps0: (0:6:0) terminated ioc 804b scsi 0 > state c xfer 0 > Jul 24 12:16:55 tila kernel: mps0: mpssas_abort_complete: abort request > on handle 0x0d SMID 699 complete > Jul 24 12:16:55 tila kernel: mps0: mpssas_complete_tm_request: sending > deferred task management request for handle 0x0d SMID 419 > Jul 24 12:16:55 tila kernel: mps0: mpssas_abort_complete: abort request > on handle 0x0d SMID 419 complete > Jul 24 12:16:55 tila kernel: mps0: mpssas_complete_tm_request: sending > deferred task management request for handle 0x0d SMID 359 > Jul 24 12:16:55 tila kernel: mps0: mpssas_abort_complete: abort request > on handle 0x0d SMID 359 complete > Jul 24 12:16:55 tila kernel: mps0: mpssas_complete_tm_request: sending > deferred task management request for handle 0x0d SMID 472 > Jul 24 12:16:55 tila kernel: mps0: mpssas_abort_complete: abort request > on handle 0x0d SMID 472 complete > Jul 24 12:16:55 tila kernel: mps0: mpssas_complete_tm_request: sending > deferred task management request for handle 0x0d SMID 124 > Jul 24 12:16:55 tila kernel: mps0: mpssas_abort_complete: abort request > on handle 0x0d SMID 124 complete > Jul 24 12:16:55 tila kernel: mps0: mpssas_complete_tm_request: sending > deferred task management request for handle 0x0d SMID 762 > Jul 24 12:16:55 tila kernel: mps0: mpssas_abort_complete: abort request > on handle 0x0d SMID 762 complete > Jul 24 12:16:55 tila kernel: mps0: mpssas_complete_tm_request: sending > deferred task management request for handle 0x0d SMID 243 > Jul 24 12:16:55 tila kernel: mps0: mpssas_abort_complete: abort request > on handle 0x0d SMID 243 complete > Jul 24 12:16:55 tila kernel: mps0: mpssas_complete_tm_request: sending > deferred task management request for handle 0x0d SMID 890 > Jul 24 12:16:55 tila kernel: mps0: mpssas_abort_complete: abort request > on handle 0x0d SMID 890 complete > Jul 24 12:16:55 tila kernel: mps0: mpssas_complete_tm_request: sending > deferred task management request for handle 0x0d SMID 204 > Jul 24 12:16:55 tila kernel: mps0: mpssas_abort_complete: abort request > on handle 0x0d SMID 204 complete > Jul 24 12:16:55 tila kernel: mps0: mpssas_complete_tm_request: sending > deferred task management request for handle 0x0d SMID 2 > Jul 24 12:16:55 tila kernel: mps0: mpssas_abort_complete: abort request > on handle 0x0d SMID 2 complete > Jul 24 12:16:55 tila kernel: mps0: mpssas_complete_tm_request: sending > deferred task management request for handle 0x0d SMID 291 > Jul 24 12:16:55 tila kernel: mps0: mpssas_abort_complete: abort request > on handle 0x0d SMID 291 complete > Jul 24 12:16:55 tila kernel: (da6:mps0:0:6:0): WRITE(16). CDB: 8a 0 0 0 > 0 1 4a 17 dc 5e 0 0 0 57 0 0 > Jul 24 12:16:55 tila kernel: (da6:mps0:0:6:0): CAM status: SCSI Status Error > Jul 24 12:16:55 tila kernel: (da6:mps0:0:6:0): SCSI status: Check Condition > Jul 24 12:16:55 tila kernel: (da6:mps0:0:6:0): SCSI sense: UNIT > ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred) > Jul 24 12:16:56 tila kernel: (da6:mps0:0:6:0): WRITE(10). CDB: 2a 0 28 0 > 85 fb 0 0 1 0 > Jul 24 12:16:56 tila kernel: (da6:mps0:0:6:0): CAM status: SCSI Status Error > Jul 24 12:16:56 tila kernel: (da6:mps0:0:6:0): SCSI status: Check Condition > Jul 24 12:16:56 tila kernel: (da6:mps0:0:6:0): SCSI sense: UNIT > ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred) > ------------------------------------------------------------------------ > > If the da0 ICRC errors would have been seen by ZFS, it should have made > a) note of that in some log? b) retried write? c) Something else? If > we assume that the disk firmware is broken and does not report these to > OS, so da0 might be corrupt. But that should still be ok with raidz2. > > We do have 3 random SCSI timeouts, which were seen by FreeBSD, and thus > should have prompted ZFS do handle the errors, and one read error on > data, which is not reported as read error in any log, other than disk's > SMART info says so. > > Is it possible to get more information, which block of the file is > corrupt, on which disks the offending blocks reside, and similar > information? > > > > > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@FreeBSD.ORG Thu Oct 18 05:20:21 2012 Return-Path: Delivered-To: FS@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 62C078B8 for ; Thu, 18 Oct 2012 05:20:21 +0000 (UTC) (envelope-from james@jrv.org) Received: from mail.jrv.org (adsl-70-243-84-11.dsl.austtx.swbell.net [70.243.84.11]) by mx1.freebsd.org (Postfix) with ESMTP id EAFED8FC08 for ; Thu, 18 Oct 2012 05:20:20 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by mail.jrv.org (Postfix) with ESMTP id D41BF6D65AF; Thu, 18 Oct 2012 00:10:18 -0500 (CDT) X-Virus-Scanned: amavisd-new at zimbra.housenet.jrv Received: from mail.jrv.org ([127.0.0.1]) by localhost (zimbra.housenet.jrv [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id XQcK+khE5SDO; Thu, 18 Oct 2012 00:09:21 -0500 (CDT) Received: from [10.0.2.15] (adsl-70-243-84-14.dsl.austtx.swbell.net [70.243.84.14]) by mail.jrv.org (Postfix) with ESMTPSA id 2B4436D603F; Thu, 18 Oct 2012 00:09:21 -0500 (CDT) Message-ID: <507F8EFF.4020609@jrv.org> Date: Thu, 18 Oct 2012 00:09:19 -0500 From: "James R. Van Artsdalen" User-Agent: Mozilla/5.0 (Windows NT 5.0; rv:12.0) Gecko/20120428 Thunderbird/12.0.1 MIME-Version: 1.0 To: Heikki Suonsivu Subject: Re: ZFS raidz2, errors in file? References: <507EED58.80409@suonsivu.net> In-Reply-To: <507EED58.80409@suonsivu.net> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: FS@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Oct 2012 05:20:21 -0000 On 10/17/2012 12:39 PM, Heikki Suonsivu wrote: > SMART data indicates problems on two other disks, but no indication of > those are seen in logs (the disks work, but SMART information > indicates problems). The problems may be in areas ZFS has not tried to read. > One disk indeed has pending sector, not unusual and should be survivable: > > ------------------------------------------------------------------------ > ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED > WHEN_FAILED RAW_VALUE > 197 Current_Pending_Sector 0x0032 200 200 000 Old_age > Always - 1 > 198 Offline_Uncorrectable 0x0030 200 200 000 Old_age > Offline - 1 That error means one sector is unreadable and a replacement is pending; replacement will happen when next as the sector is overwritten. The contents of that sector are lost (unless some future read succeeds). > In addition, there seems to be ICRC DMA errors on da0. Looks nasty, > but only show up in SMART log, not in /var/log/messages. > > ------------------------------------------------------------------------ > 199 UDMA_CRC_Error_Count 0x000a 200 200 000 Old_age > Always - 112 I believe that both of these messages refer to errors in transfers between the disk and host, not to errors within the disk. Test your cabling and enclosures. > SMART Error Log Version: 1 > ATA Error Count: 112 (device log contains only the most recent five > errors) I don't like these at all. Consider replacing that disk. > If the da0 ICRC errors would have been seen by ZFS, it should have > made a) note of that in some log? b) retried write? c) Something > else? If we assume that the disk firmware is broken and does not > report these to OS, so da0 might be corrupt. But that should still be > ok with raidz2. These errors should trigger retries in layers beneath ZFS > We do have 3 random SCSI timeouts, which were seen by FreeBSD, and > thus should have prompted ZFS do handle the errors, and one read error > on data, which is not reported as read error in any log, other than > disk's SMART info says so. The retries may have happened at layer below ZFS. ZFS does not call the disk driver directly. Other layers play a role in error handing. From owner-freebsd-fs@FreeBSD.ORG Thu Oct 18 08:30:02 2012 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 25E9C277 for ; Thu, 18 Oct 2012 08:30:02 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [8.8.178.135]) by mx1.freebsd.org (Postfix) with ESMTP id E8D728FC0C for ; Thu, 18 Oct 2012 08:30:01 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q9I8U1M2039429 for ; Thu, 18 Oct 2012 08:30:01 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q9I8U1EB039425; Thu, 18 Oct 2012 08:30:01 GMT (envelope-from gnats) Date: Thu, 18 Oct 2012 08:30:01 GMT Message-Id: <201210180830.q9I8U1EB039425@freefall.freebsd.org> To: freebsd-fs@FreeBSD.org Cc: From: Daichi GOTO Subject: Re: kern/172334: [unionfs] unionfs permits recursive union mounts; causes panic quickly X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: Daichi GOTO List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Oct 2012 08:30:02 -0000 The following reply was made to PR kern/172334; it has been noted by GNATS. From: Daichi GOTO To: bug-followup@FreeBSD.org, yanegomi@gmail.com Cc: Subject: Re: kern/172334: [unionfs] unionfs permits recursive union mounts; causes panic quickly Date: Thu, 18 Oct 2012 10:17:59 +0200 You could around this issue with a following patch. http://people.freebsd.org/~daichi/unionfs/unionfs-cross-mount3.diff And above patch is rejected at ml discussion some years ago. We need more radical changes to improve this kind of issues. -- Daichi GOTO 81-42-316-7945 | daichi@ongs.co.jp | http://www.ongs.co.jp LinkedIn: http://linkedin.com/in/daichigoto From owner-freebsd-fs@FreeBSD.ORG Thu Oct 18 08:50:27 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 53A676E6 for ; Thu, 18 Oct 2012 08:50:27 +0000 (UTC) (envelope-from alexey.w.tyurikov@gmail.com) Received: from mail-ea0-f182.google.com (mail-ea0-f182.google.com [209.85.215.182]) by mx1.freebsd.org (Postfix) with ESMTP id D86C68FC17 for ; Thu, 18 Oct 2012 08:50:26 +0000 (UTC) Received: by mail-ea0-f182.google.com with SMTP id c10so2295270eaa.13 for ; Thu, 18 Oct 2012 01:50:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:date:x-google-sender-auth:message-id:subject :from:to:content-type; bh=y+ef3O04K2BFwOm49Y/8gpxwfi2IfqlaVoPQhK08+tw=; b=SsGO3CAOJ5OKIdBKNJFLPe8BYzz8tAsfjjhaEw0NkiB9S7xGN5vkO22g/ikdSjkH9G cCedrWrcdCmDXoZ5nNZCvxNLBBrrmNIUwmn+kQ+t2JQiDsSLiPKvm86xM0hfemWG74VH y29y8mLBNQ8XBbPC5MPtFp/B2j75x9LO5NY5c9KVgYIw7plRvUgxSp5z5lBOcgK6Hs8g 4b8P6OpfKtDoMiJ1qyNCqOlWmvU0DE8s/vbONFbsXf7wjV11ZGWcNxIJ7oz8X9BAODeB 1BeEYP6lG1SRtf9t2ybHr3iapedJNULjUDuDl3urPdVkeBEj2+w86dzyBULiHWeTrgAc TfFA== MIME-Version: 1.0 Received: by 10.14.178.195 with SMTP id f43mr30641065eem.44.1350550225472; Thu, 18 Oct 2012 01:50:25 -0700 (PDT) Sender: alexey.w.tyurikov@gmail.com Received: by 10.14.179.201 with HTTP; Thu, 18 Oct 2012 01:50:25 -0700 (PDT) Date: Thu, 18 Oct 2012 10:50:25 +0200 X-Google-Sender-Auth: AXxtik0mHcvreY0OSikG2AGKuVQ Message-ID: Subject: ZFS over NFS issue with linux clients From: Alexey Tyurikov To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Oct 2012 08:50:27 -0000 Hi all, I'm looking for people using FreeBSD 9.0 amd64 ZFS over NFS (v3 and v4) with Linux clients. My clients (last CentOS6 and Debian6) don't use cache for files lying on ZFS over NFS. Tested: $ grep something /nfs_zfs/file-100MB # network activity, client read the file over NFS (slow) $ grep something /nfs_zfs/file-100MB # network activity, client read the file over NFS once again (slow) At the same time UFS over NFS works like expected: $ grep something /nfs_ufs/file-100MB # network activity, client read the file over NFS (slow) $ grep something /nfs_ufs/file-100MB # no network activity, client read the file from cache (fast) Could anyone test it? Best regards -- Alexey Tyurikov From owner-freebsd-fs@FreeBSD.ORG Thu Oct 18 09:35:50 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id E82B67E1 for ; Thu, 18 Oct 2012 09:35:50 +0000 (UTC) (envelope-from kp@sigsegv.be) Received: from mercury.codepro.be (mercury.codepro.be [IPv6:2001:4b98:dc0:51:216:3eff:feb7:3147]) by mx1.freebsd.org (Postfix) with ESMTP id 7A66E8FC08 for ; Thu, 18 Oct 2012 09:35:50 +0000 (UTC) Received: from adrastea.jupiter.sigsegv.be (adrastea.jupiter.sigsegv.be [IPv6:2001:6f8:1498:1::3]) by mercury.codepro.be (Postfix) with ESMTP id B3F3D366; Thu, 18 Oct 2012 11:35:48 +0200 (CEST) Received: from thebe.jupiter.sigsegv.be (thebe.jupiter.sigsegv.be [172.16.1.5]) by adrastea.jupiter.sigsegv.be (Postfix) with ESMTP id 69A8F6AAF; Thu, 18 Oct 2012 11:35:18 +0200 (CEST) Received: by thebe.jupiter.sigsegv.be (Postfix, from userid 1000) id 4340C40F3B; Thu, 18 Oct 2012 11:35:18 +0200 (CEST) Date: Thu, 18 Oct 2012 11:35:18 +0200 From: Kristof Provost To: Alexey Tyurikov Subject: Re: ZFS over NFS issue with linux clients Message-ID: <20121018093517.GS9028@thebe.jupiter.sigsegv.be> References: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: X-PGP-Fingerprint: E114 D9EA 909E D469 8F57 17A5 7D15 91C6 9EFA F286 User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Oct 2012 09:35:51 -0000 On 2012-10-18 10:50:25 (+0200), Alexey Tyurikov wrote: > I'm looking for people using FreeBSD 9.0 amd64 ZFS over NFS (v3 and v4) > with Linux clients. My clients (last CentOS6 and Debian6) don't use cache > for files lying on ZFS over NFS. Tested: > > $ grep something /nfs_zfs/file-100MB # network activity, client read the > file over NFS (slow) > $ grep something /nfs_zfs/file-100MB # network activity, client read the > file over NFS once again (slow) > I'm unable to reproduce this behaviour with a Debian 6.0 client (ARM, because my openrd is the only client I can get to at the moment). That's a 2.6.32-5-kirkwood kernel. The server is FreeBSD 9.1-PRERELEASE #91 r241441 (amd64). I can't easily compare to UFS, because the server is ZFS only. I see network activity on the first read, but only very little (a single getattr + reply in fact) on the second read. Regards, Kristof From owner-freebsd-fs@FreeBSD.ORG Thu Oct 18 11:31:18 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 198DF110 for ; Thu, 18 Oct 2012 11:31:18 +0000 (UTC) (envelope-from alexey.w.tyurikov@gmail.com) Received: from mail-ee0-f54.google.com (mail-ee0-f54.google.com [74.125.83.54]) by mx1.freebsd.org (Postfix) with ESMTP id 9A35A8FC19 for ; Thu, 18 Oct 2012 11:31:17 +0000 (UTC) Received: by mail-ee0-f54.google.com with SMTP id c50so4998877eek.13 for ; Thu, 18 Oct 2012 04:31:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=CbGdq374FBaG91gie8azM2d6jCuKS8acNgQdiq4pp1k=; b=reEU29cJSvODAwWeE6zDFAp1mDx3mA0OxuyOAxL6q7IHXFa+6xoHpI/ZiVVJuP/C5P vVA+U5rTEi2ECVdzYtK6PFP/74gSFkQgOc/dvuw6Q+NPGa0fJv8HFoWSV5Zmks74RpYv DnM3sBK3kkLcNnzHzJGA273/vurImaO4nI1AmmNyqvzTnwjg8Jj6s1NawnunVCvEdhqF lmuwfHtsgntaCfdjec4v9poWcyKyc4LIfxf9dH3xRq1RdB9cf9o/VFxuQcqKUPM1k775 4Vbad4ZFT5U1Rrj8k0sR0WKtM4qu74e2NX0mirdPQ0+zjmbLTZHRW+rZaWtU23iBVDbi QT8w== MIME-Version: 1.0 Received: by 10.14.203.69 with SMTP id e45mr30819295eeo.38.1350559870985; Thu, 18 Oct 2012 04:31:10 -0700 (PDT) Sender: alexey.w.tyurikov@gmail.com Received: by 10.14.179.201 with HTTP; Thu, 18 Oct 2012 04:31:10 -0700 (PDT) In-Reply-To: <20121018093517.GS9028@thebe.jupiter.sigsegv.be> References: <20121018093517.GS9028@thebe.jupiter.sigsegv.be> Date: Thu, 18 Oct 2012 13:31:10 +0200 X-Google-Sender-Auth: _E7vkIK5nbME1yGrE83ZpxAvttE Message-ID: Subject: Re: ZFS over NFS issue with linux clients From: Alexey Tyurikov To: Kristof Provost Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Oct 2012 11:31:18 -0000 Yes, I can confirm, the issue doesn't exist on FreeBSD 9.1. Affected releases: 8.3, 9.0 @Kristof: thank you for the answer! do you use 9.1 in production? Best regards Alexey 2012/10/18 Kristof Provost > On 2012-10-18 10:50:25 (+0200), Alexey Tyurikov > wrote: > > I'm looking for people using FreeBSD 9.0 amd64 ZFS over NFS (v3 and v4) > > with Linux clients. My clients (last CentOS6 and Debian6) don't use cache > > for files lying on ZFS over NFS. Tested: > > > > $ grep something /nfs_zfs/file-100MB # network activity, client read the > > file over NFS (slow) > > $ grep something /nfs_zfs/file-100MB # network activity, client read the > > file over NFS once again (slow) > > > I'm unable to reproduce this behaviour with a Debian 6.0 client (ARM, > because my openrd is the only client I can get to at the moment). That's > a 2.6.32-5-kirkwood kernel. > > The server is FreeBSD 9.1-PRERELEASE #91 r241441 (amd64). I can't > easily compare to UFS, because the server is ZFS only. > > I see network activity on the first read, but only very little (a single > getattr + reply in fact) on the second read. > > Regards, > Kristof > > -- Alexey Tyurikov From owner-freebsd-fs@FreeBSD.ORG Thu Oct 18 11:52:39 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 56580B5F for ; Thu, 18 Oct 2012 11:52:39 +0000 (UTC) (envelope-from kp@sigsegv.be) Received: from mercury.codepro.be (mercury.codepro.be [IPv6:2001:4b98:dc0:51:216:3eff:feb7:3147]) by mx1.freebsd.org (Postfix) with ESMTP id DA1338FC0C for ; Thu, 18 Oct 2012 11:52:38 +0000 (UTC) Received: from adrastea.jupiter.sigsegv.be (adrastea.jupiter.sigsegv.be [IPv6:2001:6f8:1498:1::3]) by mercury.codepro.be (Postfix) with ESMTP id 65FC9366; Thu, 18 Oct 2012 13:52:37 +0200 (CEST) Received: from thebe.jupiter.sigsegv.be (thebe.jupiter.sigsegv.be [172.16.1.5]) by adrastea.jupiter.sigsegv.be (Postfix) with ESMTP id C10926BE4; Thu, 18 Oct 2012 13:52:36 +0200 (CEST) Received: by thebe.jupiter.sigsegv.be (Postfix, from userid 1000) id 817D1B048; Thu, 18 Oct 2012 13:52:36 +0200 (CEST) Date: Thu, 18 Oct 2012 13:52:36 +0200 From: Kristof Provost To: Alexey Tyurikov Subject: Re: ZFS over NFS issue with linux clients Message-ID: <20121018115236.GT9028@thebe.jupiter.sigsegv.be> References: <20121018093517.GS9028@thebe.jupiter.sigsegv.be> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: X-PGP-Fingerprint: E114 D9EA 909E D469 8F57 17A5 7D15 91C6 9EFA F286 User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Oct 2012 11:52:39 -0000 I track stable/9 in 'production', where production is on my home gateway/file/mail server. Regards, Kristof On 2012-10-18 13:31:10 (+0200), Alexey Tyurikov wrote: > Yes, I can confirm, the issue doesn't exist on FreeBSD 9.1. Affected > releases: 8.3, 9.0 > > @Kristof: thank you for the answer! do you use 9.1 in production? > > Best regards > Alexey > > 2012/10/18 Kristof Provost > > > On 2012-10-18 10:50:25 (+0200), Alexey Tyurikov > > wrote: > > > I'm looking for people using FreeBSD 9.0 amd64 ZFS over NFS (v3 and v4) > > > with Linux clients. My clients (last CentOS6 and Debian6) don't use cache > > > for files lying on ZFS over NFS. Tested: > > > > > > $ grep something /nfs_zfs/file-100MB # network activity, client read the > > > file over NFS (slow) > > > $ grep something /nfs_zfs/file-100MB # network activity, client read the > > > file over NFS once again (slow) > > > > > I'm unable to reproduce this behaviour with a Debian 6.0 client (ARM, > > because my openrd is the only client I can get to at the moment). That's > > a 2.6.32-5-kirkwood kernel. > > > > The server is FreeBSD 9.1-PRERELEASE #91 r241441 (amd64). I can't > > easily compare to UFS, because the server is ZFS only. > > > > I see network activity on the first read, but only very little (a single > > getattr + reply in fact) on the second read. > > > > Regards, > > Kristof > > > > > > > -- > Alexey Tyurikov From owner-freebsd-fs@FreeBSD.ORG Thu Oct 18 16:16:24 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 35DB6968 for ; Thu, 18 Oct 2012 16:16:24 +0000 (UTC) (envelope-from tom@claimlynx.com) Received: from na3sys009aog102.obsmtp.com (na3sys009aog102.obsmtp.com [74.125.149.69]) by mx1.freebsd.org (Postfix) with SMTP id 9B2228FC0A for ; Thu, 18 Oct 2012 16:16:23 +0000 (UTC) Received: from mail-vb0-f54.google.com ([209.85.212.54]) (using TLSv1) by na3sys009aob102.postini.com ([74.125.148.12]) with SMTP ID DSNKUIArUVucziojDpXjw3ePmvwiiSAoqExz@postini.com; Thu, 18 Oct 2012 09:16:23 PDT Received: by mail-vb0-f54.google.com with SMTP id v11so11412662vbm.13 for ; Thu, 18 Oct 2012 09:16:16 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:date:message-id:subject:from:to:cc:content-type :x-gm-message-state; bh=bp+3PfmWT3y9BOu3zvJHknxHDiavkrYvQAVnsSdpxkw=; b=g8gEDPr1rikMiJ0gSKducF9P7DqywE0t1e+INkKYSC+W2NZdpFEamyXSgt7Yzpi7uj +RmBmJOsjtUlKX3tmKt21B3fE90CalGEpzS+9IcMMYbTbYcvlrTXtK0Ek7Tp/irncoNa Br27wHrAweSBGzTHHWbCBV8HgOMkDgB+ZDzgogC20V22GLpqfxoDcpYL5zbXTbZHuJ7O xhAUd3o4C7sdZeynEThGJtJ5b9iPY29WWuuVkyJE+N5iYxNUiMcGmvHAP0umcJW+zNnj Y19M7IDoWGTWZ83gBF6Ybb8Y/ptkICaxYg5lOWI9FSOZJ0faEk6Af8NVxm5kw46exjua AS1A== MIME-Version: 1.0 Received: by 10.220.38.73 with SMTP id a9mr6312214vce.72.1350576976347; Thu, 18 Oct 2012 09:16:16 -0700 (PDT) Received: by 10.58.28.138 with HTTP; Thu, 18 Oct 2012 09:16:16 -0700 (PDT) Date: Thu, 18 Oct 2012 11:16:16 -0500 Message-ID: Subject: Poor throughput using new NFS client (9.0) vs. old (8.2/9.0) From: Thomas Johnson To: freebsd-fs@freebsd.org X-Gm-Message-State: ALoCoQntBVkBJ+bf5j6RfKCfcEJ0awMM9kFCY5Vqc8n+gPVQK0/CtEf0zcZQCsmmwRbp1/IRTsaa Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: root X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Oct 2012 16:16:24 -0000 We recently upgraded a number of hosts from FreeBSD 8.2 to 9.0. Almost immediately, we received reports from users of poor performance. The upgraded hosts are PXE-booted, with an NFS-mounted root. Additionally, they mount a number of other NFS shares, which is where our users work from. After a week of tweaking rsize/wsize/readahead parameters (per guidance), it finally occurred to me that 9.0 defaults to the new NFS client and server. I remounted the user shares using the oldnfs file type, and users reported that performance returned to its expected level. This is obviously a workaround, rather than a solution. We would prefer to get our hosts using the newnfs client, since presumably oldnfs will be deprecated at some point in the future. Is there some change that we should have made to our NFS configuration with the upgrade to 9.0, or is it possible that our workload is exposing some deficiency with newnfs? We tend to deal with a huge number of tiny files (several KB in size). The NFS server has been running 9.0 for some time (prior to the client upgrade) without any issue. NFS is served from a zpool, backed by a Dell MD3000, populated with 15k SAS disks. Clients and server are connected with Gig-E links. The general hardware configuration has not changed in nearly 3 years. As an example of the performance difference, here is some of the testing I did while troubleshooting. Given a directory containing 5671 zip files, with an average size of 15KB. I append all files to an existing zip file. Using the newnfs mount, I found that this operation generally takes ~30 seconds (wall time). Switching the mount to oldnfs resulted in the same operation taking ~10 seconds. tom@test-1:/test-> ls 53*zip | wc -l 5671 tom@test-1:/test-> ll -h BIG* -rw-rw-r-- 1 tom claimlynx 8.9M Oct 17 14:06 BIGGER_PILE_1.zip tom@test-1:/test-> time zip BIGGER_PILE_1.zip 53*.zip 0.646u 0.826s 0:51.01 2.8% 199+2227k 0+2769io 0pf+0w ...reset and repeat... 0.501u 0.629s 0:30.49 3.6% 208+2319k 0+2772io 0pf+0w ...reset and repeat... 0.601u 0.522s 0:32.37 3.4% 220+2406k 0+2771io 0pf+0w tom@test-1:/-> cd / tom@test-1:/-> sudo umount /test tom@test-1:/-> sudo mount -t oldnfs -o rw server:/array/test /test tom@test-1:/-> mount | grep test server:/array/test on /test (oldnfs) tom@test-1:/-> cd /test ...reset and repeat... 0.470u 0.903s 0:13.09 10.4% 203+2229k 0+5107io 0pf+0w ...reset and repeat... 0.547u 0.640s 0:08.65 13.6% 231+2493k 0+5086io 0pf+0w tom@test-1:/test-> ll -h BIG* -rw-rw-r-- 1 tom claimlynx 92M Oct 17 14:14 BIGGER_PILE_1.zip Thanks! -- Thomas Johnson ClaimLynx, Inc. From owner-freebsd-fs@FreeBSD.ORG Thu Oct 18 16:37:45 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 2D6771A9 for ; Thu, 18 Oct 2012 16:37:45 +0000 (UTC) (envelope-from ronald-freebsd8@klop.yi.org) Received: from smarthost1.greenhost.nl (smarthost1.greenhost.nl [195.190.28.78]) by mx1.freebsd.org (Postfix) with ESMTP id A08848FC0A for ; Thu, 18 Oct 2012 16:37:44 +0000 (UTC) Received: from smtp.greenhost.nl ([213.108.104.138]) by smarthost1.greenhost.nl with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.69) (envelope-from ) id 1TOt6M-00065T-DR for freebsd-fs@freebsd.org; Thu, 18 Oct 2012 18:37:42 +0200 Received: from [81.21.138.17] (helo=ronaldradial.versatec.local) by smtp.greenhost.nl with esmtpsa (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.72) (envelope-from ) id 1TOt6L-0008UY-Lt for freebsd-fs@freebsd.org; Thu, 18 Oct 2012 18:37:41 +0200 Content-Type: text/plain; charset=us-ascii; format=flowed; delsp=yes To: freebsd-fs@freebsd.org Subject: Re: Poor throughput using new NFS client (9.0) vs. old (8.2/9.0) References: Date: Thu, 18 Oct 2012 18:37:40 +0200 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit From: "Ronald Klop" Message-ID: In-Reply-To: User-Agent: Opera Mail/12.02 (Win32) X-Virus-Scanned: by clamav at smarthost1.samage.net X-Spam-Level: / X-Spam-Score: 0.0 X-Spam-Status: No, score=0.0 required=5.0 tests=BAYES_50 autolearn=disabled version=3.2.5 X-Scan-Signature: 7006e789400ccdeb5065f5d065827fb5 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Oct 2012 16:37:45 -0000 On Thu, 18 Oct 2012 18:16:16 +0200, Thomas Johnson wrote: > We recently upgraded a number of hosts from FreeBSD 8.2 to 9.0. Almost > immediately, we received reports from users of poor performance. The > upgraded hosts are PXE-booted, with an NFS-mounted root. Additionally, > they > mount a number of other NFS shares, which is where our users work from. > After a week of tweaking rsize/wsize/readahead parameters (per guidance), > it finally occurred to me that 9.0 defaults to the new NFS client and > server. I remounted the user shares using the oldnfs file type, and users > reported that performance returned to its expected level. > > This is obviously a workaround, rather than a solution. We would prefer > to > get our hosts using the newnfs client, since presumably oldnfs will be > deprecated at some point in the future. Is there some change that we > should > have made to our NFS configuration with the upgrade to 9.0, or is it > possible that our workload is exposing some deficiency with newnfs? We > tend > to deal with a huge number of tiny files (several KB in size). The NFS > server has been running 9.0 for some time (prior to the client upgrade) > without any issue. NFS is served from a zpool, backed by a Dell MD3000, > populated with 15k SAS disks. Clients and server are connected with Gig-E > links. The general hardware configuration has not changed in nearly 3 > years. > > As an example of the performance difference, here is some of the testing > I > did while troubleshooting. Given a directory containing 5671 zip files, > with an average size of 15KB. I append all files to an existing zip file. > Using the newnfs mount, I found that this operation generally takes ~30 > seconds (wall time). Switching the mount to oldnfs resulted in the same > operation taking ~10 seconds. > > tom@test-1:/test-> ls 53*zip | wc -l > 5671 > tom@test-1:/test-> ll -h BIG* > -rw-rw-r-- 1 tom claimlynx 8.9M Oct 17 14:06 BIGGER_PILE_1.zip > tom@test-1:/test-> time zip BIGGER_PILE_1.zip 53*.zip > 0.646u 0.826s 0:51.01 2.8% 199+2227k 0+2769io 0pf+0w > ...reset and repeat... > 0.501u 0.629s 0:30.49 3.6% 208+2319k 0+2772io 0pf+0w > ...reset and repeat... > 0.601u 0.522s 0:32.37 3.4% 220+2406k 0+2771io 0pf+0w > > tom@test-1:/-> cd / > tom@test-1:/-> sudo umount /test > tom@test-1:/-> sudo mount -t oldnfs -o rw server:/array/test /test > tom@test-1:/-> mount | grep test > server:/array/test on /test (oldnfs) > tom@test-1:/-> cd /test > ...reset and repeat... > 0.470u 0.903s 0:13.09 10.4% 203+2229k 0+5107io 0pf+0w > ...reset and repeat... > 0.547u 0.640s 0:08.65 13.6% 231+2493k 0+5086io 0pf+0w > tom@test-1:/test-> ll -h BIG* > -rw-rw-r-- 1 tom claimlynx 92M Oct 17 14:14 BIGGER_PILE_1.zip > > Thanks! > You might find this thread from today interesting. http://lists.freebsd.org/pipermail/freebsd-fs/2012-October/015441.html Ronald. From owner-freebsd-fs@FreeBSD.ORG Thu Oct 18 17:52:00 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id BC132DDA; Thu, 18 Oct 2012 17:52:00 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: from mail-lb0-f182.google.com (mail-lb0-f182.google.com [209.85.217.182]) by mx1.freebsd.org (Postfix) with ESMTP id AA2B58FC14; Thu, 18 Oct 2012 17:51:59 +0000 (UTC) Received: by mail-lb0-f182.google.com with SMTP id b5so7517013lbd.13 for ; Thu, 18 Oct 2012 10:51:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:reply-to:sender:date:x-google-sender-auth:message-id :subject:from:to:content-type; bh=P09jwitRJATSQzHPSMZ3DWrV1HjnKkOGf3YtsmIgmfA=; b=xsrVX4T/Yc4NYObDfTA90FOIlARcTokcb0NPcRUYBHvlTeeY9/FP6JNnzz3S/63n1A MeqzIcq7yxTQikXo92ZEG2Ox/HYcO4lkW47UOSmQ4WepFQaSAmbTUYmf4ZO6oKWrcGgo /YZFQFRiPUjNFWslNufULwqY79rHTdG591ty82fu8h+2ipHr77NBlKTbftBlwFJgCNQ1 BJXUhac8TAsHTAxWovWFerdMLRUcvnOTlkuO72mpisLDl+3zP64pg4MMpTM0zQ7wrrRZ jBHNs267Smf3Olikbs64E3oCljMv/A8x49+GiHFrR4nIEd/1NNI98S2qetdOhEohSZtI 87lg== MIME-Version: 1.0 Received: by 10.152.122.11 with SMTP id lo11mr19298936lab.3.1350582712599; Thu, 18 Oct 2012 10:51:52 -0700 (PDT) Sender: asmrookie@gmail.com Received: by 10.112.30.37 with HTTP; Thu, 18 Oct 2012 10:51:52 -0700 (PDT) Date: Thu, 18 Oct 2012 18:51:52 +0100 X-Google-Sender-Auth: -HUgO22GfUGJnd8HZOGygOxrVa0 Message-ID: Subject: MPSAFE VFS -- update From: Attilio Rao To: freebsd-current@freebsd.org, FreeBSD FS , Konstantin Belousov , Peter Holm Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: attilio@FreeBSD.org List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Oct 2012 17:52:00 -0000 Following the plan reported here: http://wiki.freebsd.org/NONMPSAFE_DEORBIT_VFS We are now at the state where all non-MPSAFE filesystems are disconnected by the three. At this point we can proceed with the import of a revised kib's patch as reported in that page. This will mean effectively remove Giant from the VFS, buffer cache and GEOM layer. We expect to have this included as soon as possibe, maybe before the end of the month. Thanks, Attilio -- Peace can only be achieved by understanding - A. Einstein From owner-freebsd-fs@FreeBSD.ORG Thu Oct 18 20:59:47 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id D35972D5 for ; Thu, 18 Oct 2012 20:59:47 +0000 (UTC) (envelope-from ipejic@gmail.com) Received: from mail-ea0-f182.google.com (mail-ea0-f182.google.com [209.85.215.182]) by mx1.freebsd.org (Postfix) with ESMTP id 5D6118FC08 for ; Thu, 18 Oct 2012 20:59:47 +0000 (UTC) Received: by mail-ea0-f182.google.com with SMTP id c10so2579775eaa.13 for ; Thu, 18 Oct 2012 13:59:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=8EOM42rqiHNRX4t/5uoJjgP1XkwsF4nfPsPOrr5Pj0Y=; b=KD+kRAzLx9b8hoP2hUJsrbwsYWEho0TfqtofW2gPWI7w+RtrWBJOL0KN04bs6M0bc9 3vdJF/YvHOMl60mlk8syjpoCXpVj6D2OXKXWgZknD6w4j2dZ6O1yND4klskAUBzbXEUf /gP3K64HzUZ9mEgjzgSRdxtTuANFTeCoGoDCqTH3DcNMwO5RykrEXAatuyJzN7BMO+D2 3JIBLSKm7pqANpTw+F9WP936VLQsavrwocJzDydD3lxLnTjhrcRjplU511f2V1rZFzRd hWkPxdP68NnWuA2qep/ZxPON9vYO8J0UiixgWjrpzaFdk+0l0mj+ndV3Utp7a08q0S7h xx6g== MIME-Version: 1.0 Received: by 10.14.0.198 with SMTP id 46mr33886873eeb.21.1350593986124; Thu, 18 Oct 2012 13:59:46 -0700 (PDT) Received: by 10.14.96.13 with HTTP; Thu, 18 Oct 2012 13:59:46 -0700 (PDT) Date: Thu, 18 Oct 2012 22:59:46 +0200 Message-ID: Subject: NAND Framework supported controllers From: =?UTF-8?Q?Ivan_Peji=C4=87?= To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Oct 2012 20:59:48 -0000 Hello, Dose someone use NAND Framework on x86 platform? aka I'm sw/hw enthusiast - I'm not afraid of soldering - Latest Sa(n)dForce controller ate my data - I want DIY SSD. Best regards. From owner-freebsd-fs@FreeBSD.ORG Thu Oct 18 21:39:53 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 5F102CDE for ; Thu, 18 Oct 2012 21:39:53 +0000 (UTC) (envelope-from dustinwenz@ebureau.com) Received: from internet02.ebureau.com (internet02.tru-signal.biz [65.127.24.21]) by mx1.freebsd.org (Postfix) with ESMTP id 1DAAD8FC16 for ; Thu, 18 Oct 2012 21:39:52 +0000 (UTC) Received: from service02.office.ebureau.com (internet06.ebureau.com [65.127.24.25]) by internet02.ebureau.com (Postfix) with ESMTP id 52A8CDEDBA2; Thu, 18 Oct 2012 16:31:54 -0500 (CDT) Received: from localhost (localhost [127.0.0.1]) by service02.office.ebureau.com (Postfix) with ESMTP id 4D667A94BC6; Thu, 18 Oct 2012 16:31:54 -0500 (CDT) X-Virus-Scanned: amavisd-new at ebureau.com Received: from service02.office.ebureau.com ([127.0.0.1]) by localhost (internet06.ebureau.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id KT2M5U6xHO77; Thu, 18 Oct 2012 16:31:52 -0500 (CDT) Received: from square.office.ebureau.com (square.office.ebureau.com [10.10.20.22]) by service02.office.ebureau.com (Postfix) with ESMTPSA id DA932A94BB4; Thu, 18 Oct 2012 16:31:52 -0500 (CDT) Subject: Re: Imposing ZFS latency limits Mime-Version: 1.0 (Mac OS X Mail 6.1 \(1498\)) Content-Type: text/plain; charset=iso-8859-1 From: Dustin Wenz X-Priority: 3 In-Reply-To: <8C42CB56F50F4D8AB949BDD91970D388@multiplay.co.uk> Date: Thu, 18 Oct 2012 16:31:52 -0500 Content-Transfer-Encoding: quoted-printable Message-Id: References: <8C42CB56F50F4D8AB949BDD91970D388@multiplay.co.uk> To: Steven Hartland X-Mailer: Apple Mail (2.1498) Cc: freebsd-fs@freebsd.org, Mark Felder X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Oct 2012 21:39:53 -0000 The way I can usually detect this is by watching the operation queues = with gstat. If a disk is running slower than the others, I/O ops tend to = pile up. When that happens, I can restore performance by taking the disk = offline. It's a manual process; I think the filesystem should do better = than that. - .Dustin On Oct 17, 2012, at 8:38 AM, Steven Hartland = wrote: > ----- Original Message ----- From: "Mark Felder" >> On Tue, 16 Oct 2012 06:25:57 -0500, Olivier Smedts = wrote: >>>=20 >>> That would be great - no need for TLER drives. But if you want to >>> "drop" the drive from the bus, that would be a GEOM thing. Don't = know >>> if that's possible to implement. >> This would be GREATLY appreciated. I've seen this happen on my own = ZFS boxes as well as on a custom made SAN. It's painful but easy to = detect when you notice the symptoms... >=20 > Interesting, what metrics where you using which made it easy to = detect, > work be nice to know your process there Mark? >=20 > Regards > Steve >=20 > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > This e.mail is private and confidential between Multiplay (UK) Ltd. = and the person or entity to whom it is addressed. In the event of = misdirection, the recipient is prohibited from using, copying, printing = or otherwise disseminating it or any information contained in it.=20 > In the event of misdirection, illegible or incomplete transmission = please telephone +44 845 868 1337 > or return the E.mail to postmaster@multiplay.co.uk. >=20 From owner-freebsd-fs@FreeBSD.ORG Thu Oct 18 21:55:42 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id D3053390 for ; Thu, 18 Oct 2012 21:55:42 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-annu.mail.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id 8B5578FC18 for ; Thu, 18 Oct 2012 21:55:42 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Ap4EAG95gFCDaFvO/2dsb2JhbABFhhK7SYIgAQEEASMERwsFFhoCDRkCIzYGExuHVwMJBguqTYkxDYlUgSGJT4YYgRIDlBaBVYskhRCDC4F7 X-IronPort-AV: E=Sophos;i="4.80,610,1344225600"; d="scan'208";a="187122872" Received: from erie.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.206]) by esa-annu-pri.mail.uoguelph.ca with ESMTP; 18 Oct 2012 17:54:33 -0400 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id 055A4B3F62; Thu, 18 Oct 2012 17:54:34 -0400 (EDT) Date: Thu, 18 Oct 2012 17:54:33 -0400 (EDT) From: Rick Macklem To: Alexey Tyurikov Message-ID: <113869727.2504027.1350597273963.JavaMail.root@erie.cs.uoguelph.ca> In-Reply-To: Subject: Re: NFS using ZFS issue MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Originating-IP: [172.17.91.202] X-Mailer: Zimbra 6.0.10_GA_2692 (ZimbraWebClient - IE7 (Win)/6.0.10_GA_2692) Cc: "freebsd-fs@freebsd.org" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Oct 2012 21:55:43 -0000 Alexey Tyurikov wrote: First off, I hope you didn't mind me putting the cc for the mailing list back in. > The solution is very simple: use 9.1 :-) Unbelievable ... >=20 Not unbelievable at all. NFS performance is usually a function of the underlying file systems and network performance, at least in the server. (All an NFS server does is translate RPC requests into correspinding VOP_xxx() operations. Plus, there is DRC overhead, which has been discussed recently.) Btw, for everyone else, Alexey sent me some packet traces. The attributes for the broken cases (Change and Modify Time) looked fine (if those change, the client would flush its cache. All I could see is that it got to a point for ZFS where there were many outstanding Read requests without replies, although there was data (wireshark didn't recognize these as replies) flowing from server->client. I have no idea what might have caused this. One thought was a broken TSO for the server's network interface. Another was some sort of resource constraint. Very little has changed in the NFS server between 9.0 and 9.1. (You can look at http://svn.freebsd.org/base/stable/9, but you won't find much changed.) I'll take another look, but I can't think of any NFS change between 9.0 and 9.1 that could affect this. Anyhow, glad it works better for you for 9.1, although I'd like to know what was going on for ZFS for 9.0? rick >=20 > Best regards > Alexey >=20 >=20 > 2012/10/18 Alexey Tyurikov < alexey.tyurikov@gmail.com > >=20 >=20 >=20 > Hello Rick, >=20 >=20 > thank you for the tip, I'll try it. What I've already done, I've > installed a FreeBSD server on at least 5 machines including 3 virtual > machines. It always the same=C2=A0with the exception of FreeBSD 8.2 > (hardware and virtual machine) - it works. >=20 >=20 > Then I've tested different clients: > - FreeBSD Client works > - RedHat =C2=A05.6 (Tikanga) works (!) but only with NFSv3 > - CentOS 6 doesn't work > - Debian 6 doesn't work >=20 >=20 > Regarding non-closing file from capture output: the file has been > closed, it was just cut by tcpdump. You can see it in the=C2=A0 > test_zfs_2.pcap >=20 >=20 > 38546 0.490573 10.1.3.90 10.1.3.111 NFS 262 V4 Call (Reply In 38547) > CLOSE StateID:0x1fe0 > 38547 0.490759 10.1.3.111 10.1.3.90 NFS 202 V4 Reply (Call In 38546) > CLOSE >=20 >=20 > Rick, thank you for your help! I'll post my issue on=C2=A0freebsd-fs@. If > I'll find a solution, I'll let you know. >=20 >=20 > Best regards > Alexey >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 > 2012/10/18 Rick Macklem < rmacklem@uoguelph.ca > >=20 >=20 > Just a ranom thought. Since you still seem to have packet > loss and then there is the weird behaviour (no replies, but > data moving from server->client), it might be network > interface related. >=20 > If your network interface has TSO support, I'd try disabling > that (it has been buggy in the past). >=20 > Also, if you have a different kind (different chipset) of > network hardware, you could try switching that on the server. >=20 > Mostly though, I'll be interested to see if anyone else can > explain it, when you post it to a list (freebsd-fs@ maybe?), rick >=20 >=20 >=20 >=20 > -- > Alexey Tyurikov >=20 >=20 >=20 >=20 > -- > Alexey Tyurikov From owner-freebsd-fs@FreeBSD.ORG Thu Oct 18 22:11:27 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 286606F4 for ; Thu, 18 Oct 2012 22:11:27 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-annu.mail.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id D33318FC0A for ; Thu, 18 Oct 2012 22:11:26 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Ap8EAGJ+gFCDaFvO/2dsb2JhbABFhhK7SYIgAQEBAwEBAQEgBCcgCwUWGAICDRkCKQEJJgYIBwQBGgIEh10GC6pckxGBIYo3Gg2FCYESA5I6gQSCLYEVjx+DC4FABzQ X-IronPort-AV: E=Sophos;i="4.80,610,1344225600"; d="scan'208";a="187124894" Received: from erie.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.206]) by esa-annu-pri.mail.uoguelph.ca with ESMTP; 18 Oct 2012 18:11:22 -0400 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id 202C8B4045; Thu, 18 Oct 2012 18:11:22 -0400 (EDT) Date: Thu, 18 Oct 2012 18:11:22 -0400 (EDT) From: Rick Macklem To: Ronald Klop Message-ID: <238159534.2504674.1350598282110.JavaMail.root@erie.cs.uoguelph.ca> In-Reply-To: Subject: Re: Poor throughput using new NFS client (9.0) vs. old (8.2/9.0) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.91.203] X-Mailer: Zimbra 6.0.10_GA_2692 (ZimbraWebClient - IE7 (Win)/6.0.10_GA_2692) Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Oct 2012 22:11:27 -0000 Ronald Klop wrote: > On Thu, 18 Oct 2012 18:16:16 +0200, Thomas Johnson > wrote: > > > We recently upgraded a number of hosts from FreeBSD 8.2 to 9.0. > > Almost > > immediately, we received reports from users of poor performance. The > > upgraded hosts are PXE-booted, with an NFS-mounted root. > > Additionally, > > they > > mount a number of other NFS shares, which is where our users work > > from. > > After a week of tweaking rsize/wsize/readahead parameters (per > > guidance), > > it finally occurred to me that 9.0 defaults to the new NFS client > > and > > server. I remounted the user shares using the oldnfs file type, and > > users > > reported that performance returned to its expected level. > > > > This is obviously a workaround, rather than a solution. We would > > prefer > > to > > get our hosts using the newnfs client, since presumably oldnfs will > > be > > deprecated at some point in the future. Is there some change that we > > should > > have made to our NFS configuration with the upgrade to 9.0, or is it > > possible that our workload is exposing some deficiency with newnfs? > > We > > tend > > to deal with a huge number of tiny files (several KB in size). The > > NFS > > server has been running 9.0 for some time (prior to the client > > upgrade) > > without any issue. NFS is served from a zpool, backed by a Dell > > MD3000, > > populated with 15k SAS disks. Clients and server are connected with > > Gig-E > > links. The general hardware configuration has not changed in nearly > > 3 > > years. > > > > As an example of the performance difference, here is some of the > > testing > > I > > did while troubleshooting. Given a directory containing 5671 zip > > files, > > with an average size of 15KB. I append all files to an existing zip > > file. > > Using the newnfs mount, I found that this operation generally takes > > ~30 > > seconds (wall time). Switching the mount to oldnfs resulted in the > > same > > operation taking ~10 seconds. > > > > tom@test-1:/test-> ls 53*zip | wc -l > > 5671 > > tom@test-1:/test-> ll -h BIG* > > -rw-rw-r-- 1 tom claimlynx 8.9M Oct 17 14:06 BIGGER_PILE_1.zip > > tom@test-1:/test-> time zip BIGGER_PILE_1.zip 53*.zip > > 0.646u 0.826s 0:51.01 2.8% 199+2227k 0+2769io 0pf+0w > > ...reset and repeat... > > 0.501u 0.629s 0:30.49 3.6% 208+2319k 0+2772io 0pf+0w > > ...reset and repeat... > > 0.601u 0.522s 0:32.37 3.4% 220+2406k 0+2771io 0pf+0w > > > > tom@test-1:/-> cd / > > tom@test-1:/-> sudo umount /test > > tom@test-1:/-> sudo mount -t oldnfs -o rw server:/array/test /test > > tom@test-1:/-> mount | grep test > > server:/array/test on /test (oldnfs) > > tom@test-1:/-> cd /test > > ...reset and repeat... > > 0.470u 0.903s 0:13.09 10.4% 203+2229k 0+5107io 0pf+0w > > ...reset and repeat... > > 0.547u 0.640s 0:08.65 13.6% 231+2493k 0+5086io 0pf+0w > > tom@test-1:/test-> ll -h BIG* > > -rw-rw-r-- 1 tom claimlynx 92M Oct 17 14:14 BIGGER_PILE_1.zip > > > > Thanks! > > > > > You might find this thread from today interesting. > http://lists.freebsd.org/pipermail/freebsd-fs/2012-October/015441.html > Yes, although I can't explain why Alexey's problem went away when he went from 9.0->9.1 for his NFS server, it would be interesting if Thomas could try the same thing? About the only thing different between the old and new NFS clients is the default rsize/wsize. However, if Thomas tried rsize=32768,wsize=32768 for the default (new) NFS client, then that would be ruled out. To be honest, the new client uses code cloned from the old one for all the caching etc (which is where the clients are "smart"). They use different RPC parsing code, since the new one does NFSv4 as well, but that code is pretty straightforward, so I can't think why it would result in a factor of 3 in performance. If Thomas were to capture a packet trace of the above test for two clients and emailed them to me, I could take a look and see if I can see what is going on. (For Alexey's case, it was a whole bunch of Read RPCs without replies, but that was a Linux client, of course. It also had a significant # of TCP layer retransmits and out of order TCP segments in it.) It would be nice to figure this out, since I was thinking that the old client might go away for 10.0 (can't if these issues still exist). rick > Ronald. > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@FreeBSD.ORG Thu Oct 18 22:18:10 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id AFC799B3 for ; Thu, 18 Oct 2012 22:18:10 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-annu.mail.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id 5D3EA8FC17 for ; Thu, 18 Oct 2012 22:18:10 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Ap4EAG1/gFCDaFvO/2dsb2JhbABFhhK7SYIgAQEFIwRSGxgCAg0ZAiM2BhMbh1cDDwuqXIkwDYlUgSGJT4YYgRIDlBaBVYskhRCDC4F7 X-IronPort-AV: E=Sophos;i="4.80,610,1344225600"; d="scan'208";a="187126067" Received: from erie.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.206]) by esa-annu-pri.mail.uoguelph.ca with ESMTP; 18 Oct 2012 18:18:09 -0400 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id 68FD5B3F91; Thu, 18 Oct 2012 18:18:09 -0400 (EDT) Date: Thu, 18 Oct 2012 18:18:09 -0400 (EDT) From: Rick Macklem To: Alexey Tyurikov Message-ID: <742079930.2504900.1350598689404.JavaMail.root@erie.cs.uoguelph.ca> In-Reply-To: Subject: Re: NFS using ZFS issue MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Originating-IP: [172.17.91.203] X-Mailer: Zimbra 6.0.10_GA_2692 (ZimbraWebClient - IE7 (Win)/6.0.10_GA_2692) Cc: "freebsd-fs@freebsd.org" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Oct 2012 22:18:11 -0000 Alexey Tyurikov wrote: > The solution is very simple: use 9.1 :-) Unbelievable ... >=20 Oh, when I mention looking at the changes, I always forget to include viewvc in the url, This is what I meant to type: http://svn.freebsd.org/viewvc/base/stable/9 rick >=20 > Best regards > Alexey >=20 >=20 > 2012/10/18 Alexey Tyurikov < alexey.tyurikov@gmail.com > >=20 >=20 >=20 > Hello Rick, >=20 >=20 > thank you for the tip, I'll try it. What I've already done, I've > installed a FreeBSD server on at least 5 machines including 3 virtual > machines. It always the same=C2=A0with the exception of FreeBSD 8.2 > (hardware and virtual machine) - it works. >=20 >=20 > Then I've tested different clients: > - FreeBSD Client works > - RedHat =C2=A05.6 (Tikanga) works (!) but only with NFSv3 > - CentOS 6 doesn't work > - Debian 6 doesn't work >=20 >=20 > Regarding non-closing file from capture output: the file has been > closed, it was just cut by tcpdump. You can see it in the=C2=A0 > test_zfs_2.pcap >=20 >=20 > 38546 0.490573 10.1.3.90 10.1.3.111 NFS 262 V4 Call (Reply In 38547) > CLOSE StateID:0x1fe0 > 38547 0.490759 10.1.3.111 10.1.3.90 NFS 202 V4 Reply (Call In 38546) > CLOSE >=20 >=20 > Rick, thank you for your help! I'll post my issue on=C2=A0freebsd-fs@. If > I'll find a solution, I'll let you know. >=20 >=20 > Best regards > Alexey >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 >=20 > 2012/10/18 Rick Macklem < rmacklem@uoguelph.ca > >=20 >=20 > Just a ranom thought. Since you still seem to have packet > loss and then there is the weird behaviour (no replies, but > data moving from server->client), it might be network > interface related. >=20 > If your network interface has TSO support, I'd try disabling > that (it has been buggy in the past). >=20 > Also, if you have a different kind (different chipset) of > network hardware, you could try switching that on the server. >=20 > Mostly though, I'll be interested to see if anyone else can > explain it, when you post it to a list (freebsd-fs@ maybe?), rick >=20 >=20 >=20 >=20 > -- > Alexey Tyurikov >=20 >=20 >=20 >=20 > -- > Alexey Tyurikov From owner-freebsd-fs@FreeBSD.ORG Fri Oct 19 00:04:49 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 513F493B for ; Fri, 19 Oct 2012 00:04:49 +0000 (UTC) (envelope-from jurgen.weber@theiconic.com.au) Received: from exprod6og109.obsmtp.com (exprod6og109.obsmtp.com [64.18.1.23]) by mx1.freebsd.org (Postfix) with SMTP id AB1618FC08 for ; Fri, 19 Oct 2012 00:04:47 +0000 (UTC) Received: from mail-pa0-f72.google.com ([209.85.220.72]) (using TLSv1) by exprod6ob109.postini.com ([64.18.5.12]) with SMTP ID DSNKUICZGZpT/1DTz1sS4w+wIJJliw3mN/6i@postini.com; Thu, 18 Oct 2012 17:04:47 PDT Received: by mail-pa0-f72.google.com with SMTP id fb11so18920120pad.7 for ; Thu, 18 Oct 2012 17:04:40 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:subject :content-type:content-transfer-encoding:x-gm-message-state; bh=gqOpC+TcDLtE+9yQ/zftNcgBt0h3WLffh+Os2hbYgTY=; b=DDEEQtfA75qWbm0u6V83i6vDZxslE1fA2HU6flwKHe2823Otccg/qZX1CdPpnmwyE/ uDeA4odmxwwiVq3iymScGQHbzjV4OUzpapmi5vEG98i/Q9USpo//FJ6M4rtNm1bxdUA7 TqfpBnTPvG1ZvFdWYwt7v5eXcgwQP2QkfuWlg/T/ye93PgxlAOe3nfcmU/E7XjkbeoTm MOcR2UyUiI7chOMi2zw8c/KW6LdpX4pGdmxHUjlDwDoAQRpZBlAmHcMlyy1rBIcJCj+a voGStvP0cM9v/MPFQD9AgMjE9HWiKdR9r5lnj7tWQWjRo352GCjhwHzO+SSt0Kqald9H 2+RA== Received: by 10.68.136.231 with SMTP id qd7mr70883803pbb.3.1350604682301; Thu, 18 Oct 2012 16:58:02 -0700 (PDT) Received: by 10.68.136.231 with SMTP id qd7mr70660902pbb.3.1350602991388; Thu, 18 Oct 2012 16:29:51 -0700 (PDT) Received: from [172.20.24.157] ([202.126.107.170]) by mx.google.com with ESMTPS id i4sm27049pav.20.2012.10.18.16.29.46 (version=SSLv3 cipher=OTHER); Thu, 18 Oct 2012 16:29:47 -0700 (PDT) Message-ID: <508090E8.4010300@theiconic.com.au> Date: Fri, 19 Oct 2012 10:29:44 +1100 From: =?ISO-8859-1?Q?J=FCrgen_Weber?= User-Agent: Mozilla/5.0 (X11; Linux i686 on x86_64; rv:16.0) Gecko/20121010 Thunderbird/16.0.1 MIME-Version: 1.0 To: freebsd-fs@freebsd.org Subject: mfi0 timeout error Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit X-Gm-Message-State: ALoCoQlyvyUANTH8HAGhgnwWgOzh8BTHNtdvDQOpNrWNjH4w7i0PuVKusorTQwXUpyPZhyLr7b+3bwl1l9Vm53+Lpw0h9vMZytrnzi/n0UwogAA1sVMZUIDYSgj9VszW9g0g6rEojj3/4KrXauuWRZMIoSVn9nM/qg== X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Oct 2012 00:04:49 -0000 Team I have googled around for a solution and I see a lot of posts about firmware versions and patches for FreeBSD 8.*. I have a FreeBSD 9.1rc1 system, which was beta1 orginally and has been running for months. Now it will not boot, I get the following: "Trying to mount root from zfs:tank/root []..... mfi0: COMMAND 0Xffffff8000cb83530 TIMEOUT AFTER xxx SECONDS (this just repeats). I have not seen this error before during normal runtime, _only_ during boot. Originally when I had the problem I could boot off a USB stick (9.1beta1 or rc1), run a 'zpool import -f tank' and it would work on the livecd. Rebooting and the main system would work. This time this work around does not work for me. When I am on the USB stick I can run a 'zpool import' and all of the disk are recognised, the pool is recognised and the file system is healthy. The Card is a H700 PERC, with 12.10.3 firmware in a Dell R515. Running FreeBSD 9.1-RC1, latest zfs and zpool versions. I have tried disabling the cache (mfiutil cache xxx disable). I have also gone into the Card settings and changed under advanced settings "adaptive forward read" to "read only". Any help, appreciated. Thanks -- Jürgen Weber Systems Engineer IT Infrastructure Team Leader THE ICONIC | E jurgen.weber@theiconic.com.au | www.theiconic.com.au From owner-freebsd-fs@FreeBSD.ORG Fri Oct 19 02:42:15 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 9C9A9B0A for ; Fri, 19 Oct 2012 02:42:15 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-annu.mail.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id 4D07F8FC17 for ; Fri, 19 Oct 2012 02:42:14 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AqAEAGa9gFCDaFvO/2dsb2JhbABFFoV8u1SCIAEBAQMBAQEBIAQnIAsFFhgCAg0ZAikBCSYGCAcEARoCBIddBguqXJJ+gSGKNxoNhQmBEgOTPoItgRWPH4MLgUAHNA X-IronPort-AV: E=Sophos;i="4.80,611,1344225600"; d="scan'208";a="187154545" Received: from erie.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.206]) by esa-annu-pri.mail.uoguelph.ca with ESMTP; 18 Oct 2012 22:42:14 -0400 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id 29A90B403A; Thu, 18 Oct 2012 22:42:14 -0400 (EDT) Date: Thu, 18 Oct 2012 22:42:14 -0400 (EDT) From: Rick Macklem To: Ronald Klop Message-ID: <479626430.2512565.1350614534110.JavaMail.root@erie.cs.uoguelph.ca> In-Reply-To: Subject: Re: Poor throughput using new NFS client (9.0) vs. old (8.2/9.0) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.91.201] X-Mailer: Zimbra 6.0.10_GA_2692 (ZimbraWebClient - IE7 (Win)/6.0.10_GA_2692) Cc: freebsd-fs@freebsd.org, Alexey Tyurikov X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Oct 2012 02:42:15 -0000 Ronald Klop wrote: > On Thu, 18 Oct 2012 18:16:16 +0200, Thomas Johnson > wrote: > > > We recently upgraded a number of hosts from FreeBSD 8.2 to 9.0. > > Almost > > immediately, we received reports from users of poor performance. The > > upgraded hosts are PXE-booted, with an NFS-mounted root. > > Additionally, > > they > > mount a number of other NFS shares, which is where our users work > > from. > > After a week of tweaking rsize/wsize/readahead parameters (per > > guidance), > > it finally occurred to me that 9.0 defaults to the new NFS client > > and > > server. I remounted the user shares using the oldnfs file type, and > > users > > reported that performance returned to its expected level. > > > > This is obviously a workaround, rather than a solution. We would > > prefer > > to > > get our hosts using the newnfs client, since presumably oldnfs will > > be > > deprecated at some point in the future. Is there some change that we > > should > > have made to our NFS configuration with the upgrade to 9.0, or is it > > possible that our workload is exposing some deficiency with newnfs? > > We > > tend > > to deal with a huge number of tiny files (several KB in size). The > > NFS > > server has been running 9.0 for some time (prior to the client > > upgrade) > > without any issue. NFS is served from a zpool, backed by a Dell > > MD3000, > > populated with 15k SAS disks. Clients and server are connected with > > Gig-E > > links. The general hardware configuration has not changed in nearly > > 3 > > years. > > > > As an example of the performance difference, here is some of the > > testing > > I > > did while troubleshooting. Given a directory containing 5671 zip > > files, > > with an average size of 15KB. I append all files to an existing zip > > file. > > Using the newnfs mount, I found that this operation generally takes > > ~30 > > seconds (wall time). Switching the mount to oldnfs resulted in the > > same > > operation taking ~10 seconds. > > > > tom@test-1:/test-> ls 53*zip | wc -l > > 5671 > > tom@test-1:/test-> ll -h BIG* > > -rw-rw-r-- 1 tom claimlynx 8.9M Oct 17 14:06 BIGGER_PILE_1.zip > > tom@test-1:/test-> time zip BIGGER_PILE_1.zip 53*.zip > > 0.646u 0.826s 0:51.01 2.8% 199+2227k 0+2769io 0pf+0w > > ...reset and repeat... > > 0.501u 0.629s 0:30.49 3.6% 208+2319k 0+2772io 0pf+0w > > ...reset and repeat... > > 0.601u 0.522s 0:32.37 3.4% 220+2406k 0+2771io 0pf+0w > > > > tom@test-1:/-> cd / > > tom@test-1:/-> sudo umount /test > > tom@test-1:/-> sudo mount -t oldnfs -o rw server:/array/test /test > > tom@test-1:/-> mount | grep test > > server:/array/test on /test (oldnfs) > > tom@test-1:/-> cd /test > > ...reset and repeat... > > 0.470u 0.903s 0:13.09 10.4% 203+2229k 0+5107io 0pf+0w > > ...reset and repeat... > > 0.547u 0.640s 0:08.65 13.6% 231+2493k 0+5086io 0pf+0w > > tom@test-1:/test-> ll -h BIG* > > -rw-rw-r-- 1 tom claimlynx 92M Oct 17 14:14 BIGGER_PILE_1.zip > > > > Thanks! > > > > > You might find this thread from today interesting. > http://lists.freebsd.org/pipermail/freebsd-fs/2012-October/015441.html > There are a couple of patches you guys want to have, if you are using an NFS server on FreeBSD9.0. However, I'm not convinced either of them would cause the performance issues reported, although the second one would result in a constipated server after a while. For the server: - http://people.freebsd.org/~rmacklem/nfsd-enoent.patch Without this patch, there can be spurious ENOENT replies to lookups, because ni_topdir wasn't initialized. - http://people.freebsd.org/~rmacklem/namei-leak.patch This patch is ZFS specific and without it, each Remove RPC results in leakage of 1 lookup buffer. (This would eventually result in a constipated server.) I didn't realize these hadn't made it into 9.0 until I looked, rick > Ronald. > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@FreeBSD.ORG Fri Oct 19 13:25:44 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id ACB6244D for ; Fri, 19 Oct 2012 13:25:44 +0000 (UTC) (envelope-from freebsd-fs@m.gmane.org) Received: from plane.gmane.org (plane.gmane.org [80.91.229.3]) by mx1.freebsd.org (Postfix) with ESMTP id 600598FC0A for ; Fri, 19 Oct 2012 13:25:43 +0000 (UTC) Received: from list by plane.gmane.org with local (Exim 4.69) (envelope-from ) id 1TPCa6-0006pC-ML for freebsd-fs@freebsd.org; Fri, 19 Oct 2012 15:25:42 +0200 Received: from lara.cc.fer.hr ([161.53.72.113]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Fri, 19 Oct 2012 15:25:42 +0200 Received: from ivoras by lara.cc.fer.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Fri, 19 Oct 2012 15:25:42 +0200 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-fs@freebsd.org From: Ivan Voras Subject: unionfs and NFS? Date: Fri, 19 Oct 2012 15:25:26 +0200 Lines: 44 Message-ID: Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enig72801F85F7A5D5C801A80B62" X-Complaints-To: usenet@ger.gmane.org X-Gmane-NNTP-Posting-Host: lara.cc.fer.hr User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:14.0) Gecko/20120812 Thunderbird/14.0 X-Enigmail-Version: 1.4.3 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Oct 2012 13:25:44 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig72801F85F7A5D5C801A80B62 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hi, I'm wondering if anyone is runnning unionfs with NFS, such that the NFS (mounted read-only) is the "lower" layer, and writes go to the upper laye= r. I'm also reading this in the mount_unionfs man page: BUGS THIS FILE SYSTEM TYPE IS NOT YET FULLY SUPPORTED (READ: IT DOESN'T WORK) AND USING IT MAY, IN FACT, DESTROY DATA ON YOUR SYSTEM. USE AT YOUR OWN RISK. BEWARE OF DOG. SLIPPERY WHEN WET. BATTERIES NOT INCLUDED. This code also needs an owner in order to be less dangerous - seriou= s hackers can apply by sending mail to =E2=9F=A8freebsd-fs@FreeBSD.org= =E2=9F=A9 and announcing their intent to take it over. =2E.. is this still true? --------------enig72801F85F7A5D5C801A80B62 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAlCBVMYACgkQ/QjVBj3/HSw9aACggIwhuoE8JGSuJKZb2IDDmUhy 8GYAoIwyYeRlXqaIdjgtCb2fZ5Gt0aUU =G2WC -----END PGP SIGNATURE----- --------------enig72801F85F7A5D5C801A80B62-- From owner-freebsd-fs@FreeBSD.ORG Fri Oct 19 15:40:01 2012 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id D3D8B7F2 for ; Fri, 19 Oct 2012 15:40:01 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [8.8.178.135]) by mx1.freebsd.org (Postfix) with ESMTP id A28948FC08 for ; Fri, 19 Oct 2012 15:40:01 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q9JFe1oj026025 for ; Fri, 19 Oct 2012 15:40:01 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q9JFe1qc026024; Fri, 19 Oct 2012 15:40:01 GMT (envelope-from gnats) Date: Fri, 19 Oct 2012 15:40:01 GMT Message-Id: <201210191540.q9JFe1qc026024@freefall.freebsd.org> To: freebsd-fs@FreeBSD.org Cc: From: John Baldwin Subject: Re: kern/142489: [zfs] [lor] allproc/zfs LOR X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: John Baldwin List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Oct 2012 15:40:01 -0000 The following reply was made to PR kern/142489; it has been noted by GNATS. From: John Baldwin To: bug-followup@freebsd.org, peterjeremy@acm.org Cc: Subject: Re: kern/142489: [zfs] [lor] allproc/zfs LOR Date: Fri, 19 Oct 2012 11:20:12 -0400 I think this order is correct. Something in ZFS must be calling pfind() or the like while holding a vnode lock. I think that is probably the "wrong" order. -- John Baldwin From owner-freebsd-fs@FreeBSD.ORG Fri Oct 19 17:25:33 2012 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 172B3B9F; Fri, 19 Oct 2012 17:25:33 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 312078FC0C; Fri, 19 Oct 2012 17:25:31 +0000 (UTC) Received: from odyssey.starpoint.kiev.ua (alpha-e.starpoint.kiev.ua [212.40.38.101]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id UAA01472; Fri, 19 Oct 2012 20:25:26 +0300 (EEST) (envelope-from avg@FreeBSD.org) Message-ID: <50818D06.4010801@FreeBSD.org> Date: Fri, 19 Oct 2012 20:25:26 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:16.0) Gecko/20121014 Thunderbird/16.0.1 MIME-Version: 1.0 To: Sean Chittenden Subject: Re: ZFS crashing during snapdir lookup for non-existent snapshot... References: <5075E3E0.7060706@FreeBSD.org> <0A6567E7-3BA5-4F27-AEB2-1C00EDE00641@chittenden.org> <5075EDDD.4030008@FreeBSD.org> <5075FA8E.10200@FreeBSD.org> <50769243.2010208@FreeBSD.org> In-Reply-To: <50769243.2010208@FreeBSD.org> X-Enigmail-Version: 1.4.5 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Cc: "freebsd-fs@freebsd.org" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Oct 2012 17:25:33 -0000 Sean, do you still experience this problem? If so, you may to try the following patch: http://people.freebsd.org/~avg/zfs-vfs.diff The patch is against quite recent head. Not sure how well it would apply to the stable branches (if you use those), but it definitely needs the following two commits from head: http://svnweb.freebsd.org/base?view=revision&revision=241556 http://svnweb.freebsd.org/base?view=revision&revision=241628 -- Andriy Gapon From owner-freebsd-fs@FreeBSD.ORG Fri Oct 19 22:02:53 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 0CF3A4A1 for ; Fri, 19 Oct 2012 22:02:53 +0000 (UTC) (envelope-from jurgen.weber@theiconic.com.au) Received: from exprod6og111.obsmtp.com (exprod6og111.obsmtp.com [64.18.1.27]) by mx1.freebsd.org (Postfix) with SMTP id 8B1D08FC0A for ; Fri, 19 Oct 2012 22:02:52 +0000 (UTC) Received: from mail-da0-f72.google.com ([209.85.210.72]) (using TLSv1) by exprod6ob111.postini.com ([64.18.5.12]) with SMTP ID DSNKUIHOBrZ/TngBgCu0bJGNoqib+irzJeYA@postini.com; Fri, 19 Oct 2012 15:02:52 PDT Received: by mail-da0-f72.google.com with SMTP id r28so1510527daj.7 for ; Fri, 19 Oct 2012 15:02:45 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:subject:references :in-reply-to:content-type:content-transfer-encoding :x-gm-message-state; bh=9XAZVegsnGu2IG96E4tyX5kJX7XiWILftI5eOfSfWjk=; b=PiurRjJflrKuphzNIsn0gv4k1C/hS2E5CN7sRHu1XhP8lcbbBJs/bm1icuTA6Tz6ix TBccVuvaLyeDxqzjvjLDew4d3UzALkn5V4659/pSdHCl9k81pegPGMfgayGrSTYrsAJg LbIZwDH6+/GN7v9lwpABFnuGg8xEYPKhfa5DI07JXp/Bumt8Hf6J1B/Tw9tPxwV1YzdY 5khtL1mtQ8sFUk5/e2Br2Q/nLmMtZIuvMLaUk/+wj1gJ8FZkNMl+d7TeQSg4MIklPJUU eJbgxiJRNFFICVaLzqPTDmF5SdTzHu+DsVpVmbDwV+Qpjt6KQUllbQpUscqG1fcL0QiS nyBw== Received: by 10.66.85.40 with SMTP id e8mr7454872paz.64.1350684165904; Fri, 19 Oct 2012 15:02:45 -0700 (PDT) Received: by 10.66.85.40 with SMTP id e8mr7454854paz.64.1350684165787; Fri, 19 Oct 2012 15:02:45 -0700 (PDT) Received: from [10.29.65.2] (247.35.70.115.static.exetel.com.au. [115.70.35.247]) by mx.google.com with ESMTPS id j9sm1670814pav.15.2012.10.19.15.02.42 (version=SSLv3 cipher=OTHER); Fri, 19 Oct 2012 15:02:44 -0700 (PDT) Message-ID: <5081CE05.1010108@theiconic.com.au> Date: Sat, 20 Oct 2012 09:02:45 +1100 From: Jurgen Weber User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:16.0) Gecko/20121010 Thunderbird/16.0.1 MIME-Version: 1.0 To: freebsd-fs@freebsd.org Subject: Re: mfi0 timeout error zfs boot mount problem References: <508090E8.4010300@theiconic.com.au> In-Reply-To: <508090E8.4010300@theiconic.com.au> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit X-Gm-Message-State: ALoCoQmmhZEpLmm79ip1eORr4POK41OB8vXtWzs84bO59BKSixt0oPyjjDtPw6TERSmsmm/COXpVmLwV1BJgdUFR4JXeV4Q5F6uwEdbkC/WW79ATM1hrG94/wxW1Py7u/JSX9EQ0LcG0twEnjrtBK6u3c9aHuHFPcw== X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 19 Oct 2012 22:02:53 -0000 Guys Some more details on this, some insight would be greatly appreciated. As my day wore on trying to get this zpool to import or mount I have learnt a few things. I think over time this issue has came about as more and more data was added to the file systems. Some further details: Its a 8 disk raidz pool that the system boots from as well. The disk are all 2TB. The server has 16GB Of RAM, I notcied the day before this happen the server was struggling with its RAM griding to a halt and dumping its RAM. The issue is not hardware because I found another server (same one) swapped the harddrives out took another 8GB of RAM and I have the same problem. The main data file systems have dedup and gzip compression on. I have booted from open/Oracle Solars 11 adn attempted to import and the Solaris live CD will not import either. In the Solaris system the disk detach from the system. I get the feeling that ZFS is hitting some root limit when attempting to mount and its not finishing the job. Thanks Jurgen On 19/10/2012 10:29 AM, Jürgen Weber wrote: > Team > > I have googled around for a solution and I see a lot of posts about > firmware versions and patches for FreeBSD 8.*. > > I have a FreeBSD 9.1rc1 system, which was beta1 orginally and has been > running for months. > > Now it will not boot, I get the following: > > "Trying to mount root from zfs:tank/root []..... > mfi0: COMMAND 0Xffffff8000cb83530 TIMEOUT AFTER xxx SECONDS > (this just repeats). > > I have not seen this error before during normal runtime, _only_ during > boot. > > Originally when I had the problem I could boot off a USB stick > (9.1beta1 or rc1), run a 'zpool import -f tank' and it would work on > the livecd. Rebooting and the main system would work. > > This time this work around does not work for me. When I am on the USB > stick I can run a 'zpool import' and all of the disk are recognised, > the pool is recognised and the file system is healthy. > > The Card is a H700 PERC, with 12.10.3 firmware in a Dell R515. > Running FreeBSD 9.1-RC1, latest zfs and zpool versions. > > I have tried disabling the cache (mfiutil cache xxx disable). I have > also gone into the Card settings and changed under advanced settings > "adaptive forward read" to "read only". > > Any help, appreciated. > > Thanks > From owner-freebsd-fs@FreeBSD.ORG Sat Oct 20 01:14:05 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 9439E1CC for ; Sat, 20 Oct 2012 01:14:05 +0000 (UTC) (envelope-from z640a836aeserrofq-sf=serrofq.bet@postmaster.twitter.com) Received: from ham-cannon.twitter.com (ham-cannon.twitter.com [199.59.148.235]) by mx1.freebsd.org (Postfix) with ESMTP id 732F78FC17 for ; Sat, 20 Oct 2012 01:14:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=simple/simple; d=twitter.com; s=dkim-201205; h=Message-ID:Date:Content-Type:MIME-Version:Subject:To:From; bh=iYxJb6BVyVYSpUMC89rfVqK/f+TBcMYuPkbygBdb7DU=; b=pLuXww7pZdCckLN7Re+sX6RRI5Ni02hF3N5TJz96wA9iU9hF4/UgGMQtAch1uF+AaHLi4f5Y2r0GQdMLR+aYc+81n7R8VlFFg/iYHZf9NoDUSLI5aXQG02INFYUCSd68wgmv5nHCzkLU58K/tCMDnt0ehcxiP6M3IeTG+maBmwE=; From: Twitter To: freebsd-fs@freebsd.org Subject: shilpa kheria is still waiting for you to join Twitter... MIME-Version: 1.0 X-mailstream: default X-twfbl: 512644b3-3520-4b59-9ca7-732274f21452 Precedence: Bulk Date: Sat, 20 Oct 2012 01:13:56 +0000 Message-ID: <916886951.1059071350695636590.JavaMail.ibis@1TPNdU-000D78-l3.mta.twitter.com> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.14 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 20 Oct 2012 01:14:05 -0000 shilpa kheria is still waiting for you to join Twitter... Twitter helps you stay connected with what's happening right now and with the people and organizations you care about. Join Twitter https://twitter.com/i/2b4e88ff89e2a7f9cd00e71d10fb03c465500b6d ------------------------ This message was sent by Twitter on behalf of Twitter users who entered your email address to invite you to Twitter. Unsubscribe: https://twitter.com/i/o?c=ZNV%2BN6G7N7ihid0b6ATo0BH%2B0yO3ul3f&uid=0&iid=512644b3-3520-4b59-9ca7-732274f21452&nid=58+26+20120920 Need help? https://support.twitter.com From owner-freebsd-fs@FreeBSD.ORG Sat Oct 20 02:08:31 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id AB53BD98; Sat, 20 Oct 2012 02:08:31 +0000 (UTC) (envelope-from freebsd@pki2.com) Received: from btw.pki2.com (btw.pki2.com [IPv6:2001:470:a:6fd::2]) by mx1.freebsd.org (Postfix) with ESMTP id 614E78FC08; Sat, 20 Oct 2012 02:08:31 +0000 (UTC) Received: from [127.0.0.1] (localhost [127.0.0.1]) by btw.pki2.com (8.14.5/8.14.5) with ESMTP id q9K28P2P014322; Fri, 19 Oct 2012 19:08:25 -0700 (PDT) (envelope-from freebsd@pki2.com) Subject: ZFS hang status update From: Dennis Glatting To: Andriy Gapon Content-Type: text/plain; charset="ISO-8859-1" Date: Fri, 19 Oct 2012 19:08:25 -0700 Message-ID: <1350698905.86715.33.camel@btw.pki2.com> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port Content-Transfer-Encoding: 7bit X-yoursite-MailScanner-Information: Dennis Glatting X-yoursite-MailScanner-ID: q9K28P2P014322 X-yoursite-MailScanner: Found to be clean X-MailScanner-From: freebsd@pki2.com Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 20 Oct 2012 02:08:31 -0000 I applied your debugging patch and that system has been running under load for 43 hours. I have no idea why. That said, some of my prior batch jobs have run for over a month. There was a time when ZFS was fairly stable but took a dive some months ago. From owner-freebsd-fs@FreeBSD.ORG Sat Oct 20 05:38:38 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 3FF2C49F; Sat, 20 Oct 2012 05:38:38 +0000 (UTC) (envelope-from freebsd@pki2.com) Received: from btw.pki2.com (btw.pki2.com [IPv6:2001:470:a:6fd::2]) by mx1.freebsd.org (Postfix) with ESMTP id E2C758FC0C; Sat, 20 Oct 2012 05:38:37 +0000 (UTC) Received: from [127.0.0.1] (localhost [127.0.0.1]) by btw.pki2.com (8.14.5/8.14.5) with ESMTP id q9K5cTMo097673; Fri, 19 Oct 2012 22:38:30 -0700 (PDT) (envelope-from freebsd@pki2.com) Subject: Re: ZFS hang status update From: Dennis Glatting To: Andriy Gapon In-Reply-To: <1350698905.86715.33.camel@btw.pki2.com> References: <1350698905.86715.33.camel@btw.pki2.com> Content-Type: text/plain; charset="ISO-8859-1" Date: Fri, 19 Oct 2012 22:38:29 -0700 Message-ID: <1350711509.86715.59.camel@btw.pki2.com> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port Content-Transfer-Encoding: 7bit X-yoursite-MailScanner-Information: Dennis Glatting X-yoursite-MailScanner-ID: q9K5cTMo097673 X-yoursite-MailScanner: Found to be clean X-MailScanner-From: freebsd@pki2.com Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 20 Oct 2012 05:38:38 -0000 On Fri, 2012-10-19 at 19:08 -0700, Dennis Glatting wrote: > I applied your debugging patch and that system has been running under > load for 43 hours. I have no idea why. > > That said, some of my prior batch jobs have run for over a month. There > was a time when ZFS was fairly stable but took a dive some months ago. > Boom. Roughly 49 hours, adding a SFTP transfer (60GB off the pool disk-1) and a ls (a directory in the disk-1 pool) in a while loop. My pools: mc# zpool status pool: disk-1 state: ONLINE scan: scrub repaired 0 in 0h38m with 0 errors on Tue Oct 16 16:47:51 2012 config: NAME STATE READ WRITE CKSUM disk-1 ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 da5 ONLINE 0 0 0 da6 ONLINE 0 0 0 da7 ONLINE 0 0 0 da2 ONLINE 0 0 0 da3 ONLINE 0 0 0 da4 ONLINE 0 0 0 cache da0 ONLINE 0 0 0 errors: No known data errors pool: disk-2 state: ONLINE scan: scrub repaired 0 in 0h6m with 0 errors on Tue Oct 16 17:05:43 2012 config: NAME STATE READ WRITE CKSUM disk-2 ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 da9 ONLINE 0 0 0 da10 ONLINE 0 0 0 errors: No known data errors camcontrol output (static linked and stored in a md): mc# /mnt/camcontrol tags da0 -v (no output. session hung.) mc# /mnt/camcontrol tags da1 -v (** swap disk **) (pass1:mps0:0:5:0): dev_openings 255 (pass1:mps0:0:5:0): dev_active 0 (pass1:mps0:0:5:0): devq_openings 255 (pass1:mps0:0:5:0): devq_queued 0 (pass1:mps0:0:5:0): held 0 (pass1:mps0:0:5:0): mintags 2 (pass1:mps0:0:5:0): maxtags 255 mc# /mnt/camcontrol tags da2 -v (pass2:mps0:0:6:0): dev_openings 245 (pass2:mps0:0:6:0): dev_active 10 (pass2:mps0:0:6:0): devq_openings 245 (pass2:mps0:0:6:0): devq_queued 0 (pass2:mps0:0:6:0): held 0 (pass2:mps0:0:6:0): mintags 2 (pass2:mps0:0:6:0): maxtags 255 mc# /mnt/camcontrol tags da3 -v (pass3:mps0:0:7:0): dev_openings 245 (pass3:mps0:0:7:0): dev_active 10 (pass3:mps0:0:7:0): devq_openings 245 (pass3:mps0:0:7:0): devq_queued 0 (pass3:mps0:0:7:0): held 0 (pass3:mps0:0:7:0): mintags 2 (pass3:mps0:0:7:0): maxtags 255 mc# /mnt/camcontrol tags da4 -v (pass4:mps0:0:8:0): dev_openings 245 (pass4:mps0:0:8:0): dev_active 10 (pass4:mps0:0:8:0): devq_openings 245 (pass4:mps0:0:8:0): devq_queued 0 (pass4:mps0:0:8:0): held 0 (pass4:mps0:0:8:0): mintags 2 (pass4:mps0:0:8:0): maxtags 255 mc# /mnt/camcontrol tags da5 -v (pass5:mps0:0:9:0): dev_openings 245 (pass5:mps0:0:9:0): dev_active 10 (pass5:mps0:0:9:0): devq_openings 245 (pass5:mps0:0:9:0): devq_queued 0 (pass5:mps0:0:9:0): held 0 (pass5:mps0:0:9:0): mintags 2 (pass5:mps0:0:9:0): maxtags 255 mc# /mnt/camcontrol tags da6 -v (pass6:mps0:0:10:0): dev_openings 245 (pass6:mps0:0:10:0): dev_active 10 (pass6:mps0:0:10:0): devq_openings 245 (pass6:mps0:0:10:0): devq_queued 0 (pass6:mps0:0:10:0): held 0 (pass6:mps0:0:10:0): mintags 2 (pass6:mps0:0:10:0): maxtags 255 mc# /mnt/camcontrol tags da7 -v (pass7:mps0:0:11:0): dev_openings 245 (pass7:mps0:0:11:0): dev_active 10 (pass7:mps0:0:11:0): devq_openings 245 (pass7:mps0:0:11:0): devq_queued 0 (pass7:mps0:0:11:0): held 0 (pass7:mps0:0:11:0): mintags 2 (pass7:mps0:0:11:0): maxtags 255 mc# /mnt/camcontrol tags da8 -v (** OS hdw RAID1 **) (pass8:mps1:0:0:0): dev_openings 245 (pass8:mps1:0:0:0): dev_active 10 (pass8:mps1:0:0:0): devq_openings 245 (pass8:mps1:0:0:0): devq_queued 0 (pass8:mps1:0:0:0): held 0 (pass8:mps1:0:0:0): mintags 2 (pass8:mps1:0:0:0): maxtags 255 mc# /mnt/camcontrol tags da9 -v (pass9:mps1:0:9:0): dev_openings 251 (pass9:mps1:0:9:0): dev_active 4 (pass9:mps1:0:9:0): devq_openings 251 (pass9:mps1:0:9:0): devq_queued 0 (pass9:mps1:0:9:0): held 0 (pass9:mps1:0:9:0): mintags 2 (pass9:mps1:0:9:0): maxtags 255 mc# /mnt/camcontrol tags da10 -v (pass10:mps1:0:11:0): dev_openings 251 (pass10:mps1:0:11:0): dev_active 4 (pass10:mps1:0:11:0): devq_openings 251 (pass10:mps1:0:11:0): devq_queued 0 (pass10:mps1:0:11:0): held 0 (pass10:mps1:0:11:0): mintags 2 (pass10:mps1:0:11:0): maxtags 255 I did not run procstat before reboot. I wasn't sure if that was redundant information from my prior email. This is da0 (the cache --SSD) on which camcontrol hanged. It is on the same controller. da0 at mps0 bus 0 scbus0 target 3 lun 0 da0: Fixed Direct Access SCSI-6 device da0: 600.000MB/s transfers da0: Command Queueing enabled da0: 244198MB (500118192 512 byte sectors: 255H 63S/T 31130C) From owner-freebsd-fs@FreeBSD.ORG Sat Oct 20 07:18:13 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 4401865E for ; Sat, 20 Oct 2012 07:18:13 +0000 (UTC) (envelope-from david@wimsey.us) Received: from mail-vb0-f54.google.com (mail-vb0-f54.google.com [209.85.212.54]) by mx1.freebsd.org (Postfix) with ESMTP id E2AE78FC0C for ; Sat, 20 Oct 2012 07:18:12 +0000 (UTC) Received: by mail-vb0-f54.google.com with SMTP id v11so1666109vbm.13 for ; Sat, 20 Oct 2012 00:18:11 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=from:content-type:content-transfer-encoding:subject:message-id:date :to:mime-version:x-mailer:x-gm-message-state; bh=D3VCcTqVOTY5RKZTCtAwu78nTwxcYQVjQETJgpaK08M=; b=nfZuTetAgpnHJrvo4MLFaElLigUattrvPIycZqlu553///7HhLfzvSzMOcGFj0UCr1 XlNqlvfwpLbxAy7C+7L48SKQrzWuC/YmM5YYMwxKDaMzhkzFZX7FI/BoWvj89L7Jc4dW J8IdCtBsuj1vKp7sIJfpdjFqrgkI6ZkX6vunDOpWqfaZgAHVmmizJQCrJGsQtWbruefo D+hjbeF1D8SU7tvkrSyvKojyqu4CnHmrsSF44h2eZmN8CF+I8zt9knii/kVTn1Z2+ZYs DpqyMNSpf6vTzyHSKOq6JYGuRwz8fc+Lw1N/S1pZUErvk96tUhmEPLQAqFGkOjhm7qdT TJPA== Received: by 10.58.171.4 with SMTP id aq4mr4842162vec.14.1350717491881; Sat, 20 Oct 2012 00:18:11 -0700 (PDT) Received: from [10.27.1.242] (cpe-107-015-155-065.nc.res.rr.com. [107.15.155.65]) by mx.google.com with ESMTPS id g5sm3488104vez.6.2012.10.20.00.18.11 (version=TLSv1/SSLv3 cipher=OTHER); Sat, 20 Oct 2012 00:18:11 -0700 (PDT) From: David Wimsey Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Subject: Memory consumption after turning off dedup Message-Id: <44510CBA-2D95-4BDF-8AEE-61727760321C@wimsey.us> Date: Sat, 20 Oct 2012 03:18:10 -0400 To: "freebsd-fs@freebsd.org" Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\)) X-Mailer: Apple Mail (2.1499) X-Gm-Message-State: ALoCoQmhZQ/guFZhucxvKiYeVLIYS+sRPqjcK92Ndu+DFzetO/dPKbN6t7LfJY9fsK4Dj/EE1t5Z X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 20 Oct 2012 07:18:13 -0000 I've had dedup on for a while and everything was good until the feeble = machine I have as a file server couldn't deal with the memory = requirements of dedup. I solved the problem by adding more RAM, = imported the pools on the machine and turned off dedup. I had a ratio = of less than 2x, and the main savings were on virtual machine disks = which I want maximum performance for. Does the memory consumption due to the DDT go away when you turn dedup = off or do I need to do a send/recv on it? I assume that once the block is deduced and written to disk its not = really any different than a blocks associated with a snapshot, is that = correct? I also assume that there is no performance penalties on reads, only = writes since it (with dedup on) has to check the dedup table to see if = it is a duplicate, is that correct? From owner-freebsd-fs@FreeBSD.ORG Sat Oct 20 07:37:58 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id A36ABE03 for ; Sat, 20 Oct 2012 07:37:58 +0000 (UTC) (envelope-from daniel@digsys.bg) Received: from smtp-sofia.digsys.bg (smtp-sofia.digsys.bg [193.68.3.230]) by mx1.freebsd.org (Postfix) with ESMTP id 2070D8FC19 for ; Sat, 20 Oct 2012 07:37:56 +0000 (UTC) Received: from surfer-172-30-97-1-proxy.surfnet.iacbox ([193.43.158.229]) (authenticated bits=0) by smtp-sofia.digsys.bg (8.14.5/8.14.5) with ESMTP id q9K7UIAj002727 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO); Sat, 20 Oct 2012 10:30:25 +0300 (EEST) (envelope-from daniel@digsys.bg) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\)) Subject: Re: Memory consumption after turning off dedup From: Daniel Kalchev In-Reply-To: <44510CBA-2D95-4BDF-8AEE-61727760321C@wimsey.us> Date: Sat, 20 Oct 2012 10:30:18 +0300 Content-Transfer-Encoding: quoted-printable Message-Id: <28A26432-EB80-41DF-82B2-B1D8F998D9A5@digsys.bg> References: <44510CBA-2D95-4BDF-8AEE-61727760321C@wimsey.us> To: David Wimsey X-Mailer: Apple Mail (2.1499) Cc: "freebsd-fs@freebsd.org" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 20 Oct 2012 07:37:58 -0000 The only way I am aware of to get rid of the DDT (what is consuming = memory) is to turn dedup off, then copy all the data in again -- if you = have space, you could copy within the same zpool, then remove the old = data. Maybe it doesn't remove all the DDT entries, such as for metadata, = but at least most of it will be gone. Turning off dedup only prevents = deduplication of data you write after that act. Daniel On Oct 20, 2012, at 10:18 AM, David Wimsey wrote: > I've had dedup on for a while and everything was good until the feeble = machine I have as a file server couldn't deal with the memory = requirements of dedup. I solved the problem by adding more RAM, = imported the pools on the machine and turned off dedup. I had a ratio = of less than 2x, and the main savings were on virtual machine disks = which I want maximum performance for. >=20 > Does the memory consumption due to the DDT go away when you turn dedup = off or do I need to do a send/recv on it? >=20 > I assume that once the block is deduced and written to disk its not = really any different than a blocks associated with a snapshot, is that = correct? >=20 > I also assume that there is no performance penalties on reads, only = writes since it (with dedup on) has to check the dedup table to see if = it is a duplicate, is that correct? >=20 > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@FreeBSD.ORG Sat Oct 20 07:41:19 2012 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 8B79F356 for ; Sat, 20 Oct 2012 07:41:19 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id B705F8FC0A for ; Sat, 20 Oct 2012 07:41:18 +0000 (UTC) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id KAA07249; Sat, 20 Oct 2012 10:41:13 +0300 (EEST) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1TPTgH-000Afp-8V; Sat, 20 Oct 2012 10:41:13 +0300 Message-ID: <50825598.3070505@FreeBSD.org> Date: Sat, 20 Oct 2012 10:41:12 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:16.0) Gecko/20121013 Thunderbird/16.0.1 MIME-Version: 1.0 To: Dennis Glatting Subject: Re: ZFS hang status update References: <1350698905.86715.33.camel@btw.pki2.com> <1350711509.86715.59.camel@btw.pki2.com> In-Reply-To: <1350711509.86715.59.camel@btw.pki2.com> X-Enigmail-Version: 1.4.5 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-fs@FreeBSD.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 20 Oct 2012 07:41:19 -0000 on 20/10/2012 08:38 Dennis Glatting said the following: > This is da0 (the cache --SSD) on which camcontrol hanged. It is on the > same controller. Hmm, hanging camcontrol is a bad sign. It would be interesting to get procstat -k information just for the hanging camcontrol process. Also, is it possible to eliminate this disk from the configuration? -- Andriy Gapon From owner-freebsd-fs@FreeBSD.ORG Sat Oct 20 09:10:04 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 3B13DB67 for ; Sat, 20 Oct 2012 09:10:04 +0000 (UTC) (envelope-from dwimsey@rtsz.com) Received: from smtp.rtsz.com (rrcs-24-199-159-90.midsouth.biz.rr.com [24.199.159.90]) by mx1.freebsd.org (Postfix) with ESMTP id 066058FC0A for ; Sat, 20 Oct 2012 09:10:03 +0000 (UTC) Received: from [10.27.1.242] (cpe-107-015-155-065.nc.res.rr.com [107.15.155.65]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) by smtp.rtsz.com (Postfix) with ESMTP id E99581CEBE for ; Sat, 20 Oct 2012 04:49:01 -0400 (EDT) From: David Wimsey Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Subject: gptzfsboot very slow Message-Id: Date: Sat, 20 Oct 2012 04:50:22 -0400 To: "freebsd-fs@freebsd.org" Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\)) X-Mailer: Apple Mail (2.1499) X-RTS-MailScanner-Information: Please contact the System Administrator for more information X-RTS-MailScanner: Found to be clean X-RTS-MailScanner-MCPCheck: X-Spam-Flag: not spam (whitelisted), SpamAssassin (not cached, score=0, required 4, autolearn=not spam) X-RTS-MailScanner-From: dwimsey@rtsz.com X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 20 Oct 2012 09:10:04 -0000 My file server is configured with zfs root based on = http://wiki.freebsd.org/RootOnZFS/GPTZFSBoot/Mirror When booting, after it gets past the BIOS drive C: is disk0 (one line = for each of the 6 drives as expected) part, drops to a new line and the = rotating twiddle starts its bit. At first it moves a long at a almost = normal looking speed, then it starts only ticking away slowly, maybe = once or twice a minute. It appears that its scanning the entire drive or something odd. Its = hard to tell if its doing it on all the drives as half of them are on a = RocketRAID card (configured as JBOD) which doesn't have a LED indicator = attached to it for showing activity. There are a total of 6 drives in the machine. 2 drives are SSDs which are sliced up to provide the root mirror vdev, a = mirrored vdev for the zip if the main pool on the machine and each = provide a slice L2ARC. Some of the remaining space is in a small pool=20= 3 of the HDDs are part of a raidz vdev for my main pool. The remaining HDD is a hot spare. If I remove the HDDs from the system and just let the SSDs handle the = boot, its faster but still far longer than it should be, so when I'm in = a hurry I unplug the 4 HDDs, boot, wait for it to get to the FreeBSD = boot menu, plug the HDDs back in and send it on its marry way. This can not be expected behavior in my mind. Why is it doing so much = disk thrashing when the pools are all perfectly clean.= From owner-freebsd-fs@FreeBSD.ORG Sat Oct 20 09:48:25 2012 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id AE962B57; Sat, 20 Oct 2012 09:48:25 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id B98FE8FC0C; Sat, 20 Oct 2012 09:48:24 +0000 (UTC) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id MAA08173; Sat, 20 Oct 2012 12:48:22 +0300 (EEST) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1TPVfJ-000AoV-Pm; Sat, 20 Oct 2012 12:48:21 +0300 Message-ID: <50827364.3040204@FreeBSD.org> Date: Sat, 20 Oct 2012 12:48:20 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:16.0) Gecko/20121013 Thunderbird/16.0.1 MIME-Version: 1.0 To: freebsd-fs@FreeBSD.org Subject: Re: ZFS snapshot Folder Disappearing References: <503C5F2A.701@FreeBSD.org> <503C633B.2070508@madpilot.net> <503C66C2.5010209@FreeBSD.org> In-Reply-To: X-Enigmail-Version: 1.4.5 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Cc: Mark Felder , Matthew Seaman X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 20 Oct 2012 09:48:25 -0000 Guys, if you still have this issue and consider it a problem or an annoyance, could you please test the following patch? http://people.freebsd.org/~avg/zfs-vfs.diff I am not sure if it would fix the issue, but it certainly should improve many aspects of .zfs handling and so it may help. P.S. Please do not hurry with positive feedback unless you have a sure way to reproduce the issue with current code. -- Andriy Gapon From owner-freebsd-fs@FreeBSD.ORG Sat Oct 20 09:50:01 2012 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 80D39D46 for ; Sat, 20 Oct 2012 09:50:01 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.FreeBSD.org [8.8.178.135]) by mx1.freebsd.org (Postfix) with ESMTP id 66DCF8FC08 for ; Sat, 20 Oct 2012 09:50:01 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q9K9o1gN028134 for ; Sat, 20 Oct 2012 09:50:01 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q9K9o1SA028112; Sat, 20 Oct 2012 09:50:01 GMT (envelope-from gnats) Date: Sat, 20 Oct 2012 09:50:01 GMT Message-Id: <201210200950.q9K9o1SA028112@freefall.freebsd.org> To: freebsd-fs@FreeBSD.org Cc: From: Martin Birgmeier Subject: Re: kern/136865: [nfs] [patch] NFS exports atomic and on-the-fly atomic updates X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: Martin Birgmeier List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 20 Oct 2012 09:50:01 -0000 The following reply was made to PR kern/136865; it has been noted by GNATS. From: Martin Birgmeier To: bug-followup@FreeBSD.org, simon@comsys.ntu-kpi.kiev.ua Cc: Subject: Re: kern/136865: [nfs] [patch] NFS exports atomic and on-the-fly atomic updates Date: Sat, 20 Oct 2012 11:43:17 +0200 Andrey, I'd really like to use this. However, I need to use it with FreeBSD 7.4, 8.2, and 9.0 (and 9.1 in the near future); I tried to backport your changes, but this turned out to be too difficult for me. I believe your work could be more easily adopted (even into the core FreeBSD sources) provided that - patches for all supported branches of FreeBSD were available and - there existed a simple knob in rc.conf where one switches between the old mountd and the new nfse I guess that for the latter you'd also need to introduce some compatibility shims into your kernel changes, such that a single kernel could support both methods. What is your opinion? Regards, Martin p.s. I have seen that just a few days a workaround was committed to suspend the nfs daemons while mountd reloads the exports list, but this is just a stopgap (as mentioned in the commit message). I'd much more prefer nfse. From owner-freebsd-fs@FreeBSD.ORG Sat Oct 20 13:34:31 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 16A932EE for ; Sat, 20 Oct 2012 13:34:31 +0000 (UTC) (envelope-from hal@elizium.za.net) Received: from squishy.elizium.za.net (squishy.elizium.za.net [80.68.90.178]) by mx1.freebsd.org (Postfix) with ESMTP id C900C8FC08 for ; Sat, 20 Oct 2012 13:34:30 +0000 (UTC) Received: from squishy.elizium.za.net (squishy.elizium.za.net [80.68.90.178]) by squishy.elizium.za.net (Postfix) with ESMTPSA id 3ED2F814C; Sat, 20 Oct 2012 15:25:01 +0200 (SAST) Date: Sat, 20 Oct 2012 15:24:59 +0200 From: Hugo Lombard To: David Wimsey Subject: Re: gptzfsboot very slow Message-ID: <20121020132459.GE18399@squishy.elizium.za.net> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.20 (2009-06-14) Cc: "freebsd-fs@freebsd.org" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 20 Oct 2012 13:34:31 -0000 On Sat, Oct 20, 2012 at 04:50:22AM -0400, David Wimsey wrote: > My file server is configured with zfs root based on > http://wiki.freebsd.org/RootOnZFS/GPTZFSBoot/Mirror > > When booting, after it gets past the BIOS drive C: is disk0 (one line > for each of the 6 drives as expected) part, drops to a new line and > the rotating twiddle starts its bit. At first it moves a long at a > almost normal looking speed, then it starts only ticking away slowly, > maybe once or twice a minute. > Hi I've noticed something similar on my side, with eight disks. I've been meaning to try out these patches: http://svnweb.freebsd.org/base?view=revision&sortby=date&revision=239068 mentioned in this PR: http://www.freebsd.org/cgi/query-pr.cgi?pr=147560 Hope that helps. Regards -- Hugo Lombard From owner-freebsd-fs@FreeBSD.ORG Sat Oct 20 14:45:57 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id C798E376; Sat, 20 Oct 2012 14:45:57 +0000 (UTC) (envelope-from dg17@penx.com) Received: from btw.pki2.com (btw.pki2.com [IPv6:2001:470:a:6fd::2]) by mx1.freebsd.org (Postfix) with ESMTP id 90B7A8FC16; Sat, 20 Oct 2012 14:45:57 +0000 (UTC) Received: from [127.0.0.1] (localhost [127.0.0.1]) by btw.pki2.com (8.14.5/8.14.5) with ESMTP id q9KEjoJE031200; Sat, 20 Oct 2012 07:45:50 -0700 (PDT) (envelope-from dg17@penx.com) Subject: Re: ZFS hang status update From: Dennis Glatting To: Andriy Gapon In-Reply-To: <50825598.3070505@FreeBSD.org> References: <1350698905.86715.33.camel@btw.pki2.com> <1350711509.86715.59.camel@btw.pki2.com> <50825598.3070505@FreeBSD.org> Content-Type: text/plain; charset="us-ascii" Date: Sat, 20 Oct 2012 07:45:49 -0700 Message-ID: <1350744349.88577.10.camel@btw.pki2.com> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port Content-Transfer-Encoding: 7bit X-yoursite-MailScanner-Information: Dennis Glatting X-yoursite-MailScanner-ID: q9KEjoJE031200 X-yoursite-MailScanner: Found to be clean X-MailScanner-From: dg17@penx.com Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: dg17@penx.com List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 20 Oct 2012 14:45:57 -0000 On Sat, 2012-10-20 at 10:41 +0300, Andriy Gapon wrote: > on 20/10/2012 08:38 Dennis Glatting said the following: > > This is da0 (the cache --SSD) on which camcontrol hanged. It is on the > > same controller. > > Hmm, hanging camcontrol is a bad sign. It would be interesting to get procstat > -k information just for the hanging camcontrol process. > Also, is it possible to eliminate this disk from the configuration? > Eliminate disk: yes. However the device remains in the system (hope that is ok -- will remove if you think best). Config now: mc# zpool status disk-1 pool: disk-1 state: ONLINE scan: scrub repaired 0 in 0h38m with 0 errors on Tue Oct 16 16:47:51 2012 config: NAME STATE READ WRITE CKSUM disk-1 ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 da5 ONLINE 0 0 0 da6 ONLINE 0 0 0 da7 ONLINE 0 0 0 da2 ONLINE 0 0 0 da3 ONLINE 0 0 0 da4 ONLINE 0 0 0 errors: No known data errors I'll reboot and start the job anew. If camcontrol hangs again, I'll procstat it. From owner-freebsd-fs@FreeBSD.ORG Sat Oct 20 15:49:23 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 3277B13D; Sat, 20 Oct 2012 15:49:23 +0000 (UTC) (envelope-from freebsd@penx.com) Received: from btw.pki2.com (btw.pki2.com [IPv6:2001:470:a:6fd::2]) by mx1.freebsd.org (Postfix) with ESMTP id 8FE708FC0A; Sat, 20 Oct 2012 15:49:22 +0000 (UTC) Received: from [127.0.0.1] (localhost [127.0.0.1]) by btw.pki2.com (8.14.5/8.14.5) with ESMTP id q9KFnHXG055574; Sat, 20 Oct 2012 08:49:17 -0700 (PDT) (envelope-from freebsd@penx.com) Subject: Re: ZFS hang status update From: Dennis Glatting To: Andriy Gapon In-Reply-To: <50825598.3070505@FreeBSD.org> References: <1350698905.86715.33.camel@btw.pki2.com> <1350711509.86715.59.camel@btw.pki2.com> <50825598.3070505@FreeBSD.org> Content-Type: multipart/mixed; boundary="=-T3qbQpcvQzfHgpDvLbi3" Date: Sat, 20 Oct 2012 08:49:17 -0700 Message-ID: <1350748157.88577.15.camel@btw.pki2.com> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port X-yoursite-MailScanner-Information: Dennis Glatting X-yoursite-MailScanner-ID: q9KFnHXG055574 X-yoursite-MailScanner: Found to be clean X-MailScanner-From: freebsd@penx.com X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 20 Oct 2012 15:49:23 -0000 --=-T3qbQpcvQzfHgpDvLbi3 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit On Sat, 2012-10-20 at 10:41 +0300, Andriy Gapon wrote: > on 20/10/2012 08:38 Dennis Glatting said the following: > > This is da0 (the cache --SSD) on which camcontrol hanged. It is on the > > same controller. > > Hmm, hanging camcontrol is a bad sign. It would be interesting to get procstat > -k information just for the hanging camcontrol process. > Also, is it possible to eliminate this disk from the configuration? > Attached (I hope but also found here: http://www.pki2.com/zfs_stats_efficiency-week.png) is a munin graph of "ZFS ARC Efficiency - by week" for the server I have been talking about. You will notice before each crash the prefetch efficiency is going down. This graph is for daily graph. http://www.pki2.com/zfs_stats_efficiency-day.png --=-T3qbQpcvQzfHgpDvLbi3-- From owner-freebsd-fs@FreeBSD.ORG Sat Oct 20 16:04:35 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 520876C4; Sat, 20 Oct 2012 16:04:35 +0000 (UTC) (envelope-from artemb@gmail.com) Received: from mail-lb0-f182.google.com (mail-lb0-f182.google.com [209.85.217.182]) by mx1.freebsd.org (Postfix) with ESMTP id 8D25F8FC0C; Sat, 20 Oct 2012 16:04:34 +0000 (UTC) Received: by mail-lb0-f182.google.com with SMTP id b5so1167445lbd.13 for ; Sat, 20 Oct 2012 09:04:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=qLpQAEIqoDU3au/8LZjSGP4J3hjCg8WEp4OVma32acM=; b=tQF3OqZ0WSd0PqIk6J4ICf2LeKum8T5yBN+IwFOLyrum7B33h7LcWCCXj8wnSLN0IF ZgkEB5FbyY0hClVGJ7ScTnmF1R+Mm7jETmfoeLhgY5cBIdRDYO/DrzNFZ4xwGry09lde F3DfHFe7AC7dygZj0OUdE/YFCCcc/1tf/4Xv/PJ7CFlYmlcWwjN2MKlH7+Kt34v6r2k4 MjY2i0blwAwz+FzW8qqHQ+/nOYhzMDixfAZ3RhOoBdFMchTHXALILgWUsifdCRjBzSWc 8LslTuBWMTRlP68lHQtwkawVNgR6uuM9z7IylULu/10htgu3Xn+XOjPi6ADkGkd3J9/9 hF5A== MIME-Version: 1.0 Received: by 10.112.29.9 with SMTP id f9mr1840212lbh.22.1350749073251; Sat, 20 Oct 2012 09:04:33 -0700 (PDT) Sender: artemb@gmail.com Received: by 10.112.27.161 with HTTP; Sat, 20 Oct 2012 09:04:33 -0700 (PDT) In-Reply-To: <1350711509.86715.59.camel@btw.pki2.com> References: <1350698905.86715.33.camel@btw.pki2.com> <1350711509.86715.59.camel@btw.pki2.com> Date: Sat, 20 Oct 2012 09:04:33 -0700 X-Google-Sender-Auth: ASKwcj0ZuYF5k8Z43bcne2fE3us Message-ID: Subject: Re: ZFS hang status update From: Artem Belevich To: Dennis Glatting Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-fs@freebsd.org, Andriy Gapon X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 20 Oct 2012 16:04:35 -0000 On Fri, Oct 19, 2012 at 10:38 PM, Dennis Glatting wrote: > This is da0 (the cache --SSD) on which camcontrol hanged. It is on the > same controller. > > da0 at mps0 bus 0 scbus0 target 3 lun 0 > da0: Fixed Direct Access SCSI-6 device > da0: 600.000MB/s transfers > da0: Command Queueing enabled > da0: 244198MB (500118192 512 byte sectors: 255H 63S/T 31130C) Hmm.. It may be a coincidence, but at work I do see same 256G M4 SSDs used as L2ARC+swap hanging now and then. In my case they are connected to SATA2 on Supermicro X8DTi-F motherboard (ICH10?) running FreeBSD8-stable/amd64. It does not happen often -- few times for the last half a year or so. I was hoping that new firmware would fix the issue, but firmware rev 000f still hangs. Crucial had released new firmware (010G) few weeks back but I didn't try it yet. --Artem From owner-freebsd-fs@FreeBSD.ORG Sat Oct 20 16:32:41 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx2.freebsd.org (mx2.freebsd.org [69.147.83.53]) by hub.freebsd.org (Postfix) with ESMTP id A6119E70 for ; Sat, 20 Oct 2012 16:32:41 +0000 (UTC) (envelope-from ae@FreeBSD.org) Received: from [127.0.0.1] (hub.FreeBSD.org [8.8.178.136]) by mx2.freebsd.org (Postfix) with ESMTP id 738943B4EA5; Sat, 20 Oct 2012 16:32:40 +0000 (UTC) Message-ID: <5082D202.9010701@FreeBSD.org> Date: Sat, 20 Oct 2012 20:32:02 +0400 From: "Andrey V. Elsukov" User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:15.0) Gecko/20121010 Thunderbird/15.0.1 MIME-Version: 1.0 To: David Wimsey Subject: Re: gptzfsboot very slow References: In-Reply-To: X-Enigmail-Version: 1.4.3 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enigB57FEB64709315C762B2E4C0" Cc: "freebsd-fs@freebsd.org" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 20 Oct 2012 16:32:42 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enigB57FEB64709315C762B2E4C0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable On 20.10.2012 12:50, David Wimsey wrote: > My file server is configured with zfs root based on > http://wiki.freebsd.org/RootOnZFS/GPTZFSBoot/Mirror >=20 > When booting, after it gets past the BIOS drive C: is disk0 (one line > for each of the 6 drives as expected) part, drops to a new line and > the rotating twiddle starts its bit. At first it moves a long at a > almost normal looking speed, then it starts only ticking away slowly, > maybe once or twice a minute. >=20 > It appears that its scanning the entire drive or something odd. Its > hard to tell if its doing it on all the drives as half of them are on > a RocketRAID card (configured as JBOD) which doesn't have a LED > indicator attached to it for showing activity. >=20 > There are a total of 6 drives in the machine. >=20 > 2 drives are SSDs which are sliced up to provide the root mirror > vdev, a mirrored vdev for the zip if the main pool on the machine and > each provide a slice L2ARC. Some of the remaining space is in a > small pool 3 of the HDDs are part of a raidz vdev for my main pool.=20 > The remaining HDD is a hot spare. >=20 > If I remove the HDDs from the system and just let the SSDs handle the > boot, its faster but still far longer than it should be, so when I'm > in a hurry I unplug the 4 HDDs, boot, wait for it to get to the > FreeBSD boot menu, plug the HDDs back in and send it on its marry > way. >=20 > This can not be expected behavior in my mind. Why is it doing so > much disk thrashing when the pools are all perfectly clean. Hi, David. These lines are not from gptzfsboot, but from the loader. You can try the loader(8) from the FreeBSD 10-CURRENT, you may take it from recent snapshot: https://pub.allbsd.org/FreeBSD-snapshots/amd64-amd64/10.0-HEAD-20121006-J= PSNAP/stage/trees/boot/zfsloader --=20 WBR, Andrey V. Elsukov --=20 WBR, Andrey V. Elsukov --------------enigB57FEB64709315C762B2E4C0 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) iQEcBAEBAgAGBQJQgtIhAAoJEAHF6gQQyKF6q1IH+gOvIvfMx6TQNRJZvl2M0WFT oEsZZzBbFYCk7mLxuf92ScbCzoBhiiTgONV0uziVqIhWUNxSsNqtHVHttK0SFq1B uHG1/laZiN5hLNjRYtRAQfN6Nr0mTeH9O4VVJk9chXzhlqcF5r3whpWvHjrKgJhf mvjd3CMbxpBWmTbD0eX4SSMo2+MDS3pmhMXPGVrnoSm0fthUWxVUvqp0WOYbqRMX OkpG0hvTkQ4W4p/7CbsG9FHlyMf+c+k9bOnQcAuDyVgXN6edstH7SWxboGduba+K MKPVZrWl/bpUZdYmngL33SIVMlpDRDpFnzhd+Eij5+5h74EYNsWyPi0rp3JILuk= =JPI6 -----END PGP SIGNATURE----- --------------enigB57FEB64709315C762B2E4C0-- From owner-freebsd-fs@FreeBSD.ORG Sat Oct 20 18:01:40 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 6EEE5879 for ; Sat, 20 Oct 2012 18:01:40 +0000 (UTC) (envelope-from prvs=1640634f34=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id E5FE08FC18 for ; Sat, 20 Oct 2012 18:01:39 +0000 (UTC) Received: from r2d2 ([188.220.16.49]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50000769861.msg for ; Sat, 20 Oct 2012 19:01:31 +0100 X-Spam-Processed: mail1.multiplay.co.uk, Sat, 20 Oct 2012 19:01:31 +0100 (not processed: message from valid local sender) X-MDRemoteIP: 188.220.16.49 X-Return-Path: prvs=1640634f34=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-fs@freebsd.org Message-ID: <225C2989B8B94BE8BCA4BD437055DFA7@multiplay.co.uk> From: "Steven Hartland" To: "David Wimsey" , References: Subject: Re: gptzfsboot very slow Date: Sat, 20 Oct 2012 19:01:25 +0100 MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_NextPart_000_030C_01CDAEF5.4DC6FF00" X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 20 Oct 2012 18:01:40 -0000 This is a multi-part message in MIME format. ------=_NextPart_000_030C_01CDAEF5.4DC6FF00 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit ----- Original Message ----- From: "David Wimsey" > My file server is configured with zfs root based on http://wiki.freebsd.org/RootOnZFS/GPTZFSBoot/Mirror > > When booting, after it gets past the BIOS drive C: is > disk0 (one line for each of the 6 drives as expected) > part, drops to a new line and the rotating twiddle > starts its bit. At first it moves a long at a almost > normal looking speed, then it starts only ticking away > slowly, maybe once or twice a minute. > ... You don't say what version your running but we had the same issue and the attached patch fixes it for us by reducing the amount of slices tasted during the boot process. I believe there's some on going work to improve the boot process which includes reduction in the amount of tastes of disks / partitions which may be a better option but the patch works for us on 8.3-RELEASE. In addition if you have a large amount of memory setting the following in /boot/loader.conf can also significantly reduced your boot time. hw.memtest.tests="0" Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. ------=_NextPart_000_030C_01CDAEF5.4DC6FF00 Content-Type: text/plain; format=flowed; name="zfs-slice-boot.txt"; reply-type=original Content-Transfer-Encoding: quoted-printable Content-Disposition: attachment; filename="zfs-slice-boot.txt" --- sys/boot/zfs/zfs.c.orig 2011-10-20 18:15:29.966685430 +0000=0A= +++ sys/boot/zfs/zfs.c 2011-10-20 18:18:22.291033636 +0000=0A= @@ -45,6 +45,12 @@=0A= =0A= #include "zfsimpl.c"=0A= =0A= +/*=0A= + * For GPT this should be 128 but leads to 50+ second delay in BTX = loader so=0A= + * we use the original 4 pre r198420 by default for the boot process=0A= + */=0A= +#define ZFS_MAX_SLICES 4=0A= +=0A= static int zfs_open(const char *path, struct open_file *f);=0A= static int zfs_write(struct open_file *f, void *buf, size_t size, = size_t *resid);=0A= static int zfs_close(struct open_file *f);=0A= @@ -415,7 +421,7 @@=0A= if (vdev_probe(vdev_read, (void*) (uintptr_t) fd, 0))=0A= close(fd);=0A= =0A= - for (slice =3D 1; slice <=3D 128; slice++) {=0A= + for (slice =3D 1; slice <=3D ZFS_MAX_SLICES; slice++) {=0A= sprintf(devname, "disk%dp%d:", unit, slice);=0A= fd =3D open(devname, O_RDONLY);=0A= if (fd =3D=3D -1) {=0A= ------=_NextPart_000_030C_01CDAEF5.4DC6FF00-- From owner-freebsd-fs@FreeBSD.ORG Sat Oct 20 20:58:04 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 53505D84 for ; Sat, 20 Oct 2012 20:58:04 +0000 (UTC) (envelope-from jurgen.weber@theiconic.com.au) Received: from exprod6og114.obsmtp.com (exprod6og114.obsmtp.com [64.18.1.33]) by mx1.freebsd.org (Postfix) with SMTP id CACF48FC08 for ; Sat, 20 Oct 2012 20:58:03 +0000 (UTC) Received: from mail-da0-f72.google.com ([209.85.210.72]) (using TLSv1) by exprod6ob114.postini.com ([64.18.5.12]) with SMTP ID DSNKUIMQVG/zGAQHLq4US6UjIFyYxr4G0WxS@postini.com; Sat, 20 Oct 2012 13:58:03 PDT Received: by mail-da0-f72.google.com with SMTP id r28so2621112daj.7 for ; Sat, 20 Oct 2012 13:57:56 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:subject:references :in-reply-to:content-type:content-transfer-encoding :x-gm-message-state; bh=JwzE6fQ54GAtt6sOaQMzpgvMXb0THROMaEtDC4rbAWc=; b=YT6oy323CQ/tNmUDTZpvbdvhkjAxpiGJjfEthrKW9PhevmsD2EZrkv7HTkqkYJ/zN3 /TKt4lWZGiw3OZSTc9ObYLFbqaah2IFPtzxkfTYagDoYBl3XQUTx5Jy0XwjnkJIst7lw 89bS3myHwbKa7pHPPANsE0x+E8hapDL/En11yDcn2r0t3u/ho6IqY06jRDV74Q1bZN4d fGcuTawrs9g8ynaBWvRnufAzvUt7zK2vtl6F1p4Dnj8i016DfqptdKpGxA3ECtzS6QPQ r50eN3vKT38/eraiYe40cHTb0rllk9r+hMmEqg2JBweZCm4uz1y2QvBKLxTqPV4OUjdC GQdQ== Received: by 10.69.0.40 with SMTP id av8mr17751003pbd.117.1350766252529; Sat, 20 Oct 2012 13:50:52 -0700 (PDT) Received: by 10.69.0.40 with SMTP id av8mr17750991pbd.117.1350766252390; Sat, 20 Oct 2012 13:50:52 -0700 (PDT) Received: from [10.29.65.2] (247.35.70.115.static.exetel.com.au. [115.70.35.247]) by mx.google.com with ESMTPS id nc10sm3293166pbc.17.2012.10.20.13.50.40 (version=SSLv3 cipher=OTHER); Sat, 20 Oct 2012 13:50:50 -0700 (PDT) Message-ID: <50830EA3.6020001@theiconic.com.au> Date: Sun, 21 Oct 2012 07:50:43 +1100 From: Jurgen Weber User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:16.0) Gecko/20121010 Thunderbird/16.0.1 MIME-Version: 1.0 To: freebsd-fs@freebsd.org Subject: Re: mfi0 timeout error zfs boot mount problem References: <508090E8.4010300@theiconic.com.au> <5081CE05.1010108@theiconic.com.au> In-Reply-To: <5081CE05.1010108@theiconic.com.au> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit X-Gm-Message-State: ALoCoQkHw1WjN2/w7AM1lUitu+VeIJ/M0fW1NiORfGieqhXKjdhHqKpZSN3omhtz4kD1sluiBaJtafhlzVz9uj/nbDmqNTPl5zCAkOo6NcEo61wQagyS0ZM7Bs3+yAuYj1YaMSBK6riJocG46f5TIG9EgvnF47NupQ== X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 20 Oct 2012 20:58:04 -0000 Hi Lastly, is there a way at boot time, some sysctl's or something I can set to bring zfs to a minimalistic state? Turn off features, etc to get this to mount? Any ideas appreciated. Thanks Jurgen On 20/10/2012 9:02 AM, Jurgen Weber wrote: > Guys > > Some more details on this, some insight would be greatly appreciated. > > As my day wore on trying to get this zpool to import or mount I have > learnt a few things. I think over time this issue has came about as > more and more data was added to the file systems. > > Some further details: > > Its a 8 disk raidz pool that the system boots from as well. The disk > are all 2TB. > The server has 16GB Of RAM, I notcied the day before this happen the > server was struggling with its RAM griding to a halt and dumping its RAM. > The issue is not hardware because I found another server (same one) > swapped the harddrives out took another 8GB of RAM and I have the same > problem. > The main data file systems have dedup and gzip compression on. > > I have booted from open/Oracle Solars 11 adn attempted to import and > the Solaris live CD will not import either. In the Solaris system the > disk detach from the system. > > I get the feeling that ZFS is hitting some root limit when attempting > to mount and its not finishing the job. > > Thanks > > Jurgen > > On 19/10/2012 10:29 AM, Jürgen Weber wrote: >> Team >> >> I have googled around for a solution and I see a lot of posts about >> firmware versions and patches for FreeBSD 8.*. >> >> I have a FreeBSD 9.1rc1 system, which was beta1 orginally and has >> been running for months. >> >> Now it will not boot, I get the following: >> >> "Trying to mount root from zfs:tank/root []..... >> mfi0: COMMAND 0Xffffff8000cb83530 TIMEOUT AFTER xxx SECONDS >> (this just repeats). >> >> I have not seen this error before during normal runtime, _only_ >> during boot. >> >> Originally when I had the problem I could boot off a USB stick >> (9.1beta1 or rc1), run a 'zpool import -f tank' and it would work on >> the livecd. Rebooting and the main system would work. >> >> This time this work around does not work for me. When I am on the USB >> stick I can run a 'zpool import' and all of the disk are recognised, >> the pool is recognised and the file system is healthy. >> >> The Card is a H700 PERC, with 12.10.3 firmware in a Dell R515. >> Running FreeBSD 9.1-RC1, latest zfs and zpool versions. >> >> I have tried disabling the cache (mfiutil cache xxx disable). I have >> also gone into the Card settings and changed under advanced settings >> "adaptive forward read" to "read only". >> >> Any help, appreciated. >> >> Thanks >> > From owner-freebsd-fs@FreeBSD.ORG Sat Oct 20 22:02:59 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id D8CABABF; Sat, 20 Oct 2012 22:02:59 +0000 (UTC) (envelope-from dwimsey@rtsz.com) Received: from smtp.rtsz.com (rrcs-24-199-159-90.midsouth.biz.rr.com [24.199.159.90]) by mx1.freebsd.org (Postfix) with ESMTP id 9F2A68FC08; Sat, 20 Oct 2012 22:02:59 +0000 (UTC) Received: from [10.27.1.242] (cpe-107-015-155-065.nc.res.rr.com [107.15.155.65]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) by smtp.rtsz.com (Postfix) with ESMTP id B0A2C1CEBD; Sat, 20 Oct 2012 18:01:33 -0400 (EDT) Content-Type: text/plain; charset=iso-8859-1 Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\)) Subject: Re: gptzfsboot very slow From: David Wimsey In-Reply-To: <5082D202.9010701@FreeBSD.org> Date: Sat, 20 Oct 2012 18:02:55 -0400 Content-Transfer-Encoding: quoted-printable Message-Id: <211EBAB0-5105-4106-A3CF-30E4D08301DF@rtsz.com> References: <5082D202.9010701@FreeBSD.org> To: "Andrey V. Elsukov" X-Mailer: Apple Mail (2.1499) X-RTS-MailScanner-Information: Please contact the System Administrator for more information X-RTS-MailScanner: Found to be clean X-RTS-MailScanner-MCPCheck: X-Spam-Flag: not spam (whitelisted), SpamAssassin (not cached, score=0, required 4, autolearn=not spam) X-RTS-MailScanner-From: dwimsey@rtsz.com Cc: "freebsd-fs@freebsd.org" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 20 Oct 2012 22:03:00 -0000 On Oct 20, 2012, at 12:32 PM, "Andrey V. Elsukov" = wrote: > On 20.10.2012 12:50, David Wimsey wrote: >> My file server is configured with zfs root based on >> http://wiki.freebsd.org/RootOnZFS/GPTZFSBoot/Mirror >>=20 >> When booting, after it gets past the BIOS drive C: is disk0 (one line >> for each of the 6 drives as expected) part, drops to a new line and >> the rotating twiddle starts its bit. At first it moves a long at a >> almost normal looking speed, then it starts only ticking away slowly, >> maybe once or twice a minute. >>=20 >> It appears that its scanning the entire drive or something odd. Its >> hard to tell if its doing it on all the drives as half of them are on >> a RocketRAID card (configured as JBOD) which doesn't have a LED >> indicator attached to it for showing activity. >>=20 >> There are a total of 6 drives in the machine. >>=20 >> 2 drives are SSDs which are sliced up to provide the root mirror >> vdev, a mirrored vdev for the zip if the main pool on the machine and >> each provide a slice L2ARC. Some of the remaining space is in a >> small pool 3 of the HDDs are part of a raidz vdev for my main pool.=20= >> The remaining HDD is a hot spare. >>=20 >> If I remove the HDDs from the system and just let the SSDs handle the >> boot, its faster but still far longer than it should be, so when I'm >> in a hurry I unplug the 4 HDDs, boot, wait for it to get to the >> FreeBSD boot menu, plug the HDDs back in and send it on its marry >> way. >>=20 >> This can not be expected behavior in my mind. Why is it doing so >> much disk thrashing when the pools are all perfectly clean. >=20 > Hi, David. >=20 > These lines are not from gptzfsboot, but from the loader. You can try > the loader(8) from the FreeBSD 10-CURRENT, you may take it from = recent > snapshot: > = https://pub.allbsd.org/FreeBSD-snapshots/amd64-amd64/10.0-HEAD-20121006-JP= SNAP/stage/trees/boot/zfsloader >=20 > --=20 > WBR, Andrey V. Elsukov >=20 > --=20 > WBR, Andrey V. Elsukov >=20 Ding! We have a winner! The new zfsloader fixed the problem, the twiddle keeps spinning fast = and only takes a few seconds before jumping to the boot menu. Thanks! Just out of curiosity, do you know what change fixed it or what exactly = the old loader was doing?= From owner-freebsd-fs@FreeBSD.ORG Sat Oct 20 22:17:24 2012 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id DEEBCEF7 for ; Sat, 20 Oct 2012 22:17:24 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 30A538FC12 for ; Sat, 20 Oct 2012 22:17:23 +0000 (UTC) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id BAA12453; Sun, 21 Oct 2012 01:17:17 +0300 (EEST) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1TPhM5-000BrZ-CP; Sun, 21 Oct 2012 01:17:17 +0300 Message-ID: <508322EC.4080700@FreeBSD.org> Date: Sun, 21 Oct 2012 01:17:16 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:16.0) Gecko/20121013 Thunderbird/16.0.1 MIME-Version: 1.0 To: Dennis Glatting Subject: Re: ZFS hang (system #2) References: <1350698905.86715.33.camel@btw.pki2.com> <1350711509.86715.59.camel@btw.pki2.com> <50825598.3070505@FreeBSD.org> <1350744349.88577.10.camel@btw.pki2.com> <1350765093.86715.69.camel@btw.pki2.com> In-Reply-To: <1350765093.86715.69.camel@btw.pki2.com> X-Enigmail-Version: 1.4.5 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Cc: freebsd-fs@FreeBSD.org, dg17@penx.com X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 20 Oct 2012 22:17:25 -0000 on 20/10/2012 23:31 Dennis Glatting said the following: > The following is from a second system working on a 7TB file (started this > morning), which also hung. However, an important difference is this system's CPU > is slightly over clocked from 3.6GHz to 4.0GHz; However, prior not over clocking > made no difference -- it still hanged. > > This system has a Gigabyte GA-990FXA-UD7 board. To me this again looks like an issue with a stuck zio/bio, and not a deadlock. > bd3# /mnt/camcontrol tags da7 -v (** OS - RAID1 **) > (pass7:mps1:0:0:0): dev_openings 215 > (pass7:mps1:0:0:0): dev_active 40 > (pass7:mps1:0:0:0): devq_openings 215 > (pass7:mps1:0:0:0): devq_queued 0 > (pass7:mps1:0:0:0): held 0 > (pass7:mps1:0:0:0): mintags 2 > (pass7:mps1:0:0:0): maxtags 255 Of all the disks this one looks the most suspicious, of course. Do you have the zio/bio debug patch there and usable kgdb? -- Andriy Gapon From owner-freebsd-fs@FreeBSD.ORG Sat Oct 20 22:38:12 2012 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id CA62AF8; Sat, 20 Oct 2012 22:38:12 +0000 (UTC) (envelope-from freebsd@pki2.com) Received: from btw.pki2.com (btw.pki2.com [IPv6:2001:470:a:6fd::2]) by mx1.freebsd.org (Postfix) with ESMTP id 8B5F48FC0A; Sat, 20 Oct 2012 22:38:12 +0000 (UTC) Received: from [127.0.0.1] (localhost [127.0.0.1]) by btw.pki2.com (8.14.5/8.14.5) with ESMTP id q9KMc40b019100; Sat, 20 Oct 2012 15:38:05 -0700 (PDT) (envelope-from freebsd@pki2.com) Subject: Re: ZFS hang (system #2) From: Dennis Glatting To: Andriy Gapon In-Reply-To: <508322EC.4080700@FreeBSD.org> References: <1350698905.86715.33.camel@btw.pki2.com> <1350711509.86715.59.camel@btw.pki2.com> <50825598.3070505@FreeBSD.org> <1350744349.88577.10.camel@btw.pki2.com> <1350765093.86715.69.camel@btw.pki2.com> <508322EC.4080700@FreeBSD.org> Content-Type: text/plain; charset="ISO-8859-1" Date: Sat, 20 Oct 2012 15:38:04 -0700 Message-ID: <1350772684.86715.72.camel@btw.pki2.com> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port Content-Transfer-Encoding: 7bit X-yoursite-MailScanner-Information: Dennis Glatting X-yoursite-MailScanner-ID: q9KMc40b019100 X-yoursite-MailScanner: Found to be clean X-MailScanner-From: freebsd@pki2.com Cc: freebsd-fs@FreeBSD.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 20 Oct 2012 22:38:12 -0000 On Sun, 2012-10-21 at 01:17 +0300, Andriy Gapon wrote: > on 20/10/2012 23:31 Dennis Glatting said the following: > > The following is from a second system working on a 7TB file (started this > > morning), which also hung. However, an important difference is this system's CPU > > is slightly over clocked from 3.6GHz to 4.0GHz; However, prior not over clocking > > made no difference -- it still hanged. > > > > This system has a Gigabyte GA-990FXA-UD7 board. > > To me this again looks like an issue with a stuck zio/bio, and not a deadlock. > > > bd3# /mnt/camcontrol tags da7 -v (** OS - RAID1 **) > > (pass7:mps1:0:0:0): dev_openings 215 > > (pass7:mps1:0:0:0): dev_active 40 > > (pass7:mps1:0:0:0): devq_openings 215 > > (pass7:mps1:0:0:0): devq_queued 0 > > (pass7:mps1:0:0:0): held 0 > > (pass7:mps1:0:0:0): mintags 2 > > (pass7:mps1:0:0:0): maxtags 255 > > Of all the disks this one looks the most suspicious, of course. > > Do you have the zio/bio debug patch there and usable kgdb? > No but I'll apply it and restart. I was curious whether it would hang on the SSD. The work/stuck process is named "sep". From owner-freebsd-fs@FreeBSD.ORG Sat Oct 20 23:37:47 2012 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 2E7AB1FF; Sat, 20 Oct 2012 23:37:47 +0000 (UTC) (envelope-from freebsd@pki2.com) Received: from btw.pki2.com (btw.pki2.com [IPv6:2001:470:a:6fd::2]) by mx1.freebsd.org (Postfix) with ESMTP id C45E28FC08; Sat, 20 Oct 2012 23:37:46 +0000 (UTC) Received: from [127.0.0.1] (localhost [127.0.0.1]) by btw.pki2.com (8.14.5/8.14.5) with ESMTP id q9KNbd7Q043186; Sat, 20 Oct 2012 16:37:39 -0700 (PDT) (envelope-from freebsd@pki2.com) Subject: Re: ZFS hang (system #2) From: Dennis Glatting To: Andriy Gapon In-Reply-To: <508322EC.4080700@FreeBSD.org> References: <1350698905.86715.33.camel@btw.pki2.com> <1350711509.86715.59.camel@btw.pki2.com> <50825598.3070505@FreeBSD.org> <1350744349.88577.10.camel@btw.pki2.com> <1350765093.86715.69.camel@btw.pki2.com> <508322EC.4080700@FreeBSD.org> Content-Type: text/plain; charset="ISO-8859-1" Date: Sat, 20 Oct 2012 16:37:39 -0700 Message-ID: <1350776259.86715.83.camel@btw.pki2.com> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port Content-Transfer-Encoding: 7bit X-yoursite-MailScanner-Information: Dennis Glatting X-yoursite-MailScanner-ID: q9KNbd7Q043186 X-yoursite-MailScanner: Found to be clean X-MailScanner-From: freebsd@pki2.com Cc: freebsd-fs@FreeBSD.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 20 Oct 2012 23:37:47 -0000 On Sun, 2012-10-21 at 01:17 +0300, Andriy Gapon wrote: > on 20/10/2012 23:31 Dennis Glatting said the following: > > The following is from a second system working on a 7TB file (started this > > morning), which also hung. However, an important difference is this system's CPU > > is slightly over clocked from 3.6GHz to 4.0GHz; However, prior not over clocking > > made no difference -- it still hanged. > > > > This system has a Gigabyte GA-990FXA-UD7 board. > > To me this again looks like an issue with a stuck zio/bio, and not a deadlock. > > > bd3# /mnt/camcontrol tags da7 -v (** OS - RAID1 **) > > (pass7:mps1:0:0:0): dev_openings 215 > > (pass7:mps1:0:0:0): dev_active 40 > > (pass7:mps1:0:0:0): devq_openings 215 > > (pass7:mps1:0:0:0): devq_queued 0 > > (pass7:mps1:0:0:0): held 0 > > (pass7:mps1:0:0:0): mintags 2 > > (pass7:mps1:0:0:0): maxtags 255 > > Of all the disks this one looks the most suspicious, of course. > > Do you have the zio/bio debug patch there and usable kgdb? > In src/sys/dev/mps/mps.c are the following tunables. /* Grab the unit-instance variables */ snprintf(tmpstr, sizeof(tmpstr), "dev.mps.%d.debug_level", device_get_unit(sc->mps_dev)); TUNABLE_INT_FETCH(tmpstr, &sc->mps_debug); snprintf(tmpstr, sizeof(tmpstr), "dev.mps.%d.disable_msix", device_get_unit(sc->mps_dev)); TUNABLE_INT_FETCH(tmpstr, &sc->disable_msix); snprintf(tmpstr, sizeof(tmpstr), "dev.mps.%d.disable_msi", device_get_unit(sc->mps_dev)); TUNABLE_INT_FETCH(tmpstr, &sc->disable_msi); snprintf(tmpstr, sizeof(tmpstr), "dev.mps.%d.max_chains", device_get_unit(sc->mps_dev)); TUNABLE_INT_FETCH(tmpstr, &sc->max_chains); Whose after boot values are: dev.mps.0.debug_level: 4 dev.mps.0.disable_msix: 0 dev.mps.0.disable_msi: 0 dev.mps.0.firmware_version: 14.00.00.00 dev.mps.0.driver_version: 14.00.00.01-fbsd dev.mps.0.io_cmds_active: 0 dev.mps.0.io_cmds_highwater: 60 dev.mps.0.chain_free: 2048 dev.mps.0.chain_free_lowwater: 2047 dev.mps.0.max_chains: 2048 dev.mps.0.chain_alloc_fail: 0 Is there any value to tweaking these?