From owner-freebsd-current@FreeBSD.ORG Mon Oct 11 20:38:13 2004 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E687C16A4CE for ; Mon, 11 Oct 2004 20:38:13 +0000 (GMT) Received: from apollo.backplane.com (apollo.backplane.com [216.240.41.2]) by mx1.FreeBSD.org (Postfix) with ESMTP id 9E78B43D41 for ; Mon, 11 Oct 2004 20:38:13 +0000 (GMT) (envelope-from dillon@apollo.backplane.com) Received: from apollo.backplane.com (localhost [127.0.0.1]) i9BKcDvA051304; Mon, 11 Oct 2004 13:38:13 -0700 (PDT) (envelope-from dillon@apollo.backplane.com) Received: (from dillon@localhost) by apollo.backplane.com (8.12.9p2/8.12.9/Submit) id i9BKcCWt051290; Mon, 11 Oct 2004 13:38:12 -0700 (PDT) (envelope-from dillon) Date: Mon, 11 Oct 2004 13:38:12 -0700 (PDT) From: Matthew Dillon Message-Id: <200410112038.i9BKcCWt051290@apollo.backplane.com> To: Mikhail Teterin References: <416AE7D7.3030502@murex.com> cc: freebsd-current@FreeBSD.org cc: bde@zeta.org.au Subject: Re: hangs in nbufkv X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Oct 2004 20:38:14 -0000 :Hello! : :While investigating the server's hanging, I noticed some processes in :the `nbufkv' state (even a graceful reboot becomes impossible: "some :processes would not die..."). Quick search brought up links like: :... : :One of our file systems here does, indeed, use large block size (64K, I :think, not sure, how to verify it) -- it is used for storing large :database dumps. Are the bugs, Bruce and Matt are talking about, supposed :to be gone by now (in which case, I can provide more debugging info), or :does this remain a "known problem" and I should simply adopt the :workaround suggested by Bruce in the first link above -- increase :BKVASIZE? Should I also merge the patch posted by Bruce in the last of :the links above, or are there good reasons, it is not in the official tree? : :In the former case, what would anyone need to know to help fix this :problem(s)? : :In the latter -- what is a good BKVASIZE value for an amd64 opteron with :2Gb of memory, intended, primarily, to keep database archives online and :handy? : :Thanks! : : -mi Well, this sort of deadlock ought to be easy to debug if you can obtain a kernel core (and have the associated kernel.debug), but one of the FreeBSD developers would have to track it down, I'm hip deep in other things. The most likely scenario is that either vfs.lobufspace/vfs.hibufspace needs tuning, or vfs.lofreebuffers/vfs.hifreebuffers needs tuning to overcome the fragmentation issue. You could try reducing both vfs.lobufspace and hibufspace somewhat plus increase their spread, and you could also try increasing vfs.lofreebuffers and hifreebuffers and increasing their spread. You can also try reducing vfs.lodirtybuffers and vfs.hidirtybuffers but it is unlike that those are the cause unless they were specifically tuned up. But to be absolutely safe, I would follow Bruce's original suggestion and increase BKVASIZE to 64K, for your particular system. The only caveat with doing that is that is that it drastically reduces the number of buffers available in the system. You can compensate somewhat by increasing the number of buffers in the system (kern.nbuf boot-time kernel environment variable), but then you may run the kernel out of KVM (this is especially true on FreeBSD due to the fact that kmem_map still exists). -- Ultimately these will become non-issues once the buffer cache is moved to a default non-mapping mode for situations where no mapping is needed (e.g. file data buffer -> DMA to/from disk), but you'd have to ask PHK about that vis-a-vie FreeBSD. I have similar plans for DragonFly but nothing is finished yet. -Matt Matthew Dillon