From owner-freebsd-questions@FreeBSD.ORG Mon May 17 07:56:54 2004 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id C545616A4CE for ; Mon, 17 May 2004 07:56:54 -0700 (PDT) Received: from dan.emsphone.com (dan.emsphone.com [199.67.51.101]) by mx1.FreeBSD.org (Postfix) with ESMTP id C353A43D2F for ; Mon, 17 May 2004 07:56:52 -0700 (PDT) (envelope-from dan@dan.emsphone.com) Received: (from dan@localhost) by dan.emsphone.com (8.12.10/8.12.10) id i4HEup8E024462; Mon, 17 May 2004 09:56:51 -0500 (CDT) (envelope-from dan) Date: Mon, 17 May 2004 09:56:51 -0500 From: Dan Nelson To: Eric Anderson Message-ID: <20040517145650.GI80376@dan.emsphone.com> References: <40A82BAB.7030005@centtech.com> <20040517032438.GE80376@dan.emsphone.com> <40A8A942.7090504@centtech.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <40A8A942.7090504@centtech.com> X-OS: FreeBSD 5.2-CURRENT X-message-flag: Outlook Error User-Agent: Mutt/1.5.6i cc: freebsd-questions@freebsd.org Subject: Re: Disk full / NFS, df, and du X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 17 May 2004 14:56:54 -0000 In the last episode (May 17), Eric Anderson said: > Dan Nelson wrote: > >In the last episode (May 16), Eric Anderson said: > >>I have a few large NFS file servers, holding about 1Tb of diskspace > >>each. I break those logical disks (it's on a hardware RAID) into > >>partitions, and share them. My users fill up the partitions often > >>enough, and when they do, they rm entire directory trees to free > >>the space. They use du to determine how much space is in a > >>directory and how much they are hogging. > >> > >>The problem I'm having is, after they do the rm's, it doesn't free > >>the disk space. df shows it still being used, but du claims their > >>directories are empty. > >> > >>If I reboot the file server, the space magically appears. > > > >Does a du on server itself show files? How about "lsof +L1"? The > >NFS protocol doesn't allow clients to unlink files they have open, > >so FreeBSD clients (at least) rename open files that are unlinked to > >.nfs##### until the last process closes the file, and then they delete > >it. If you've got unlinked files held open, it's got to be on the > >server itself. > > lsof +L1 shows nothing.. any more ideas? Actually now that I think about it I have seen similar symptoms on my 4.8 servers where a volume would fill up but no amount of deleting would help until a reboot. It happens very infrequently (maybe twice a year), and I assume it's due to a filesystem-full bug in the softupdates code since on 4.x free space isn't really free until softupdates has flushed its updates to disk, and that could take a while. Try dropping the kern.{file,dir,meta}delay sysctls down to like {7,6,5} seconds and see if that does anything for you. -- Dan Nelson dnelson@allantgroup.com