From owner-freebsd-questions@FreeBSD.ORG Thu Jun 19 04:47:13 2008 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 82F9810657BD for ; Thu, 19 Jun 2008 04:47:13 +0000 (UTC) (envelope-from pschmehl_lists@tx.rr.com) Received: from mail.stovebolt.com (mail.stovebolt.com [66.221.101.249]) by mx1.freebsd.org (Postfix) with ESMTP id 1D0298FC1E for ; Thu, 19 Jun 2008 04:47:12 +0000 (UTC) (envelope-from pschmehl_lists@tx.rr.com) Received: from [192.168.2.102] (cpe-24-175-90-48.tx.res.rr.com [24.175.90.48]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.stovebolt.com (Postfix) with ESMTPSA id C821011438F for ; Wed, 18 Jun 2008 23:47:04 -0500 (CDT) Date: Wed, 18 Jun 2008 23:47:11 -0500 From: Paul Schmehl To: freebsd-questions@freebsd.org Message-ID: In-Reply-To: <20080619035949.GB8205@shepherd> References: <20080619035949.GB8205@shepherd> X-Mailer: Mulberry/4.0.8 (Mac OS X) X-Munged-Reply-To: To reply - figure it out MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline Subject: Re: /var full X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Paul Schmehl List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Jun 2008 04:47:14 -0000 --On June 18, 2008 11:59:49 PM -0400 Sahil Tandon wrote: > Paul Schmehl wrote: > >> At 10PM (local time) this evening, a server started reporting that /var >> was full. When I ssh'd in to the server to investigate, df said /var >> was at 2% full (5.1G) and dh reported the same (5.1G). >> /var/log/dmesg.today is full of messages listing multiple entries with >> the same inode number followed by one entry listing dd as the culprit. >> >> +pid 730 (mysqld), uid 88 inumber 7089166 on /var: filesystem full > > [...] > >> Was this some sort of temporary glitch? Or something more ominous? >> Why would toor be running dd? Is it some sort of file recovery >> routine triggered by filesystem full messages? > > This appears to be mysql-related: > I gathered that from the error messages. > http://dev.mysql.com/doc/refman/5.0/en/temporary-files.html Hmmm..tmpdir is not defined in /etc/my.cnf, so if I'm reading this right, mysqld *should* use /tmp for its temporary files. This server has a /tmp partition that is 3.2GB, so that should be more than ample space *if* mysqld is really using it. It appears that it may be using /var/tmp instead, which would be incorrect behavior *if* I'm reading their docs right. But this /var partition is 300GB, so that's a really, really huge temporary file. > > Also, what is the output of 'df -i /var'? > # df -i /var/ Filesystem 1K-blocks Used Avail Capacity iused ifree %iused Mounted on /dev/da1s1d 283737842 5397568 255641248 2% 20350 36673664 0% /var > See recent thread on FreeBSD Forums for context: > > http://www.freebsdforums.org/forums/printthread.php?t=58071 Thanks. At least I know I'm not the only one to have run into this oddity. I'm not that knowledgeable of inodes. My understanding is they are destroyed once a file is no longer in use. Is that correct? Is there any sort of history kept of file system activity that would identify what filename was identified by the inumbers listed in dmesg.today? Or is that vain hope? This is a 6.2 RELEASE system. (Looks like it's time to upgrade to 7.0 STABLE.) Paul Schmehl If it isn't already obvious, my opinions are my own and not those of my employer.