From owner-freebsd-fs@FreeBSD.ORG Mon May 21 19:16:35 2007 Return-Path: X-Original-To: freebsd-fs@freebsd.org Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 75BF016A468 for ; Mon, 21 May 2007 19:16:35 +0000 (UTC) (envelope-from gore_jarold@yahoo.com) Received: from web63006.mail.re1.yahoo.com (web63006.mail.re1.yahoo.com [69.147.96.217]) by mx1.freebsd.org (Postfix) with SMTP id 3D2FA13C4AD for ; Mon, 21 May 2007 19:16:35 +0000 (UTC) (envelope-from gore_jarold@yahoo.com) Received: (qmail 33803 invoked by uid 60001); 21 May 2007 19:16:33 -0000 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding:Message-ID; b=N3cDez3AF3mzdt4BYATqleKfRsT6r1BxgTZ+/f9KyitlECL+FPRxSRXvynThOMQ3/uI9lFPAnc+ShPbFHk/UEjgmtuAW5teNYa5v4tPnCW8tlc8AQqQR6iVOFvG9HASPzi6AEapXyTHBuakOB6Ny932gclJJrR3yFvZzDNw6CWU=; X-YMail-OSG: 9Xolaz8VM1nIxZskhUo7f81KaIN.4V_ayoOMFAyG9rkJkC4XzHcSC0qFtgM20.J6SGhMfzKtaavgp3ZnSP8r17xwzcjFPVESi2gE Received: from [24.118.228.153] by web63006.mail.re1.yahoo.com via HTTP; Mon, 21 May 2007 12:16:33 PDT Date: Mon, 21 May 2007 12:16:33 -0700 (PDT) From: Gore Jarold To: Brooks Davis In-Reply-To: <20070521174818.GA64826@lor.one-eyed-alien.net> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit Message-ID: <475187.33232.qm@web63006.mail.re1.yahoo.com> Cc: freebsd-fs@freebsd.org Subject: Re: VERY frustrated with FreeBSD/UFS stability - please help or comment... X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 May 2007 19:16:35 -0000 --- Brooks Davis wrote: > > a) am I really the only person in the world that > moves > > around millions of inodes throughout the day ? Am > I > > the only person in the world that has ever filled > up a > > snapshotted FS (or a quota'd FS, for that matter) > ? > > Am I the only person in the world that does a mass > > deletion of several hundred thousand inodes > several > > times per day ? > > > > OR: > > > > b) am I just stupid ? Is everyone doing this, and > > there is 3 pages of sysctls and kernel tunes that > > everyone does to their system when they are going > to > > use it this way ? Am I just naive for taking a > > release and paring down GENERIC and attempting to > run > > as-is out of the box without major tuning ? > > > > If so, can I see those tunes/sysctls ? > > > > I am _really_ hoping that it is (b) ... I would > much > > rather look back on all of this frustration as my > own > > fault than have the burden of proving all of this > (as > > I will no doubt be called upon to do). (1) > > > > Thanks. Please add your comments... > > I'd say it's certaintly (a). Consider that a full > source tree contains > a few under 85K files so that's a reasionable bound > on average > workloads. Deliberatly producing a kernel that > required tuning to just > us the APIs without crashing would be stupid and we > wouldn't go it > without a very good reason and very large warnings > all over the place. > Lousy performance might be expected, but crashing > wouldn't be. Ok - your initial comments / impression are reassuring. It's hard to believe that the simple file movements I do are so alien to mainstream use, but I'll accept your judgement. > > (1) just load up 6.2 and cp/rm a few million > inodes > > around. Or turn on quotas and fill your > filesystem > > up. Kaboom. > > It's not clear to me what you mean by "cp/rm a few > million inodes > around." The organization of those inodes into > files and directories > could conceviably have a major impact on the > problem. If you could > provide a script that fails for you, that would > really help. Specifically, I have private departmental fileservers that other fileservers rsync to using Mike Rubel-style rsync snapshots: http://www.mikerubel.org/computers/rsync_snapshots/ This means that the remote system runs a script like this: ssh user@host rm -rf backup.2 ssh user@host mv backup.1 backup.2 ssh user@host cp -al backup.0 backup.1 rsync /files user@host:/backup.0 The /files in question range from .2 to 2.2 million files, all told. This means that when this script runs, it first either deletes OR unlinks up to 2 million items. Then it does a (presumably) zero cost move operation. Then it does a hard-link-creating cp of the same (up to 2 million) items. As I write this, I realize this isn't _totally_ generic, since I am using GNU cp rather than the built-in FreeBSD cp, but that is _truly_ the extent of customization on this system. So that's that. Run that a few times from a few servers and 6.2 locks up. Can't ping. For trivias sake, it does the same thing if you fill up one of its filesystems with quotas turned on. For further trivias sake, 6.1 is (seemingly) not susceptible to this, but it is certainly susceptible to other situations that I regularly produce. What do you think of this (more specific) workload, and are there any tunings that immediately jump to mind ? ____________________________________________________________________________________Yahoo! oneSearch: Finally, mobile search that gives answers, not web links. http://mobile.yahoo.com/mobileweb/onesearch?refer=1ONXIC