From owner-freebsd-fs@FreeBSD.ORG Mon Jun 27 23:44:16 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A41BA106564A for ; Mon, 27 Jun 2011 23:44:16 +0000 (UTC) (envelope-from gosand1982@yahoo.com) Received: from nm3-vm1.bullet.mail.ne1.yahoo.com (nm3-vm1.bullet.mail.ne1.yahoo.com [98.138.91.53]) by mx1.freebsd.org (Postfix) with SMTP id 6729C8FC1B for ; Mon, 27 Jun 2011 23:44:16 +0000 (UTC) Received: from [98.138.90.52] by nm3.bullet.mail.ne1.yahoo.com with NNFMP; 27 Jun 2011 23:30:50 -0000 Received: from [98.138.87.10] by tm5.bullet.mail.ne1.yahoo.com with NNFMP; 27 Jun 2011 23:30:50 -0000 Received: from [127.0.0.1] by omp1010.mail.ne1.yahoo.com with NNFMP; 27 Jun 2011 23:30:50 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 469577.46038.bm@omp1010.mail.ne1.yahoo.com Received: (qmail 45749 invoked by uid 60001); 27 Jun 2011 23:30:50 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1309217450; bh=Zju11+sUkKL4/rYQxtbBpq3BC1dPBr/aOO0PgWOf+VQ=; h=X-YMail-OSG:Received:X-Mailer:Message-ID:Date:From:Subject:To:MIME-Version:Content-Type; b=JT3T2Mg+iJ1sSgq8+GsO5RLyUG57fZ5uKy6xXPmkO/ttjCNN7A8Xxj34wjeWkC/bjlU+ZkKBojlc2Sd2dYArQZ7ginQ10px3sxKiTBsB+aPkw8v0dycEHDcR8l9xuJIPCqCIm1UfhvVi5lMx/T+ZpBIgcVLSrt9LJuMkIB8cnWQ= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:X-Mailer:Message-ID:Date:From:Subject:To:MIME-Version:Content-Type; b=e/VBA3i8b/g5vjs7B4CLsb02rl762DhVr5uB9yKffFMY/68F9/SQNEookibvsH10rtUBAfubU/eegKlUP1N430kUKRftub9TqU0sMS4aWy3ARUDhFNa7HYZHwH6dgUnSye5O5ZxGM+ECGZDixaq/iyVLknw8c3mLuUIlQ8Ld9ik=; X-YMail-OSG: qcBcqzwVM1lPbZZPNDFr1dDc1hDw0ihWVQd77R9scqkWYNF nU62GG9dOwQ4qi5yIgluf22oe4Jolklcem80fDSircqcrzq9jOWf2catp_xz O3gO8Kjdk4Vj7W.jkEMjoj67CfVuyjvQEaRbKLJ2j36hFeny3u5olJSwh6p3 d3sVjIx0KI5op3NFCHRXaRr1JCPtGI2aWhVJ69OXMUrKtSBJpePYm1NnpGWB ELS3oLurO8N1HKqW.xNH9xXxIol0ALCoMDINQAUvarxyZxBrlSsSqmgjUtSJ u7SmITLrTFW5_NcnaFu3_qFf00Ef7t.bJ6zwTwadsqX6J5Orc8xc- Received: from [173.164.238.34] by web120014.mail.ne1.yahoo.com via HTTP; Mon, 27 Jun 2011 16:30:50 PDT X-Mailer: YahooMailRC/572 YahooMailWebService/0.8.112.307740 Message-ID: <1309217450.43651.YahooMailRC@web120014.mail.ne1.yahoo.com> Date: Mon, 27 Jun 2011 16:30:50 -0700 (PDT) From: George Sanders To: freebsd-fs@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Subject: Improving old-fashioned UFS2 performance with lots of inodes... X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Jun 2011 23:44:16 -0000 I have a very old-fashioned file server running a 12-disk raid6 array on a 3ware 9650SE. 2TB disks, so the size comes out to 18TB. I newfs the raw device with: newfs -i 65535 /dev/xxx and I would consider jumping to 131072 ... that way my fsck should not take any longer than it would with a smaller disk, since there are not any more total inodes. BUT ... with over 100 million inodes on the filesystem, things go slow. Overall throughput is fine, and I have no complaints there, but doing any kind of operations with the files is quite slow. Building a file list with rsync, or doing a cp, or a ln -s of a big dir tree, etc. Let's assume that the architecture is not changing ... it's going to be FreeBSD 8.x, using UFS2, and raid6 on actual spinning (7200rpm) disks. What can I do to speed things up ? Right now I have these in my loader.conf: kern.maxdsiz="4096000000"# for fsck vm.kmem_size="1610612736"# for big rsyncs vm.kmem_size_max="1610612736"# for big rsyncs and I also set: vfs.ufs.dirhash_maxmem=64000000 but that's it. What bugs me is, the drives have 64M cache, and the 3ware controller has 224 MB (or so) but the system itself has 64 GB of RAM ... is there no way to use the RAM to increase performance ? I don't see a way to actually throw hardware resources at UFS2, other than faster disks which are uneconomical for this application ... Yes, 3ware write cache is turned on, and storsave is set to "balanced". Is there anything that can be done ?