From owner-freebsd-hackers@FreeBSD.ORG Tue Sep 2 16:29:06 2003 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2C07C16A4BF; Tue, 2 Sep 2003 16:29:06 -0700 (PDT) Received: from dan.emsphone.com (dan.emsphone.com [199.67.51.101]) by mx1.FreeBSD.org (Postfix) with ESMTP id 1C05343FB1; Tue, 2 Sep 2003 16:29:05 -0700 (PDT) (envelope-from dan@dan.emsphone.com) Received: (from dan@localhost) by dan.emsphone.com (8.12.9/8.12.9) id h82NT2nR044828; Tue, 2 Sep 2003 18:29:02 -0500 (CDT) (envelope-from dan) Date: Tue, 2 Sep 2003 18:29:02 -0500 From: Dan Nelson To: Max Clark Message-ID: <20030902232902.GB98381@dan.emsphone.com> References: <20030902224136.GA98381@dan.emsphone.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-OS: FreeBSD 5.1-CURRENT X-message-flag: Outlook Error User-Agent: Mutt/1.5.4i cc: freebsd-hackers@freebsd.org cc: freebsd-performance@freebsd.org cc: freebsd-questions@freebsd.org Subject: Re: 20TB Storage System X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Sep 2003 23:29:06 -0000 In the last episode (Sep 02), Max Clark said: [ quoting format manually recovered ] > Dan Nelson wrote > > Depends on whether you plan on crashing or not :) According to > > http://lists.freebsd.org/pipermail/freebsd-fs/2003-July/000181.html, > > you may not want to create filesystems over 3TB if you want fsck to > > succeed. I don't know if that's using the default newfs settings > > (which would create an insane number of inodes), though. > > This is a big problem (no pun intended), my smallest requirement is > still 5TB... what would you recommend? The smallest file on the > storage will be 500MB. I'd say try formatting a 5TB filesystem with the values you'd use (use a very large -i; 1048576 maybe?) and see how much memory fsck consumes. I don't know what UFS2's max blocksize is, but a larger blocksize would help too. You should be able to fake enough storage to do the test with mdconfig and some large sparse files. > > To sustain only 30MByte/s across the entire set? Doesn't really > > matter, since even a single disk could do that. > > What would I see better performance with ccd or vinum? So a better > question isn't if I can sustain with 30MByte/s but what would I > expect to maintain? For sequential access to mirrored arrays, your bottleneck will probably be the ATA->FC bridges, since they claim to only do 100MBytes/sec. If your three HBAs are 1gbit, then those will be your bottleneck and you'll be able to do 300MB/s reads, and 150MB/s writes (50% mirror penalty). If they're 2gbit and you have 6 bridges, you'll max out at 600MB/s and 300MB/s. If you want to use vinum raid5, cut those write speeds in half again (25% raid-5 penalty). Theoretically, assuming you can max your FC links and your server can handle the load :) I do mrtg graphs of my fibre switches, and I haven't seen it peak over 80MB/sec through a 1gbit link, but I regularly see 70MB/sec sustained to some Tru64 Alpha servers. I only have external hardware raid, though, so I don't know what kind of penalty ccd/vinum will add on top of that. Shouldn't be too much. -- Dan Nelson dnelson@allantgroup.com