From owner-freebsd-fs@FreeBSD.ORG Wed Feb 5 05:01:15 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B62AADA1 for ; Wed, 5 Feb 2014 05:01:15 +0000 (UTC) Received: from mail.physics.umn.edu (smtp.spa.umn.edu [128.101.220.4]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 8A6F21D1A for ; Wed, 5 Feb 2014 05:01:15 +0000 (UTC) Received: from c-66-41-25-68.hsd1.mn.comcast.net ([66.41.25.68] helo=[192.168.0.138]) by mail.physics.umn.edu with esmtpsa (TLSv1:CAMELLIA256-SHA:256) (Exim 4.77 (FreeBSD)) (envelope-from ) id 1WAu53-0008Sk-5m for freebsd-fs@freebsd.org; Tue, 04 Feb 2014 22:27:22 -0600 Message-ID: <52F1BDA4.6090504@physics.umn.edu> Date: Tue, 04 Feb 2014 22:27:16 -0600 From: Graham Allan User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20100101 Thunderbird/24.2.0 MIME-Version: 1.0 To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on mrmachenry.spa.umn.edu X-Spam-Level: X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED autolearn=unavailable version=3.3.2 Subject: practical maximum number of drives X-SA-Exim-Version: 4.2 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Feb 2014 05:01:15 -0000 This may well be a question with no real answer but since we're speccing out a new ZFS-based storage system, I've been asked what the maximum number of drives it can support would be (for a hypothetical expansion option). While there are some obvious limits such as SAS addressing, I assume there must be more fundamental ones in the kernel or drivers, and the practical limits will be very different from the hypothetical ones. So far the largest system we've built is using three 45-drive chassis on one SAS2008 (mps) controller, so 135 drives total. Over many months of running we had several drives fail and be replaced, and eventually the OS (9.1) failed to assign new da devices. It was time to patch the system and reboot anyway, which solved it, but we did wonder if we were running into some kind of limit around 150 drives - though I don't see why. Interestingly we initially built this system with each drive chassis on its own SAS2008 HBA, but it ultimately behaved better daisy-chained with only one. I think I saw a hint somewhere this could be to do with interrupt sharing... Thanks for any insights, Graham