From owner-freebsd-stable@FreeBSD.ORG Tue Sep 30 00:17:12 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 8D649116 for ; Tue, 30 Sep 2014 00:17:12 +0000 (UTC) Received: from mail-wg0-x233.google.com (mail-wg0-x233.google.com [IPv6:2a00:1450:400c:c00::233]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 18E32DD5 for ; Tue, 30 Sep 2014 00:17:11 +0000 (UTC) Received: by mail-wg0-f51.google.com with SMTP id b13so5234845wgh.10 for ; Mon, 29 Sep 2014 17:17:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=ZYMQH0zWEdPKPuaUEjsG0hAC3CNTrq8hsPFYx8F9NYM=; b=SZQq/ARN9Bgxbb9ypiz5Sn362dyJ9l5tOIYXZqp5xwdEQGxZyHbiSU9xv+Jr8Abv7t JxZjiuAEwOKbd/NJDHYvCYOtbspDichWMC+kGmXmSFbro3EjBPj7fpggTq8ez7GcIjNw oXcS6azma+KfhbiaWCAiHjuiEb5bNhUzXuPiclM//V2wNdFEQq47e8mc/k/7naJTQGd8 nOaZ1r1f1y5PdrX67mega5PcB9zPrpnvWrjHZJJepyjdqF7/xD+pr4hBogbAuaQRCHix YKdOWd+uRiS6RbqEJCt5YmqeA8cU9uCzfZuIGMCKj+lNmp5nbuwVCzEYeYv0FkEp3231 +ELA== MIME-Version: 1.0 X-Received: by 10.180.107.194 with SMTP id he2mr1483629wib.72.1412036230363; Mon, 29 Sep 2014 17:17:10 -0700 (PDT) Sender: kob6558@gmail.com Received: by 10.27.86.81 with HTTP; Mon, 29 Sep 2014 17:17:10 -0700 (PDT) In-Reply-To: <018BC41041EE4DF589EF5D834AD45BAD@multiplay.co.uk> References: <5429BB41.8080609@denninger.net> <018BC41041EE4DF589EF5D834AD45BAD@multiplay.co.uk> Date: Mon, 29 Sep 2014 17:17:10 -0700 X-Google-Sender-Auth: NFIbLB4zenvgcT8f-HaGrXvTOAM Message-ID: Subject: Re: MPS From: Kevin Oberman To: Steven Hartland Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: FreeBSD-STABLE Mailing List , Karl Denninger X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 30 Sep 2014 00:17:12 -0000 On Mon, Sep 29, 2014 at 1:25 PM, Steven Hartland wrote: > I seem to remember detection being made parallel for speed, as there > can be lots of drives. The Avago guys may be able to shed more light. > > Regards > Steve > > ----- Original Message ----- From: "Karl Denninger" > To: > Sent: Monday, September 29, 2014 9:04 PM > Subject: MPS > > > > Has anyone found a set of configuration settings for the mps driver (LSI > SAS series cards) that gets the following situation under control? > > I have a machine here with one of these cards and a SAS expander. The > expander has 16 ports, the card has 8 -- one connector attached to the > expander. So I have 20 potential drives associated with this adapter. > > The issue is that the driver makes utterly no sense when it comes to > assigning drive designations. In theory it should enumerate in series; > that is, target 0 on the card should be da0, target 1 da1, etc. This > assumes there are no gaps in the attached drives; if there are then > things may move around, which is why some people wire drives. > > Ok so far, but it doesn't behave that way! > > This is the excerpt of what it returns: > > root@Dbms-10:/boot # dmesg|grep mps > mps0: port 0xc000-0xc0ff mem > 0xfbb3c000-0xfbb3ffff,0xfbb40000-0xfbb7ffff irq 30 at device 0.0 on pci3 > mps0: Firmware: 19.00.00.00, Driver: 19.00.00.00-fbsd > mps0: IOCCapabilities: > 1285c > ses0 at mps0 bus 0 scbus0 target 8 lun 0 > da0 at mps0 bus 0 scbus0 target 0 lun 0 > da1 at mps0 bus 0 scbus0 target 1 lun 0 > da2 at mps0 bus 0 scbus0 target 2 lun 0 > da3 at mps0 bus 0 scbus0 target 3 lun 0 > *da4 at mps0 bus 0 scbus0 target 6 lun 0** > **da5 at mps0 bus 0 scbus0 target 7 lun 0* > > Those last two are the drives the card identifies as being on targets 0 > and 1 when looked at in the BIOS. The other four drives (0-3) are on > the SAS expander -- and there's a two drive "gap" in the targets > identified as well which has zippo for logic associated with it. > > Even more-oddly if I swap the plugs -- that is, plug the two boot disks > into the OTHER connector (putatively targets 4-7) and put the SAS > expander on the 0-3 port connector the BIOS *does* show the target shift > but the machine, when it boots, still places those two drives on targets > 6 and 7! > > There is nothing in /boot/loader.conf or /boot/device.hints that relates > to these devices at all. > > I would like to wire down the various ports so Drive Bay #x corresponds > to da[x] but how can you when you don't know what order they will come > up in predicated on either the physical port they're occupying or, for > that matter, what the card's BIOS shows them as? > > I looked in the driver's man page and saw nothing related to this; I get > around it by labeling the drives and then using "glabel list" to see > what da[x] number it grabbed if I need to pull a disk while the machine > is up, but this is a rather-unfriendly way to handle that. > > -- > Karl Denninger > karl@denninger.net > /The Market Ticker/ > Karl, This problem seems to crop up with some controllers, especially when port expanders are used. If at all possible, I really recommend labeling them and using the labels in lieu of device names. I stopped using device names some time ago just because of this. I do recommend using the labeling for the file system involved as opposed to glabel, if it has one. Certainly UFS does and I thought ZFS did, as well, but I am less sure of that as ZFS started to stabilize about the time I stopped dealing with big systems. I still find UFS quite adequate for my baby server and laptop. I do use glabel for swap. In any case, using labeling will give you stable names to put into things like your fstab and not make you worry about the actual device number. And, of course, talking to the manufacturer is also a good idea, espacially when they don't say "FreeBSD? We don't support that.". -- R. Kevin Oberman, Network Engineer, Retired E-mail: rkoberman@gmail.com