Date: Mon, 29 Sep 2014 17:17:10 -0700 From: Kevin Oberman <rkoberman@gmail.com> To: Steven Hartland <killing@multiplay.co.uk> Cc: FreeBSD-STABLE Mailing List <freebsd-stable@freebsd.org>, Karl Denninger <karl@denninger.net> Subject: Re: MPS Message-ID: <CAN6yY1v2MyOMahcTEagjxKQ33q56MmSwHx87_ZqyVYR5RDH8Mw@mail.gmail.com> In-Reply-To: <018BC41041EE4DF589EF5D834AD45BAD@multiplay.co.uk> References: <5429BB41.8080609@denninger.net> <018BC41041EE4DF589EF5D834AD45BAD@multiplay.co.uk>
next in thread | previous in thread | raw e-mail | index | archive | help
On Mon, Sep 29, 2014 at 1:25 PM, Steven Hartland <killing@multiplay.co.uk> wrote: > I seem to remember detection being made parallel for speed, as there > can be lots of drives. The Avago guys may be able to shed more light. > > Regards > Steve > > ----- Original Message ----- From: "Karl Denninger" <karl@denninger.net> > To: <freebsd-stable@freebsd.org> > Sent: Monday, September 29, 2014 9:04 PM > Subject: MPS > > > > Has anyone found a set of configuration settings for the mps driver (LSI > SAS series cards) that gets the following situation under control? > > I have a machine here with one of these cards and a SAS expander. The > expander has 16 ports, the card has 8 -- one connector attached to the > expander. So I have 20 potential drives associated with this adapter. > > The issue is that the driver makes utterly no sense when it comes to > assigning drive designations. In theory it should enumerate in series; > that is, target 0 on the card should be da0, target 1 da1, etc. This > assumes there are no gaps in the attached drives; if there are then > things may move around, which is why some people wire drives. > > Ok so far, but it doesn't behave that way! > > This is the excerpt of what it returns: > > root@Dbms-10:/boot # dmesg|grep mps > mps0: <LSI SAS2008> port 0xc000-0xc0ff mem > 0xfbb3c000-0xfbb3ffff,0xfbb40000-0xfbb7ffff irq 30 at device 0.0 on pci3 > mps0: Firmware: 19.00.00.00, Driver: 19.00.00.00-fbsd > mps0: IOCCapabilities: > 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc> > ses0 at mps0 bus 0 scbus0 target 8 lun 0 > da0 at mps0 bus 0 scbus0 target 0 lun 0 > da1 at mps0 bus 0 scbus0 target 1 lun 0 > da2 at mps0 bus 0 scbus0 target 2 lun 0 > da3 at mps0 bus 0 scbus0 target 3 lun 0 > *da4 at mps0 bus 0 scbus0 target 6 lun 0** > **da5 at mps0 bus 0 scbus0 target 7 lun 0* > > Those last two are the drives the card identifies as being on targets 0 > and 1 when looked at in the BIOS. The other four drives (0-3) are on > the SAS expander -- and there's a two drive "gap" in the targets > identified as well which has zippo for logic associated with it. > > Even more-oddly if I swap the plugs -- that is, plug the two boot disks > into the OTHER connector (putatively targets 4-7) and put the SAS > expander on the 0-3 port connector the BIOS *does* show the target shift > but the machine, when it boots, still places those two drives on targets > 6 and 7! > > There is nothing in /boot/loader.conf or /boot/device.hints that relates > to these devices at all. > > I would like to wire down the various ports so Drive Bay #x corresponds > to da[x] but how can you when you don't know what order they will come > up in predicated on either the physical port they're occupying or, for > that matter, what the card's BIOS shows them as? > > I looked in the driver's man page and saw nothing related to this; I get > around it by labeling the drives and then using "glabel list" to see > what da[x] number it grabbed if I need to pull a disk while the machine > is up, but this is a rather-unfriendly way to handle that. > > -- > Karl Denninger > karl@denninger.net <mailto:karl@denninger.net> > /The Market Ticker/ > Karl, This problem seems to crop up with some controllers, especially when port expanders are used. If at all possible, I really recommend labeling them and using the labels in lieu of device names. I stopped using device names some time ago just because of this. I do recommend using the labeling for the file system involved as opposed to glabel, if it has one. Certainly UFS does and I thought ZFS did, as well, but I am less sure of that as ZFS started to stabilize about the time I stopped dealing with big systems. I still find UFS quite adequate for my baby server and laptop. I do use glabel for swap. In any case, using labeling will give you stable names to put into things like your fstab and not make you worry about the actual device number. And, of course, talking to the manufacturer is also a good idea, espacially when they don't say "FreeBSD? We don't support that.". -- R. Kevin Oberman, Network Engineer, Retired E-mail: rkoberman@gmail.com
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAN6yY1v2MyOMahcTEagjxKQ33q56MmSwHx87_ZqyVYR5RDH8Mw>