From owner-freebsd-proliant@FreeBSD.ORG Wed Jun 17 10:10:36 2009 Return-Path: Delivered-To: freebsd-proliant@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 31B2F106564A for ; Wed, 17 Jun 2009 10:10:36 +0000 (UTC) (envelope-from gallasch@free.de) Received: from smtp.free.de (smtp.free.de [91.204.6.103]) by mx1.freebsd.org (Postfix) with ESMTP id 8FF0E8FC0A for ; Wed, 17 Jun 2009 10:10:35 +0000 (UTC) (envelope-from gallasch@free.de) Received: (qmail 35992 invoked from network); 17 Jun 2009 11:43:53 +0200 Received: from smtp.free.de (HELO orwell.free.de) (gallasch@free.de@[91.204.4.103]) (envelope-sender ) by smtp.free.de (qmail-ldap-1.03) with AES256-SHA encrypted SMTP for ; 17 Jun 2009 11:43:53 +0200 Message-ID: <4A38BAD8.8070900@free.de> Date: Wed, 17 Jun 2009 11:43:52 +0200 From: Kai Gallasch User-Agent: Thunderbird 2.0.0.21 (Macintosh/20090302) MIME-Version: 1.0 To: freebsd-proliant@freebsd.org X-Enigmail-Version: 0.95.7 OpenPGP: id=1254A186; url=http://home.free.de/kai/1254A186.asc Content-Type: text/plain; charset=ISO-8859-15 Content-Transfer-Encoding: 7bit Subject: ciss and mpt disk devices numbering priority on a DL385 G2 X-BeenThere: freebsd-proliant@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion of FreeBSD on HP ProLiant server platforms." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Jun 2009 10:10:36 -0000 Hello, on a DL385 G2 I needed some JBOD disks and built a SAS controller into the server, because the builtin Compaq P400 controller does not support JBOD. Luckily the 8 x SAS 2.5" drive cage of the server has two SAS connectors on the back, with one addressing the first four SAS disks (A) and the other one disks five to eight (B) So I had the P400 connected to A and the LSI SAS Controller 3442E-R to connector B of the drive cage. The problem now is, that with 7.2 Release the SCSI disks of the LSI SAS controller show up as da0, da1, etc. What I wanted was that disk devices of the ciss controller take precedence over disk devices of the LSI SAS controller (mpt driver) so I have da0 as boot disk on the P400 So I have right now.. mpt0: da0 mpt0: da1 ciss0: da3 ciss0: da4 and not as wanted: ciss0: da0 ciss0: da1 ciss0: da2 mpt0: da3 mpt0: da4 What I already tried: - Putting the 3442E-R into two different PCIe slots What options do I have to get the ciss controlled da0-da3 show up before the mpt devices? Any hint appreciated, Cheers, Kai. ----------------- devices in dmesg ------------- mpt0: port 0x4000-0x40ff mem 0xfddf0000-0xfddf3fff,0xfdde0000-0xfddeffff irq 36 at device 0.0 on pci67 mpt0: [ITHREAD] mpt0: MPI Version=1.5.19.0 mpt0: Capabilities: ( RAID-0 RAID-1E RAID-1 ) mpt0: 0 Active Volumes (2 Max) mpt0: 0 Hidden Drive Members (14 Max) ciss0: port 0x5000-0x50ff mem 0xfdf00000-0xfdffffff,0xfdef0000-0xfdef0fff irq 32 at device 0.0 on pci70 ciss0: [ITHREAD] da0 at mpt0 bus 0 target 8 lun 0 da0: Fixed Direct Access SCSI-5 device da0: 300.000MB/s transfers da0: Command Queueing Enabled da0: 140014MB (286749488 512 byte sectors: 255H 63S/T 17849C) da2 at ciss0 bus 0 target 0 lun 0 da2: Fixed Direct Access SCSI-5 device da2: 135.168MB/s transfers da2: Command Queueing Enabled da2: 26109MB (53472480 512 byte sectors: 255H 32S/T 6553C) da3 at ciss0 bus 0 target 1 lun 0 da3: Fixed Direct Access SCSI-5 device da3: 135.168MB/s transfers da3: Command Queueing Enabled da3: 143357MB (293596800 512 byte sectors: 255H 32S/T 35980C) da1 at mpt0 bus 0 target 9 lun 0 da1: Fixed Direct Access SCSI-5 device da1: 300.000MB/s transfers da1: Command Queueing Enabled da1: 140014MB (286749488 512 byte sectors: 255H 63S/T 17849C) da4 at ciss0 bus 0 target 2 lun 0 da4: Fixed Direct Access SCSI-5 device da4: 135.168MB/s transfers da4: Command Queueing Enabled From owner-freebsd-proliant@FreeBSD.ORG Wed Jun 17 16:07:10 2009 Return-Path: Delivered-To: freebsd-proliant@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A48321065670 for ; Wed, 17 Jun 2009 16:07:10 +0000 (UTC) (envelope-from jcagle@gmail.com) Received: from mail-qy0-f173.google.com (mail-qy0-f173.google.com [209.85.221.173]) by mx1.freebsd.org (Postfix) with ESMTP id 5B73A8FC16 for ; Wed, 17 Jun 2009 16:07:10 +0000 (UTC) (envelope-from jcagle@gmail.com) Received: by qyk3 with SMTP id 3so540702qyk.3 for ; Wed, 17 Jun 2009 09:07:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:cc:content-type; bh=4JI8BY4pFJxDmdpFGonbyE+V6q2sNKsluYw6L5Mq1KA=; b=fpxbpsQxsCMgumCSLxnBXt+ZN41mWFbVzOth/7yL7kShN+sKltLMFQJAr+Df0cyyNS KAxmlpLziqgcAx0KWkmGex6/EhWf9nY7ZN8ybvr561ktFixzE2S9NnAiv91Upv6WyKXf +I6TA9mGW0TDkBrMACuyM4h7ibzteXtR6gqws= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=c5nSPSgY/dzyR7O1SKRWfZiXV/BpndVB3yU1uy119ghgvHieI8yN5lfvq9Ro7w3GYQ OODgbVZtkyVihzzu71vx7gCoGe6Bb+2EVjdohPJCfKHfjxla7qcwpL3h6Jk8G1p1KsGe stVbvnZUY41xHllAQ9MZZZ7QhvpRmrWMDN7Zw= MIME-Version: 1.0 Received: by 10.229.91.13 with SMTP id k13mr44570qcm.98.1245253008587; Wed, 17 Jun 2009 08:36:48 -0700 (PDT) In-Reply-To: <4A38BAD8.8070900@free.de> References: <4A38BAD8.8070900@free.de> Date: Wed, 17 Jun 2009 10:36:48 -0500 Message-ID: <6863f0c90906170836g6ee87f8eq9880a9c462be572b@mail.gmail.com> From: John Cagle To: Kai Gallasch Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-proliant@freebsd.org Subject: Re: ciss and mpt disk devices numbering priority on a DL385 G2 X-BeenThere: freebsd-proliant@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion of FreeBSD on HP ProLiant server platforms." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Jun 2009 16:07:11 -0000 Hello Kai, Have you looked at the options in the BIOS setup utility (RBSU)? There should be an option to select the boot controller (which should be the P400 in your case). If that is already set to the P400, then the problem may be one of PCI enumeration. If FreeBSD loads the mpt driver first, then it may grab da0. Maybe there's a way to force it to load the ciss driver first? John On Wed, Jun 17, 2009 at 4:43 AM, Kai Gallasch wrote: > Hello, > > on a DL385 G2 I needed some JBOD disks and built a SAS controller into > the server, because the builtin Compaq P400 controller does not support > JBOD. > > Luckily the 8 x SAS 2.5" drive cage of the server has two SAS connectors > on the back, with one addressing the first four SAS disks (A) and the > other one disks five to eight (B) > > So I had the P400 connected to A and the LSI SAS Controller 3442E-R to > connector B of the drive cage. > > The problem now is, that with 7.2 Release the SCSI disks of the LSI SAS > controller show up as da0, da1, etc. What I wanted was that disk devices > of the ciss controller take precedence over disk devices of the LSI SAS > controller (mpt driver) so I have da0 as boot disk on the P400 > > So I have right now.. > > mpt0: da0 > mpt0: da1 > ciss0: da3 > ciss0: da4 > > and not as wanted: > > ciss0: da0 > ciss0: da1 > ciss0: da2 > mpt0: da3 > mpt0: da4 > > What I already tried: > > - Putting the 3442E-R into two different PCIe slots > > What options do I have to get the ciss controlled da0-da3 show up before > the mpt devices? > > Any hint appreciated, > > Cheers, > Kai. > > ----------------- devices in dmesg ------------- > > mpt0: port 0x4000-0x40ff mem > 0xfddf0000-0xfddf3fff,0xfdde0000-0xfddeffff irq 36 at device 0.0 on pci67 > mpt0: [ITHREAD] > mpt0: MPI Version=1.5.19.0 > mpt0: Capabilities: ( RAID-0 RAID-1E RAID-1 ) > mpt0: 0 Active Volumes (2 Max) > mpt0: 0 Hidden Drive Members (14 Max) > > ciss0: port 0x5000-0x50ff mem > 0xfdf00000-0xfdffffff,0xfdef0000-0xfdef0fff irq 32 at device 0.0 on pci70 > ciss0: [ITHREAD] > > da0 at mpt0 bus 0 target 8 lun 0 > da0: Fixed Direct Access SCSI-5 device > da0: 300.000MB/s transfers > da0: Command Queueing Enabled > da0: 140014MB (286749488 512 byte sectors: 255H 63S/T 17849C) > > da2 at ciss0 bus 0 target 0 lun 0 > da2: Fixed Direct Access SCSI-5 device > da2: 135.168MB/s transfers > da2: Command Queueing Enabled > da2: 26109MB (53472480 512 byte sectors: 255H 32S/T 6553C) > > da3 at ciss0 bus 0 target 1 lun 0 > da3: Fixed Direct Access SCSI-5 device > da3: 135.168MB/s transfers > da3: Command Queueing Enabled > da3: 143357MB (293596800 512 byte sectors: 255H 32S/T 35980C) > > da1 at mpt0 bus 0 target 9 lun 0 > da1: Fixed Direct Access SCSI-5 device > da1: 300.000MB/s transfers > da1: Command Queueing Enabled > da1: 140014MB (286749488 512 byte sectors: 255H 63S/T 17849C) > > da4 at ciss0 bus 0 target 2 lun 0 > da4: Fixed Direct Access SCSI-5 device > da4: 135.168MB/s transfers > da4: Command Queueing Enabled > > _______________________________________________ > freebsd-proliant@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-proliant > To unsubscribe, send any mail to "freebsd-proliant-unsubscribe@freebsd.org > " > From owner-freebsd-proliant@FreeBSD.ORG Thu Jun 18 16:21:14 2009 Return-Path: Delivered-To: freebsd-proliant@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D16781065675 for ; Thu, 18 Jun 2009 16:21:14 +0000 (UTC) (envelope-from roy@net-vantage.com) Received: from fantasia.lunarpages.com (fantasia.lunarpages.com [67.210.124.90]) by mx1.freebsd.org (Postfix) with ESMTP id B33F68FC13 for ; Thu, 18 Jun 2009 16:21:14 +0000 (UTC) (envelope-from roy@net-vantage.com) Received: from dpc674466036.direcpc.com ([67.44.66.36] helo=[192.168.2.155]) by fantasia.lunarpages.com with esmtpa (Exim 4.69) (envelope-from ) id 1MHK1O-0008HH-GK for freebsd-proliant@freebsd.org; Thu, 18 Jun 2009 08:59:27 -0700 Message-ID: <4A3A648B.9070807@net-vantage.com> Date: Thu, 18 Jun 2009 12:00:11 -0400 From: RA Cohen Organization: Network Advantage LLC User-Agent: Thunderbird 2.0.0.21 (Windows/20090302) MIME-Version: 1.0 To: freebsd-proliant@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - fantasia.lunarpages.com X-AntiAbuse: Original Domain - freebsd.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - net-vantage.com Subject: kernel: ciss0: *** SCSI bus speed downshifted, SCSI port X-BeenThere: freebsd-proliant@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: roy@net-vantage.com List-Id: "Technical discussion of FreeBSD on HP ProLiant server platforms." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Jun 2009 16:21:15 -0000 I'm new to this list but probably should have joined several years ago. I'm an independent consultant and I have placed many off-lease Proliants (lately DL 360 and 380 G3's) in the field running FBSD. They are generally problem free, but I've had one recent case that still puzzles me: I had a large ML 530 G2 in service populated with 36 gb drives, in the same server room as 2 DL380 G3's. I train my clients to observe the drive lights daily if possible. One morning the client noticed one drive on the ML530 showing red, and two drives on one of the DL380's as red (this server was build with two mirrored drives (for os) in one array, and the remaining 4 striped drives (for data) in the other). The third server was normal with no failed drives. I replaced the drive on the 530, the array rebuilt and all was and continues to be, well. Not so for the DL380. The failed drives were each part of different arrays, that was good I thought. They were replaced, the array rebuilt, and we had six green drives. Until the next day when there were one or two failures - different drives - I kept playing around shuffling drives (at this point I knew I had to rebuild the machine and we had very good tape backups thanks to bacula and SDLT drive). The arrays would rebuild, but as soon as almost all the data was restored the array would fail. Long story short, I concluded there were environmental problems that caused the original failures, my feeling was there very well could be damage to the embedded 5i SCSI controller. The server was replaced with a new (used) DL380 G3 and all seems ok until this morning (about a week after replacement) when I see this in the log: Jun 17 18:03:16 dodson1 kernel: ciss0: *** SCSI bus speed downshifted, SCSI port 2 Jun 17 18:08:21 dodson1 kernel: ciss0: *** SCSI bus speed downshifted, SCSI port 2 I've google'd and searched for this kind of message and am still unsure what it means exactly. Does the bus downshift in reaction to a specific problem and then upshift? Or is there a problem lurking waiting to rear its ugly head? I'm also uneducated as to HP/Compaq's pro-active drive monitoring that is integral to these servers. I wish there were a comprehensive summary of hardware-related messages for FreeBSD on Proliants. The two seem to go together quite well in most situations. Thanks in advance for any and all help, -Roy -- Roy A Cohen Network Advantage LLC www.net-vantage.com 413.223.9007 -------------------------------------------------- "Bringing Cost-Saving, State-of-the-Art Technology Solutions to Small and Mid-Size Organizations" From owner-freebsd-proliant@FreeBSD.ORG Sat Jun 20 10:03:05 2009 Return-Path: Delivered-To: freebsd-proliant@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8E6B2106568D for ; Sat, 20 Jun 2009 10:03:05 +0000 (UTC) (envelope-from gallasch@free.de) Received: from smtp.free.de (smtp.free.de [91.204.6.103]) by mx1.freebsd.org (Postfix) with ESMTP id EA8B78FC18 for ; Sat, 20 Jun 2009 10:03:04 +0000 (UTC) (envelope-from gallasch@free.de) Received: (qmail 82593 invoked from network); 20 Jun 2009 12:03:03 +0200 Received: from smtp.free.de (HELO orwell.free.de) (gallasch@free.de@[91.204.4.103]) (envelope-sender ) by smtp.free.de (qmail-ldap-1.03) with AES256-SHA encrypted SMTP for ; 20 Jun 2009 12:03:03 +0200 Message-ID: <4A3CB3D6.6040002@free.de> Date: Sat, 20 Jun 2009 12:03:02 +0200 From: Kai Gallasch User-Agent: Thunderbird 2.0.0.21 (Macintosh/20090302) MIME-Version: 1.0 To: freebsd-proliant@freebsd.org References: <4A38BAD8.8070900@free.de> <6863f0c90906170836g6ee87f8eq9880a9c462be572b@mail.gmail.com> In-Reply-To: <6863f0c90906170836g6ee87f8eq9880a9c462be572b@mail.gmail.com> X-Enigmail-Version: 0.95.7 OpenPGP: id=1254A186; url=http://home.free.de/kai/1254A186.asc Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Subject: Re: ciss and mpt disk devices numbering priority on a DL385 G2 X-BeenThere: freebsd-proliant@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion of FreeBSD on HP ProLiant server platforms." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 20 Jun 2009 10:03:05 -0000 John Cagle wrote: > Have you looked at the options in the BIOS setup utility (RBSU)? There > should be an option to select the boot controller (which should be the > P400 in your case). > > If that is already set to the P400, then the problem may be one of PCI > enumeration. If FreeBSD loads the mpt driver first, then it may grab > da0. Maybe there's a way to force it to load the ciss driver first? Hi John. In the end I came up with some "solution". I built a kernel without support for the LSI HBA (mpt driver). When the server has booted I do a "kldload mpt" and "camcontrol rescan" to bring up the mpt scsi devices. The scsi devices of the mpt controller then show up after those of the smart array (ciss). In my case I need the LSI controller to give me some JBOD devices, so I can go and play with zfs on the server. (hp/compaq smart array offers no jbod) It is easy to patch /etc/rc.d/zfs to load the mpt kernel module and do the rescan of the scsi bus: Index: zfs =================================================================== RCS file: /common/config/server/ROOT/etc/rc.d/zfs,v retrieving revision 1.1 retrieving revision 1.2 diff -r1.1 -r1.2 15c15 < required_modules="zfs" --- > required_modules="mpt:zfs" 25a26,29 > # required module mpt is on bus 2. do a rescan, so the > # devices are found and become available. > camcontrol rescan 2 So when zfs starts, the rest happens automatically. --Kai.