Date: Tue, 10 Jul 2018 09:27:58 -0600 From: Stephen Mcconnell <stephen.mcconnell@broadcom.com> To: Oliver Sech <crimsonthunder@gmx.net>, FreeBSD-scsi <freebsd-scsi@freebsd.org> Subject: RE: problems with SAS JBODs 2 Message-ID: <3caf8ccd6fde8cfc4db25bae5327c46b@mail.gmail.com> In-Reply-To: <b785fe02-9242-c95f-56cb-2130f90e17f5@gmx.net> References: <trinity-14d18077-ea73-40f6-9e87-d2d4000b1f7e-1530620937871@3c-app-gmx-bs01> <CAOtMX2h8r31AeNCKyckK2P0VLn1CKFogo9bWom2So1x2ngpa4A@mail.gmail.com> <237f77ab-89e2-188b-b2b1-84c6d88609b0@gmx.net> <b785fe02-9242-c95f-56cb-2130f90e17f5@gmx.net>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi Oliver, I can't get to your links. Can you try to send the logs in another way? Steve > -----Original Message----- > From: owner-freebsd-scsi@freebsd.org [mailto:owner-freebsd- > scsi@freebsd.org] On Behalf Of Oliver Sech > Sent: Tuesday, July 10, 2018 9:14 AM > To: FreeBSD-scsi > Subject: Re: problems with SAS JBODs 2 > > I tested a few additional things. I don't think this is a multipath, daisy > chain > nor a SAS wide ports problem. > I can reproduce the problem with just a single connection to an > Expander/JBOD. > > Test: > * physically disconnect all shelves > * reboot system > * connect one shelf via SAS cable > * check number of disks (after a reboot everything always shows up) > * disconnect the shelf and wait (geom disk list still shows most disks.) > * connect the shelf (missing disks) > > Tested Hardware: > * Supermicro SAS3 847E2C-R1K28JBOD + SAS3 LSI 9305-16e ( internal > daisy > chain + wide links) > * Supermicro SAS3 847E2C-R1K28JBOD + SAS3 LSI 9305-16e (straight HBA > <- > > EXPANDER connection. (no wide links, no daisy chain)) > * Supermicro SAS2 SC847E26-RJBOD1 + SAS3 LSI 9305-16e (internal daisy > chain) > * Promise SAS2 VTrak 830 + SAS3 LSI 9305-16e (straight HBA > <-> > EXPANDER connection.) > > > > On 07/04/2018 12:15 PM, Oliver Sech wrote: > >> 1) Are the expanders daisy chained? Some SAS expanders don't work > reliably > >> when daisy chained. Best to direct connect each one to the server. > > At the moment I have 1 JBOD connected to 1 HBA Port with 1 cable (4 > lanes?). > > Unfortunately the JBOD has 24 slots in the front and 20 in the back and, > those are connected via a internal SAS daisy chaining. > > I could rewire and connect each backplane directly to the server, but > unfortunately I do not have enough ports.. > > > > JOBD Model: Supermicro 847E2C-R1K28JBOD > > > >> 2) Are the expanders connected in multipath or single path? You need > >> geom_multipath if you're going to do that. > > See answer 1. There is a single path from the host to the first > > expander. > > > >> 3) Are you attempting to use wide ports (two SAS cables connecting each > >> expander to the HBA). If do, you'll need to make sure that each pair > >> of > >> SAS cables goes to the same HBA chip (not merely the same card, as some > >> cards contain two HBA chips). > > see 1. The last time I opened one of those JBODs there were 8 SAS cables > between the Front and Back expander. I assume that wide ports are being > used. > > (2 expanders per backplane as well) > > > >> 4) Are you trying to remove an expander while ZFS is active on that > >> expander? That will suspend your pool, and ZFS doesn't always recover > from > >> a suspended state. > > I'm testing with a new unused disk shelf that was never part of the ZFS > pool. There were > > _______________________________________________ > > freebsd-scsi@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-scsi > > To unsubscribe, send any mail to "freebsd-scsi-unsubscribe@freebsd.org" > _______________________________________________ > freebsd-scsi@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-scsi > To unsubscribe, send any mail to "freebsd-scsi-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?3caf8ccd6fde8cfc4db25bae5327c46b>