Date: Tue, 30 May 2017 11:31:33 -0600 From: Alan Somers <asomers@freebsd.org> To: Dustin Wenz <dustinwenz@ebureau.com> Cc: FreeBSD-scsi <freebsd-scsi@freebsd.org> Subject: Re: Inferring SAS expander topology Message-ID: <CAOtMX2h-E_GG9wopPH5Zz2jTZ%2BrqBmyJRSVvU9wxr4fczoHBiA@mail.gmail.com> In-Reply-To: <18E31C87-AB53-491F-9E40-F496AE31E305@ebureau.com> References: <18E31C87-AB53-491F-9E40-F496AE31E305@ebureau.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Tue, May 30, 2017 at 11:00 AM, Dustin Wenz <dustinwenz@ebureau.com> wrot=
e:
> I'm working on a server that has 6 LSI SAS expanders connected to it. Two=
of them are throwing intermittent command errors and need some attention. =
Because the host only has four external 8088 ports, I know some of the expa=
nders are chained through each other.
>
> Before I can send parts for replacement, I need to determine if the two p=
roblematic expanders are either daisy-chained or directly connected to the =
host. On Linux, this is trivial; I would just browse the device topology in=
/sys/. However, this machine is running FreeBSD 10.3, and I am unable to f=
ind a way to do this. I have sg3_utils available, as well as any build-in F=
reeBSD tools. Using this software, is it possible to determine if a SAS exp=
ander is chained off of another?
>
> - .Dustin
I don't think sg3_utils will help you. You want sysutils/smp_utils
instead. If you install that and then run "smp_discover /dev/ses0",
it will show you what each phy is connected to. Usually, there will
be four phys connected to the upstream port. You can tell by their
EUI64s whether they're connected directly to the HBA or to another
expander. And you can tell which expander by comparing the exact
EUI64 to each other expander's SEP phy. Note that with LSI expanders,
the SEP's address usually differs from the expander's address by a few
bits in the last byte. For example, if the expander's address is
0x50000000000000ff, then the SEP's address might by
0x50000000000000ffd. Here's some example output from one of my
systems:
# smp_discover /dev/ses3
phy 0:U:attached:[500093d23000a000:00 t(SATA)] 6 Gbps
phy 1:U:attached:[500093d23000a001:00 t(SATA)] 6 Gbps
phy 2:U:attached:[500093d23000a002:00 t(SATA)] 6 Gbps
phy 3:U:attached:[500093d23000a003:00 t(SATA)] 6 Gbps
phy 4:U:attached:[500093d23000a17f:11 exp t(SMP)] 6 Gbps
<- Connected to the other expander
phy 5:U:attached:[500093d23000a17f:10 exp t(SMP)] 6 Gbps
<- Connected to the other expander
phy 6:U:attached:[500093d23000a17f:09 exp t(SMP)] 6 Gbps
<- Connected to the other expander
phy 7:U:attached:[500093d23000a17f:08 exp t(SMP)] 6 Gbps
<- Connected to the other expander
phy 8:U:attached:[500093d23000a1bf:11 exp t(SMP)] 6 Gbps
<- Connected to a third expander
phy 9:U:attached:[500093d23000a1bf:10 exp t(SMP)] 6 Gbps
<- Connected to a third expander
phy 10:U:attached:[500093d23000a1bf:09 exp t(SMP)] 6 Gbps
<- Connected to a third expander
phy 11:U:attached:[500093d23000a1bf:08 exp t(SMP)] 6 Gbps
<- Connected to a third expander
phy 16:U:attached:[500093d23000a010:00 t(SATA)] 6 Gbps
phy 17:U:attached:[500093d23000a011:00 t(SATA)] 6 Gbps
phy 18:U:attached:[500093d23000a012:00 t(SATA)] 6 Gbps
phy 19:U:attached:[500093d23000a013:00 t(SATA)] 6 Gbps
phy 20:U:attached:[500093d23000a014:00 t(SATA)] 6 Gbps
phy 21:U:attached:[500093d23000a015:00 t(SATA)] 6 Gbps
phy 22:U:attached:[500093d23000a016:00 t(SATA)] 6 Gbps
phy 23:U:attached:[500093d23000a017:00 t(SATA)] 6 Gbps
phy 25:U:attached:[500093d23000a019:00 t(SATA)] 6 Gbps
phy 26:U:attached:[500093d23000a01a:00 t(SATA)] 6 Gbps
phy 29:U:attached:[500093d23000a01d:00 t(SATA)] 6 Gbps
phy 30:U:attached:[500093d23000a01e:00 t(SATA)] 6 Gbps
phy 31:U:attached:[500093d23000a01f:00 t(SATA)] 6 Gbps
phy 33:U:attached:[500093d23000a021:00 t(SATA)] 6 Gbps
phy 34:U:attached:[500093d23000a022:00 t(SATA)] 6 Gbps
phy 35:U:attached:[500093d23000a023:00 t(SATA)] 6 Gbps
phy 36:U:attached:[500093d23000a024:00 t(SATA)] 6 Gbps
phy 37:U:attached:[500093d23000a025:00 t(SATA)] 6 Gbps
phy 38:U:attached:[500093d23000a026:00 t(SATA)] 6 Gbps
phy 39:U:attached:[500093d23000a027:00 t(SATA)] 6 Gbps
phy 40:U:attached:[500605b008a93990:04 i(SSP+STP+SMP)] 6 Gbps
<- Connected to an LSI HBA
phy 41:U:attached:[500605b008a93990:07 i(SSP+STP+SMP)] 6 Gbps
<- Connected to an LSI HBA
phy 42:U:attached:[500605b008a93990:05 i(SSP+STP+SMP)] 6 Gbps
<- Connected to an LSI HBA
phy 43:U:attached:[500605b008a93990:06 i(SSP+STP+SMP)] 6 Gbps
<- Connected to an LSI HBA
phy 48:D:attached:[500093d23000a03d:00 V i(SMP) t(SSP)] 12 Gbps
<- This expander's SEP
# smp_discover /dev/pass121
phy 0:U:attached:[500093d23000a140:00 t(SATA)] 6 Gbps
phy 1:U:attached:[500093d23000a141:00 t(SATA)] 6 Gbps
phy 2:U:attached:[500093d23000a142:00 t(SATA)] 6 Gbps
phy 3:U:attached:[500093d23000a143:00 t(SATA)] 6 Gbps
phy 4:U:attached:[500093d23000a144:00 t(SATA)] 6 Gbps
phy 5:U:attached:[500093d23000a145:00 t(SATA)] 6 Gbps
phy 6:U:attached:[500093d23000a146:00 t(SATA)] 6 Gbps
phy 7:U:attached:[500093d23000a147:00 t(SATA)] 6 Gbps
phy 8:U:attached:[500093d23000a03f:07 exp t(SMP)] 6 Gbps
<- Connected to the other expander
phy 9:U:attached:[500093d23000a03f:06 exp t(SMP)] 6 Gbps
<- Connected to the other expander
phy 10:U:attached:[500093d23000a03f:05 exp t(SMP)] 6 Gbps
<- Connected to the other expander
phy 11:U:attached:[500093d23000a03f:04 exp t(SMP)] 6 Gbps
<- Connected to the other expander
phy 12:U:attached:[500093d23000a14c:00 t(SATA)] 6 Gbps
phy 14:U:attached:[500093d23000a14e:00 t(SATA)] 6 Gbps
phy 15:U:attached:[500093d23000a14f:00 t(SATA)] 6 Gbps
phy 16:U:attached:[500093d23000a150:00 t(SATA)] 6 Gbps
phy 17:U:attached:[500093d23000a151:00 t(SATA)] 6 Gbps
phy 18:U:attached:[500093d23000a152:00 t(SATA)] 6 Gbps
phy 19:U:attached:[500093d23000a153:00 t(SATA)] 6 Gbps
phy 20:U:attached:[500093d23000a154:00 t(SATA)] 6 Gbps
phy 21:U:attached:[500093d23000a155:00 t(SATA)] 6 Gbps
phy 22:U:attached:[500093d23000a156:00 t(SATA)] 6 Gbps
phy 23:U:attached:[500093d23000a157:00 t(SATA)] 6 Gbps
phy 24:U:attached:[500093d23000a158:00 t(SATA)] 6 Gbps
phy 25:U:attached:[500093d23000a159:00 t(SATA)] 6 Gbps
phy 26:U:attached:[500093d23000a15a:00 t(SATA)] 6 Gbps
phy 27:U:attached:[500093d23000a15b:00 t(SATA)] 6 Gbps
phy 28:U:attached:[500093d23000a15c:00 t(SATA)] 6 Gbps
phy 29:U:attached:[500093d23000a15d:00 t(SATA)] 6 Gbps
phy 30:U:attached:[500093d23000a15e:00 t(SATA)] 6 Gbps
phy 32:U:attached:[500093d23000a160:00 t(SATA)] 6 Gbps
phy 33:U:attached:[500093d23000a161:00 t(SATA)] 6 Gbps
phy 34:U:attached:[500093d23000a162:00 t(SATA)] 6 Gbps
phy 35:U:attached:[500093d23000a163:00 t(SATA)] 6 Gbps
phy 36:U:attached:[500093d23000a164:00 t(SATA)] 6 Gbps
phy 37:U:attached:[500093d23000a165:00 t(SATA)] 6 Gbps
phy 38:U:attached:[500093d23000a166:00 t(SATA)] 6 Gbps
phy 39:U:attached:[500093d23000a167:00 t(SATA)] 6 Gbps
phy 40:D:attached:[500093d23000a17d:00 V i(SMP) t(SSP)] 12 Gbps
<- This expander's SEP
-Alan
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAOtMX2h-E_GG9wopPH5Zz2jTZ%2BrqBmyJRSVvU9wxr4fczoHBiA>