Date: Mon, 15 Jan 2018 10:56:34 -0600 (CST) From: "Valeri Galtsev" <galtsev@kicp.uchicago.edu> To: "Warner Losh" <imp@bsdimp.com> Cc: "Grzegorz Junka" <list1@gjunka.com>, freebsd-questions@freebsd.org, freebsd-drivers@freebsd.org Subject: Re: Server doesn't boot when 3 PCIe slots are populated Message-ID: <57933.108.68.169.115.1516035394.squirrel@cosmo.uchicago.edu> In-Reply-To: <CANCZdfqZ-dogHXBdoyMPLOPs_R-vD%2BwLM-r6sm6ypesd0Nvp4A@mail.gmail.com> References: <ecce3fa6-3909-0947-685c-8a412684e99c@gjunka.com> <CAOgwaMsf9zByJYhL3KqpUMW5qKAzQEHpDWcwejY-uK=9swWbUQ@mail.gmail.com> <3d0ad00c-5214-71b0-017b-c2d5ba608e37@gjunka.com> <CAOgwaMsOKrGfGNmRt-C9Skjssj8JPAtFpk8bwG9v55LmaWdoVw@mail.gmail.com> <8df1e967-01e0-d3c2-e14c-64c7fc8c66b0@gjunka.com> <CANCZdfqZ-dogHXBdoyMPLOPs_R-vD%2BwLM-r6sm6ypesd0Nvp4A@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Mon, January 15, 2018 12:18 am, Warner Losh wrote: > On Jan 14, 2018 11:05 PM, "Grzegorz Junka" <list1@gjunka.com> wrote: > > > On 14/01/2018 16:18, Mehmet Erol Sanliturk wrote: > >> >> >> On Sun, Jan 14, 2018 at 5:46 PM, Grzegorz Junka <list1@gjunka.com >> <mailto: >> list1@gjunka.com>> wrote: >> >> >> On 13/01/2018 17:56, Mehmet Erol Sanliturk wrote: >> >> >> >> On Sat, Jan 13, 2018 at 7:21 PM, Grzegorz Junka >> <list1@gjunka.com <mailto:list1@gjunka.com> >> <mailto:list1@gjunka.com <mailto:list1@gjunka.com>>> wrote: >> >> Hello, >> >> I am installing a FreeBSD server based on Supermicro >> H8SML-iF. >> There are three PCIe slots to which I installed 2 NVMe >> drives and >> one network card Intel I350-T4 (with 4 Ethernet slots). >> >> I am observing a strange behavior where the system doesn't >> boot if >> all three PCIe slots are populated. It shows this message: >> >> nvme0: <Generic NVMe Device> mem 0xfd8fc000-0xfd8fffff irq >> 24 at >> device 0.0 on pci1 >> nvme0: controller ready did not become 1 within 30000 ms >> nvme0: did not complete shutdown within 5 seconds of >> notification >> >> The I see a kernel panic/dump and the system reboots after >> 15 seconds. >> >> If I remove one card, either one of the NVMe drives or the >> network >> card, the system boots fine. Also, if in BIOS I set PnP OS >> to YES >> then sometimes it boots (but not always). If I set PnP OS >> to NO, >> and all three cards are installed, the system never boots. >> >> When the system boots OK I can see that the network card is >> reported as 4 separate devices on one of the PCIe slots. I >> tried >> different NVMe drives as well as changing which device is >> installed to which slot but the result seems to be the >> same in any >> case. >> >> What may be the issue? Amount of power drawn by the >> hardware? Too >> many devices not supported by the motherboard? Too many >> interrupts >> for the FreeBSD kernel to handle? >> >> Any help would be greatly appreciated. >> >> GregJ >> >> _______________________________________________ >> >> >> >> >> >> From my experience from other trade marked main boards , an >> action may be to check manual of your server board to see >> whether there are rules about use of these slots : Sometimes >> differently shaped slots are supplied with same ports : If one >> slot is occupied , the other slot should be left open , or >> rules about not to insert such a kind of device into a slot , >> for example , graphic cards . >> >> >> Mehmet Erol Sanliturk >> >> >> I checked the manual but couldn't find any restrictions regarding >> PCIe ports. It only says how many lanes are available in each >> slot. Would there be any obvious BIOS setting that could cause >> this issue? I tried after resetting BIOS to default settings but >> maybe something is set incorrectly by default? >> >> GregJ >> _______________________________________________ >> >> >> >> >> >> http://www.supermicro.com/Aplus/motherboard/Opteron3000/SR56 >> x0/H8SML-iF.cfm >> H8SML-iF >> >> >> On the above page , click "OS Compatibility" >> >> >> On the following page , click "SR5650" >> >> http://www.supermicro.com/Aplus/support/resources/OS/OS_Comp_SR5650.cfm >> OS Compatibility Chart >> >> >> On the column ( third ) >> >> H8SML-7F >> H8SML-7 >> H8SML-iF >> H8SML-i >> >> >> there listed only * >> * >> ** >> * >> * >> * >> * >> >> FreeBSD 8.0 >> FreeBSD 9.1 >> >> From this list , it may be said that , this mother board date is old , >> means , it seems that the new OS versions are not tested after currently >> tested OS versions . >> >> >> To check interaction between operating system and your Supermicro >> H8SML-iF >> , select one of the suitable operating system ( Unix class OSes are more >> suitable ) for you and tested on this card , and try to install it as >> you >> like your installed components . If it boots successfully , it means >> that >> there is an incompatibility between your FreeBSD and the main board . If >> no >> one of them boots , then you may conclude that , there is a problem in >> your >> settings . >> >> >> BIOS settings are important , because , OS communicates with the main >> board through these settings . >> >> >> In manual ( downloaded from the above page : >> Manual Revision 1.0c >> Release Date: March 12, 2014 ) , page 4-9 , "PCI/PnP Configuration" is >> defined . >> If PnP is selected YES. OS adjusts some device settings . If NO is >> selected , BIOS adjusts some device settings . When BIOS adjusted device >> settings are not conforming to OS parameters , the result will be "FAIL" >> . >> >> Therefore , more suitable selection is YES . >> >> >> Another point is that , there are many more BIOS selectable parameters >> and >> jumpers about PCI slots and others . >> There are some BIOS settings for PCI slots : >> >> PCI X4 Slot 6 ( page 4-9 ) >> PCI x8 Slot 7 ( page 4-10 ) >> >> >> >> Please review these BIOS settings in your manual and set them with >> respect >> to your requirements . >> >> > Thanks Mehmet for looking into this. It's an old motherboard but my point > is that it boots fine when either: one NVMe and the network card, or both > NVMe are installed, but not when all three are installed. How would that > be > related to FreeBSD compatibility? The chipset and all devices that I am > trying to install are supported by FreeBSD 11.x. > > I just tried booting into a Debian live system and it also didn't > enumerate > NVMe drives properly. This means that it's not FreeBSD related and is no > longer relevant for this list. This sounds to me as having something to do with allocation of PCI address space to talk to devices. Many devices can alternatively use different ranges of addresses, so more than one such device can be attached to the same PCI bus. These two particular devices seem to not be successfully negotiated to use different (not overlapping) ranges of addresses (in presence of some particular third device). Maybe it only happens like that in this particular system board ("motherboard"). I would try the same on different machine. But it is likely that all these three devices do not have non-overlapping address ranges. I hope, someone more knowledgeable that I will chime in. Valeri > I will try to play with BIOS settings to > see > if I can make it work that way. Thanks for all the help. > > > > Nvme drives are weird about power. I distrust the power estimate of 5-9w > earlier in the thread... given the oddity with debian, it's not too crazy > to think that. How far does FreeBSD boot though? > > Warner > > Warner > > _______________________________________________ > freebsd-drivers@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-drivers > To unsubscribe, send any mail to "freebsd-drivers-unsubscribe@freebsd.org" > _______________________________________________ > freebsd-questions@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-questions > To unsubscribe, send any mail to > "freebsd-questions-unsubscribe@freebsd.org" > ++++++++++++++++++++++++++++++++++++++++ Valeri Galtsev Sr System Administrator Department of Astronomy and Astrophysics Kavli Institute for Cosmological Physics University of Chicago Phone: 773-702-4247 ++++++++++++++++++++++++++++++++++++++++
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?57933.108.68.169.115.1516035394.squirrel>