From owner-freebsd-drivers@freebsd.org Sun Jan 14 14:34:15 2018 Return-Path: Delivered-To: freebsd-drivers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C3F2AE72C7E; Sun, 14 Jan 2018 14:34:15 +0000 (UTC) (envelope-from list1@gjunka.com) Received: from msa1.earth.yoonka.com (yoonka.com [88.98.225.149]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "msa1.earth.yoonka.com", Issuer "msa1.earth.yoonka.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 5C6D37E52C; Sun, 14 Jan 2018 14:34:14 +0000 (UTC) (envelope-from list1@gjunka.com) Received: from crayon2.yoonka.com (crayon2.yoonka.com [10.70.7.20]) (authenticated bits=0) by msa1.earth.yoonka.com (8.15.2/8.15.2) with ESMTPSA id w0EEYBk8024846 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO); Sun, 14 Jan 2018 14:34:11 GMT (envelope-from list1@gjunka.com) Subject: Re: Server doesn't boot when 3 PCIe slots are populated To: freebsd-questions@freebsd.org, freebsd-drivers@freebsd.org References: <061ccfb3-ee6a-71a7-3926-372bb17b3171@kicp.uchicago.edu> From: Grzegorz Junka Message-ID: <4cd39c52-9bf0-ef44-8335-9b4cf6eb6a6b@gjunka.com> Date: Sun, 14 Jan 2018 14:34:11 +0000 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.3.0 MIME-Version: 1.0 In-Reply-To: <061ccfb3-ee6a-71a7-3926-372bb17b3171@kicp.uchicago.edu> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-GB-large X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 14 Jan 2018 14:34:15 -0000 On 13/01/2018 18:31, Valeri Galtsev wrote: > > > On 01/13/18 10:21, Grzegorz Junka wrote: >> Hello, >> >> I am installing a FreeBSD server based on Supermicro H8SML-iF. There >> are three PCIe slots to which I installed 2 NVMe drives and one >> network card Intel I350-T4 (with 4 Ethernet slots). >> >> I am observing a strange behavior where the system doesn't boot if >> all three PCIe slots are populated. It shows this message: >> >> nvme0: mem 0xfd8fc000-0xfd8fffff irq 24 at >> device 0.0 on pci1 >> nvme0: controller ready did not become 1 within 30000 ms >> nvme0: did not complete shutdown within 5 seconds of notification >> >> The I see a kernel panic/dump and the system reboots after 15 seconds. >> >> If I remove one card, either one of the NVMe drives or the network >> card, the system boots fine. Also, if in BIOS I set PnP OS to YES >> then sometimes it boots (but not always). If I set PnP OS to NO, and >> all three cards are installed, the system never boots. >> >> When the system boots OK I can see that the network card is reported >> as 4 separate devices on one of the PCIe slots. I tried different >> NVMe drives as well as changing which device is installed to which >> slot but the result seems to be the same in any case. >> >> What may be the issue? Amount of power drawn by the hardware? Too >> many devices not supported by the motherboard? Too many interrupts >> for the FreeBSD kernel to handle? > > That would be my first suspicion. Either total power drawn off the > power supply. Or total power drawn off the PCI[whichever it is] bus > power leads. Check if any of the add-on cards have extra power port > (many video cards do). Card likely will work without extra power > connected to it, but connecting extra power on the card may solve your > problem. Next: borrow more powerful power supply and see if that > resolves the issue. Or temporarily disconnect everything else (like > all hard drives), and boot with all three cards off live CD, and see > if that doesn't crash, then it is marginally insufficient power supply. Thanks for the suggestion. The power supply was able to power two NVMe disks and 6 spinning HDD disks without issues in another server. So the total power should be fine. It may be the PCI bus power leads is causing problems but then, two NVMe drives wouldn't take more than 5-9W and the network card even less. PCI Express specification allows much more to be drawn from each slot. In total the server shouldn't take more than 50-70W, I am not saying that it's not because of the power supply, but I think it would be the least likely at this point. I will try with another power supply when I find one. GregJ From owner-freebsd-drivers@freebsd.org Sun Jan 14 14:46:21 2018 Return-Path: Delivered-To: freebsd-drivers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 90D2DE736CD; Sun, 14 Jan 2018 14:46:21 +0000 (UTC) (envelope-from list1@gjunka.com) Received: from msa1.earth.yoonka.com (yoonka.com [88.98.225.149]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "msa1.earth.yoonka.com", Issuer "msa1.earth.yoonka.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id E80467EAF9; Sun, 14 Jan 2018 14:46:20 +0000 (UTC) (envelope-from list1@gjunka.com) Received: from crayon2.yoonka.com (crayon2.yoonka.com [10.70.7.20]) (authenticated bits=0) by msa1.earth.yoonka.com (8.15.2/8.15.2) with ESMTPSA id w0EEkIPT025002 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO); Sun, 14 Jan 2018 14:46:18 GMT (envelope-from list1@gjunka.com) Subject: Re: Server doesn't boot when 3 PCIe slots are populated Cc: FreeBSD Questions Mailing List , freebsd-drivers@freebsd.org References: From: Grzegorz Junka Message-ID: <3d0ad00c-5214-71b0-017b-c2d5ba608e37@gjunka.com> Date: Sun, 14 Jan 2018 14:46:18 +0000 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.3.0 MIME-Version: 1.0 In-Reply-To: Content-Language: en-GB-large Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 14 Jan 2018 14:46:21 -0000 On 13/01/2018 17:56, Mehmet Erol Sanliturk wrote: > > > On Sat, Jan 13, 2018 at 7:21 PM, Grzegorz Junka > wrote: > > Hello, > > I am installing a FreeBSD server based on Supermicro H8SML-iF. > There are three PCIe slots to which I installed 2 NVMe drives and > one network card Intel I350-T4 (with 4 Ethernet slots). > > I am observing a strange behavior where the system doesn't boot if > all three PCIe slots are populated. It shows this message: > > nvme0: mem 0xfd8fc000-0xfd8fffff irq 24 at > device 0.0 on pci1 > nvme0: controller ready did not become 1 within 30000 ms > nvme0: did not complete shutdown within 5 seconds of notification > > The I see a kernel panic/dump and the system reboots after 15 seconds. > > If I remove one card, either one of the NVMe drives or the network > card, the system boots fine. Also, if in BIOS I set PnP OS to YES > then sometimes it boots (but not always). If I set PnP OS to NO, > and all three cards are installed, the system never boots. > > When the system boots OK I can see that the network card is > reported as 4 separate devices on one of the PCIe slots. I tried > different NVMe drives as well as changing which device is > installed to which slot but the result seems to be the same in any > case. > > What may be the issue? Amount of power drawn by the hardware? Too > many devices not supported by the motherboard? Too many interrupts > for the FreeBSD kernel to handle? > > Any help would be greatly appreciated. > > GregJ > > _______________________________________________ > > > > > > From my experience from other trade marked main boards , an action may > be to check manual of your server board to see whether there are rules > about use of these slots : Sometimes differently shaped slots are > supplied with same ports : If one slot is occupied , the other slot > should be left open , or rules about not to insert such a kind of > device into a slot , for example , graphic cards . > > > Mehmet Erol Sanliturk > I checked the manual but couldn't find any restrictions regarding PCIe ports. It only says how many lanes are available in each slot. Would there be any obvious BIOS setting that could cause this issue? I tried after resetting BIOS to default settings but maybe something is set incorrectly by default? GregJ From owner-freebsd-drivers@freebsd.org Sun Jan 14 14:50:17 2018 Return-Path: Delivered-To: freebsd-drivers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 79FA5E73B80; Sun, 14 Jan 2018 14:50:17 +0000 (UTC) (envelope-from galtsev@kicp.uchicago.edu) Received: from cosmo.uchicago.edu (cosmo.uchicago.edu [128.135.20.71]) by mx1.freebsd.org (Postfix) with ESMTP id 305437ED5D; Sun, 14 Jan 2018 14:50:16 +0000 (UTC) (envelope-from galtsev@kicp.uchicago.edu) Received: by cosmo.uchicago.edu (Postfix, from userid 48) id 7E4D1CB8D3A; Sun, 14 Jan 2018 08:50:10 -0600 (CST) Received: from 108.68.169.115 (SquirrelMail authenticated user valeri) by cosmo.uchicago.edu with HTTP; Sun, 14 Jan 2018 08:50:10 -0600 (CST) Message-ID: <60145.108.68.169.115.1515941410.squirrel@cosmo.uchicago.edu> In-Reply-To: <4cd39c52-9bf0-ef44-8335-9b4cf6eb6a6b@gjunka.com> References: <061ccfb3-ee6a-71a7-3926-372bb17b3171@kicp.uchicago.edu> <4cd39c52-9bf0-ef44-8335-9b4cf6eb6a6b@gjunka.com> Date: Sun, 14 Jan 2018 08:50:10 -0600 (CST) Subject: Re: Server doesn't boot when 3 PCIe slots are populated From: "Valeri Galtsev" To: "Grzegorz Junka" Cc: freebsd-questions@freebsd.org, freebsd-drivers@freebsd.org Reply-To: galtsev@kicp.uchicago.edu User-Agent: SquirrelMail/1.4.8-5.el5.centos.7 MIME-Version: 1.0 Content-Type: text/plain;charset=iso-8859-1 Content-Transfer-Encoding: 8bit X-Priority: 3 (Normal) Importance: Normal X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 14 Jan 2018 14:50:17 -0000 On Sun, January 14, 2018 8:34 am, Grzegorz Junka wrote: > > On 13/01/2018 18:31, Valeri Galtsev wrote: >> >> >> On 01/13/18 10:21, Grzegorz Junka wrote: >>> Hello, >>> >>> I am installing a FreeBSD server based on Supermicro H8SML-iF. There >>> are three PCIe slots to which I installed 2 NVMe drives and one >>> network card Intel I350-T4 (with 4 Ethernet slots). >>> >>> I am observing a strange behavior where the system doesn't boot if >>> all three PCIe slots are populated. It shows this message: >>> >>> nvme0: mem 0xfd8fc000-0xfd8fffff irq 24 at >>> device 0.0 on pci1 >>> nvme0: controller ready did not become 1 within 30000 ms >>> nvme0: did not complete shutdown within 5 seconds of notification >>> >>> The I see a kernel panic/dump and the system reboots after 15 seconds. >>> >>> If I remove one card, either one of the NVMe drives or the network >>> card, the system boots fine. Also, if in BIOS I set PnP OS to YES >>> then sometimes it boots (but not always). If I set PnP OS to NO, and >>> all three cards are installed, the system never boots. >>> >>> When the system boots OK I can see that the network card is reported >>> as 4 separate devices on one of the PCIe slots. I tried different >>> NVMe drives as well as changing which device is installed to which >>> slot but the result seems to be the same in any case. >>> >>> What may be the issue? Amount of power drawn by the hardware? Too >>> many devices not supported by the motherboard? Too many interrupts >>> for the FreeBSD kernel to handle? >> >> That would be my first suspicion. Either total power drawn off the >> power supply. Or total power drawn off the PCI[whichever it is] bus >> power leads. Check if any of the add-on cards have extra power port >> (many video cards do). Card likely will work without extra power >> connected to it, but connecting extra power on the card may solve your >> problem. Next: borrow more powerful power supply and see if that >> resolves the issue. Or temporarily disconnect everything else (like >> all hard drives), and boot with all three cards off live CD, and see >> if that doesn't crash, then it is marginally insufficient power supply. > > Thanks for the suggestion. The power supply was able to power two NVMe > disks and 6 spinning HDD disks without issues in another server. So the > total power should be fine. It may be the PCI bus power leads is causing > problems but then, two NVMe drives wouldn't take more than 5-9W and the > network card even less. PCI Express specification allows much more to be > drawn from each slot. In total the server shouldn't take more than 50-70W, > > I am not saying that it's not because of the power supply, but I think > it would be the least likely at this point. I will try with another > power supply when I find one. Another shot in the dark: some PCI-express slots may be "a pair", i.e. they can only take cards with the same number of signal lanes. Then you may have trouble if one of the cards is, say, x8 another is x4. System board ("motherboard") manual may shed light on this. Incidentally, PS powering successfully different machine not necessarily is also sufficient to power this one. As you said. Good luck! Valeri ++++++++++++++++++++++++++++++++++++++++ Valeri Galtsev Sr System Administrator Department of Astronomy and Astrophysics Kavli Institute for Cosmological Physics University of Chicago Phone: 773-702-4247 ++++++++++++++++++++++++++++++++++++++++ From owner-freebsd-drivers@freebsd.org Sun Jan 14 15:01:30 2018 Return-Path: Delivered-To: freebsd-drivers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D20EDE74466; Sun, 14 Jan 2018 15:01:30 +0000 (UTC) (envelope-from list1@gjunka.com) Received: from msa1.earth.yoonka.com (yoonka.com [88.98.225.149]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "msa1.earth.yoonka.com", Issuer "msa1.earth.yoonka.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 87EBE7F480; Sun, 14 Jan 2018 15:01:29 +0000 (UTC) (envelope-from list1@gjunka.com) Received: from crayon2.yoonka.com (crayon2.yoonka.com [10.70.7.20]) (authenticated bits=0) by msa1.earth.yoonka.com (8.15.2/8.15.2) with ESMTPSA id w0EF1SpL025287 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO); Sun, 14 Jan 2018 15:01:28 GMT (envelope-from list1@gjunka.com) Subject: Re: Server doesn't boot when 3 PCIe slots are populated Cc: freebsd-questions@freebsd.org, freebsd-drivers@freebsd.org References: <061ccfb3-ee6a-71a7-3926-372bb17b3171@kicp.uchicago.edu> <4cd39c52-9bf0-ef44-8335-9b4cf6eb6a6b@gjunka.com> <60145.108.68.169.115.1515941410.squirrel@cosmo.uchicago.edu> From: Grzegorz Junka Message-ID: Date: Sun, 14 Jan 2018 15:01:28 +0000 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.3.0 MIME-Version: 1.0 In-Reply-To: <60145.108.68.169.115.1515941410.squirrel@cosmo.uchicago.edu> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-GB-large X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 14 Jan 2018 15:01:30 -0000 On 14/01/2018 14:50, Valeri Galtsev wrote: > On Sun, January 14, 2018 8:34 am, Grzegorz Junka wrote: >> On 13/01/2018 18:31, Valeri Galtsev wrote: >>> >>> On 01/13/18 10:21, Grzegorz Junka wrote: >>>> Hello, >>>> >>>> I am installing a FreeBSD server based on Supermicro H8SML-iF. There >>>> are three PCIe slots to which I installed 2 NVMe drives and one >>>> network card Intel I350-T4 (with 4 Ethernet slots). >>>> >>>> I am observing a strange behavior where the system doesn't boot if >>>> all three PCIe slots are populated. It shows this message: >>>> >>>> nvme0: mem 0xfd8fc000-0xfd8fffff irq 24 at >>>> device 0.0 on pci1 >>>> nvme0: controller ready did not become 1 within 30000 ms >>>> nvme0: did not complete shutdown within 5 seconds of notification >>>> >>>> The I see a kernel panic/dump and the system reboots after 15 seconds. >>>> >>>> If I remove one card, either one of the NVMe drives or the network >>>> card, the system boots fine. Also, if in BIOS I set PnP OS to YES >>>> then sometimes it boots (but not always). If I set PnP OS to NO, and >>>> all three cards are installed, the system never boots. >>>> >>>> When the system boots OK I can see that the network card is reported >>>> as 4 separate devices on one of the PCIe slots. I tried different >>>> NVMe drives as well as changing which device is installed to which >>>> slot but the result seems to be the same in any case. >>>> >>>> What may be the issue? Amount of power drawn by the hardware? Too >>>> many devices not supported by the motherboard? Too many interrupts >>>> for the FreeBSD kernel to handle? >>> That would be my first suspicion. Either total power drawn off the >>> power supply. Or total power drawn off the PCI[whichever it is] bus >>> power leads. Check if any of the add-on cards have extra power port >>> (many video cards do). Card likely will work without extra power >>> connected to it, but connecting extra power on the card may solve your >>> problem. Next: borrow more powerful power supply and see if that >>> resolves the issue. Or temporarily disconnect everything else (like >>> all hard drives), and boot with all three cards off live CD, and see >>> if that doesn't crash, then it is marginally insufficient power supply. >> Thanks for the suggestion. The power supply was able to power two NVMe >> disks and 6 spinning HDD disks without issues in another server. So the >> total power should be fine. It may be the PCI bus power leads is causing >> problems but then, two NVMe drives wouldn't take more than 5-9W and the >> network card even less. PCI Express specification allows much more to be >> drawn from each slot. In total the server shouldn't take more than 50-70W, >> >> I am not saying that it's not because of the power supply, but I think >> it would be the least likely at this point. I will try with another >> power supply when I find one. > Another shot in the dark: some PCI-express slots may be "a pair", i.e. > they can only take cards with the same number of signal lanes. Then you > may have trouble if one of the cards is, say, x8 another is x4. System > board ("motherboard") manual may shed light on this. > > Incidentally, PS powering successfully different machine not necessarily > is also sufficient to power this one. As you said. > The manual states: Slot 7: One (1) PCI-Express x8 (in x16) Gen. 2 Slot 6: One (1) PCI-Express x4 (in x8 slot) Gen. 2 Slot 5: One (1) PCI-Express x8 Gen. 2 I tried all combinations (the pair of NVMe drives in slots 5/6, 5/7, 6/7, the network card in the third slot) but none worked. Considering the manual, slots 5/7 for NVMe and slot 6 for the network card should be the safest bet? Is there any utility to verify how many CPU lanes are in use/free? GregJ From owner-freebsd-drivers@freebsd.org Sun Jan 14 16:18:11 2018 Return-Path: Delivered-To: freebsd-drivers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 190ABE786BE; Sun, 14 Jan 2018 16:18:11 +0000 (UTC) (envelope-from m.e.sanliturk@gmail.com) Received: from mail-ot0-x236.google.com (mail-ot0-x236.google.com [IPv6:2607:f8b0:4003:c0f::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id CE2238205A; Sun, 14 Jan 2018 16:18:10 +0000 (UTC) (envelope-from m.e.sanliturk@gmail.com) Received: by mail-ot0-x236.google.com with SMTP id 53so8824148otj.2; Sun, 14 Jan 2018 08:18:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=KsWqC5QJddch9AjI/Cx9DJO284czsu8Nll3HIyfKXbY=; b=gMYGVUk0Ujbf3so6HtVXZzH4Uhv7h9vFZm6j6D5dunxKeNndPxdyzRSFskSY6cikL9 qqbqvfKwDnoXnLzeioE/UfFesZed2r+EacKrPOS3C6orjpZWW+HXeAAbBhP3q5y/i5Cn cUtxb3N7sfsQT/TI1SulliJGLmAfMYjXNfvT8qBp/Y/0GAi9AYhsCXtrjBatiqplCO6S YJT8J6C4/Y66uaQeDvOwaDjXzr0oTTfOLbDAjweI8KwXqFRlfgtYOvZhSEUd2k/MJcTS BKy7gPGirBFFh+3Fp9KwVZpRA7Q/2m3c/klvxZJEv1kVfK4y37m5n5tuc9jsMA4CKe+V te4A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=KsWqC5QJddch9AjI/Cx9DJO284czsu8Nll3HIyfKXbY=; b=CUL4pN7Ioy7bK4y8hef9RxkUdS+8BJuLGVXJvd536HMx8O6Td1Jcx2ajifnYTUqFkq WG+JvYtbX4VKXEBlTQSCHtBI4O8gvXKR4eTnQNK/rqjC6q20qtRkB6m8uFJqcVIUhxpP pN80Mw+l5nKEpp2002lk/Zr0D9JdJu9E6uRFTRN3RTYqjQI7BS+qlhgfXkGM4sNVpOur 9fD3hVAEkeI7YUVjSXy9VA0T/zsN033GTnEVSz1ouS38rPgYW9Ms7/vlrUxbuQ0UGhzG aMlNlElkKmH9EjUMkqg23ErHwLpcuChTwJsUTg95r7AcSo0aOsu6xevPMPSoHCtbgVQr zdQg== X-Gm-Message-State: AKwxytcM5UBF8UcwglrlPXktsVnOwZlGb22pKFeo+a0Dwpsal3q9GbW8 w4f/QYNDBbwPoCpH3lgZqWiycMrzFx7nE+KysKxnuA== X-Google-Smtp-Source: ACJfBovlv3K0zJazMWg9DWTu3iZCdlqHwm2xRWCPmXyf8nGcfJk75ZH3qpz+Bupu1wvLgAVYds4Zcb2pTetwkG2eK6U= X-Received: by 10.157.82.96 with SMTP id q32mr6745373otg.376.1515946690099; Sun, 14 Jan 2018 08:18:10 -0800 (PST) MIME-Version: 1.0 Received: by 10.157.0.2 with HTTP; Sun, 14 Jan 2018 08:18:09 -0800 (PST) In-Reply-To: <3d0ad00c-5214-71b0-017b-c2d5ba608e37@gjunka.com> References: <3d0ad00c-5214-71b0-017b-c2d5ba608e37@gjunka.com> From: Mehmet Erol Sanliturk Date: Sun, 14 Jan 2018 19:18:09 +0300 Message-ID: Subject: Re: Server doesn't boot when 3 PCIe slots are populated To: Grzegorz Junka Cc: FreeBSD Questions Mailing List , freebsd-drivers@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 14 Jan 2018 16:18:11 -0000 On Sun, Jan 14, 2018 at 5:46 PM, Grzegorz Junka wrote: > > On 13/01/2018 17:56, Mehmet Erol Sanliturk wrote: > >> >> >> On Sat, Jan 13, 2018 at 7:21 PM, Grzegorz Junka > > wrote: >> >> Hello, >> >> I am installing a FreeBSD server based on Supermicro H8SML-iF. >> There are three PCIe slots to which I installed 2 NVMe drives and >> one network card Intel I350-T4 (with 4 Ethernet slots). >> >> I am observing a strange behavior where the system doesn't boot if >> all three PCIe slots are populated. It shows this message: >> >> nvme0: mem 0xfd8fc000-0xfd8fffff irq 24 at >> device 0.0 on pci1 >> nvme0: controller ready did not become 1 within 30000 ms >> nvme0: did not complete shutdown within 5 seconds of notification >> >> The I see a kernel panic/dump and the system reboots after 15 seconds. >> >> If I remove one card, either one of the NVMe drives or the network >> card, the system boots fine. Also, if in BIOS I set PnP OS to YES >> then sometimes it boots (but not always). If I set PnP OS to NO, >> and all three cards are installed, the system never boots. >> >> When the system boots OK I can see that the network card is >> reported as 4 separate devices on one of the PCIe slots. I tried >> different NVMe drives as well as changing which device is >> installed to which slot but the result seems to be the same in any >> case. >> >> What may be the issue? Amount of power drawn by the hardware? Too >> many devices not supported by the motherboard? Too many interrupts >> for the FreeBSD kernel to handle? >> >> Any help would be greatly appreciated. >> >> GregJ >> >> _______________________________________________ >> >> >> >> >> >> From my experience from other trade marked main boards , an action may be >> to check manual of your server board to see whether there are rules about >> use of these slots : Sometimes differently shaped slots are supplied with >> same ports : If one slot is occupied , the other slot should be left open , >> or rules about not to insert such a kind of device into a slot , for >> example , graphic cards . >> >> >> Mehmet Erol Sanliturk >> >> > I checked the manual but couldn't find any restrictions regarding PCIe > ports. It only says how many lanes are available in each slot. Would there > be any obvious BIOS setting that could cause this issue? I tried after > resetting BIOS to default settings but maybe something is set incorrectly > by default? > > GregJ > _______________________________________________ > > http://www.supermicro.com/Aplus/motherboard/Opteron3000/SR56x0/H8SML-iF.cfm H8SML-iF On the above page , click "OS Compatibility" On the following page , click "SR5650" http://www.supermicro.com/Aplus/support/resources/OS/OS_Comp_SR5650.cfm OS Compatibility Chart On the column ( third ) H8SML-7F H8SML-7 H8SML-iF H8SML-i there listed only FreeBSD 8.0 FreeBSD 9.1 >From this list , it may be said that , this mother board date is old , means , it seems that the new OS versions are not tested after currently tested OS versions . To check interaction between operating system and your Supermicro H8SML-iF , select one of the suitable operating system ( Unix class OSes are more suitable ) for you and tested on this card , and try to install it as you like your installed components . If it boots successfully , it means that there is an incompatibility between your FreeBSD and the main board . If no one of them boots , then you may conclude that , there is a problem in your settings . BIOS settings are important , because , OS communicates with the main board through these settings . In manual ( downloaded from the above page : Manual Revision 1.0c Release Date: March 12, 2014 ) , page 4-9 , "PCI/PnP Configuration" is defined . If PnP is selected YES. OS adjusts some device settings . If NO is selected , BIOS adjusts some device settings . When BIOS adjusted device settings are not conforming to OS parameters , the result will be "FAIL" . Therefore , more suitable selection is YES . Another point is that , there are many more BIOS selectable parameters and jumpers about PCI slots and others . There are some BIOS settings for PCI slots : PCI X4 Slot 6 ( page 4-9 ) PCI x8 Slot 7 ( page 4-10 ) Please review these BIOS settings in your manual and set them with respect to your requirements . Mehmet Erol Sanliturk From owner-freebsd-drivers@freebsd.org Mon Jan 15 06:04:57 2018 Return-Path: Delivered-To: freebsd-drivers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6F338EA688A; Mon, 15 Jan 2018 06:04:57 +0000 (UTC) (envelope-from list1@gjunka.com) Received: from msa1.earth.yoonka.com (yoonka.com [88.98.225.149]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "msa1.earth.yoonka.com", Issuer "msa1.earth.yoonka.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id F3C1280E8F; Mon, 15 Jan 2018 06:04:56 +0000 (UTC) (envelope-from list1@gjunka.com) Received: from ultrabook.yoonka.com (188.29.165.65.threembb.co.uk [188.29.165.65]) (authenticated bits=0) by msa1.earth.yoonka.com (8.15.2/8.15.2) with ESMTPSA id w0F64hOV040719 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO); Mon, 15 Jan 2018 06:04:47 GMT (envelope-from list1@gjunka.com) X-Authentication-Warning: msa1.earth.yoonka.com: Host 188.29.165.65.threembb.co.uk [188.29.165.65] claimed to be ultrabook.yoonka.com From: Grzegorz Junka Subject: Re: Server doesn't boot when 3 PCIe slots are populated Cc: FreeBSD Questions Mailing List , freebsd-drivers@freebsd.org References: <3d0ad00c-5214-71b0-017b-c2d5ba608e37@gjunka.com> Message-ID: <8df1e967-01e0-d3c2-e14c-64c7fc8c66b0@gjunka.com> Date: Mon, 15 Jan 2018 06:04:36 +0000 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.3.0 MIME-Version: 1.0 In-Reply-To: Content-Language: en-GB-large Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Jan 2018 06:04:57 -0000 On 14/01/2018 16:18, Mehmet Erol Sanliturk wrote: > > > On Sun, Jan 14, 2018 at 5:46 PM, Grzegorz Junka > wrote: > > > On 13/01/2018 17:56, Mehmet Erol Sanliturk wrote: > > > > On Sat, Jan 13, 2018 at 7:21 PM, Grzegorz Junka > > >> wrote: > >     Hello, > >     I am installing a FreeBSD server based on Supermicro H8SML-iF. >     There are three PCIe slots to which I installed 2 NVMe > drives and >     one network card Intel I350-T4 (with 4 Ethernet slots). > >     I am observing a strange behavior where the system doesn't > boot if >     all three PCIe slots are populated. It shows this message: > >     nvme0: mem 0xfd8fc000-0xfd8fffff irq > 24 at >     device 0.0 on pci1 >     nvme0: controller ready did not become 1 within 30000 ms >     nvme0: did not complete shutdown within 5 seconds of > notification > >     The I see a kernel panic/dump and the system reboots after > 15 seconds. > >     If I remove one card, either one of the NVMe drives or the > network >     card, the system boots fine. Also, if in BIOS I set PnP OS > to YES >     then sometimes it boots (but not always). If I set PnP OS > to NO, >     and all three cards are installed, the system never boots. > >     When the system boots OK I can see that the network card is >     reported as 4 separate devices on one of the PCIe slots. I > tried >     different NVMe drives as well as changing which device is >     installed to which slot but the result seems to be the > same in any >     case. > >     What may be the issue? Amount of power drawn by the > hardware? Too >     many devices not supported by the motherboard? Too many > interrupts >     for the FreeBSD kernel to handle? > >     Any help would be greatly appreciated. > >     GregJ > >     _______________________________________________ > > > > > > From my experience from other trade marked main boards , an > action may be to check manual of your server board to see > whether there are rules about use of these slots : Sometimes > differently shaped slots are supplied with same ports : If one > slot is occupied , the other slot should be left open , or > rules about not to insert such a kind of device into a slot , > for example , graphic cards . > > > Mehmet Erol Sanliturk > > > I checked the manual but couldn't find any restrictions regarding > PCIe ports. It only says how many lanes are available in each > slot. Would there be any obvious BIOS setting that could cause > this issue? I tried after resetting BIOS to default settings but > maybe something is set incorrectly by default? > > GregJ > _______________________________________________ > > > > > > http://www.supermicro.com/Aplus/motherboard/Opteron3000/SR56x0/H8SML-iF.cfm > H8SML-iF > > > On the above page , click "OS Compatibility" > > > On the following page , click "SR5650" > > http://www.supermicro.com/Aplus/support/resources/OS/OS_Comp_SR5650.cfm > OS Compatibility Chart > > > On the column ( third ) > > H8SML-7F > H8SML-7 > H8SML-iF > H8SML-i > > > there listed only * > * > ** > * > * > * > * > FreeBSD 8.0 > FreeBSD 9.1 > > From this list , it may be said that , this mother board date is old , > means , it seems that the new OS versions are not tested after > currently tested OS versions . > > > To check interaction between operating system and your Supermicro > H8SML-iF , select one of the suitable operating system ( Unix class > OSes are more suitable ) for you and tested on this card , and try to > install it as you like your installed components . If it boots > successfully , it means that there is an incompatibility between your > FreeBSD and the main board . If no one of them boots , then you may > conclude that , there is a problem in your settings . > > > BIOS settings are important , because , OS communicates with the main > board through these settings . > > > In manual ( downloaded from the above page : > Manual Revision 1.0c > Release Date: March 12, 2014 ) , page 4-9  , "PCI/PnP Configuration" > is defined . > If PnP is selected YES. OS adjusts some device settings  . If NO is > selected , BIOS adjusts some device settings . When BIOS adjusted > device settings are not conforming to OS parameters , the result will > be "FAIL" . > > Therefore , more suitable selection is YES . > > > Another point is that , there are many more BIOS selectable parameters > and jumpers about PCI slots and others  . > There are some BIOS settings for PCI slots : > > PCI X4 Slot 6 ( page 4-9 ) > PCI x8 Slot 7 ( page 4-10 ) > > > > Please review these BIOS settings in your manual and set them with > respect to your requirements . > Thanks Mehmet for looking into this. It's an old motherboard but my point is that it boots fine when either: one NVMe and the network card, or both NVMe are installed, but not when all three are installed. How would that be related to FreeBSD compatibility? The chipset and all devices that I am trying to install are supported by FreeBSD 11.x. I just tried booting into a Debian live system and it also didn't enumerate NVMe drives properly. This means that it's not FreeBSD related and is no longer relevant for this list. I will try to play with BIOS settings to see if I can make it work that way. Thanks for all the help. Regards GregJ From owner-freebsd-drivers@freebsd.org Mon Jan 15 06:18:46 2018 Return-Path: Delivered-To: freebsd-drivers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CD3A1EA7578 for ; Mon, 15 Jan 2018 06:18:46 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-io0-x22a.google.com (mail-io0-x22a.google.com [IPv6:2607:f8b0:4001:c06::22a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8FC708156E for ; Mon, 15 Jan 2018 06:18:46 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-io0-x22a.google.com with SMTP id f89so3912519ioj.4 for ; Sun, 14 Jan 2018 22:18:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc; bh=SeA1p6h8YVvM3DjMdJjKDSZOihp2kVMiXdxVn+1yjfY=; b=Lx0aLmpFRudzETNrez002rQAxjTslxqv2u3FtkxJ+sqc427V5XRjnYd5aP9FeIB04H /6UR5EEWX+GFhYAPpEQcZRvLggPRcMDmMWJWr1YgO86AOyWcojB3go+fyyZWTdaCFyU/ YKAhQzntkSrc8MxGuhP9PjRhfYrV+T/nM5j1+9lFoXE0+aTf3qb02QJaV89mqTLJgpR5 4J9ST1xA6VlbAP2leY+6jGdk7WvoyGC7i943lUuUipYbhERL2NWWzYDryf/1k73zJ4oM 0Q0gmHqlQA65SpFGSDF3kw4BQJAIOSwShD1RjZEXWZHyqsynQCUd/MHkXYDhAHemWp0T yneg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc; bh=SeA1p6h8YVvM3DjMdJjKDSZOihp2kVMiXdxVn+1yjfY=; b=ujGTYb7UFZk3zipmCLzXboHze7ch7vq1xviXVfriwmiG0jP9Q4UD78ymcOzJIO48B2 SG+esZENAwhX47S0u8fAnhxi0X+E3nEe5l2mDHdkYXumE0/pPNmtBWX1hcdhK6Hxgdgc Ul+7qokSSKGN4S7im4bpPj8wWDZloBjH5MpBZ7ab1A5Hw+AVugq7FqKYjT+OUgj7NVM3 gdcXMQisg7rSsx2gRgU/bNLXrCBGbZ+LWiAucdy8iZfdVy5OQO0tNs8KXEYTCo0MW3na 5X+jlgBZXtgEoiAKsD8dD9KoPCga10Q6Yc+fT0MxC8qQ29CO3P7zuaaGWEI6Kaph+VRS pW/Q== X-Gm-Message-State: AKwxytd0reBvG0S+mx33jPwblxl0t0vM3th6A5UZDJ9trkNmDbOAGX/o /oPD/0zr1a9+CBrcRNLdbm4xlkwGD85iFjiTexNbrA== X-Google-Smtp-Source: ACJfBovRPQL83JH6pgJ17lFvWTsBxcx/Ug+jI9/EfEz51DNA93ZZ9UwqbRzCEQA1kMJhXaxoZ4TdJ4FfQRe9Fu/5GKM= X-Received: by 10.107.167.69 with SMTP id q66mr882720ioe.130.1515997125642; Sun, 14 Jan 2018 22:18:45 -0800 (PST) MIME-Version: 1.0 Sender: wlosh@bsdimp.com Received: by 10.79.199.131 with HTTP; Sun, 14 Jan 2018 22:18:44 -0800 (PST) X-Originating-IP: [2603:300b:6:5100:7125:ff5f:2cdf:ed98] Received: by 10.79.199.131 with HTTP; Sun, 14 Jan 2018 22:18:44 -0800 (PST) In-Reply-To: <8df1e967-01e0-d3c2-e14c-64c7fc8c66b0@gjunka.com> References: <3d0ad00c-5214-71b0-017b-c2d5ba608e37@gjunka.com> <8df1e967-01e0-d3c2-e14c-64c7fc8c66b0@gjunka.com> From: Warner Losh Date: Sun, 14 Jan 2018 23:18:44 -0700 X-Google-Sender-Auth: FWKg1inrzo-sp7Mj9wNnu26UKqU Message-ID: Subject: Re: Server doesn't boot when 3 PCIe slots are populated To: Grzegorz Junka Cc: freebsd-questions@freebsd.org, freebsd-drivers@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Jan 2018 06:18:46 -0000 On Jan 14, 2018 11:05 PM, "Grzegorz Junka" wrote: On 14/01/2018 16:18, Mehmet Erol Sanliturk wrote: > > > On Sun, Jan 14, 2018 at 5:46 PM, Grzegorz Junka list1@gjunka.com>> wrote: > > > On 13/01/2018 17:56, Mehmet Erol Sanliturk wrote: > > > > On Sat, Jan 13, 2018 at 7:21 PM, Grzegorz Junka > > >> wrote: > > Hello, > > I am installing a FreeBSD server based on Supermicro H8SML-iF. > There are three PCIe slots to which I installed 2 NVMe > drives and > one network card Intel I350-T4 (with 4 Ethernet slots). > > I am observing a strange behavior where the system doesn't > boot if > all three PCIe slots are populated. It shows this message: > > nvme0: mem 0xfd8fc000-0xfd8fffff irq > 24 at > device 0.0 on pci1 > nvme0: controller ready did not become 1 within 30000 ms > nvme0: did not complete shutdown within 5 seconds of > notification > > The I see a kernel panic/dump and the system reboots after > 15 seconds. > > If I remove one card, either one of the NVMe drives or the > network > card, the system boots fine. Also, if in BIOS I set PnP OS > to YES > then sometimes it boots (but not always). If I set PnP OS > to NO, > and all three cards are installed, the system never boots. > > When the system boots OK I can see that the network card is > reported as 4 separate devices on one of the PCIe slots. I > tried > different NVMe drives as well as changing which device is > installed to which slot but the result seems to be the > same in any > case. > > What may be the issue? Amount of power drawn by the > hardware? Too > many devices not supported by the motherboard? Too many > interrupts > for the FreeBSD kernel to handle? > > Any help would be greatly appreciated. > > GregJ > > _______________________________________________ > > > > > > From my experience from other trade marked main boards , an > action may be to check manual of your server board to see > whether there are rules about use of these slots : Sometimes > differently shaped slots are supplied with same ports : If one > slot is occupied , the other slot should be left open , or > rules about not to insert such a kind of device into a slot , > for example , graphic cards . > > > Mehmet Erol Sanliturk > > > I checked the manual but couldn't find any restrictions regarding > PCIe ports. It only says how many lanes are available in each > slot. Would there be any obvious BIOS setting that could cause > this issue? I tried after resetting BIOS to default settings but > maybe something is set incorrectly by default? > > GregJ > _______________________________________________ > > > > > > http://www.supermicro.com/Aplus/motherboard/Opteron3000/SR56 > x0/H8SML-iF.cfm > H8SML-iF > > > On the above page , click "OS Compatibility" > > > On the following page , click "SR5650" > > http://www.supermicro.com/Aplus/support/resources/OS/OS_Comp_SR5650.cfm > OS Compatibility Chart > > > On the column ( third ) > > H8SML-7F > H8SML-7 > H8SML-iF > H8SML-i > > > there listed only * > * > ** > * > * > * > * > > FreeBSD 8.0 > FreeBSD 9.1 > > From this list , it may be said that , this mother board date is old , > means , it seems that the new OS versions are not tested after currently > tested OS versions . > > > To check interaction between operating system and your Supermicro H8SML-iF > , select one of the suitable operating system ( Unix class OSes are more > suitable ) for you and tested on this card , and try to install it as you > like your installed components . If it boots successfully , it means that > there is an incompatibility between your FreeBSD and the main board . If no > one of them boots , then you may conclude that , there is a problem in your > settings . > > > BIOS settings are important , because , OS communicates with the main > board through these settings . > > > In manual ( downloaded from the above page : > Manual Revision 1.0c > Release Date: March 12, 2014 ) , page 4-9 , "PCI/PnP Configuration" is > defined . > If PnP is selected YES. OS adjusts some device settings . If NO is > selected , BIOS adjusts some device settings . When BIOS adjusted device > settings are not conforming to OS parameters , the result will be "FAIL" . > > Therefore , more suitable selection is YES . > > > Another point is that , there are many more BIOS selectable parameters and > jumpers about PCI slots and others . > There are some BIOS settings for PCI slots : > > PCI X4 Slot 6 ( page 4-9 ) > PCI x8 Slot 7 ( page 4-10 ) > > > > Please review these BIOS settings in your manual and set them with respect > to your requirements . > > Thanks Mehmet for looking into this. It's an old motherboard but my point is that it boots fine when either: one NVMe and the network card, or both NVMe are installed, but not when all three are installed. How would that be related to FreeBSD compatibility? The chipset and all devices that I am trying to install are supported by FreeBSD 11.x. I just tried booting into a Debian live system and it also didn't enumerate NVMe drives properly. This means that it's not FreeBSD related and is no longer relevant for this list. I will try to play with BIOS settings to see if I can make it work that way. Thanks for all the help. Nvme drives are weird about power. I distrust the power estimate of 5-9w earlier in the thread... given the oddity with debian, it's not too crazy to think that. How far does FreeBSD boot though? Warner Warner _______________________________________________ freebsd-drivers@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-drivers To unsubscribe, send any mail to "freebsd-drivers-unsubscribe@freebsd.org" From owner-freebsd-drivers@freebsd.org Mon Jan 15 06:44:19 2018 Return-Path: Delivered-To: freebsd-drivers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D9B1FEB2DE0; Mon, 15 Jan 2018 06:44:19 +0000 (UTC) (envelope-from list1@gjunka.com) Received: from msa1.earth.yoonka.com (yoonka.com [88.98.225.149]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "msa1.earth.yoonka.com", Issuer "msa1.earth.yoonka.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 6EA788253F; Mon, 15 Jan 2018 06:44:18 +0000 (UTC) (envelope-from list1@gjunka.com) Received: from ultrabook.yoonka.com (188.29.165.65.threembb.co.uk [188.29.165.65]) (authenticated bits=0) by msa1.earth.yoonka.com (8.15.2/8.15.2) with ESMTPSA id w0F6iDb7041285 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO); Mon, 15 Jan 2018 06:44:15 GMT (envelope-from list1@gjunka.com) X-Authentication-Warning: msa1.earth.yoonka.com: Host 188.29.165.65.threembb.co.uk [188.29.165.65] claimed to be ultrabook.yoonka.com Subject: Re: Server doesn't boot when 3 PCIe slots are populated To: Warner Losh Cc: freebsd-questions@freebsd.org, freebsd-drivers@freebsd.org References: <3d0ad00c-5214-71b0-017b-c2d5ba608e37@gjunka.com> <8df1e967-01e0-d3c2-e14c-64c7fc8c66b0@gjunka.com> From: Grzegorz Junka Message-ID: <0e582bdb-e1f9-438c-3da2-2bcdc950aab5@gjunka.com> Date: Mon, 15 Jan 2018 06:44:06 +0000 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.3.0 MIME-Version: 1.0 In-Reply-To: Content-Language: en-GB-large Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Jan 2018 06:44:20 -0000 On 15/01/2018 06:18, Warner Losh wrote: > > > On Jan 14, 2018 11:05 PM, "Grzegorz Junka" > wrote: > > > On 14/01/2018 16:18, Mehmet Erol Sanliturk wrote: > > > > On Sun, Jan 14, 2018 at 5:46 PM, Grzegorz Junka > > >> wrote: > > >     On 13/01/2018 17:56, Mehmet Erol Sanliturk wrote: > > > >         On Sat, Jan 13, 2018 at 7:21 PM, Grzegorz Junka >         > > >         > >>> wrote: > >             Hello, > >             I am installing a FreeBSD server based on > Supermicro H8SML-iF. >             There are three PCIe slots to which I installed 2 NVMe >         drives and >             one network card Intel I350-T4 (with 4 Ethernet > slots). > >             I am observing a strange behavior where the system > doesn't >         boot if >             all three PCIe slots are populated. It shows this > message: > >             nvme0: mem > 0xfd8fc000-0xfd8fffff irq >         24 at >             device 0.0 on pci1 >             nvme0: controller ready did not become 1 within > 30000 ms >             nvme0: did not complete shutdown within 5 seconds of >         notification > >             The I see a kernel panic/dump and the system > reboots after >         15 seconds. > >             If I remove one card, either one of the NVMe > drives or the >         network >             card, the system boots fine. Also, if in BIOS I > set PnP OS >         to YES >             then sometimes it boots (but not always). If I set > PnP OS >         to NO, >             and all three cards are installed, the system > never boots. > >             When the system boots OK I can see that the > network card is >             reported as 4 separate devices on one of the PCIe > slots. I >         tried >             different NVMe drives as well as changing which > device is >             installed to which slot but the result seems to be the >         same in any >             case. > >             What may be the issue? Amount of power drawn by the >         hardware? Too >             many devices not supported by the motherboard? Too > many >         interrupts >             for the FreeBSD kernel to handle? > >             Any help would be greatly appreciated. > >             GregJ > >             _______________________________________________ > > > > > >         From my experience from other trade marked main boards > , an >         action may be to check manual of your server board to see >         whether there are rules about use of these slots : > Sometimes >         differently shaped slots are supplied with same ports > : If one >         slot is occupied , the other slot should be left open , or >         rules about not to insert such a kind of device into a > slot , >         for example , graphic cards . > > >         Mehmet Erol Sanliturk > > >     I checked the manual but couldn't find any restrictions > regarding >     PCIe ports. It only says how many lanes are available in each >     slot. Would there be any obvious BIOS setting that could cause >     this issue? I tried after resetting BIOS to default > settings but >     maybe something is set incorrectly by default? > >     GregJ >     _______________________________________________ > > > > > > http://www.supermicro.com/Aplus/motherboard/Opteron3000/SR56x0/H8SML-iF.cfm > > H8SML-iF > > > On the above page , click "OS Compatibility" > > > On the following page , click "SR5650" > > http://www.supermicro.com/Aplus/support/resources/OS/OS_Comp_SR5650.cfm > > OS Compatibility Chart > > > On the column ( third ) > > H8SML-7F > H8SML-7 > H8SML-iF > H8SML-i > > > there listed only * > * > ** > * > * > * > * > > FreeBSD 8.0 > FreeBSD 9.1 > > From this list , it may be said that , this mother board date > is old , means , it seems that the new OS versions are not > tested after currently tested OS versions . > > > To check interaction between operating system and your > Supermicro H8SML-iF , select one of the suitable operating > system ( Unix class OSes are more suitable ) for you and > tested on this card , and try to install it as you like your > installed components . If it boots successfully , it means > that there is an incompatibility between your FreeBSD and the > main board . If no one of them boots , then you may conclude > that , there is a problem in your settings . > > > BIOS settings are important , because , OS communicates with > the main board through these settings . > > > In manual ( downloaded from the above page : > Manual Revision 1.0c > Release Date: March 12, 2014 ) , page 4-9  , "PCI/PnP > Configuration" is defined . > If PnP is selected YES. OS adjusts some device settings  . If > NO is selected , BIOS adjusts some device settings . When BIOS > adjusted device settings are not conforming to OS parameters , > the result will be "FAIL" . > > Therefore , more suitable selection is YES . > > > Another point is that , there are many more BIOS selectable > parameters and jumpers about PCI slots and others  . > There are some BIOS settings for PCI slots : > > PCI X4 Slot 6 ( page 4-9 ) > PCI x8 Slot 7 ( page 4-10 ) > > > > Please review these BIOS settings in your manual and set them > with respect to your requirements . > > > Thanks Mehmet for looking into this. It's an old motherboard but > my point is that it boots fine when either: one NVMe and the > network card, or both NVMe are installed, but not when all three > are installed. How would that be related to FreeBSD compatibility? > The chipset and all devices that I am trying to install are > supported by FreeBSD 11.x. > > I just tried booting into a Debian live system and it also didn't > enumerate NVMe drives properly. This means that it's not FreeBSD > related and is no longer relevant for this list. I will try to > play with BIOS settings to see if I can make it work that way. > Thanks for all the help. > > > > Nvme drives are weird about power. I distrust the power estimate of > 5-9w earlier in the thread... given the oddity with debian, it's not > too crazy to think that. How far does FreeBSD boot though? > I tried with a different power supply but the outcome was exactly the same. Sometimes FreeBSD boots fine but one of the NVMe drives is not visible (i.e. dmesg grep shows only one NVMe). When it doesn't work it boots up to the point of enumerating drives (SATA, USB, NVMe). Then it stops at the first NVMe and reboots. The funny thing is that very often it's enough to pull out one of the cards and put it back in. Then the system boots fine with all three cards. I had that a few times. Once it's booted it works, I can restart the system and it boots every time. As soon as I power off, unplug from the power main, wait a few minutes and power it on again, the issue comes back - can't boot as NVMe can't be enumerated. I though it might be caused by the hardware being too cold. I left the server once overnight but it didn't boot up, it was trying and restarting the whole night. GregJ From owner-freebsd-drivers@freebsd.org Mon Jan 15 09:44:34 2018 Return-Path: Delivered-To: freebsd-drivers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E77EDE6D3BF; Mon, 15 Jan 2018 09:44:34 +0000 (UTC) (envelope-from m.e.sanliturk@gmail.com) Received: from mail-ot0-x22b.google.com (mail-ot0-x22b.google.com [IPv6:2607:f8b0:4003:c0f::22b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A577F69992; Mon, 15 Jan 2018 09:44:34 +0000 (UTC) (envelope-from m.e.sanliturk@gmail.com) Received: by mail-ot0-x22b.google.com with SMTP id r4so6312468oti.12; Mon, 15 Jan 2018 01:44:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=xJChbAZw+czmM6zsoXu3xJIyn72fbHSPMvojhv++0eU=; b=AOl4vVfpnyOjFd9xReJI6UtTo3BQaGe7EbQCIQHiN9WkAFK+mNwxZYAdfi1vvlJuS4 z0IApI+gnMz/SNMi66SfHWW1Dq6t8KZKjAXcC715XrSNbcdN6/CNjuB3Y7kHDNiPsK1l m3oNQje/sOJ4e0ZbDtKZDJLgw6MCe7Rskg3fgNtDIhQdsaBw58SpM7tf/uZcYIGuvZIm MjT77k3N0IvL7wDucHknC0gZjnGlp28Fk58eCh2UT/yqJqdANGIxgcd8b89Q2RYJt7Zz Wm3dIvdIDDZ9W8unlKlDhUcSjtxsaX5rebHGm+FwOQF74N2mQsMmPPnJOI6NaBWY9hr7 +nNg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=xJChbAZw+czmM6zsoXu3xJIyn72fbHSPMvojhv++0eU=; b=HIngeHLIc4Pz6rIuYdJl5Iz8ziOqKgDgk5Q+vhZmIulxJX+0uveEYPyxHhtbXcC0mP xqWzHLdLCATx41RPpTyswY1ac/k7sN6kIAWa7jH5wSUQbiTMvO30OqgiTrg11vF7sDAg FLcPGZu82mU7dbYoCAy6LtxOlbcOELGgx+1PNqqkCtVIKVSxL7duyGP4wQIZ9RAw9qfF bfHkz5N+/Xo+YvrfPj2rp9hZHFFrdcQ5x+wZ0PWH6XqYxxP84umkF4kJHFPDBwz4UJx5 tIEYMyO2mdf4cJP2AVjYUZb2QKNs84JoY/jvyATiLJWL0K2NeTAl1IhugiF2hDwbSo1F pU0w== X-Gm-Message-State: AKwxytfn+iCmquY4hzFEJN06/Ezvqhixi3Plfl4xNW2fnYkcAohDUbAR G8oaRgQ/ONpUEK3l412y9t8F+mhpRKPW/6ceFJI= X-Google-Smtp-Source: ACJfBouAZQnlrFR3Cu+bVwl1dTlZC3D91ShLoWxj/1FHnlUwlRJqEiqXD9OeTBwqpDMtF4tgtzBHVQ3CvrDufX3O4lc= X-Received: by 10.157.74.76 with SMTP id d12mr15010220otj.231.1516009473882; Mon, 15 Jan 2018 01:44:33 -0800 (PST) MIME-Version: 1.0 Received: by 10.157.0.2 with HTTP; Mon, 15 Jan 2018 01:44:33 -0800 (PST) In-Reply-To: <0e582bdb-e1f9-438c-3da2-2bcdc950aab5@gjunka.com> References: <3d0ad00c-5214-71b0-017b-c2d5ba608e37@gjunka.com> <8df1e967-01e0-d3c2-e14c-64c7fc8c66b0@gjunka.com> <0e582bdb-e1f9-438c-3da2-2bcdc950aab5@gjunka.com> From: Mehmet Erol Sanliturk Date: Mon, 15 Jan 2018 12:44:33 +0300 Message-ID: Subject: Re: Server doesn't boot when 3 PCIe slots are populated To: Grzegorz Junka Cc: Warner Losh , FreeBSD Questions Mailing List , freebsd-drivers@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Jan 2018 09:44:35 -0000 On Mon, Jan 15, 2018 at 9:44 AM, Grzegorz Junka wrote: > > On 15/01/2018 06:18, Warner Losh wrote: > >> >> >> On Jan 14, 2018 11:05 PM, "Grzegorz Junka" > list1@gjunka.com>> wrote: >> >> >> On 14/01/2018 16:18, Mehmet Erol Sanliturk wrote: >> >> >> >> On Sun, Jan 14, 2018 at 5:46 PM, Grzegorz Junka >> >> >> wrote: >> >> >> On 13/01/2018 17:56, Mehmet Erol Sanliturk wrote: >> >> >> >> On Sat, Jan 13, 2018 at 7:21 PM, Grzegorz Junka >> >> > >> >> >>> wrote: >> >> Hello, >> >> I am installing a FreeBSD server based on >> Supermicro H8SML-iF. >> There are three PCIe slots to which I installed 2 NVMe >> drives and >> one network card Intel I350-T4 (with 4 Ethernet >> slots). >> >> I am observing a strange behavior where the system >> doesn't >> boot if >> all three PCIe slots are populated. It shows this >> message: >> >> nvme0: mem >> 0xfd8fc000-0xfd8fffff irq >> 24 at >> device 0.0 on pci1 >> nvme0: controller ready did not become 1 within >> 30000 ms >> nvme0: did not complete shutdown within 5 seconds of >> notification >> >> The I see a kernel panic/dump and the system >> reboots after >> 15 seconds. >> >> If I remove one card, either one of the NVMe >> drives or the >> network >> card, the system boots fine. Also, if in BIOS I >> set PnP OS >> to YES >> then sometimes it boots (but not always). If I set >> PnP OS >> to NO, >> and all three cards are installed, the system >> never boots. >> >> When the system boots OK I can see that the >> network card is >> reported as 4 separate devices on one of the PCIe >> slots. I >> tried >> different NVMe drives as well as changing which >> device is >> installed to which slot but the result seems to be the >> same in any >> case. >> >> What may be the issue? Amount of power drawn by the >> hardware? Too >> many devices not supported by the motherboard? Too >> many >> interrupts >> for the FreeBSD kernel to handle? >> >> Any help would be greatly appreciated. >> >> GregJ >> >> _______________________________________________ >> >> >> >> >> >> From my experience from other trade marked main boards >> , an >> action may be to check manual of your server board to see >> whether there are rules about use of these slots : >> Sometimes >> differently shaped slots are supplied with same ports >> : If one >> slot is occupied , the other slot should be left open , or >> rules about not to insert such a kind of device into a >> slot , >> for example , graphic cards . >> >> >> Mehmet Erol Sanliturk >> >> >> I checked the manual but couldn't find any restrictions >> regarding >> PCIe ports. It only says how many lanes are available in each >> slot. Would there be any obvious BIOS setting that could cause >> this issue? I tried after resetting BIOS to default >> settings but >> maybe something is set incorrectly by default? >> >> GregJ >> _______________________________________________ >> >> >> >> >> >> http://www.supermicro.com/Aplus/motherboard/Opteron3000/SR56 >> x0/H8SML-iF.cfm >> > 6x0/H8SML-iF.cfm> >> H8SML-iF >> >> >> On the above page , click "OS Compatibility" >> >> >> On the following page , click "SR5650" >> >> http://www.supermicro.com/Aplus/support/resources/OS/OS_Comp >> _SR5650.cfm >> > p_SR5650.cfm> >> OS Compatibility Chart >> >> >> On the column ( third ) >> >> H8SML-7F >> H8SML-7 >> H8SML-iF >> H8SML-i >> >> >> there listed only * >> * >> ** >> * >> * >> * >> * >> >> FreeBSD 8.0 >> FreeBSD 9.1 >> >> From this list , it may be said that , this mother board date >> is old , means , it seems that the new OS versions are not >> tested after currently tested OS versions . >> >> >> To check interaction between operating system and your >> Supermicro H8SML-iF , select one of the suitable operating >> system ( Unix class OSes are more suitable ) for you and >> tested on this card , and try to install it as you like your >> installed components . If it boots successfully , it means >> that there is an incompatibility between your FreeBSD and the >> main board . If no one of them boots , then you may conclude >> that , there is a problem in your settings . >> >> >> BIOS settings are important , because , OS communicates with >> the main board through these settings . >> >> >> In manual ( downloaded from the above page : >> Manual Revision 1.0c >> Release Date: March 12, 2014 ) , page 4-9 , "PCI/PnP >> Configuration" is defined . >> If PnP is selected YES. OS adjusts some device settings . If >> NO is selected , BIOS adjusts some device settings . When BIOS >> adjusted device settings are not conforming to OS parameters , >> the result will be "FAIL" . >> >> Therefore , more suitable selection is YES . >> >> >> Another point is that , there are many more BIOS selectable >> parameters and jumpers about PCI slots and others . >> There are some BIOS settings for PCI slots : >> >> PCI X4 Slot 6 ( page 4-9 ) >> PCI x8 Slot 7 ( page 4-10 ) >> >> >> >> Please review these BIOS settings in your manual and set them >> with respect to your requirements . >> >> >> Thanks Mehmet for looking into this. It's an old motherboard but >> my point is that it boots fine when either: one NVMe and the >> network card, or both NVMe are installed, but not when all three >> are installed. How would that be related to FreeBSD compatibility? >> The chipset and all devices that I am trying to install are >> supported by FreeBSD 11.x. >> >> I just tried booting into a Debian live system and it also didn't >> enumerate NVMe drives properly. This means that it's not FreeBSD >> related and is no longer relevant for this list. I will try to >> play with BIOS settings to see if I can make it work that way. >> Thanks for all the help. >> >> >> >> Nvme drives are weird about power. I distrust the power estimate of 5-9w >> earlier in the thread... given the oddity with debian, it's not too crazy >> to think that. How far does FreeBSD boot though? >> >> > I tried with a different power supply but the outcome was exactly the > same. Sometimes FreeBSD boots fine but one of the NVMe drives is not > visible (i.e. dmesg grep shows only one NVMe). When it doesn't work it > boots up to the point of enumerating drives (SATA, USB, NVMe). Then it > stops at the first NVMe and reboots. > > The funny thing is that very often it's enough to pull out one of the > cards and put it back in. Then the system boots fine with all three cards. > I had that a few times. Once it's booted it works, I can restart the system > and it boots every time. As soon as I power off, unplug from the power > main, wait a few minutes and power it on again, the issue comes back - > can't boot as NVMe can't be enumerated. > > I though it might be caused by the hardware being too cold. I left the > server once overnight but it didn't boot up, it was trying and restarting > the whole night. > > GregJ > > > _______________________________________________ > > The above explanation brings mind to the "impedance mismatch in electronics" problem . ( Please search impedance mismatch in electronics impedance matching in electronics in Internet if you want explanations about them . ) When all of these cards are inserted into slots simultaneously , their accumulated electronic effect may distort behaviour of your mother board circuits or attached card circuit(s) . Therefore , if you can find another NVMe and/or network card , please test their effect . Such tests may be inconclusive because mother board circuits may be affected negatively from "properly" operating add on cards when they are inserted together . If it is feasible for you , you may use USB attached network card(s) to eliminate network card attachment . Or you may use a more capable one NVMe card instead of two smaller NVMe cards , or you may use only one of them , or/and select an SATA SSD . Such a choice would save your investment and produces a working server with a "little" loss when compared to "all" . Mehmet Erol Sanliturk From owner-freebsd-drivers@freebsd.org Mon Jan 15 15:43:34 2018 Return-Path: Delivered-To: freebsd-drivers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2BF1BEB5AFC for ; Mon, 15 Jan 2018 15:43:34 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-wm0-x234.google.com (mail-wm0-x234.google.com [IPv6:2a00:1450:400c:c09::234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9FEC97D2C1 for ; Mon, 15 Jan 2018 15:43:33 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-wm0-x234.google.com with SMTP id f71so2727162wmf.0 for ; Mon, 15 Jan 2018 07:43:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc; bh=xpkZ0s0FxVolTsPohexcaEKh55rzTd67Ktl6Xz4e5VQ=; b=Q/Re+Fv84WBMsF4VCwzd6Ekitrq0rarT5sw0JtUryT5GUXeDM5JQJoup8PLbJLScd5 YJ8XWbt2d3i/3J6xM4qpObVDSvEfdEQ/tnwY6EabltxBgxDGaWt1iTKjM5mBz7yeCkZC ZJUpEXEeZutfMLTDDl02d+sxUw/KDrsWNuqurwfyz6Bw//v5DaNvBQ3P8EH0fY0ErnkH puFgzvyd/TBxqBecE92LXztSJ2rglLti0wd/HNih9uQk8CHUXiilY8bUfAe6jOYMQ0bp Z0YJt5DMoZ0HJV7CTSEthfkK1zJS5gwTLTS5uPgT16bSuTUJ90dDQSsjKsdk6Ph7YWWe kDsA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc; bh=xpkZ0s0FxVolTsPohexcaEKh55rzTd67Ktl6Xz4e5VQ=; b=Q+9oi7kWd/p4wEx3lzjvBcNDZ4lP1z0KZVyvecfQPDN4oedeTNk68K5LnfG6oh0sls yZ6wWUi5F8BHvouUmW2ck+9VbZ6lUH7g5Nw4bzgY116H0qtV8nmApjbD0go2QlbBXly8 Q94IN4i7VbkV5N0V9cpXyVHuYSXOZqlQZWPNK20CFrkgyl3Q7KKZ7fUO1/APOpDigSSx 88/xbxNhXNhleGsk4aXDc1tpUe7kXLbQWdHYlBZ/tNLGXMGGUDcZfLVQB2DPQwmDuTzu uvF3kUV8tVYKXwTMClp3MYtLMu7mMtmUOvf3Ct4G+qmMeSZIzWfxPOaO80YFz6v5Gpux rwwg== X-Gm-Message-State: AKwxytfzZ5731K2p+71x7DKVExp+RixtW4fPxYIE9UPiMAHzTbETS40d S1jYx5+p6HXDQbXT+481GJBFz46VsStdX8NSYePAqQ== X-Google-Smtp-Source: ACJfBosw16dtbWsJ+Zzh1ipMai0TESu72BTk+FASzn6OZWQx63o6Rb4F64NFITdmmgapqaW+EfTtD2bo2jeV2awtKJ8= X-Received: by 10.80.171.165 with SMTP id u34mr16165080edc.167.1516031011872; Mon, 15 Jan 2018 07:43:31 -0800 (PST) MIME-Version: 1.0 Sender: wlosh@bsdimp.com Received: by 10.80.195.88 with HTTP; Mon, 15 Jan 2018 07:43:31 -0800 (PST) X-Originating-IP: [50.253.99.174] In-Reply-To: <0e582bdb-e1f9-438c-3da2-2bcdc950aab5@gjunka.com> References: <3d0ad00c-5214-71b0-017b-c2d5ba608e37@gjunka.com> <8df1e967-01e0-d3c2-e14c-64c7fc8c66b0@gjunka.com> <0e582bdb-e1f9-438c-3da2-2bcdc950aab5@gjunka.com> From: Warner Losh Date: Mon, 15 Jan 2018 08:43:31 -0700 X-Google-Sender-Auth: ZEHAZfwlV7egHwVthvP0vKDWJY4 Message-ID: Subject: Re: Server doesn't boot when 3 PCIe slots are populated To: Grzegorz Junka Cc: freebsd-questions@freebsd.org, freebsd-drivers@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Jan 2018 15:43:34 -0000 On Sun, Jan 14, 2018 at 11:44 PM, Grzegorz Junka wrote: > > On 15/01/2018 06:18, Warner Losh wrote: > > > > On Jan 14, 2018 11:05 PM, "Grzegorz Junka" wrote: > > > On 14/01/2018 16:18, Mehmet Erol Sanliturk wrote: > >> >> >> On Sun, Jan 14, 2018 at 5:46 PM, Grzegorz Junka > > wrote: >> >> >> On 13/01/2018 17:56, Mehmet Erol Sanliturk wrote: >> >> >> >> On Sat, Jan 13, 2018 at 7:21 PM, Grzegorz Junka >> >> >> wrote: >> >> Hello, >> >> I am installing a FreeBSD server based on Supermicro H8SML-iF. >> There are three PCIe slots to which I installed 2 NVMe >> drives and >> one network card Intel I350-T4 (with 4 Ethernet slots). >> >> I am observing a strange behavior where the system doesn't >> boot if >> all three PCIe slots are populated. It shows this message: >> >> nvme0: mem 0xfd8fc000-0xfd8fffff irq >> 24 at >> device 0.0 on pci1 >> nvme0: controller ready did not become 1 within 30000 ms >> nvme0: did not complete shutdown within 5 seconds of >> notification >> >> The I see a kernel panic/dump and the system reboots after >> 15 seconds. >> >> If I remove one card, either one of the NVMe drives or the >> network >> card, the system boots fine. Also, if in BIOS I set PnP OS >> to YES >> then sometimes it boots (but not always). If I set PnP OS >> to NO, >> and all three cards are installed, the system never boots. >> >> When the system boots OK I can see that the network card is >> reported as 4 separate devices on one of the PCIe slots. I >> tried >> different NVMe drives as well as changing which device is >> installed to which slot but the result seems to be the >> same in any >> case. >> >> What may be the issue? Amount of power drawn by the >> hardware? Too >> many devices not supported by the motherboard? Too many >> interrupts >> for the FreeBSD kernel to handle? >> >> Any help would be greatly appreciated. >> >> GregJ >> >> _______________________________________________ >> >> >> >> >> >> From my experience from other trade marked main boards , an >> action may be to check manual of your server board to see >> whether there are rules about use of these slots : Sometimes >> differently shaped slots are supplied with same ports : If one >> slot is occupied , the other slot should be left open , or >> rules about not to insert such a kind of device into a slot , >> for example , graphic cards . >> >> >> Mehmet Erol Sanliturk >> >> >> I checked the manual but couldn't find any restrictions regarding >> PCIe ports. It only says how many lanes are available in each >> slot. Would there be any obvious BIOS setting that could cause >> this issue? I tried after resetting BIOS to default settings but >> maybe something is set incorrectly by default? >> >> GregJ >> _______________________________________________ >> >> >> >> >> >> http://www.supermicro.com/Aplus/motherboard/Opteron3000/SR56 >> x0/H8SML-iF.cfm >> H8SML-iF >> >> >> On the above page , click "OS Compatibility" >> >> >> On the following page , click "SR5650" >> >> http://www.supermicro.com/Aplus/support/resources/OS/OS_Comp_SR5650.cfm >> OS Compatibility Chart >> >> >> On the column ( third ) >> >> H8SML-7F >> H8SML-7 >> H8SML-iF >> H8SML-i >> >> >> there listed only * >> * >> ** >> * >> * >> * >> * >> >> FreeBSD 8.0 >> FreeBSD 9.1 >> >> From this list , it may be said that , this mother board date is old , >> means , it seems that the new OS versions are not tested after currently >> tested OS versions . >> >> >> To check interaction between operating system and your Supermicro >> H8SML-iF , select one of the suitable operating system ( Unix class OSes >> are more suitable ) for you and tested on this card , and try to install it >> as you like your installed components . If it boots successfully , it means >> that there is an incompatibility between your FreeBSD and the main board . >> If no one of them boots , then you may conclude that , there is a problem >> in your settings . >> >> >> BIOS settings are important , because , OS communicates with the main >> board through these settings . >> >> >> In manual ( downloaded from the above page : >> Manual Revision 1.0c >> Release Date: March 12, 2014 ) , page 4-9 , "PCI/PnP Configuration" is >> defined . >> If PnP is selected YES. OS adjusts some device settings . If NO is >> selected , BIOS adjusts some device settings . When BIOS adjusted device >> settings are not conforming to OS parameters , the result will be "FAIL" . >> >> Therefore , more suitable selection is YES . >> >> >> Another point is that , there are many more BIOS selectable parameters >> and jumpers about PCI slots and others . >> There are some BIOS settings for PCI slots : >> >> PCI X4 Slot 6 ( page 4-9 ) >> PCI x8 Slot 7 ( page 4-10 ) >> >> >> >> Please review these BIOS settings in your manual and set them with >> respect to your requirements . >> >> > Thanks Mehmet for looking into this. It's an old motherboard but my point > is that it boots fine when either: one NVMe and the network card, or both > NVMe are installed, but not when all three are installed. How would that be > related to FreeBSD compatibility? The chipset and all devices that I am > trying to install are supported by FreeBSD 11.x. > > I just tried booting into a Debian live system and it also didn't > enumerate NVMe drives properly. This means that it's not FreeBSD related > and is no longer relevant for this list. I will try to play with BIOS > settings to see if I can make it work that way. Thanks for all the help. > > > > Nvme drives are weird about power. I distrust the power estimate of 5-9w > earlier in the thread... given the oddity with debian, it's not too crazy > to think that. How far does FreeBSD boot though? > > > I tried with a different power supply but the outcome was exactly the > same. Sometimes FreeBSD boots fine but one of the NVMe drives is not > visible (i.e. dmesg grep shows only one NVMe). When it doesn't work it > boots up to the point of enumerating drives (SATA, USB, NVMe). Then it > stops at the first NVMe and reboots. > Any panic message / traceback, or just a system reset? > The funny thing is that very often it's enough to pull out one of the > cards and put it back in. Then the system boots fine with all three cards. > I had that a few times. Once it's booted it works, I can restart the system > and it boots every time. As soon as I power off, unplug from the power > main, wait a few minutes and power it on again, the issue comes back - > can't boot as NVMe can't be enumerated. > Sounds like misaligned cards then... Warner > I though it might be caused by the hardware being too cold. I left the > server once overnight but it didn't boot up, it was trying and restarting > the whole night. > > GregJ > > > From owner-freebsd-drivers@freebsd.org Mon Jan 15 16:09:55 2018 Return-Path: Delivered-To: freebsd-drivers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E1E6BEB7623; Mon, 15 Jan 2018 16:09:55 +0000 (UTC) (envelope-from galtsev@kicp.uchicago.edu) Received: from cosmo.uchicago.edu (cosmo.uchicago.edu [128.135.20.71]) by mx1.freebsd.org (Postfix) with ESMTP id B498F7E8AF; Mon, 15 Jan 2018 16:09:55 +0000 (UTC) (envelope-from galtsev@kicp.uchicago.edu) Received: by cosmo.uchicago.edu (Postfix, from userid 48) id C4BDACB8D20; Mon, 15 Jan 2018 10:09:48 -0600 (CST) Received: from 108.68.169.115 (SquirrelMail authenticated user valeri) by cosmo.uchicago.edu with HTTP; Mon, 15 Jan 2018 10:09:48 -0600 (CST) Message-ID: <57380.108.68.169.115.1516032588.squirrel@cosmo.uchicago.edu> In-Reply-To: <0e582bdb-e1f9-438c-3da2-2bcdc950aab5@gjunka.com> References: <3d0ad00c-5214-71b0-017b-c2d5ba608e37@gjunka.com> <8df1e967-01e0-d3c2-e14c-64c7fc8c66b0@gjunka.com> <0e582bdb-e1f9-438c-3da2-2bcdc950aab5@gjunka.com> Date: Mon, 15 Jan 2018 10:09:48 -0600 (CST) Subject: Re: Server doesn't boot when 3 PCIe slots are populated From: "Valeri Galtsev" To: "Grzegorz Junka" Cc: "Warner Losh" , freebsd-questions@freebsd.org, freebsd-drivers@freebsd.org Reply-To: galtsev@kicp.uchicago.edu User-Agent: SquirrelMail/1.4.8-5.el5.centos.7 MIME-Version: 1.0 Content-Type: text/plain;charset=iso-8859-1 Content-Transfer-Encoding: 8bit X-Priority: 3 (Normal) Importance: Normal X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Jan 2018 16:09:56 -0000 On Mon, January 15, 2018 12:44 am, Grzegorz Junka wrote: > > On 15/01/2018 06:18, Warner Losh wrote: >> >> >> On Jan 14, 2018 11:05 PM, "Grzegorz Junka" > > wrote: >> >> >> On 14/01/2018 16:18, Mehmet Erol Sanliturk wrote: >> >> >> >> On Sun, Jan 14, 2018 at 5:46 PM, Grzegorz Junka >> >> >> wrote: >> >> >>     On 13/01/2018 17:56, Mehmet Erol Sanliturk wrote: >> >> >> >>         On Sat, Jan 13, 2018 at 7:21 PM, Grzegorz Junka >>         >> > >>         >> >>> wrote: >> >>             Hello, >> >>             I am installing a FreeBSD server based on >> Supermicro H8SML-iF. >>             There are three PCIe slots to which I >> installed 2 NVMe >>         drives and >>             one network card Intel I350-T4 (with 4 >> Ethernet >> slots). >> >>             I am observing a strange behavior where the >> system >> doesn't >>         boot if >>             all three PCIe slots are populated. It shows >> this >> message: >> >>             nvme0: mem >> 0xfd8fc000-0xfd8fffff irq >>         24 at >>             device 0.0 on pci1 >>             nvme0: controller ready did not become 1 >> within >> 30000 ms >>             nvme0: did not complete shutdown within 5 >> seconds of >>         notification >> >>             The I see a kernel panic/dump and the system >> reboots after >>         15 seconds. >> >>             If I remove one card, either one of the NVMe >> drives or the >>         network >>             card, the system boots fine. Also, if in BIOS >> I >> set PnP OS >>         to YES >>             then sometimes it boots (but not always). If I >> set >> PnP OS >>         to NO, >>             and all three cards are installed, the system >> never boots. >> >>             When the system boots OK I can see that the >> network card is >>             reported as 4 separate devices on one of the >> PCIe >> slots. I >>         tried >>             different NVMe drives as well as changing >> which >> device is >>             installed to which slot but the result seems >> to be the >>         same in any >>             case. >> >>             What may be the issue? Amount of power drawn >> by the >>         hardware? Too >>             many devices not supported by the motherboard? >> Too >> many >>         interrupts >>             for the FreeBSD kernel to handle? >> >>             Any help would be greatly appreciated. >> >>             GregJ >> >>             >> _______________________________________________ >> >> >> >> >> >>         From my experience from other trade marked main >> boards >> , an >>         action may be to check manual of your server board >> to see >>         whether there are rules about use of these slots : >> Sometimes >>         differently shaped slots are supplied with same >> ports >> : If one >>         slot is occupied , the other slot should be left >> open , or >>         rules about not to insert such a kind of device into >> a >> slot , >>         for example , graphic cards . >> >> >>         Mehmet Erol Sanliturk >> >> >>     I checked the manual but couldn't find any restrictions >> regarding >>     PCIe ports. It only says how many lanes are available in >> each >>     slot. Would there be any obvious BIOS setting that could >> cause >>     this issue? I tried after resetting BIOS to default >> settings but >>     maybe something is set incorrectly by default? >> >>     GregJ >>     _______________________________________________ >> >> >> >> >> >> http://www.supermicro.com/Aplus/motherboard/Opteron3000/SR56x0/H8SML-iF.cfm >> >> H8SML-iF >> >> >> On the above page , click "OS Compatibility" >> >> >> On the following page , click "SR5650" >> >> http://www.supermicro.com/Aplus/support/resources/OS/OS_Comp_SR5650.cfm >> >> OS Compatibility Chart >> >> >> On the column ( third ) >> >> H8SML-7F >> H8SML-7 >> H8SML-iF >> H8SML-i >> >> >> there listed only * >> * >> ** >> * >> * >> * >> * >> >> FreeBSD 8.0 >> FreeBSD 9.1 >> >> From this list , it may be said that , this mother board date >> is old , means , it seems that the new OS versions are not >> tested after currently tested OS versions . >> >> >> To check interaction between operating system and your >> Supermicro H8SML-iF , select one of the suitable operating >> system ( Unix class OSes are more suitable ) for you and >> tested on this card , and try to install it as you like your >> installed components . If it boots successfully , it means >> that there is an incompatibility between your FreeBSD and the >> main board . If no one of them boots , then you may conclude >> that , there is a problem in your settings . >> >> >> BIOS settings are important , because , OS communicates with >> the main board through these settings . >> >> >> In manual ( downloaded from the above page : >> Manual Revision 1.0c >> Release Date: March 12, 2014 ) , page 4-9  , "PCI/PnP >> Configuration" is defined . >> If PnP is selected YES. OS adjusts some device settings  . If >> NO is selected , BIOS adjusts some device settings . When BIOS >> adjusted device settings are not conforming to OS parameters , >> the result will be "FAIL" . >> >> Therefore , more suitable selection is YES . >> >> >> Another point is that , there are many more BIOS selectable >> parameters and jumpers about PCI slots and others  . >> There are some BIOS settings for PCI slots : >> >> PCI X4 Slot 6 ( page 4-9 ) >> PCI x8 Slot 7 ( page 4-10 ) >> >> >> >> Please review these BIOS settings in your manual and set them >> with respect to your requirements . >> >> >> Thanks Mehmet for looking into this. It's an old motherboard but >> my point is that it boots fine when either: one NVMe and the >> network card, or both NVMe are installed, but not when all three >> are installed. How would that be related to FreeBSD compatibility? >> The chipset and all devices that I am trying to install are >> supported by FreeBSD 11.x. >> >> I just tried booting into a Debian live system and it also didn't >> enumerate NVMe drives properly. This means that it's not FreeBSD >> related and is no longer relevant for this list. I will try to >> play with BIOS settings to see if I can make it work that way. >> Thanks for all the help. >> >> >> >> Nvme drives are weird about power. I distrust the power estimate of >> 5-9w earlier in the thread... given the oddity with debian, it's not >> too crazy to think that. How far does FreeBSD boot though? >> > > I tried with a different power supply but the outcome was exactly the > same. I assume different power supply is spec'ed at some 30% higher power. Otherwise this is inconclusive. > Sometimes FreeBSD boots fine but one of the NVMe drives is not > visible (i.e. dmesg grep shows only one NVMe). When it doesn't work it > boots up to the point of enumerating drives (SATA, USB, NVMe). Then it > stops at the first NVMe and reboots. It sounds like without regard to third card, only two NVMe cards when they both are plugged in do cause this problem. Am I right? If it is not so, i.e. two NVMe cards in the absence of third card do work, then it may have something to do with usage of PCI address space and inability to allocate such for whatever reason. Anyway at this point I would try to experiment more attempting to boot off live CDs/DVDs with different systems in configuration in which FreeBSD has problem. Linux (Debian, CentOS, Fedora, Ubuntu, Knoppix - any one of them will probably speak for all of them, as it is Linux kernel...), OpenBSD, NetBSD, MS Windows (you don't have to install the last, or have license or register it, just test if you can boot it off installation or recovery disk). Good luck! Valeri > > The funny thing is that very often it's enough to pull out one of the > cards and put it back in. Then the system boots fine with all three > cards. I had that a few times. Once it's booted it works, I can restart > the system and it boots every time. As soon as I power off, unplug from > the power main, wait a few minutes and power it on again, the issue > comes back - can't boot as NVMe can't be enumerated. > > I though it might be caused by the hardware being too cold. I left the > server once overnight but it didn't boot up, it was trying and > restarting the whole night. > > GregJ > > > _______________________________________________ > freebsd-questions@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-questions > To unsubscribe, send any mail to > "freebsd-questions-unsubscribe@freebsd.org" > ++++++++++++++++++++++++++++++++++++++++ Valeri Galtsev Sr System Administrator Department of Astronomy and Astrophysics Kavli Institute for Cosmological Physics University of Chicago Phone: 773-702-4247 ++++++++++++++++++++++++++++++++++++++++ From owner-freebsd-drivers@freebsd.org Mon Jan 15 16:31:06 2018 Return-Path: Delivered-To: freebsd-drivers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 04891EB8905; Mon, 15 Jan 2018 16:31:06 +0000 (UTC) (envelope-from galtsev@kicp.uchicago.edu) Received: from cosmo.uchicago.edu (cosmo.uchicago.edu [128.135.20.71]) by mx1.freebsd.org (Postfix) with ESMTP id BBACB7F55C; Mon, 15 Jan 2018 16:31:05 +0000 (UTC) (envelope-from galtsev@kicp.uchicago.edu) Received: by cosmo.uchicago.edu (Postfix, from userid 48) id 672FECB8D3A; Mon, 15 Jan 2018 10:31:04 -0600 (CST) Received: from 108.68.169.115 (SquirrelMail authenticated user valeri) by cosmo.uchicago.edu with HTTP; Mon, 15 Jan 2018 10:31:04 -0600 (CST) Message-ID: <57715.108.68.169.115.1516033864.squirrel@cosmo.uchicago.edu> In-Reply-To: References: <3d0ad00c-5214-71b0-017b-c2d5ba608e37@gjunka.com> <8df1e967-01e0-d3c2-e14c-64c7fc8c66b0@gjunka.com> <0e582bdb-e1f9-438c-3da2-2bcdc950aab5@gjunka.com> Date: Mon, 15 Jan 2018 10:31:04 -0600 (CST) Subject: Re: Server doesn't boot when 3 PCIe slots are populated From: "Valeri Galtsev" To: "Mehmet Erol Sanliturk" Cc: "Grzegorz Junka" , "FreeBSD Questions Mailing List" , "Warner Losh" , freebsd-drivers@freebsd.org Reply-To: galtsev@kicp.uchicago.edu User-Agent: SquirrelMail/1.4.8-5.el5.centos.7 MIME-Version: 1.0 Content-Type: text/plain;charset=iso-8859-1 Content-Transfer-Encoding: 8bit X-Priority: 3 (Normal) Importance: Normal X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Jan 2018 16:31:06 -0000 On Mon, January 15, 2018 3:44 am, Mehmet Erol Sanliturk wrote: > On Mon, Jan 15, 2018 at 9:44 AM, Grzegorz Junka wrote: > >> >> On 15/01/2018 06:18, Warner Losh wrote: >> >>> >>> >>> On Jan 14, 2018 11:05 PM, "Grzegorz Junka" >> list1@gjunka.com>> wrote: >>> >>> >>> On 14/01/2018 16:18, Mehmet Erol Sanliturk wrote: >>> >>> >>> >>> On Sun, Jan 14, 2018 at 5:46 PM, Grzegorz Junka >>> >>> >> wrote: >>> >>> >>> On 13/01/2018 17:56, Mehmet Erol Sanliturk wrote: >>> >>> >>> >>> On Sat, Jan 13, 2018 at 7:21 PM, Grzegorz Junka >>> >>> > >>> >>> >>> wrote: >>> >>> Hello, >>> >>> I am installing a FreeBSD server based on >>> Supermicro H8SML-iF. >>> There are three PCIe slots to which I installed 2 >>> NVMe >>> drives and >>> one network card Intel I350-T4 (with 4 Ethernet >>> slots). >>> >>> I am observing a strange behavior where the system >>> doesn't >>> boot if >>> all three PCIe slots are populated. It shows this >>> message: >>> >>> nvme0: mem >>> 0xfd8fc000-0xfd8fffff irq >>> 24 at >>> device 0.0 on pci1 >>> nvme0: controller ready did not become 1 within >>> 30000 ms >>> nvme0: did not complete shutdown within 5 seconds >>> of >>> notification >>> >>> The I see a kernel panic/dump and the system >>> reboots after >>> 15 seconds. >>> >>> If I remove one card, either one of the NVMe >>> drives or the >>> network >>> card, the system boots fine. Also, if in BIOS I >>> set PnP OS >>> to YES >>> then sometimes it boots (but not always). If I set >>> PnP OS >>> to NO, >>> and all three cards are installed, the system >>> never boots. >>> >>> When the system boots OK I can see that the >>> network card is >>> reported as 4 separate devices on one of the PCIe >>> slots. I >>> tried >>> different NVMe drives as well as changing which >>> device is >>> installed to which slot but the result seems to be >>> the >>> same in any >>> case. >>> >>> What may be the issue? Amount of power drawn by the >>> hardware? Too >>> many devices not supported by the motherboard? Too >>> many >>> interrupts >>> for the FreeBSD kernel to handle? >>> >>> Any help would be greatly appreciated. >>> >>> GregJ >>> >>> _______________________________________________ >>> >>> >>> >>> >>> >>> From my experience from other trade marked main boards >>> , an >>> action may be to check manual of your server board to >>> see >>> whether there are rules about use of these slots : >>> Sometimes >>> differently shaped slots are supplied with same ports >>> : If one >>> slot is occupied , the other slot should be left open , >>> or >>> rules about not to insert such a kind of device into a >>> slot , >>> for example , graphic cards . >>> >>> >>> Mehmet Erol Sanliturk >>> >>> >>> I checked the manual but couldn't find any restrictions >>> regarding >>> PCIe ports. It only says how many lanes are available in >>> each >>> slot. Would there be any obvious BIOS setting that could >>> cause >>> this issue? I tried after resetting BIOS to default >>> settings but >>> maybe something is set incorrectly by default? >>> >>> GregJ >>> _______________________________________________ >>> >>> >>> >>> >>> >>> http://www.supermicro.com/Aplus/motherboard/Opteron3000/SR56 >>> x0/H8SML-iF.cfm >>> >> 6x0/H8SML-iF.cfm> >>> H8SML-iF >>> >>> >>> On the above page , click "OS Compatibility" >>> >>> >>> On the following page , click "SR5650" >>> >>> http://www.supermicro.com/Aplus/support/resources/OS/OS_Comp >>> _SR5650.cfm >>> >> p_SR5650.cfm> >>> OS Compatibility Chart >>> >>> >>> On the column ( third ) >>> >>> H8SML-7F >>> H8SML-7 >>> H8SML-iF >>> H8SML-i >>> >>> >>> there listed only * >>> * >>> ** >>> * >>> * >>> * >>> * >>> >>> FreeBSD 8.0 >>> FreeBSD 9.1 >>> >>> From this list , it may be said that , this mother board date >>> is old , means , it seems that the new OS versions are not >>> tested after currently tested OS versions . >>> >>> >>> To check interaction between operating system and your >>> Supermicro H8SML-iF , select one of the suitable operating >>> system ( Unix class OSes are more suitable ) for you and >>> tested on this card , and try to install it as you like your >>> installed components . If it boots successfully , it means >>> that there is an incompatibility between your FreeBSD and the >>> main board . If no one of them boots , then you may conclude >>> that , there is a problem in your settings . >>> >>> >>> BIOS settings are important , because , OS communicates with >>> the main board through these settings . >>> >>> >>> In manual ( downloaded from the above page : >>> Manual Revision 1.0c >>> Release Date: March 12, 2014 ) , page 4-9 , "PCI/PnP >>> Configuration" is defined . >>> If PnP is selected YES. OS adjusts some device settings . If >>> NO is selected , BIOS adjusts some device settings . When BIOS >>> adjusted device settings are not conforming to OS parameters , >>> the result will be "FAIL" . >>> >>> Therefore , more suitable selection is YES . >>> >>> >>> Another point is that , there are many more BIOS selectable >>> parameters and jumpers about PCI slots and others . >>> There are some BIOS settings for PCI slots : >>> >>> PCI X4 Slot 6 ( page 4-9 ) >>> PCI x8 Slot 7 ( page 4-10 ) >>> >>> >>> >>> Please review these BIOS settings in your manual and set them >>> with respect to your requirements . >>> >>> >>> Thanks Mehmet for looking into this. It's an old motherboard but >>> my point is that it boots fine when either: one NVMe and the >>> network card, or both NVMe are installed, but not when all three >>> are installed. How would that be related to FreeBSD compatibility? >>> The chipset and all devices that I am trying to install are >>> supported by FreeBSD 11.x. >>> >>> I just tried booting into a Debian live system and it also didn't >>> enumerate NVMe drives properly. This means that it's not FreeBSD >>> related and is no longer relevant for this list. I will try to >>> play with BIOS settings to see if I can make it work that way. >>> Thanks for all the help. >>> >>> >>> >>> Nvme drives are weird about power. I distrust the power estimate of >>> 5-9w >>> earlier in the thread... given the oddity with debian, it's not too >>> crazy >>> to think that. How far does FreeBSD boot though? >>> >>> >> I tried with a different power supply but the outcome was exactly the >> same. Sometimes FreeBSD boots fine but one of the NVMe drives is not >> visible (i.e. dmesg grep shows only one NVMe). When it doesn't work it >> boots up to the point of enumerating drives (SATA, USB, NVMe). Then it >> stops at the first NVMe and reboots. >> >> The funny thing is that very often it's enough to pull out one of the >> cards and put it back in. Then the system boots fine with all three >> cards. >> I had that a few times. Once it's booted it works, I can restart the >> system >> and it boots every time. As soon as I power off, unplug from the power >> main, wait a few minutes and power it on again, the issue comes back - >> can't boot as NVMe can't be enumerated. >> >> I though it might be caused by the hardware being too cold. I left the >> server once overnight but it didn't boot up, it was trying and >> restarting >> the whole night. >> >> GregJ >> >> >> _______________________________________________ >> >> > > > > The above explanation brings mind to the "impedance mismatch in > electronics" problem . Hm, I wouldn't say so. First of all, I will seriously doubt that sane cards are out of specs as far as impedance is concerned. But before going further, let's make sure we talk about the same thing. I assume impedance mismatch is what is related to impedance of the load attached to transmission line to be different from impedance of transmission line itself. In such case part of transmitted signal is reflected from the load back into transmission line. This can make mess as transmitted signal is mixed with this reflected at different positions of the loads along the same transmission line. One has to have really large mismatch (over 20% at least) to make that matter. Many of us remember this in at least two computer related cases: 1. we used terminators at the end of SCSI cables (or attached "self-terminating SCSI device to the end of line). 2. In some system boards in which memory buses had no terminators the manual would say to populate slots beginning from the fartherst away from CPU (to defeat reflection from open end of memory bus lines). I have never heard of anything like that on PCI express bus. If I am wrong, could you give some pointer so I can read about it. Thanks in advance for pointers! (I know: you learn something every day - which I bet I am about to ;-) Valeri > > ( Please search > > > impedance mismatch in electronics > impedance matching in electronics > > > in Internet if you want explanations about them . ) > > > When all of these cards are inserted into slots simultaneously , their > accumulated electronic effect may distort behaviour of your mother board > circuits or attached card circuit(s) . > > > Therefore , if you can find another NVMe and/or network card , please test > their effect . > Such tests may be inconclusive because mother board circuits may be > affected negatively from "properly" operating add on cards when they are > inserted together . > > > If it is feasible for you , you may use USB attached network card(s) to > eliminate network card attachment . > Or you may use a more capable one NVMe card instead of two smaller NVMe > cards , or you may use only one of them , or/and select an SATA SSD . > Such a choice would save your investment and produces a working server > with > a "little" loss when compared to "all" . > > > > > Mehmet Erol Sanliturk > _______________________________________________ > freebsd-questions@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-questions > To unsubscribe, send any mail to > "freebsd-questions-unsubscribe@freebsd.org" > ++++++++++++++++++++++++++++++++++++++++ Valeri Galtsev Sr System Administrator Department of Astronomy and Astrophysics Kavli Institute for Cosmological Physics University of Chicago Phone: 773-702-4247 ++++++++++++++++++++++++++++++++++++++++ From owner-freebsd-drivers@freebsd.org Mon Jan 15 16:56:35 2018 Return-Path: Delivered-To: freebsd-drivers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CDF3CEBA593; Mon, 15 Jan 2018 16:56:35 +0000 (UTC) (envelope-from galtsev@kicp.uchicago.edu) Received: from cosmo.uchicago.edu (cosmo.uchicago.edu [128.135.20.71]) by mx1.freebsd.org (Postfix) with ESMTP id A1A5B80F29; Mon, 15 Jan 2018 16:56:35 +0000 (UTC) (envelope-from galtsev@kicp.uchicago.edu) Received: by cosmo.uchicago.edu (Postfix, from userid 48) id 8984ECB8D3A; Mon, 15 Jan 2018 10:56:34 -0600 (CST) Received: from 108.68.169.115 (SquirrelMail authenticated user valeri) by cosmo.uchicago.edu with HTTP; Mon, 15 Jan 2018 10:56:34 -0600 (CST) Message-ID: <57933.108.68.169.115.1516035394.squirrel@cosmo.uchicago.edu> In-Reply-To: References: <3d0ad00c-5214-71b0-017b-c2d5ba608e37@gjunka.com> <8df1e967-01e0-d3c2-e14c-64c7fc8c66b0@gjunka.com> Date: Mon, 15 Jan 2018 10:56:34 -0600 (CST) Subject: Re: Server doesn't boot when 3 PCIe slots are populated From: "Valeri Galtsev" To: "Warner Losh" Cc: "Grzegorz Junka" , freebsd-questions@freebsd.org, freebsd-drivers@freebsd.org Reply-To: galtsev@kicp.uchicago.edu User-Agent: SquirrelMail/1.4.8-5.el5.centos.7 MIME-Version: 1.0 Content-Type: text/plain;charset=iso-8859-1 Content-Transfer-Encoding: 8bit X-Priority: 3 (Normal) Importance: Normal X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Jan 2018 16:56:35 -0000 On Mon, January 15, 2018 12:18 am, Warner Losh wrote: > On Jan 14, 2018 11:05 PM, "Grzegorz Junka" wrote: > > > On 14/01/2018 16:18, Mehmet Erol Sanliturk wrote: > >> >> >> On Sun, Jan 14, 2018 at 5:46 PM, Grzegorz Junka > > list1@gjunka.com>> wrote: >> >> >> On 13/01/2018 17:56, Mehmet Erol Sanliturk wrote: >> >> >> >> On Sat, Jan 13, 2018 at 7:21 PM, Grzegorz Junka >> >> >> wrote: >> >> Hello, >> >> I am installing a FreeBSD server based on Supermicro >> H8SML-iF. >> There are three PCIe slots to which I installed 2 NVMe >> drives and >> one network card Intel I350-T4 (with 4 Ethernet slots). >> >> I am observing a strange behavior where the system doesn't >> boot if >> all three PCIe slots are populated. It shows this message: >> >> nvme0: mem 0xfd8fc000-0xfd8fffff irq >> 24 at >> device 0.0 on pci1 >> nvme0: controller ready did not become 1 within 30000 ms >> nvme0: did not complete shutdown within 5 seconds of >> notification >> >> The I see a kernel panic/dump and the system reboots after >> 15 seconds. >> >> If I remove one card, either one of the NVMe drives or the >> network >> card, the system boots fine. Also, if in BIOS I set PnP OS >> to YES >> then sometimes it boots (but not always). If I set PnP OS >> to NO, >> and all three cards are installed, the system never boots. >> >> When the system boots OK I can see that the network card is >> reported as 4 separate devices on one of the PCIe slots. I >> tried >> different NVMe drives as well as changing which device is >> installed to which slot but the result seems to be the >> same in any >> case. >> >> What may be the issue? Amount of power drawn by the >> hardware? Too >> many devices not supported by the motherboard? Too many >> interrupts >> for the FreeBSD kernel to handle? >> >> Any help would be greatly appreciated. >> >> GregJ >> >> _______________________________________________ >> >> >> >> >> >> From my experience from other trade marked main boards , an >> action may be to check manual of your server board to see >> whether there are rules about use of these slots : Sometimes >> differently shaped slots are supplied with same ports : If one >> slot is occupied , the other slot should be left open , or >> rules about not to insert such a kind of device into a slot , >> for example , graphic cards . >> >> >> Mehmet Erol Sanliturk >> >> >> I checked the manual but couldn't find any restrictions regarding >> PCIe ports. It only says how many lanes are available in each >> slot. Would there be any obvious BIOS setting that could cause >> this issue? I tried after resetting BIOS to default settings but >> maybe something is set incorrectly by default? >> >> GregJ >> _______________________________________________ >> >> >> >> >> >> http://www.supermicro.com/Aplus/motherboard/Opteron3000/SR56 >> x0/H8SML-iF.cfm >> H8SML-iF >> >> >> On the above page , click "OS Compatibility" >> >> >> On the following page , click "SR5650" >> >> http://www.supermicro.com/Aplus/support/resources/OS/OS_Comp_SR5650.cfm >> OS Compatibility Chart >> >> >> On the column ( third ) >> >> H8SML-7F >> H8SML-7 >> H8SML-iF >> H8SML-i >> >> >> there listed only * >> * >> ** >> * >> * >> * >> * >> >> FreeBSD 8.0 >> FreeBSD 9.1 >> >> From this list , it may be said that , this mother board date is old , >> means , it seems that the new OS versions are not tested after currently >> tested OS versions . >> >> >> To check interaction between operating system and your Supermicro >> H8SML-iF >> , select one of the suitable operating system ( Unix class OSes are more >> suitable ) for you and tested on this card , and try to install it as >> you >> like your installed components . If it boots successfully , it means >> that >> there is an incompatibility between your FreeBSD and the main board . If >> no >> one of them boots , then you may conclude that , there is a problem in >> your >> settings . >> >> >> BIOS settings are important , because , OS communicates with the main >> board through these settings . >> >> >> In manual ( downloaded from the above page : >> Manual Revision 1.0c >> Release Date: March 12, 2014 ) , page 4-9 , "PCI/PnP Configuration" is >> defined . >> If PnP is selected YES. OS adjusts some device settings . If NO is >> selected , BIOS adjusts some device settings . When BIOS adjusted device >> settings are not conforming to OS parameters , the result will be "FAIL" >> . >> >> Therefore , more suitable selection is YES . >> >> >> Another point is that , there are many more BIOS selectable parameters >> and >> jumpers about PCI slots and others . >> There are some BIOS settings for PCI slots : >> >> PCI X4 Slot 6 ( page 4-9 ) >> PCI x8 Slot 7 ( page 4-10 ) >> >> >> >> Please review these BIOS settings in your manual and set them with >> respect >> to your requirements . >> >> > Thanks Mehmet for looking into this. It's an old motherboard but my point > is that it boots fine when either: one NVMe and the network card, or both > NVMe are installed, but not when all three are installed. How would that > be > related to FreeBSD compatibility? The chipset and all devices that I am > trying to install are supported by FreeBSD 11.x. > > I just tried booting into a Debian live system and it also didn't > enumerate > NVMe drives properly. This means that it's not FreeBSD related and is no > longer relevant for this list. This sounds to me as having something to do with allocation of PCI address space to talk to devices. Many devices can alternatively use different ranges of addresses, so more than one such device can be attached to the same PCI bus. These two particular devices seem to not be successfully negotiated to use different (not overlapping) ranges of addresses (in presence of some particular third device). Maybe it only happens like that in this particular system board ("motherboard"). I would try the same on different machine. But it is likely that all these three devices do not have non-overlapping address ranges. I hope, someone more knowledgeable that I will chime in. Valeri > I will try to play with BIOS settings to > see > if I can make it work that way. Thanks for all the help. > > > > Nvme drives are weird about power. I distrust the power estimate of 5-9w > earlier in the thread... given the oddity with debian, it's not too crazy > to think that. How far does FreeBSD boot though? > > Warner > > Warner > > _______________________________________________ > freebsd-drivers@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-drivers > To unsubscribe, send any mail to "freebsd-drivers-unsubscribe@freebsd.org" > _______________________________________________ > freebsd-questions@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-questions > To unsubscribe, send any mail to > "freebsd-questions-unsubscribe@freebsd.org" > ++++++++++++++++++++++++++++++++++++++++ Valeri Galtsev Sr System Administrator Department of Astronomy and Astrophysics Kavli Institute for Cosmological Physics University of Chicago Phone: 773-702-4247 ++++++++++++++++++++++++++++++++++++++++ From owner-freebsd-drivers@freebsd.org Mon Jan 15 16:59:37 2018 Return-Path: Delivered-To: freebsd-drivers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 27AC9EBAA4E; Mon, 15 Jan 2018 16:59:37 +0000 (UTC) (envelope-from m.e.sanliturk@gmail.com) Received: from mail-oi0-x22b.google.com (mail-oi0-x22b.google.com [IPv6:2607:f8b0:4003:c06::22b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D77D781278; Mon, 15 Jan 2018 16:59:36 +0000 (UTC) (envelope-from m.e.sanliturk@gmail.com) Received: by mail-oi0-x22b.google.com with SMTP id y141so8726859oia.0; Mon, 15 Jan 2018 08:59:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=kMU1BoMb/CNJbDtWuC+uyq9T9y4nDbibz/tfD5oTwZ4=; b=Z83rXIdTUDRCriepEog8mUDbljcVIlGHAuQjDmuLmWmAsj1pH55sZejDhkyH/QnKzi TknTlr72bXb7K1blkzfzoZkyPw7cuBmGdiQi2Ou+eEOjJbbtVtZolG+bvBhX8HbKC1i2 HiZUypqtWluzLVvnz6+MgGqTtVEIXzz9wgiLFrnduI+TJ7w8aDaz1WH3sc9wFb/ap+mC EZeJiZAbDDVfab0V2emY1PdZrqNRV5KOksRFVD/xFu+2I/PHiGsO9fwC/ipBRRrAkkw1 NiDo95/NXEkSZFyy9RzEoFY4dD4SOeg55x4lvIYTp+3vec97z/8iqWeylPOt8rJjNAZU Ks0A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=kMU1BoMb/CNJbDtWuC+uyq9T9y4nDbibz/tfD5oTwZ4=; b=FafyIohmppLV7/FqOvpQ5vdmt7x1MSAkOcKEidz4ISQ4OSlOv95CzHLYb7Mf/R8c/v uaLjf8ryC1AEKq0bkt29l2YGH12BoKpvrnfL+ROPqgNTW2LPOena7rEJdVHqoswW8/3B dpVQJ6oEWZevUby9dzvU2PNPcwcaCwDqCOEXhF8T3HvKLCs8wrUq1PXEzfiZ9fmwToik wL+hsBvR3g8L916YRdfohAL0b33njpv9kEJdojlICL8g/QtrW9Dylba+3Tq/0E+35Luh 104RYZ5+q+hsqN1csbnjydW9GzCzB3b2p5ozNlm1WJDfdTKE0SHtc9zKdI2QcDH90O1F GODA== X-Gm-Message-State: AKwxytdXu+Dft8EUaqroSAMSoFPyijq/p1HEZWJvvzYjN3UkC0xW1KfW eMsGpQE2ZDWuHIW4DQJnbFR3mH8WVvCJdllmv1Uqdw== X-Google-Smtp-Source: ACJfBotqd4ZREZr77vEpbscig2EcwQHQORUZ3ha5QamaENqJznZeul+/xUdBxiiWbWzhpcQdxhAwg5Dpid05GaE1Hjk= X-Received: by 10.202.60.134 with SMTP id j128mr1055082oia.268.1516035575973; Mon, 15 Jan 2018 08:59:35 -0800 (PST) MIME-Version: 1.0 Received: by 10.157.0.2 with HTTP; Mon, 15 Jan 2018 08:59:35 -0800 (PST) In-Reply-To: <57715.108.68.169.115.1516033864.squirrel@cosmo.uchicago.edu> References: <3d0ad00c-5214-71b0-017b-c2d5ba608e37@gjunka.com> <8df1e967-01e0-d3c2-e14c-64c7fc8c66b0@gjunka.com> <0e582bdb-e1f9-438c-3da2-2bcdc950aab5@gjunka.com> <57715.108.68.169.115.1516033864.squirrel@cosmo.uchicago.edu> From: Mehmet Erol Sanliturk Date: Mon, 15 Jan 2018 19:59:35 +0300 Message-ID: Subject: Re: Server doesn't boot when 3 PCIe slots are populated To: galtsev@kicp.uchicago.edu Cc: Grzegorz Junka , FreeBSD Questions Mailing List , Warner Losh , freebsd-drivers@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Jan 2018 16:59:37 -0000 On Mon, Jan 15, 2018 at 7:31 PM, Valeri Galtsev wrote: > > On Mon, January 15, 2018 3:44 am, Mehmet Erol Sanliturk wrote: > > On Mon, Jan 15, 2018 at 9:44 AM, Grzegorz Junka > wrote: > > > >> > >> On 15/01/2018 06:18, Warner Losh wrote: > >> > >>> > >>> > >>> On Jan 14, 2018 11:05 PM, "Grzegorz Junka" >>> list1@gjunka.com>> wrote: > >>> > >>> > >>> On 14/01/2018 16:18, Mehmet Erol Sanliturk wrote: > >>> > >>> > >>> > >>> On Sun, Jan 14, 2018 at 5:46 PM, Grzegorz Junka > >>> > >>> >> wrote: > >>> > >>> > >>> On 13/01/2018 17:56, Mehmet Erol Sanliturk wrote: > >>> > >>> > >>> > >>> On Sat, Jan 13, 2018 at 7:21 PM, Grzegorz Junka > >>> > >>> > > >>> > >>> >>> wrote: > >>> > >>> Hello, > >>> > >>> I am installing a FreeBSD server based on > >>> Supermicro H8SML-iF. > >>> There are three PCIe slots to which I installed 2 > >>> NVMe > >>> drives and > >>> one network card Intel I350-T4 (with 4 Ethernet > >>> slots). > >>> > >>> I am observing a strange behavior where the system > >>> doesn't > >>> boot if > >>> all three PCIe slots are populated. It shows this > >>> message: > >>> > >>> nvme0: mem > >>> 0xfd8fc000-0xfd8fffff irq > >>> 24 at > >>> device 0.0 on pci1 > >>> nvme0: controller ready did not become 1 within > >>> 30000 ms > >>> nvme0: did not complete shutdown within 5 seconds > >>> of > >>> notification > >>> > >>> The I see a kernel panic/dump and the system > >>> reboots after > >>> 15 seconds. > >>> > >>> If I remove one card, either one of the NVMe > >>> drives or the > >>> network > >>> card, the system boots fine. Also, if in BIOS I > >>> set PnP OS > >>> to YES > >>> then sometimes it boots (but not always). If I set > >>> PnP OS > >>> to NO, > >>> and all three cards are installed, the system > >>> never boots. > >>> > >>> When the system boots OK I can see that the > >>> network card is > >>> reported as 4 separate devices on one of the PCIe > >>> slots. I > >>> tried > >>> different NVMe drives as well as changing which > >>> device is > >>> installed to which slot but the result seems to be > >>> the > >>> same in any > >>> case. > >>> > >>> What may be the issue? Amount of power drawn by the > >>> hardware? Too > >>> many devices not supported by the motherboard? Too > >>> many > >>> interrupts > >>> for the FreeBSD kernel to handle? > >>> > >>> Any help would be greatly appreciated. > >>> > >>> GregJ > >>> > >>> _______________________________________________ > >>> > >>> > >>> > >>> > >>> > >>> From my experience from other trade marked main boards > >>> , an > >>> action may be to check manual of your server board to > >>> see > >>> whether there are rules about use of these slots : > >>> Sometimes > >>> differently shaped slots are supplied with same ports > >>> : If one > >>> slot is occupied , the other slot should be left open , > >>> or > >>> rules about not to insert such a kind of device into a > >>> slot , > >>> for example , graphic cards . > >>> > >>> > >>> Mehmet Erol Sanliturk > >>> > >>> > >>> I checked the manual but couldn't find any restrictions > >>> regarding > >>> PCIe ports. It only says how many lanes are available in > >>> each > >>> slot. Would there be any obvious BIOS setting that could > >>> cause > >>> this issue? I tried after resetting BIOS to default > >>> settings but > >>> maybe something is set incorrectly by default? > >>> > >>> GregJ > >>> _______________________________________________ > >>> > >>> > >>> > >>> > >>> > >>> http://www.supermicro.com/Aplus/motherboard/Opteron3000/SR56 > >>> x0/H8SML-iF.cfm > >>> >>> 6x0/H8SML-iF.cfm> > >>> H8SML-iF > >>> > >>> > >>> On the above page , click "OS Compatibility" > >>> > >>> > >>> On the following page , click "SR5650" > >>> > >>> http://www.supermicro.com/Aplus/support/resources/OS/OS_Comp > >>> _SR5650.cfm > >>> >>> p_SR5650.cfm> > >>> OS Compatibility Chart > >>> > >>> > >>> On the column ( third ) > >>> > >>> H8SML-7F > >>> H8SML-7 > >>> H8SML-iF > >>> H8SML-i > >>> > >>> > >>> there listed only * > >>> * > >>> ** > >>> * > >>> * > >>> * > >>> * > >>> > >>> FreeBSD 8.0 > >>> FreeBSD 9.1 > >>> > >>> From this list , it may be said that , this mother board date > >>> is old , means , it seems that the new OS versions are not > >>> tested after currently tested OS versions . > >>> > >>> > >>> To check interaction between operating system and your > >>> Supermicro H8SML-iF , select one of the suitable operating > >>> system ( Unix class OSes are more suitable ) for you and > >>> tested on this card , and try to install it as you like your > >>> installed components . If it boots successfully , it means > >>> that there is an incompatibility between your FreeBSD and the > >>> main board . If no one of them boots , then you may conclude > >>> that , there is a problem in your settings . > >>> > >>> > >>> BIOS settings are important , because , OS communicates with > >>> the main board through these settings . > >>> > >>> > >>> In manual ( downloaded from the above page : > >>> Manual Revision 1.0c > >>> Release Date: March 12, 2014 ) , page 4-9 , "PCI/PnP > >>> Configuration" is defined . > >>> If PnP is selected YES. OS adjusts some device settings . If > >>> NO is selected , BIOS adjusts some device settings . When BIOS > >>> adjusted device settings are not conforming to OS parameters , > >>> the result will be "FAIL" . > >>> > >>> Therefore , more suitable selection is YES . > >>> > >>> > >>> Another point is that , there are many more BIOS selectable > >>> parameters and jumpers about PCI slots and others . > >>> There are some BIOS settings for PCI slots : > >>> > >>> PCI X4 Slot 6 ( page 4-9 ) > >>> PCI x8 Slot 7 ( page 4-10 ) > >>> > >>> > >>> > >>> Please review these BIOS settings in your manual and set them > >>> with respect to your requirements . > >>> > >>> > >>> Thanks Mehmet for looking into this. It's an old motherboard but > >>> my point is that it boots fine when either: one NVMe and the > >>> network card, or both NVMe are installed, but not when all three > >>> are installed. How would that be related to FreeBSD compatibility? > >>> The chipset and all devices that I am trying to install are > >>> supported by FreeBSD 11.x. > >>> > >>> I just tried booting into a Debian live system and it also didn't > >>> enumerate NVMe drives properly. This means that it's not FreeBSD > >>> related and is no longer relevant for this list. I will try to > >>> play with BIOS settings to see if I can make it work that way. > >>> Thanks for all the help. > >>> > >>> > >>> > >>> Nvme drives are weird about power. I distrust the power estimate of > >>> 5-9w > >>> earlier in the thread... given the oddity with debian, it's not too > >>> crazy > >>> to think that. How far does FreeBSD boot though? > >>> > >>> > >> I tried with a different power supply but the outcome was exactly the > >> same. Sometimes FreeBSD boots fine but one of the NVMe drives is not > >> visible (i.e. dmesg grep shows only one NVMe). When it doesn't work it > >> boots up to the point of enumerating drives (SATA, USB, NVMe). Then it > >> stops at the first NVMe and reboots. > >> > >> The funny thing is that very often it's enough to pull out one of the > >> cards and put it back in. Then the system boots fine with all three > >> cards. > >> I had that a few times. Once it's booted it works, I can restart the > >> system > >> and it boots every time. As soon as I power off, unplug from the power > >> main, wait a few minutes and power it on again, the issue comes back - > >> can't boot as NVMe can't be enumerated. > >> > >> I though it might be caused by the hardware being too cold. I left the > >> server once overnight but it didn't boot up, it was trying and > >> restarting > >> the whole night. > >> > >> GregJ > >> > >> > >> _______________________________________________ > >> > >> > > > > > > > > The above explanation brings mind to the "impedance mismatch in > > electronics" problem . > > Hm, I wouldn't say so. First of all, I will seriously doubt that sane > cards are out of specs as far as impedance is concerned. > > But before going further, let's make sure we talk about the same thing. I > assume impedance mismatch is what is related to impedance of the load > attached to transmission line to be different from impedance of > transmission line itself. In such case part of transmitted signal is > reflected from the load back into transmission line. This can make mess as > transmitted signal is mixed with this reflected at different positions of > the loads along the same transmission line. One has to have really large > mismatch (over 20% at least) to make that matter. Many of us remember this > in at least two computer related cases: 1. we used terminators at the end > of SCSI cables (or attached "self-terminating SCSI device to the end of > line). 2. In some system boards in which memory buses had no terminators > the manual would say to populate slots beginning from the fartherst away > from CPU (to defeat reflection from open end of memory bus lines). > > I have never heard of anything like that on PCI express bus. If I am > wrong, could you give some pointer so I can read about it. > > Thanks in advance for pointers! (I know: you learn something every day - > which I bet I am about to ;-) > > Valeri > > > > > ( Please search > > > > > > impedance mismatch in electronics > > impedance matching in electronics > > > > > > in Internet if you want explanations about them . ) > > > > > > When all of these cards are inserted into slots simultaneously , their > > accumulated electronic effect may distort behaviour of your mother board > > circuits or attached card circuit(s) . > > > > > > Therefore , if you can find another NVMe and/or network card , please > test > > their effect . > > Such tests may be inconclusive because mother board circuits may be > > affected negatively from "properly" operating add on cards when they are > > inserted together . > > > > > > If it is feasible for you , you may use USB attached network card(s) to > > eliminate network card attachment . > > Or you may use a more capable one NVMe card instead of two smaller NVMe > > cards , or you may use only one of them , or/and select an SATA SSD . > > Such a choice would save your investment and produces a working server > > with > > a "little" loss when compared to "all" . > > > > > > > > > > Mehmet Erol Sanliturk > > _______________________________________________ > > freebsd-questions@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-questions > > To unsubscribe, send any mail to > > "freebsd-questions-unsubscribe@freebsd.org" > > > > > ++++++++++++++++++++++++++++++++++++++++ > Valeri Galtsev > Sr System Administrator > Department of Astronomy and Astrophysics > Kavli Institute for Cosmological Physics > University of Chicago > Phone: 773-702-4247 > ++++++++++++++++++++++++++++++++++++++++ > The problem of "impedance matching" occurs between any two interacting circuits : When a circuit gives its "output" to another circuit as "input" there exists this problem irrespective of subjects and kinds of circuits . Obviously , behaviours are not exactly the same . If you search the following phrase in Internet , you will find a large amount of links : impedance matching circuit design If we think a computer main board slots , the following may occur : Assume a slot has a voltage level for triggering input into an add on card , i.e. , add on card is affected when it senses a voltage level equal or greater than that level . The lower level values will not trigger the add on card . Assume an add on card is working . Assume a new add on card is also working alone . When both of these add on cards are inserted into slots , the power drawn will lower the voltage level of the surrounding circuit more than a single card . If this lowered voltage level is less than threshold level of the added cards ( one of them , or both of them ) it ( they ) will not sense the signals from the surrounding circuits . Therefore , it (they) will not respond to the action requesting signals . In one of the previous messages , https://lists.freebsd.org/pipermail/freebsd-questions/2018-January/280455.html it is said that " I am observing a strange behavior where the system doesn't boot if all three PCIe slots are populated. It shows this message: nvme0: mem 0xfd8fc000-0xfd8fffff irq 24 at device 0.0 on pci1 nvme0: controller ready did not become 1 within 30000 ms nvme0: did not complete shutdown within 5 seconds of notification The I see a kernel panic/dump and the system reboots after 15 seconds. If I remove one card, either one of the NVMe drives or the network card, the system boots fine. " A good example may be the above message . Mehmet Erol Sanliturk From owner-freebsd-drivers@freebsd.org Mon Jan 15 22:30:28 2018 Return-Path: Delivered-To: freebsd-drivers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C2843E7B864; Mon, 15 Jan 2018 22:30:28 +0000 (UTC) (envelope-from list1@gjunka.com) Received: from msa1.earth.yoonka.com (yoonka.com [88.98.225.149]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "msa1.earth.yoonka.com", Issuer "msa1.earth.yoonka.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 56D4E72F7B; Mon, 15 Jan 2018 22:30:27 +0000 (UTC) (envelope-from list1@gjunka.com) Received: from ultrabook.yoonka.com (pD9E429BE.dip0.t-ipconnect.de [217.228.41.190]) (authenticated bits=0) by msa1.earth.yoonka.com (8.15.2/8.15.2) with ESMTPSA id w0FMUO99053958 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO); Mon, 15 Jan 2018 22:30:24 GMT (envelope-from list1@gjunka.com) X-Authentication-Warning: msa1.earth.yoonka.com: Host pD9E429BE.dip0.t-ipconnect.de [217.228.41.190] claimed to be ultrabook.yoonka.com Subject: Re: Server doesn't boot when 3 PCIe slots are populated Cc: FreeBSD Questions Mailing List , freebsd-drivers@freebsd.org References: <3d0ad00c-5214-71b0-017b-c2d5ba608e37@gjunka.com> <8df1e967-01e0-d3c2-e14c-64c7fc8c66b0@gjunka.com> <0e582bdb-e1f9-438c-3da2-2bcdc950aab5@gjunka.com> <57715.108.68.169.115.1516033864.squirrel@cosmo.uchicago.edu> From: Grzegorz Junka Message-ID: <8fee9df3-c40b-addb-b3c9-bedd90683d62@gjunka.com> Date: Mon, 15 Jan 2018 22:30:18 +0000 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.3.0 MIME-Version: 1.0 In-Reply-To: Content-Language: en-GB-large Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Jan 2018 22:30:28 -0000 On 15/01/2018 16:59, Mehmet Erol Sanliturk wrote: > > > On Mon, Jan 15, 2018 at 7:31 PM, Valeri Galtsev > > wrote: > > [cut] > > >> > >> The funny thing is that very often it's enough to pull out one > of the > >> cards and put it back in. Then the system boots fine with all three > >> cards. > >> I had that a few times. Once it's booted it works, I can > restart the > >> system > >> and it boots every time. As soon as I power off, unplug from > the power > >> main, wait a few minutes and power it on again, the issue comes > back - > >> can't boot as NVMe can't be enumerated. > >> > >> I though it might be caused by the hardware being too cold. I > left the > >> server once overnight but it didn't boot up, it was trying and > >> restarting > >> the whole night. > >> > > > > The above explanation brings mind to the "impedance mismatch  in > > electronics" problem . > > Hm, I wouldn't say so. First of all, I will seriously doubt that sane > cards are out of specs as far as impedance is concerned. > > But before going further, let's make sure we talk about the same > thing. I > assume impedance mismatch is what is related to impedance of the load > attached to transmission line to be different from impedance of > transmission line itself. In such case part of transmitted signal is > reflected from the load back into transmission line. This can make > mess as > transmitted signal is mixed with this reflected at different > positions of > the loads along the same transmission line. One has to have really > large > mismatch (over 20% at least) to make that matter. Many of us > remember this > in at least two computer related cases: 1. we used terminators at > the end > of SCSI cables (or attached "self-terminating SCSI device to the > end of > line). 2. In some system boards in which memory buses had no > terminators > the manual would say to populate slots beginning from the > fartherst away > from CPU (to defeat reflection from open end of memory bus lines). > > I have never heard of anything like that on PCI express bus. If I am > wrong, could you give some pointer so I can read about it. > > Thanks in advance for pointers! (I know: you learn something every > day - > which I bet I am about to ;-) > > Valeri > > > > > ( Please search > > > > impedance mismatch  in electronics > > impedance matching  in electronics > > > > in Internet if you want explanations about them . ) > > > > When all of these cards are inserted into slots simultaneously , > their > > accumulated electronic effect may distort behaviour of your > mother board > > circuits or attached card circuit(s) . > > > > Therefore , if you can find another NVMe and/or network card , > please test > > their effect . > > Such tests may be inconclusive because mother board circuits may be > > affected negatively from "properly" operating add on cards when > they are > > inserted together . > > > > If it is feasible for you , you may use USB attached network > card(s) to > > eliminate network card attachment . > > Or you may use a more capable one NVMe card instead of two > smaller NVMe > > cards , or you may use only one of them , or/and select an SATA > SSD . > > Such a choice would save your investment and produces a working > server > > with > > a "little" loss when compared to "all" . > > > > Mehmet Erol Sanliturk > > _______________________________________________ > > freebsd-questions@freebsd.org > mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-questions > > > To unsubscribe, send any mail to > > "freebsd-questions-unsubscribe@freebsd.org > " > > ++++++++++++++++++++++++++++++++++++++++ > Valeri Galtsev > Sr System Administrator > Department of Astronomy and Astrophysics > Kavli Institute for Cosmological Physics > University of Chicago > Phone: 773-702-4247 > ++++++++++++++++++++++++++++++++++++++++ > > > > The problem of "impedance matching" occurs between any two interacting > circuits : When a circuit gives its "output" to another circuit as > "input" there exists this problem irrespective of subjects and kinds > of circuits . Obviously , behaviours are not exactly the same . > > If you search the following phrase in Internet , you will find a large > amount of links : > > impedance matching circuit design > > If we think a computer main board slots , the following may occur : > > Assume a slot has a voltage level for triggering input into an add on > card , i.e. , add on card is affected when it senses a voltage level > equal or greater than that level . The lower level values will not > trigger the add on card . > > Assume an add on card is working . > Assume a new add on card is also working alone . > > When both of these add on cards are inserted into slots , the power > drawn will lower the voltage level of the surrounding circuit more > than a single card . > If this lowered voltage level is less than threshold level of the > added cards ( one of them , or both of them ) it ( they ) will not > sense the signals from the surrounding circuits . Therefore , it > (they) will not respond to the action requesting signals . > > In one of the previous messages , > > https://lists.freebsd.org/pipermail/freebsd-questions/2018-January/280455.html > > > it is said that > > " > I am observing a strange behavior where the system doesn't boot if all > three PCIe slots are populated. It shows this message: > > nvme0: mem 0xfd8fc000-0xfd8fffff irq 24 at device > 0.0 on pci1 > nvme0: controller ready did not become 1 within 30000 ms > nvme0: did not complete shutdown within 5 seconds of notification > > The I see a kernel panic/dump and the system reboots after 15 seconds. > > If I remove one card, either one of the NVMe drives or the network card, > the system boots fine. > " > > A good example may be the above message . > > > Mehmet Erol Sanliturk > > I tried a different pair of NVMe cards (different adapters with different SSD disks) and the result was exactly the same. Note, that the pair that I tried was previously working in this motherboard without problems for many months, so it's safe to assume that the addition of the network card is causing this problem. But then again, the network card with one of the NVMe drives works fine too. Could be that all three cause some sort of impedance mismatch but that's kind of hard to believe - these are simple cards, there is almost no circuits on the NVMe adapter and the network card is just a chipset with 4 slots. I will have to look into other solutions, e.g. using SATA drives instead, but neither card was cheap, especially the pair of NVMe drives, so I am trying to figure out if there is anything I could do to make them cooperate before giving up. BTW Is there any way to verify from which group you received this thread so that I can remove either freebsd-drivers or freebsd-questions from this cross-post? GregJ From owner-freebsd-drivers@freebsd.org Mon Jan 15 22:36:45 2018 Return-Path: Delivered-To: freebsd-drivers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 42DA6E7BEB0; Mon, 15 Jan 2018 22:36:45 +0000 (UTC) (envelope-from list1@gjunka.com) Received: from msa1.earth.yoonka.com (yoonka.com [88.98.225.149]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "msa1.earth.yoonka.com", Issuer "msa1.earth.yoonka.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id D82D473472; Mon, 15 Jan 2018 22:36:44 +0000 (UTC) (envelope-from list1@gjunka.com) Received: from ultrabook.yoonka.com (pD9E429BE.dip0.t-ipconnect.de [217.228.41.190]) (authenticated bits=0) by msa1.earth.yoonka.com (8.15.2/8.15.2) with ESMTPSA id w0FMafiT054057 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO); Mon, 15 Jan 2018 22:36:42 GMT (envelope-from list1@gjunka.com) X-Authentication-Warning: msa1.earth.yoonka.com: Host pD9E429BE.dip0.t-ipconnect.de [217.228.41.190] claimed to be ultrabook.yoonka.com Subject: Re: Server doesn't boot when 3 PCIe slots are populated Cc: freebsd-questions@freebsd.org, freebsd-drivers@freebsd.org References: <3d0ad00c-5214-71b0-017b-c2d5ba608e37@gjunka.com> <8df1e967-01e0-d3c2-e14c-64c7fc8c66b0@gjunka.com> <57933.108.68.169.115.1516035394.squirrel@cosmo.uchicago.edu> From: Grzegorz Junka Message-ID: Date: Mon, 15 Jan 2018 22:36:36 +0000 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.3.0 MIME-Version: 1.0 In-Reply-To: <57933.108.68.169.115.1516035394.squirrel@cosmo.uchicago.edu> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-GB-large X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Jan 2018 22:36:45 -0000 [cut] On 15/01/2018 16:56, Valeri Galtsev wrote: > On Mon, January 15, 2018 12:18 am, Warner Losh wrote: >> >> Thanks Mehmet for looking into this. It's an old motherboard but my point >> is that it boots fine when either: one NVMe and the network card, or both >> NVMe are installed, but not when all three are installed. How would that >> be >> related to FreeBSD compatibility? The chipset and all devices that I am >> trying to install are supported by FreeBSD 11.x. >> >> I just tried booting into a Debian live system and it also didn't >> enumerate >> NVMe drives properly. This means that it's not FreeBSD related and is no >> longer relevant for this list. > This sounds to me as having something to do with allocation of PCI address > space to talk to devices. Many devices can alternatively use different > ranges of addresses, so more than one such device can be attached to the > same PCI bus. These two particular devices seem to not be successfully > negotiated to use different (not overlapping) ranges of addresses (in > presence of some particular third device). Maybe it only happens like that > in this particular system board ("motherboard"). I would try the same on > different machine. But it is likely that all these three devices do not > have non-overlapping address ranges. > > I hope, someone more knowledgeable that I will chime in. That was exactly my suspicion, that the network card somehow takes away some resources that otherwise would be available to the NVMe cards. Not sure if that's possible but would love to hear an informed opinion on this. Note, that as stated somewhere else, a live Debian showed the same issue - after booting from live USB only one NVMe and the network card were visible. The other NVMe was ignored. This would suggest either hardware or BIOS problems, so if I had some more information I could try to change some of the BIOS settings to see if that would help. GregJ > Valeri > >> I will try to play with BIOS settings to >> see >> if I can make it work that way. Thanks for all the help. >> >> >> >> Nvme drives are weird about power. I distrust the power estimate of 5-9w >> earlier in the thread... given the oddity with debian, it's not too crazy >> to think that. How far does FreeBSD boot though? >> >> Warner >> From owner-freebsd-drivers@freebsd.org Mon Jan 15 22:47:17 2018 Return-Path: Delivered-To: freebsd-drivers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5C92BE7C845; Mon, 15 Jan 2018 22:47:17 +0000 (UTC) (envelope-from galtsev@kicp.uchicago.edu) Received: from cosmo.uchicago.edu (cosmo.uchicago.edu [128.135.20.71]) by mx1.freebsd.org (Postfix) with ESMTP id 2596A73D3B; Mon, 15 Jan 2018 22:47:16 +0000 (UTC) (envelope-from galtsev@kicp.uchicago.edu) Received: by cosmo.uchicago.edu (Postfix, from userid 48) id C7777CB8D3A; Mon, 15 Jan 2018 16:47:15 -0600 (CST) Received: from 108.68.169.115 (SquirrelMail authenticated user valeri) by cosmo.uchicago.edu with HTTP; Mon, 15 Jan 2018 16:47:15 -0600 (CST) Message-ID: <60974.108.68.169.115.1516056435.squirrel@cosmo.uchicago.edu> In-Reply-To: References: <3d0ad00c-5214-71b0-017b-c2d5ba608e37@gjunka.com> <8df1e967-01e0-d3c2-e14c-64c7fc8c66b0@gjunka.com> <57933.108.68.169.115.1516035394.squirrel@cosmo.uchicago.edu> Date: Mon, 15 Jan 2018 16:47:15 -0600 (CST) Subject: Re: Server doesn't boot when 3 PCIe slots are populated From: "Valeri Galtsev" To: "Grzegorz Junka" Cc: freebsd-questions@freebsd.org, freebsd-drivers@freebsd.org Reply-To: galtsev@kicp.uchicago.edu User-Agent: SquirrelMail/1.4.8-5.el5.centos.7 MIME-Version: 1.0 Content-Type: text/plain;charset=iso-8859-1 Content-Transfer-Encoding: 8bit X-Priority: 3 (Normal) Importance: Normal X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 15 Jan 2018 22:47:17 -0000 On Mon, January 15, 2018 4:36 pm, Grzegorz Junka wrote: > [cut] > > On 15/01/2018 16:56, Valeri Galtsev wrote: >> On Mon, January 15, 2018 12:18 am, Warner Losh wrote: >>> >>> Thanks Mehmet for looking into this. It's an old motherboard but my >>> point >>> is that it boots fine when either: one NVMe and the network card, or >>> both >>> NVMe are installed, but not when all three are installed. How would >>> that >>> be >>> related to FreeBSD compatibility? The chipset and all devices that I am >>> trying to install are supported by FreeBSD 11.x. >>> >>> I just tried booting into a Debian live system and it also didn't >>> enumerate >>> NVMe drives properly. This means that it's not FreeBSD related and is >>> no >>> longer relevant for this list. >> This sounds to me as having something to do with allocation of PCI >> address >> space to talk to devices. Many devices can alternatively use different >> ranges of addresses, so more than one such device can be attached to the >> same PCI bus. These two particular devices seem to not be successfully >> negotiated to use different (not overlapping) ranges of addresses (in >> presence of some particular third device). Maybe it only happens like >> that >> in this particular system board ("motherboard"). I would try the same on >> different machine. But it is likely that all these three devices do not >> have non-overlapping address ranges. >> >> I hope, someone more knowledgeable that I will chime in. > > That was exactly my suspicion, that the network card somehow takes away > some resources that otherwise would be available to the NVMe cards. Not > sure if that's possible but would love to hear an informed opinion on > this. > > Note, that as stated somewhere else, a live Debian showed the same issue > - after booting from live USB only one NVMe and the network card were > visible. The other NVMe was ignored. Right. Taking as hypothesis inability to de-tangle these three devices I would try to replace the most generic one: network card. Hopefully you have access to some batch of these by different manufacturers with different chipsets. See if you will find one that will play in the band with your two NVMe-s. Good luck! Valeri > This would suggest either hardware > or BIOS problems, so if I had some more information I could try to > change some of the BIOS settings to see if that would help. > > GregJ > >> Valeri >> >>> I will try to play with BIOS settings to >>> see >>> if I can make it work that way. Thanks for all the help. >>> >>> >>> >>> Nvme drives are weird about power. I distrust the power estimate of >>> 5-9w >>> earlier in the thread... given the oddity with debian, it's not too >>> crazy >>> to think that. How far does FreeBSD boot though? >>> >>> Warner >>> > > _______________________________________________ > freebsd-questions@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-questions > To unsubscribe, send any mail to > "freebsd-questions-unsubscribe@freebsd.org" > ++++++++++++++++++++++++++++++++++++++++ Valeri Galtsev Sr System Administrator Department of Astronomy and Astrophysics Kavli Institute for Cosmological Physics University of Chicago Phone: 773-702-4247 ++++++++++++++++++++++++++++++++++++++++