From owner-freebsd-questions@freebsd.org Sat Jan 13 16:21:33 2018 Return-Path: Delivered-To: freebsd-questions@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 41062E713D6; Sat, 13 Jan 2018 16:21:33 +0000 (UTC) (envelope-from list1@gjunka.com) Received: from msa1.earth.yoonka.com (yoonka.com [88.98.225.149]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "msa1.earth.yoonka.com", Issuer "msa1.earth.yoonka.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 3A8B16B9BF; Sat, 13 Jan 2018 16:21:31 +0000 (UTC) (envelope-from list1@gjunka.com) Received: from crayon2.yoonka.com (crayon2.yoonka.com [10.70.7.20]) (authenticated bits=0) by msa1.earth.yoonka.com (8.15.2/8.15.2) with ESMTPSA id w0DGLOw9003337 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO); Sat, 13 Jan 2018 16:21:24 GMT (envelope-from list1@gjunka.com) To: freebsd-questions@freebsd.org, freebsd-drivers@freebsd.org From: Grzegorz Junka Subject: Server doesn't boot when 3 PCIe slots are populated Message-ID: Date: Sat, 13 Jan 2018 16:21:24 +0000 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.3.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-GB-large X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 13 Jan 2018 16:21:33 -0000 Hello, I am installing a FreeBSD server based on Supermicro H8SML-iF. There are three PCIe slots to which I installed 2 NVMe drives and one network card Intel I350-T4 (with 4 Ethernet slots). I am observing a strange behavior where the system doesn't boot if all three PCIe slots are populated. It shows this message: nvme0: mem 0xfd8fc000-0xfd8fffff irq 24 at device 0.0 on pci1 nvme0: controller ready did not become 1 within 30000 ms nvme0: did not complete shutdown within 5 seconds of notification The I see a kernel panic/dump and the system reboots after 15 seconds. If I remove one card, either one of the NVMe drives or the network card, the system boots fine. Also, if in BIOS I set PnP OS to YES then sometimes it boots (but not always). If I set PnP OS to NO, and all three cards are installed, the system never boots. When the system boots OK I can see that the network card is reported as 4 separate devices on one of the PCIe slots. I tried different NVMe drives as well as changing which device is installed to which slot but the result seems to be the same in any case. What may be the issue? Amount of power drawn by the hardware? Too many devices not supported by the motherboard? Too many interrupts for the FreeBSD kernel to handle? Any help would be greatly appreciated. GregJ