From nobody Wed Jun 9 15:23:53 2021 X-Original-To: freebsd-hardware@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 32B607E925D for ; Wed, 9 Jun 2021 15:24:07 +0000 (UTC) (envelope-from csforgeron@gmail.com) Received: from mail-il1-x136.google.com (mail-il1-x136.google.com [IPv6:2607:f8b0:4864:20::136]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4G0W9L354jz4ntW for ; Wed, 9 Jun 2021 15:24:06 +0000 (UTC) (envelope-from csforgeron@gmail.com) Received: by mail-il1-x136.google.com with SMTP id h3so7783104ilc.9 for ; Wed, 09 Jun 2021 08:24:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=Jhr5N/9lw+HzFlhm6i6+uhQgxHRL8fbdp97UMp/qxCs=; b=BRv2ddBfNsa22TNvQVukgrwU7FNx2ROT1ERxLQpsHj77TUSaxNfe+9afufwIMciQVR DyWZHnLOYdYby0crt6xRNEL34zNmu07VEWDUUMwL1cGwwnGht1F7S9Q8Y+9hX02am+gh uTmcINBMQjG41twGcA3mZkLyS0FyosT50bNXhHPnI0qqOS0GKjP2dkn25MyGVgd5XnZs Q2XJt8BX9HL4XDOcqfve58tw8dH8BHYK/OQQtkPcO1/UuWMCQYTdthoohYHT78FZlwzW O9EpjCEWZG99Zne7MD2+qJCFbP5AFqY3wxB44qgqr4fmpCRvyOHQUQ5EtpWaKbmHyHhr eW8A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=Jhr5N/9lw+HzFlhm6i6+uhQgxHRL8fbdp97UMp/qxCs=; b=kQIWpXDjC5euMe+QqVGdea44OSJpiyIslrRZ+eEH/E3DseFS8B2ME0+I0tPZZE25B7 0PNvIFTfrwpaXdkXVncRoqjQXPVsfm8ndJvMLN2bJfWLv5d2RKqmpXyfStS8RlaGRhtr +1exQ1Si87shUnw5Hb013Vj9seNt3N/GXpdCKvnz71fkWXiIG68xhuvkElGP2zlnm+kd wZ5XHi5ZgV4y1rWK8/O1kQ6lNFPml5UHtvJHW3cpNTlDpT+zHC7dmGK/xC6YdAaZ7IUE ZGdxE4kTHzs2Zl311CgE3QXv46P5QKDL/YbrH7yApNYnEUds59HbEsR9KwWd1nQ4UjWu qd9g== X-Gm-Message-State: AOAM531/cC6wZfykaeeEx6/lyHQTDYZ49xmxjy7cyOtwMileMgI8NZx+ D1ZYZdgZ3TMC0khu8XLEPKJD/NcXMmVectVI9WCUNcK7Wl4= X-Google-Smtp-Source: ABdhPJybW4oCUbAbr14WrcXFPEdXC+6749LGQQQLw6g8MZrqLMwlPL494Ko2H2HiUu5sQZJsQ9+4HRUFa/oiB8XY77s= X-Received: by 2002:a6b:e50c:: with SMTP id y12mr61950ioc.20.1623252244038; Wed, 09 Jun 2021 08:24:04 -0700 (PDT) List-Id: General discussion of FreeBSD hardware List-Archive: https://lists.freebsd.org/archives/freebsd-hardware List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hardware@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Christopher Forgeron Date: Wed, 9 Jun 2021 12:23:53 -0300 Message-ID: Subject: Re: Devices behind PEX 8664 PCIe Switch not detected since 11.0-RELEASE To: freebsd-hardware@freebsd.org Content-Type: multipart/alternative; boundary="00000000000063073c05c456dd21" X-Rspamd-Queue-Id: 4G0W9L354jz4ntW X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=BRv2ddBf; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of csforgeron@gmail.com designates 2607:f8b0:4864:20::136 as permitted sender) smtp.mailfrom=csforgeron@gmail.com X-Spamd-Result: default: False [0.99 / 15.00]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; FREEMAIL_FROM(0.00)[gmail.com]; TO_DN_NONE(0.00)[]; URI_COUNT_ODD(1.00)[1]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RBL_DBL_DONT_QUERY_IPS(0.00)[2607:f8b0:4864:20::136:from]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hardware@freebsd.org]; NEURAL_SPAM_MEDIUM(1.00)[1.000]; RCPT_COUNT_ONE(0.00)[1]; SPAMHAUS_ZRD(0.00)[2607:f8b0:4864:20::136:from:127.0.2.255]; NEURAL_SPAM_SHORT(0.99)[0.995]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::136:from]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-hardware] X-ThisMailContainsUnwantedMimeParts: Y --00000000000063073c05c456dd21 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hello, I'm not sure if I should try posting on a different mail list? The problem is still here, it's beyond my ability to fix, and I'm willing to actively help out anyone who can point me in the right direction - Anyone with any ideas? On Tue, Jun 1, 2021 at 10:44 PM Christopher Forgeron wrote: > Hello, > > I have not been able to upgrade past FreeBSD 10.3 as since versions 11.0= -> 13.0 devices behind my fileserver's PCIe Switch are not detected. The s= witch itself is detected, which is a PEX 8664 PCIe Switch, link below. > > I'm using a Dell M610x blade server, which has a PEX/Broadcom 8664 Switc= h in it to provide 2 PCIe slots, which I use to run SAS controllers to a dr= ive array. > > This arrangement has worked perfectly since the early days of 7/8.0, and= I've pretty much ran each version of FreeBSD since then, until 11.0 came o= ut. > > This is when I could no longer see/detect devices plugged into the PCIe = slots. Things were busy, FreeBSD 10.2 worked well, so it was just left this= way for years. > > Now I do need to upgrade software-wise to 13.0, so I've invested some ti= me tracking down what is happening, with the hopes that a knowledgeable per= son can fix this up. > > - I have confirmed with 10.3-RELEASE and a 11.0-RELEASE mini-iso boot th= at the problem starts in 11.0, and I can confirm it's still an issue in 12.= 0 and 13.0. I also boot into a 10.2-RELEASE because I happen to have that h= andy. Everything works/detects fine in 10.3 or lower. > > - These are GENERIC kernels, fresh from the iso/img I boot from, nothing= custom. > > - I currently have a 10.2 (or 10.3) boot and a 13.0 boot ready to swap b= ack and forth to try and isolate / probe info. I'll attach a few lspci dump= s after this message. > > - The card in question is a LSI 2008 Chipset, mps0 - But it's nothing do= do with that card/brand/config. I have also installed an Intel X540 Networ= k card in the PCIe slots and find the same issue - There's nothing behind t= he PEC 8664 in FreeBSD 13.0, and no problems in 10.2. I have also confirmed= that 13.0 boots up and detects the mps0 without issue when I use a differe= nt system without the PEX 8664 > > - At some point in my research I came across a similar issue where the d= eveloper mentioned that for some reason they were not scanning behind the s= witch. It was a different chipset and issue, and I don't have that thread = handy at the moment, I will post if I come across it again. > > Ref:https://www.broadcom.com/products/pcie-switches-bridges/pcie-switches= /pex8664 > > I'm rusty, but fairly knowledgeable and useful if you need me to compile = something, install patches, etc. I could give SSH access, but sending me a = patch to test may be quicker. > > lspci dumps are in my bug report here: > > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D256264 > > Thanks. > > --00000000000063073c05c456dd21--