Date: Tue, 16 Apr 2024 17:52:09 +0000 From: bugzilla-noreply@freebsd.org To: bugs@FreeBSD.org Subject: [Bug 269133] bnxt(4): BCM57416 - HWRM_CFA_L2_SET_RX_MASK command returned RESOURCE_ALLOC_ERROR error Message-ID: <bug-269133-227-NjQ7AY2C0N@https.bugs.freebsd.org/bugzilla/> In-Reply-To: <bug-269133-227@https.bugs.freebsd.org/bugzilla/> References: <bug-269133-227@https.bugs.freebsd.org/bugzilla/>
next in thread | previous in thread | raw e-mail | index | archive | help
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D269133 Kenneth D. Merry <ken@FreeBSD.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |ken@FreeBSD.org --- Comment #64 from Kenneth D. Merry <ken@FreeBSD.org> --- I have a machine with a BCM57414 on the motherboard: bnxt0: <Broadcom BCM57414 NetXtreme-E 10Gb/25Gb Ethernet> mem 0xd1d10000-0xd1d1ffff,0xd1c00000-0xd1cfffff,0xd1d22000-0xd1d23fff irq 36 at device 0.0 numa-domain 0 on pci3 bnxt0: Using 256 TX descriptors and 256 RX descriptors bnxt0: Using 16 RX queues 16 TX queues bnxt0: Using MSI-X interrupts with 17 vectors bnxt0: Ethernet address: 9c:6b:00:46:a2:0c bnxt1: <Broadcom BCM57414 NetXtreme-E 10Gb/25Gb Ethernet> mem 0xd1d00000-0xd1d0ffff,0xd1b00000-0xd1bfffff,0xd1d20000-0xd1d21fff irq 37 at device 0.1 numa-domain 0 on pci3 bnxt1: Using 256 TX descriptors and 256 RX descriptors bnxt1: Using 16 RX queues 16 TX queues bnxt1: Using MSI-X interrupts with 17 vectors bnxt1: Ethernet address: 9c:6b:00:46:a2:0d Here is the firmware information: dev.bnxt.0.nvram.available_size: 4173824 dev.bnxt.0.nvram.reserved_size: 16384 dev.bnxt.0.nvram.size: 8388608 dev.bnxt.0.nvram.sector_size: 4096 dev.bnxt.0.nvram.device_id: 16407 dev.bnxt.0.nvram.mfg_id: 239 dev.bnxt.0.ver.hwrm_min_ver: 1.10.2 dev.bnxt.0.ver.package_ver: <unknown> dev.bnxt.0.ver.chip_type: ASIC dev.bnxt.0.ver.chip_bond_id: 0 dev.bnxt.0.ver.chip_metal: 1 dev.bnxt.0.ver.chip_rev: 1 dev.bnxt.0.ver.chip_num: 5847 dev.bnxt.0.ver.phy_partnumber: MCP7F00-A002 dev.bnxt.0.ver.phy_vendor: Mellanox dev.bnxt.0.ver.roce_fw_name: BONO_FW dev.bnxt.0.ver.netctrl_fw_name: KONG_FW dev.bnxt.0.ver.mgmt_fw_name: AFW_226.0.145.0 dev.bnxt.0.ver.hwrm_fw_name: CHIMP_FW dev.bnxt.0.ver.phy: 13.1.11 dev.bnxt.0.ver.fw_ver: 226.0.145.0/pkg N/A dev.bnxt.0.ver.roce_fw: 226.0.145 dev.bnxt.0.ver.netctrl_fw: 226.0.145 dev.bnxt.0.ver.mgmt_fw: 226.0.145 dev.bnxt.0.ver.hwrm_fw: 226.0.145 dev.bnxt.0.ver.driver_hwrm_if: 1.10.2.34 dev.bnxt.0.ver.hwrm_if: 1.10.2 dev.bnxt.0.%domain: 0 dev.bnxt.0.%parent: pci3 dev.bnxt.0.%pnpinfo: vendor=3D0x14e4 device=3D0x16d7 subvendor=3D0x1849 subdevice=3D0x1402 class=3D0x020000 dev.bnxt.0.%location: slot=3D0 function=3D0 dbsf=3Dpci0:195:0:0 dev.bnxt.0.%driver: bnxt dev.bnxt.0.%desc: Broadcom BCM57414 NetXtreme-E 10Gb/25Gb Ethernet I have no VLANs configured. I'm running stable/13 from mid-2023, but I've tried the driver from the latest FreeBSD/head and FreeBSD stable/13 with no success. I still get: bnxt0: HWRM_RING_ALLOC command returned RESOURCE_ALLOC_ERROR error. bnxt0: HWRM_CFA_L2_SET_RX_MASK command returned RESOURCE_ALLOC_ERROR error. bnxt0: set_multi: rx_mask set failed bnxt0: HWRM_CFA_L2_SET_RX_MASK command returned RESOURCE_ALLOC_ERROR error. bnxt0: set_multi: rx_mask set failed bnxt0: HWRM_CFA_L2_SET_RX_MASK command returned RESOURCE_ALLOC_ERROR error. bnxt0: set_multi: rx_mask set failed bnxt0: HWRM_CFA_L2_SET_RX_MASK command returned RESOURCE_ALLOC_ERROR error. bnxt0: set_multi: rx_mask set failed bnxt0: HWRM_CFA_L2_SET_RX_MASK command returned RESOURCE_ALLOC_ERROR error. bnxt0: set_multi: rx_mask set failed bnxt0: Link is UP full duplex, FC - none - 25000 Mbps=20 bnxt0: link state changed to UP bnxt0: HWRM_CFA_L2_SET_RX_MASK command returned RESOURCE_ALLOC_ERROR error. bnxt0: set_multi: rx_mask set failed bnxt1: Link is UP full duplex, FC - none - 25000 Mbps=20 bnxt1: link state changed to UP bnxt0: HWRM_CFA_L2_SET_RX_MASK command returned RESOURCE_ALLOC_ERROR error. bnxt0: set_multi: rx_mask set failed bnxt0: HWRM_CFA_L2_SET_RX_MASK command returned RESOURCE_ALLOC_ERROR error. bnxt0: set_multi: rx_mask set failed bnxt0: HWRM_CFA_L2_SET_RX_MASK command returned RESOURCE_ALLOC_ERROR error. bnxt0: set_multi: rx_mask set failed Strangely enough, though, the driver works fine if I first PXE boot the mac= hine off of the chip. If I do that, it works normally. But if I boot off disk,= I get the RESOURCE_ALLOC_ERROR messages above. This suggests that there is some kind of initialization issue that the PXE = boot environment takes care of, but the driver does not. Also, bnxtnvm and niccli don't work with the driver in my kernel. But it isn't, apparently, because of the state of the driver, it is because of the ioctl definition in the driver. The ioctl call doesn't even get to bnxt_mgmt_ioctl. I've verified that with dtrace, but here is the ioctl call from ktrace/kdump: 7818 bnxtnvm CALL openat(AT_FDCWD,0x46199d,0x2<O_RDWR>) 7818 bnxtnvm NAMI "/dev/bnxt_mgmt" 7818 bnxtnvm RET openat 4 7818 bnxtnvm CALL ioctl(0x4,0x80000000,0x821199a70) 7818 bnxtnvm RET ioctl -1 errno 25 Inappropriate ioctl for device 7818 bnxtnvm CALL close(0x4) Using this Dtrace script: #pragma D option cleanrate=3D5000hz #pragma D option dynvarsize=3D8192000 fbt::bnxt_mgmt_ioctl:entry { printf("cmd =3D %#x\n", args[1]); } fbt::bnxt_mgmt_open:entry { printf("opened bnxt mgmt device\n"); } fbt::sys_ioctl:entry /args[1]->com =3D=3D 0x80000000/ { printf("got ioctl command 0x80000000\n"); } I verified that it isn't getting down to bnxt_mgmt_ioctl: # dtrace -s bnxt.d=20 dtrace: script 'bnxt.d' matched 3 probes CPU ID FUNCTION:NAME 31 2108 bnxt_mgmt_open:entry opened bnxt mgmt device 31 22882 sys_ioctl:entry got ioctl command 0x80000000 31 2108 bnxt_mgmt_open:entry opened bnxt mgmt device 31 22882 sys_ioctl:entry got ioctl command 0x80000000 ^C When I boot the machine via PXE, though, bnxtnvm listdev shows the device: # ./bnxtnvm listdev N/A #1 Device Interface Name : bnxt0 MACAddress : 9c:6b:00:46:a2:0c PCI Device Name : 0000:c3:00.0 And strangely the ioctl works, although from my reading of sys_ioctl(), it shouldn't but I think I've discovered why it does: 1904 bnxtnvm CALL openat(AT_FDCWD,0x46199d,0x2<O_RDWR>) 1904 bnxtnvm NAMI "/dev/bnxt_mgmt" 1904 bnxtnvm RET openat 3 1904 bnxtnvm CALL ioctl(0x3,0x80000000,0x8212303d0) 1904 bnxtnvm RET ioctl 0 1904 bnxtnvm CALL close(0x3) This code is from sys_ioctl(): /* * Interpret high order word to find amount of data to be * copied to/from the user's address space. */ size =3D IOCPARM_LEN(com); if ((size > IOCPARM_MAX) || ((com & (IOC_VOID | IOC_IN | IOC_OUT)) =3D=3D 0) || #if defined(COMPAT_FREEBSD5) || defined(COMPAT_FREEBSD4) || defined(COMPAT_= 43) ((com & IOC_OUT) && size =3D=3D 0) || #else ((com & (IOC_IN | IOC_OUT)) && size =3D=3D 0) || #endif ((com & IOC_VOID) && size > 0 && size !=3D sizeof(int))) return (ENOTTY); My regular kernel config file doesn't have COMPAT_FREEBSD4/5, but the PXE kernel config file does. Here are the bit definitions from sys/ioccom.h: #ifndef _SYS_IOCCOM_H_ #define _SYS_IOCCOM_H_ /* * Ioctl's have the command encoded in the lower word, and the size of * any in or out parameters in the upper word. The high 3 bits of the * upper word are used to encode the in/out status of the parameter. * * 31 29 28 16 15 8 7 0 * +---------------------------------------------------------------+ * | I/O | Parameter Length | Command Group | Command | * +---------------------------------------------------------------+ */ #define IOCPARM_SHIFT 13 /* number of bits for ioctl size */ #define IOCPARM_MASK ((1 << IOCPARM_SHIFT) - 1) /* parameter length mask= */ #define IOCPARM_LEN(x) (((x) >> 16) & IOCPARM_MASK) #define IOCBASECMD(x) ((x) & ~(IOCPARM_MASK << 16)) #define IOCGROUP(x) (((x) >> 8) & 0xff) #define IOCPARM_MAX (1 << IOCPARM_SHIFT) /* max size of ioctl */ #define IOC_VOID 0x20000000UL /* no parameters */ #define IOC_OUT 0x40000000UL /* copy out parameters */ #define IOC_IN 0x80000000UL /* copy in parameters */ #define IOC_INOUT (IOC_IN|IOC_OUT)/* copy parameters in and out */ #define IOC_DIRMASK (IOC_VOID|IOC_OUT|IOC_IN)/* mask for IN/OUT/VOID */ Because the BNXT_MGMT_OPCODE_GET_DEV_INFO ioctl the same as the IOC_IN bit definition, the ioctl breaks if the old compat stuff isn't built into the kernel. The ioctls for the bnxt(4) driver need to be changed to use the usual _IOW/_IOWR macros from sys/ioccom.h. I realize that will break the managem= ent tools. Perhaps they can have a version check and a fallback to the old ioc= tls if need be. --=20 You are receiving this mail because: You are the assignee for the bug.=
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?bug-269133-227-NjQ7AY2C0N>