From owner-freebsd-net@freebsd.org Thu Feb 1 14:56:46 2018 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 36834ED5DBE for ; Thu, 1 Feb 2018 14:56:46 +0000 (UTC) (envelope-from marius.h@lden.org) Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D7AAC6FC4A for ; Thu, 1 Feb 2018 14:56:45 +0000 (UTC) (envelope-from marius.h@lden.org) Received: from compute7.internal (compute7.nyi.internal [10.202.2.47]) by mailout.nyi.internal (Postfix) with ESMTP id 5983520E45 for ; Thu, 1 Feb 2018 09:56:45 -0500 (EST) Received: from web1 ([10.202.2.211]) by compute7.internal (MEProxy); Thu, 01 Feb 2018 09:56:45 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lden.org; h= content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-sender :x-me-sender:x-sasl-enc; s=fm2; bh=4SEE3lPAOJUJbMmLGnxc6ilwAByen +H/pD0HOQkBJys=; b=jSHYyS1A4qlJqARQO2G7XF//RbBjNd+QmAXcDxRiO6vmS z5IC1cQBTFnKvZKyd3O8wrHlpxooxqXEbQXoyFJ7ReFo03jdy5caZJi8ywaBcF9o fVDo42YhpcO2K2xV6sCmzsprUr5Sd7Nvn1OxfUaXUZsQBZmisF9SdE5dn0cRlBMx gVLpE7KUMpY083OaBhyItsbM2xRnfip7jyYnuHmJErKE+H1xKAQN2nDRBw9xdyXi 6aY4dN182YG4mATwKuQpWTHAC/8CeQN/99k7ChhGEvMM6qhet0rRCYsyuK7ZOjz/ lz1duQCQubAaVKP5NVSadJkOJubOI1PLQxQzVPBSg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-sender:x-me-sender:x-sasl-enc; s=fm1; bh=4SEE3l PAOJUJbMmLGnxc6ilwAByen+H/pD0HOQkBJys=; b=oNDtKfpA0vBwo++cgDjibA Ur/NCEPtLF8xhO3aFuiAuOSUpk0Op6TMEtOgbhaEf8o0VLE8xrpiHTn76c9sTY8X JN4k7cZ3GIEopNLdIMWLx/nj3fsyJ00B4ytcPxmPqjacP5zupBB7sfslpztNaQ53 zBjvgvqn8qDg4cReEzFfZOoCDzQr9Lf5Zg0+CzcOFEh/+I2IWLCpQ+vbJSRTWYIJ zmCgrbMqpxqtIUOISW7kaP40dHpHT2xYPmnvJzBAAd480FwoPxh+K2yuxQae1NYO 9OF+RiMX3jpIhQZx+pvmdfFrR8plWgS5jsk7Bxt6RbZtruNrXZ4vkYBjIPc5iNcg == X-ME-Sender: Received: by mailuser.nyi.internal (Postfix, from userid 99) id 389C3940F1; Thu, 1 Feb 2018 09:56:45 -0500 (EST) Message-Id: <1517497005.912910.1255997344.7F24E8FC@webmail.messagingengine.com> From: Marius Halden To: freebsd-net@freebsd.org MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="utf-8" X-Mailer: MessagingEngine.com Webmail Interface - ajax-fde26eb3 Date: Thu, 01 Feb 2018 15:56:45 +0100 In-Reply-To: <1517494674.4068747.1255940744.6E4465CF@webmail.messagingengine.com> Subject: Re: Issues with bxe NIC References: <1517494674.4068747.1255940744.6E4465CF@webmail.messagingengine.com> X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Feb 2018 14:56:46 -0000 On Thu, Feb 1, 2018, at 15:17, Marius Halden wrote: > Hi, > > We have two routers running FreeBSD r320487 (BSDRP) with bxe NICs, they > are connected together with a twinax cable and to our ISP with 1Gbit > fiber. The NICs in question are Broadcom BCM57840S (Supermicro AOC-STG- > B4S). > > The link between the routers will go down on one side not long after > it's brought up while staying up on the other side. I am not able to > bring the link back up without either unplugging the cable or powering > the routers off and on again, if I try to bring them up by taking them > down and up with ifconfig they will stay down no matter what the state > was before. > > After the routers have been up some time the console will be flooded > with messages like this: > > bxe0: ERROR: resource (0x0) in use (status 0xffffffff bit 0x1) > bxe0: ELINK EVENT LOG (3) > bxe0: ELINK EVENT LOG (3) > bxe0: ERROR: resource (0x0) in use (status 0xffffffff bit 0x1) > bxe0: ELINK EVENT LOG (3) > bxe0: ELINK EVENT LOG (3) > bxe0: ERROR: resource (0x0) in use (status 0xffffffff bit 0x1) > bxe0: ELINK EVENT LOG (3) > bxe0: ELINK EVENT LOG (3) > bxe0: ERROR: resource (0x0) in use (status 0xffffffff bit 0x1) > bxe0: ELINK EVENT LOG (3) > bxe0: ELINK EVENT LOG (3) > bxe0: ERROR: resource (0x0) in use (status 0xffffffff bit 0x1) > bxe0: ELINK EVENT LOG (3) > bxe0: ELINK EVENT LOG (3) > bxe0: ERROR: resource (0x0) in use (status 0xffffffff bit 0x1) > bxe0: ELINK EVENT LOG (3) > bxe0: ELINK EVENT LOG (3) > bxe0: ERROR: resource (0x0) in use (status 0xffffffff bit 0x1) > bxe0: ELINK EVENT LOG (3) > bxe0: ELINK EVENT LOG (3) > bxe0: ERROR: resource (0x0) in use (status 0xffffffff bit 0x1) > bxe0: ELINK EVENT LOG (3) > bxe0: ELINK EVENT LOG (3) > bxe0: ERROR: resource (0x0) in use (status 0xffffffff bit 0x1) > bxe0: ELINK EVENT LOG (3) > bxe0: ELINK EVENT LOG (3) > bxe0: ERROR: resource (0x0) in use (status 0xffffffff bit 0x1) > bxe0: ELINK EVENT LOG (3) > bxe0: ELINK EVENT LOG (3) > bxe0: ERROR: resource (0x0) in use (status 0xffffffff bit 0x1) > bxe0: ELINK EVENT LOG (3) > bxe0: ELINK EVENT LOG (3) > bxe0: ERROR: resource (0x0) in use (status 0xffffffff bit 0x1) > bxe0: ELINK EVENT LOG (3) > bxe0: ELINK EVENT LOG (3) > bxe0: ERROR: resource (0x0) in use (status 0xffffffff bit 0x1) > bxe0: ELINK EVENT LOG (3) > bxe0: ELINK EVENT LOG (3) > bxe0: ERROR: resource (0x0) in use (status 0xffffffff bit 0x1) > bxe0: ELINK EVENT LOG (3) > bxe0: ELINK EVENT LOG (3) > bxe0: ERROR: resource (0x0) in use (status 0xffffffff bit 0x1) > bxe0: ELINK EVENT LOG (3) > bxe0: ELINK EVENT LOG (3) > > This is not always limited to just one interface, but can include all > interfaces which are in use. Trying to bringing down the interface with ifconfig, the following was logged: bxe0: ERROR: ECORE: timeout waiting for state 0 bxe0: ERROR: Failed to delete MACs (-4) mac_type 0 wait_for_comp 0x1 bxe0: ERROR: Failed to delete UC MACs list (-4) bxe0: ERROR: FW failed to respond! bxe0: ERROR: ECORE: timeout waiting for state 7 bxe0: ERROR: ECORE: timeout waiting for state 2 bxe0: ERROR: FUNC_STOP ramrod failed. Running a dry transaction (-4) bxe0: ERROR: proper val not read from IGU! bxe0: ERROR: ECORE: Unknown reset_phase (0x0) from MCP bxe0: ERROR: FW failed to respond! bxe0: ERROR: Parity error: HW block parity attention: [0]:0x55540000 [1]:0x55555555 [2]:0x00005555 [3]:0xf0000000 [4]:0x00000028 bxe0: ERROR: resource (0xb) in use (status 0xffffffff bit 0x800) bxe0: ERROR: resource (0xb) in use (status 0xffffffff bit 0x800) bxe0: ERROR: resource (0xb) in use (status 0xffffffff bit 0x800) Is this a hardware or a driver issue? -- Marius Halden