From owner-freebsd-net@freebsd.org Tue Apr 19 05:47:17 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 06370B02D80 for ; Tue, 19 Apr 2016 05:47:17 +0000 (UTC) (envelope-from lars@netapp.com) Received: from mx144.netapp.com (mx144.netapp.com [216.240.21.25]) (using TLSv1.2 with cipher RC4-SHA (128/128 bits)) (Client CN "mx141.netapp.com", Issuer "Symantec Class 3 Secure Server CA - G4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id DB769136E for ; Tue, 19 Apr 2016 05:47:16 +0000 (UTC) (envelope-from lars@netapp.com) X-IronPort-AV: E=Sophos;i="5.24,505,1455004800"; d="asc'?scan'208";a="111191908" Received: from hioexcmbx07-prd.hq.netapp.com ([10.122.105.40]) by mx144-out.netapp.com with ESMTP; 18 Apr 2016 22:45:23 -0700 Received: from HIOEXCMBX07-PRD.hq.netapp.com (10.122.105.40) by hioexcmbx07-prd.hq.netapp.com (10.122.105.40) with Microsoft SMTP Server (TLS) id 15.0.1156.6; Mon, 18 Apr 2016 22:45:21 -0700 Received: from HIOEXCMBX07-PRD.hq.netapp.com ([::1]) by hioexcmbx07-prd.hq.netapp.com ([fe80::c07c:8fcd:f7e4:f32b%21]) with mapi id 15.00.1156.000; Mon, 18 Apr 2016 22:45:21 -0700 From: "Eggert, Lars" To: Dustin Marquess CC: "freebsd-net@freebsd.org" Subject: Re: Issues with ixl(4) Thread-Topic: Issues with ixl(4) Thread-Index: AQHRmfDiNfycsBH8CEW32UntV96nZp+RPsUA Date: Tue, 19 Apr 2016 05:45:19 +0000 Message-ID: <2C78DBCF-26F2-44D0-A45E-6EE8918648EA@netapp.com> References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: yes X-MS-TNEF-Correlator: x-mailer: Apple Mail (2.3124) x-ms-exchange-messagesentrepresentingtype: 1 x-ms-exchange-transport-fromentityheader: Hosted x-originating-ip: [10.120.60.35] Content-Type: multipart/signed; boundary="Apple-Mail=_E29563A1-48A5-410C-A19F-5F15BA3125E7"; protocol="application/pgp-signature"; micalg=pgp-sha256 MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 19 Apr 2016 05:47:17 -0000 --Apple-Mail=_E29563A1-48A5-410C-A19F-5F15BA3125E7 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii I haven't played with lagg+vlan+bridge, but I briefly evaluated XL710 = boards last year = (https://lists.freebsd.org/pipermail/freebsd-net/2015-October/043584.html)= and saw very poor throughputs and latencies even in very simple setups. = As far as I could figure it out, TSO/LRO wasn't being performed = (although enabled) and so I ran into packet-rate issues. I basically gave up and went with a different vendor. FWIW, the XL710 = boards in the same machines booted into Linux performed fine. Lars > On 2016-04-19, at 6:06, Dustin Marquess wrote: >=20 > I'm having some strange issues with ixl(4) and a X710-DA4 card in a = new-ish > Intel-based server. I'm pretty much replicating an existing setup = from an > older AMD machine that used 2 x X520-DA2 cards and ixgbe(4). This is = all > on -CURRENT. >=20 > It's meant to be a bhyve server, so the 4x10GE ports are put into a > LACP-based lagg(4), then vlan(4) interfaces are bound to the lagg, and = then > if_bridge(4) interfaces are created to bind the vlan and tap = interfaces > together. >=20 > The X710-DA4 is running the latest NVM from Intel (5.02): >=20 > dev.ixl.3.fw_version: nvm 5.02 etid 80002284 oem 0.0.0 > dev.ixl.2.fw_version: nvm 5.02 etid 80002284 oem 0.0.0 > dev.ixl.1.fw_version: nvm 5.02 etid 80002284 oem 0.0.0 > dev.ixl.0.fw_version: nvm 5.02 etid 80002284 oem 0.0.0 >=20 > I've tried both the ixl driver that comes with -CURRENT (1.4.3?) and = the > 1.4.27 driver from Intel and am having the same problem. The problem = is > this exactly (sorry it's taken me so long to get to it!): >=20 > Using just one interface, one interface + VLANs, the lagg without = VLANs, > etc, everything works perfectly fine. As soon as I combine > lagg+vlan+bridge, all hell breaks loose. One machine can ping one = alias on > the server but not the other while other machines can. The server = itself > can't ping the DNS server nor the default route, but can ping things > through the default route, etc. The behavior is very unpredictable. = ssh > can take a few times to get in, and then once it, "svn update" will = work > for a few seconds and then bomb out, etc. >=20 > He is the working config from the X520-DA2 system: >=20 > ifconfig_ix0=3D"-lro -tso -txcsum up" > ifconfig_ix1=3D"-lro -tso -txcsum up" > ifconfig_ix2=3D"-lro -tso -txcsum up" > ifconfig_ix3=3D"-lro -tso -txcsum up" > cloned_interfaces=3D"lagg0 tap0 tap1 bridge0 bridge1 vlan1 vlan2" > ifconfig_lagg0=3D"laggproto lacp laggport ix0 laggport ix1 laggport = ix2 > laggport ix3" > ifconfig_vlan1=3D"vlan 1 vlandev lagg0" > ifconfig_vlan2=3D"vlan 2 vlandev lagg0" > ifconfig_bridge0=3D"inet 192.168.1.100/24 addm vlan1 addm tap0" > ifconfig_bridge1=3D"addm vlan2 addm tap1" > defaultrouter=3D"192.168.1.1" >=20 > Here is the "broken" config from the X710-DA4 system: >=20 > ifconfig_ixl0=3D"-rxcsum -txcsum -lro -tso -vlanmtu -vlanhwtag = -vlanhwfilter > -vlanhwtso -vlanhwcsum up" > ifconfig_ixl1=3D"-rxcsum -txcsum -lro -tso -vlanmtu -vlanhwtag = -vlanhwfilter > -vlanhwtso -vlanhwcsum up" > ifconfig_ixl2=3D"-rxcsum -txcsum -lro -tso -vlanmtu -vlanhwtag = -vlanhwfilter > -vlanhwtso -vlanhwcsum up" > ifconfig_ixl3=3D"-rxcsum -txcsum -lro -tso -vlanmtu -vlanhwtag = -vlanhwfilter > -vlanhwtso -vlanhwcsum up" > cloned_interfaces=3D"lagg0 tap0 tap1 bridge0 bridge1 vlan1 vlan2" > ifconfig_lagg0=3D"laggproto lacp laggport ixl0 laggport ixl1 laggport = ixl2 > laggport ixl3" > ifconfig_vlan1=3D"vlan 1 vlandev lagg0" > ifconfig_vlan2=3D"vlan 2 vlandev lagg0" > ifconfig_bridge0=3D"inet 192.168.1.101/24 addm vlan1 addm tap0" > ifconfig_bridge1=3D"addm vlan2 addm tap1" > defaultrouter=3D"192.168.1.1" >=20 > I've changed the various flags in the ifconfig_ixl# lines without any > obvious differences. Both machines are connected to the same HPe = 5820X > switch with the same exact config, so I don't believe it's a switch = issue. >=20 > Any ideas? Has anybody seen something like this before? >=20 > Thanks! > -Dustin > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" --Apple-Mail=_E29563A1-48A5-410C-A19F-5F15BA3125E7 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="signature.asc" Content-Type: application/pgp-signature; name="signature.asc" Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- iQIcBAEBCAAGBQJXFcXuAAoJEFS1wwm/cMFXKusQAMlS/HUcRwWmE+PZhtrohMCe W2BkMqrTjsvjQf25pS6BOhsC1q42mXWpUgw3JvfPTgbKMm4aEFTrUcyeBECc8gt4 oZ5mwtWy+5rdWt+8hep+LXW64He3jQjGaC3e2SAA/GmkLtwCQlSSAtrrD2acxlJF yv810XaIz5sg8lQ768oxk/Ks6hJep3UIhwAD+d/lEpuCBvmwrixbyV0OwIatcdnG dNNoI14qx54z8E+cs0Y1dDd0CpeB27JGBA7t9NSTrrWu04F8JOz1RqNZEeuQw/2h Y04zYkgs9ZQqEuIvnCsjGPajeMAlP+lEVTp1wqgWW4bNVaKjbxGnIecAboTw2ELN Ud4dj4Yij2yIYQuRKCOD9Mj/gcXTv2nKQfz2zz3Nqm7tSvHDmzuP0ikf3tlrYMKs 7tewBT0CBsz6W5Kn7AywOD7tbBMvIAn+3Vdz16j8Qn40wEfbobhMsp2gUwj2Gg76 pmKztq6knkcTCfvmgb5uNXFZKfEFr/wjwIUkO+2bs8HnfqxUYts+PPMXuXYvR57t E2iLbkjBM+ksug/5BvxTcfA0Snw1d926oamjg66Fp5hzKl5rdTUa/SuQmRKW4y25 QbEoPWBIJxCke40/jloZFD1tCIue8ZJrSUfnZFq8tTMScmDB84+9vf0HzpGzEyNM ET8SoSghDgdvaNYQ+rIE =4q1w -----END PGP SIGNATURE----- --Apple-Mail=_E29563A1-48A5-410C-A19F-5F15BA3125E7--