From owner-freebsd-hackers@FreeBSD.ORG Tue May 19 22:04:21 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 65DCD910; Tue, 19 May 2015 22:04:21 +0000 (UTC) Received: from na01-bl2-obe.outbound.protection.outlook.com (mail-bl2on0066.outbound.protection.outlook.com [65.55.169.66]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9494B1C85; Tue, 19 May 2015 22:04:19 +0000 (UTC) Received: from CY1PR08MB1803.namprd08.prod.outlook.com (25.162.218.25) by SN1PR0801MB1566.namprd08.prod.outlook.com (25.163.133.156) with Microsoft SMTP Server (TLS) id 15.1.160.19; Tue, 19 May 2015 22:04:11 +0000 Received: from CY1PR08MB1803.namprd08.prod.outlook.com ([25.162.218.25]) by CY1PR08MB1803.namprd08.prod.outlook.com ([25.162.218.25]) with mapi id 15.01.0160.009; Tue, 19 May 2015 22:04:11 +0000 From: "Pokala, Ravi" To: "freebsd-net@freebsd.org" , "jfv@freebsd.org" , "erj@freebsd.org" CC: "freebsd-hackers@freebsd.org" , "Lewis, Fred" , "Sundararajan, Lakshmi" Subject: Performance issues with Intel Fortville (XL710/ixl(4)) Thread-Topic: Performance issues with Intel Fortville (XL710/ixl(4)) Thread-Index: AQHQkn+7WBa4imHGwkeMI3kSTJaMlQ== Date: Tue, 19 May 2015 22:04:10 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: user-agent: Microsoft-MacOutlook/14.4.9.150325 authentication-results: spf=none (sender IP is ) smtp.mailfrom=rpokala@panasas.com; x-ms-exchange-messagesentrepresentingtype: 1 x-originating-ip: [64.80.217.3] x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:SN1PR0801MB1566; x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:; x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(601004)(5005006)(3002001); SRVR:SN1PR0801MB1566; BCL:0; PCL:0; RULEID:; SRVR:SN1PR0801MB1566; x-forefront-prvs: 0581B5AB35 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(6009001)(189002)(199003)(164054003)(5001960100002)(189998001)(5001830100001)(5001860100001)(5001770100001)(106356001)(106116001)(102836002)(2900100001)(83506001)(229853001)(36756003)(50986999)(54356999)(101416001)(64706001)(4001540100001)(86362001)(2656002)(87936001)(66066001)(2201001)(97736004)(81156007)(92566002)(40100003)(77156002)(46102003)(2501003)(4001350100001)(122556002)(68736005)(62966003)(105586002)(99286002); DIR:OUT; SFP:1101; SCL:1; SRVR:SN1PR0801MB1566; H:CY1PR08MB1803.namprd08.prod.outlook.com; FPR:; SPF:None; PTR:InfoNoRecords; MX:1; A:1; LANG:en; received-spf: None (protection.outlook.com: panasas.com does not designate permitted sender hosts) Content-Type: text/plain; charset="us-ascii" Content-ID: Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: panasas.com X-MS-Exchange-CrossTenant-originalarrivaltime: 19 May 2015 22:04:10.2974 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: acf01c9d-c699-42af-bdbb-44bf582e60b0 X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN1PR0801MB1566 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 19 May 2015 22:04:21 -0000 Hi folks, At Panasas, we are working with the Intel XL710 40G NIC (aka Fortville), and we're seeing some performance issues w/ 11-CURRENT (r282653). Motherboard: Intel S2600KP (aka Kennedy Pass) CPU: E5-2660 v3 @ 2.6GHz (aka Haswell Xeon) (1 socket x 10 physical cores x 2 SMT threads) =3D 20 logical cores NIC: Intel XL710, 2x40Gbps QSFP, configured in 4x10Gbps mode RAM: 4x 16GB DDR4 DIMMs What we've seen so far: - TX performance is pretty consistently lower than RX performance. All numbers below are for unidrectional tests using `iperf': 10Gbps links threads/link TX Gbps RX Gbps TX/RX 1 1 9.02 9.85 91.57% 1 8 8.49 9.91 85.67% 1 16 7.00 9.91 70.63% 1 32 6.68 9.92 67.40% - With multiple active links, both TX and RX performance suffer greatly; the aggregate bandwidth tops out at about a third of the theoretical 40Gbps implied by 4x 10Gbps. 10Gbps links threads/link TX Gbps RX Gbps % of 40Gbps 4 1 13.39 13.38 33.4% - Multi-link bidirectional throughput is absolutely terrible; the aggregate is less than a tenth of the theoretical 40Gbps. 10Gbps links threads/link TX Gbps RX Gbps % of 40Gbps 4 1 3.83 2.96 9.6% / 7.4% - Occasional interrupt storm messages are seen from the IRQs associated with the NICs. Since that can impact performance, those runs were not included in the data listed above. Our questions: - How stable is ixl(4) in -CURRENT? By that, we mean both how quickly is the driver changing, and does the driver cause any system instability? - What type of performance have others been getting w/ Fortville? In 40Gbps mode? In 4x10Gbps mode? - Does anyone have any tuning parameters they can recommend for this card? - We did our testing w/ 11-CURRENT, but we will initially ship Fortville running on 10.1-RELEASE or 10.2-RELEASE. The presence of RSS - even though it is disabled by default - makes the driver back-port non-trivial. Is there an estimate on when the 11-CURRENT version of the driver (1.4.1) will get MFCed to 10-STABLE? My colleagues Lakshmi and Fred (CCed) are working on this; please make sure to include them if you have any comments. Thanks, Ravi