From owner-freebsd-net@freebsd.org Sat Oct 10 14:03:49 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BDEDF9D23B6 for ; Sat, 10 Oct 2015 14:03:49 +0000 (UTC) (envelope-from bmerry@ska.ac.za) Received: from mail-ig0-f170.google.com (mail-ig0-f170.google.com [209.85.213.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9736AA97 for ; Sat, 10 Oct 2015 14:03:49 +0000 (UTC) (envelope-from bmerry@ska.ac.za) Received: by igcrk20 with SMTP id rk20so52205855igc.1 for ; Sat, 10 Oct 2015 07:03:49 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:date:message-id:subject:from:to :content-type; bh=cp9CD3htycbl4suo8z3Ks9ZXyU40yzZI35elzlfcmEw=; b=BMGxSFgp+QQd66JNnsVuxUxsMpZqL007U8MmuGkT4yOleB/TzIeXiGsiFHUwnKiIaX FqAtBdTmKv0zCzi1LnIISp31rKvXw96l2mqozn92DusZa99fEMI6pVsotThkSz4PUMuz utD8EDVIu7/lhqmsafMErwFi5k7SCVrAxXG6DzqL/uBIxVZx4kBsIaqqOG57u9vYDAS2 ZhyQr7gR7ufSEm/VpGvudlSwQQpex2EnvY1p2NexQ4r05v6FRctEDPTKWeRVGK8WZRpl aPfJ958BsqgaAuvBVFcTXZPjODGE38N1a3RE3Y99vouIX+3RCt5HirV4ai3QsW8q40O/ vUlw== X-Gm-Message-State: ALoCoQmYjnD+c/XvFx6kQ/Xlnf1WMP39H1QTnxm2SxZAdQbR+Zj/umf+cTuvE/Q+8ajBuCZvqYTZ MIME-Version: 1.0 X-Received: by 10.50.23.80 with SMTP id k16mr4347915igf.62.1444485828817; Sat, 10 Oct 2015 07:03:48 -0700 (PDT) Received: by 10.107.46.103 with HTTP; Sat, 10 Oct 2015 07:03:48 -0700 (PDT) Date: Sat, 10 Oct 2015 16:03:48 +0200 Message-ID: Subject: netmap: recommended NIC for 40GbE capture on Linux? From: Bruce Merry To: freebsd-net@freebsd.org Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 10 Oct 2015 14:03:49 -0000 Hi We're aim to do some high-speed data capture and processing on data being sent from an FPGA over Ethernet. The data is about 3.3Mpps with 512 bytes of application-level data per packet (plus somewhere around 64 bytes of app-level headers, plus UDP+IP+Ethernet headers). I'm looking for some advice on what hardware to use. We currently have Mellanox ConnectX-3 NICs with QSFP+. My experiments with pkt-gen on a single flow (generated by pkt-gen on another machine) are topping out at receiving 2.4 Mpps, with ksortirqd pushing one core at 100%. This is with netmap trunk, which does not build a custom mlx4 driver. I've also tried https://github.com/caldweba/netmap, which does, but which is broken (receives only 1023 pps, confirmed broken by the repo owner). Before we spend money, I'd love to hear someone report success with capturing a single flow at >4Mpps, >20Gbps using netmap on Linux and what NIC they use. Ideally this would include transparent mode working so that ARP, ICMP etc can be handled by the host stack. Alternatively, if anyone is managing this on ConnectX-3, I'd love to know how (I have already disabled flow control and adaptive interrupt mitigation on the receiver). Thanks Bruce -- Bruce Merry Senior Science Processing Developer SKA South Africa