From owner-freebsd-net@FreeBSD.ORG Wed Oct 24 00:01:22 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 0EAE9EB1; Wed, 24 Oct 2012 00:01:22 +0000 (UTC) (envelope-from yanegomi@gmail.com) Received: from mail-la0-f54.google.com (mail-la0-f54.google.com [209.85.215.54]) by mx1.freebsd.org (Postfix) with ESMTP id 5828A8FC1B; Wed, 24 Oct 2012 00:01:21 +0000 (UTC) Received: by mail-la0-f54.google.com with SMTP id e12so3474907lag.13 for ; Tue, 23 Oct 2012 17:01:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:cc:content-type; bh=lapaKTo9ILgAtkfty+y6rCSWegvYV6STYr85tSmsAe4=; b=BQOsef7Xv2zpgIYT9mpi30O6DdHy95ot/O6Z613SMaNRNpyDiP+P2TGslmV3FDd/9H PXui5cywYA//XKPw6KjkFRux6rMVDQDPK0iEKAkHxFpMXpE0QpR0nJPuntFwAIMD4+5n DjMMF+GZg+x+PJXNT+XI8HjcPJyU37md1aaLYK/6dpht14uJwq0cIcaKJJR+PwKJL+3H t6Y3kHD7icpEzavb+LSOytsqWSIaYp3BllY8VsRcH+XUZhqwEi0nRX8TpgMMM1EEwfxg klexH3lSIe0x99LT3ywpjXQcDvn/ej7xgDj65+WVPiS9m6bp1HFGAZzNqApS3lI5Y3+F 343w== MIME-Version: 1.0 Received: by 10.112.42.201 with SMTP id q9mr5576255lbl.28.1351036880303; Tue, 23 Oct 2012 17:01:20 -0700 (PDT) Received: by 10.112.43.232 with HTTP; Tue, 23 Oct 2012 17:01:20 -0700 (PDT) Date: Tue, 23 Oct 2012 17:01:20 -0700 Message-ID: Subject: Re: ixgbe TSO performance degrades by ~30% between 7.4 and 8.2/9.0/9.1 From: Garrett Cooper To: Jack Vogel Content-Type: text/plain; charset=ISO-8859-1 Cc: Jack F Vogel , freebsd-net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Oct 2012 00:01:22 -0000 On Tue, Oct 23, 2012 at 4:58 PM, Jack Vogel wrote: > It you mean the ixgbe driver please call it that, the IS an ixgb driver > which is for > VERY old PCI cards, from the context I assume you mean the newer hardware :) Yeah... I meant ixgbe. Subject line fixed :). Thanks! -Garrett > On Tue, Oct 23, 2012 at 3:57 PM, Garrett Cooper wrote: >> >> Hi, >> >> Doing some poking around at the ixgb driver with a card I have at >> $work using netperf and two machines hooked up over crossover, I >> discovered that while ixgb's throughput performance was fantastic on >> 7.3/7.4, thoughput performance of the card is degraded on 8.2/9.0/9.1 >> by ~30% (9400Mbps on 7.4 -> 6294Mbps on 9.0 for example). LRO >> performance on the other hand is fantastic and doesn't degrade with >> the card across FreeBSD versions. Performance remains constant with >> ixgb across 8.2/9.0/9.1. I didn't observe the CPU usage. >> >> More details: >> >> The machines are hooked up in the following configuration: >> >> ----------------------- >> -------------------- >> | Machine 1 | cxgb | <- 10Gbit fibre -> | ix1 | Machine 2 | >> ----------------------- >> --------------------- >> >> Machine Configuration: >> >> The card in Machine 2 is an 82599EB card according to pciconf -lv. >> >> /boot/loader.conf tunables (most of these are set according to 9.x >> defaults in order to establish a sane baseline): >> >> kern.ipc.nmbjumbo9=262144 >> kern.ipc.nmbjumbo16=262144 >> kern.ipc.nmbclusters=262144 >> kern.ipc.nmbjumbop=262144 >> kern.ipc.maxsockbuf=2097152 >> >> /etc/sysctl.conf tunables: >> >> net.inet.tcp.recvspace=65536 >> net.inet.tcp.recvspace_inc=16384 >> net.inet.tcp.recvspace_max=2097152 >> net.inet.tcp.sendspace=32768 >> net.inet.tcp.sendbuf_max=2097152 >> net.inet.tcp.sendbuf_inc=8192 >> >> Kernel Config: >> >> Machine 1 is running a custom version of FreeBSD. The version has been >> constant over the course of my testing. Can give vague details on the >> config, but can't give some specific details. >> Machine 2 is running 7.4/8.2/9.0/9.1 with a GENERIC kernel. >> >> Networking configuration: >> >> - Machine 1 has an IPv4 address of 10.10.10.1; IPv6 is not configured. >> The interface mtu is 1500. >> - Machine 2 has an IPv4 address of 10.10.10.2; IPv6 is not configured. >> The interface mtu is 1500. >> >> Netperf configuration: >> >> - netserver is run on both machines; I don't add any additional >> arguments to the netserver invocation so it just goes off and forks. >> - netperf is run like: netperf -cCjt TCP_STREAM -H >> >> I was wondering if this was a known issue and/or others had seen >> similar problems with this card. I haven't gone into profiling the >> kernel yet with DTrace, but if no one gets back to me before sometime >> later on this week/next week that will be my next course of action for >> tracking down the source of the performance problem.