From owner-freebsd-net@freebsd.org Tue Jul 19 13:36:47 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2BFBDB9E1E1 for ; Tue, 19 Jul 2016 13:36:47 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 1A443169A for ; Tue, 19 Jul 2016 13:36:47 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u6JDakle058209 for ; Tue, 19 Jul 2016 13:36:46 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-net@FreeBSD.org Subject: [Bug 211031] [panic] in ng_uncallout when argument is NULL Date: Tue, 19 Jul 2016 13:36:47 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: crash, needs-qa, patch X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: dim@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-net@FreeBSD.org X-Bugzilla-Flags: mfc-stable10? mfc-stable11? X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 19 Jul 2016 13:36:47 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D211031 Dimitry Andric changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dim@FreeBSD.org --- Comment #2 from Dimitry Andric --- Similar panic here on stable/11 r302854, with a slightly different backtrac= e: #0 __curthread () at ./machine/pcpu.h:221 #1 doadump (textdump=3D) at /home/dim/stable-11/sys/kern/kern_shutdown.c:298 #2 0xffffffff80ae663a in kern_reboot (howto=3D260) at /home/dim/stable-11/sys/kern/kern_shutdown.c:366 #3 0xffffffff80ae6beb in vpanic (fmt=3D, ap=3D0xfffffe04538= b3180) at /home/dim/stable-11/sys/kern/kern_shutdown.c:759 #4 0xffffffff80ae6a23 in panic (fmt=3D) at /home/dim/stable-11/sys/kern/kern_shutdown.c:690 #5 0xffffffff80fb3020 in trap_fatal (frame=3D0xfffffe04538b3480, eva=3D16)= at /home/dim/stable-11/sys/amd64/amd64/trap.c:841 #6 0xffffffff80fb3213 in trap_pfault (frame=3D0xfffffe04538b3480, usermode= =3D0) at /home/dim/stable-11/sys/amd64/amd64/trap.c:691 #7 0xffffffff80fb27bd in trap (frame=3D0xfffffe04538b3480) at /home/dim/stable-11/sys/amd64/amd64/trap.c:442 #8 #9 ng_uncallout (c=3D0xfffff80019f21e60, node=3D0xfffff80019db3000) at /home/dim/stable-11/sys/modules/netgraph/netgraph/../../../netgraph/ng_base= .c:3817 #10 0xffffffff82fbebbc in ng_pptpgre_reset (hpriv=3D0xfffff80019f21e00) at /home/dim/stable-11/sys/modules/netgraph/pptpgre/../../../netgraph/ng_pptpg= re.c:966 #11 ng_pptpgre_disconnect (hook=3D) at /home/dim/stable-11/sys/modules/netgraph/pptpgre/../../../netgraph/ng_pptpg= re.c:493 #12 0xffffffff82f9f928 in ng_destroy_hook (hook=3D0xfffff80019e4fe80) at /home/dim/stable-11/sys/modules/netgraph/netgraph/../../../netgraph/ng_base= .c:1219 #13 0xffffffff82f9f635 in ng_rmnode (node=3D, dummy1=3D, dummy2=3D, dummy3=3D) at /home/dim/stable-11/sys/modules/netgraph/netgraph/../../../netgraph/ng_base= .c:744 #14 0xffffffff82fa1843 in ng_generic_msg (here=3D0xfffff80019db3000, item=3D0xfffff801ab2a7e80, lasthook=3D) at /home/dim/stable-11/sys/modules/netgraph/netgraph/../../../netgraph/ng_base= .c:2523 #15 ng_apply_item (node=3D0xfffff80019db3000, item=3D0xfffff801ab2a7e80, rw= =3D1) at /home/dim/stable-11/sys/modules/netgraph/netgraph/../../../netgraph/ng_base= .c:2437 #16 0xffffffff82fa11b3 in ng_snd_item (item=3D, flags=3D) at /home/dim/stable-11/sys/modules/netgraph/netgraph/../../../netgraph/ng_base= .c:2320 #17 0xffffffff82f9bc4e in ngc_send (so=3D, flags=3D, m=3D, addr=3D, control=3D, td= =3D) at /home/dim/stable-11/sys/modules/netgraph/socket/../../../netgraph/ng_socket= .c:338 #18 0xffffffff80b80ea7 in sosend_generic (so=3D, addr=3D, uio=3D, top=3D, control=3D, flags=3D, td=3D) at /home/dim/stable-11/sys/kern/uipc_socket.c:1359 #19 0xffffffff80b88b7b in kern_sendit (td=3D, s=3D, mp=3D, flags=3D0, control=3D0x0, segflg=3DUIO_USERSPACE) at /home/dim/stable-11/sys/kern/uipc_syscalls.c:848 #20 0xffffffff80b88f7f in sendit (td=3D0xfffff80019f01500, s=3D, mp=3D0xfffffe04538b3960, flags=3D) at /home/dim/stable-11/sys/kern/uipc_syscalls.c:775 #21 0xffffffff80b88dcd in sys_sendto (td=3D0x0, uap=3D) at /home/dim/stable-11/sys/kern/uipc_syscalls.c:899 #22 0xffffffff80fb397e in syscallenter (td=3D, sa=3D) at /home/dim/stable-11/sys/amd64/amd64/../../kern/subr_syscall.c:135 #23 amd64_syscall (td=3D, traced=3D0) at /home/dim/stable-11/sys/amd64/amd64/trap.c:942 #24 #25 0x0000000802518caa in ?? () --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-net@freebsd.org Tue Jul 19 16:16:58 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4967EB9E165 for ; Tue, 19 Jul 2016 16:16:58 +0000 (UTC) (envelope-from patfbsd@davenulle.org) Received: from sender163-mail.zoho.com (sender163-mail.zoho.com [74.201.84.163]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2A45015AF for ; Tue, 19 Jul 2016 16:16:57 +0000 (UTC) (envelope-from patfbsd@davenulle.org) Received: from mr185083 (mr185083.univ-rennes1.fr [129.20.185.83]) by mx.zohomail.com with SMTPS id 1468945013578333.6112500281413; Tue, 19 Jul 2016 09:16:53 -0700 (PDT) Date: Tue, 19 Jul 2016 18:16:44 +0200 From: Patrick Lamaiziere To: Patrick Lamaiziere Cc: freebsd-net@freebsd.org Subject: Re: 10/stable pfsync bulk fail Message-ID: <20160719181644.4d9997c1@mr185083> In-Reply-To: <20160713153523.1640e0e0@mr185083> References: <20160713153523.1640e0e0@mr185083> X-Mailer: Claws Mail 3.13.2 (GTK+ 2.24.29; amd64-portbld-freebsd10.3) MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit X-ZohoMail: Z_59798685 SPT_1 Z_59799992 SPT_1 SLF_D X-Zoho-Virus-Status: 2 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 19 Jul 2016 16:16:58 -0000 Le Wed, 13 Jul 2016 15:35:23 +0200, Patrick Lamaiziere a écrit : Hello, > 10/stable rev 302560 > > I'm building a pair of firewalls with pf and carp and the states are > well synchronized between the firewalls. But at startup or using > "service pfsync restart" pfsync fails the bulk update. > > In rare situations the bulk is successful but I don't know why. I've made some progress on this problem and I think there are several issues. The most one is that pfsync is started by rc(8) before pf starts. And the first thing "/etc/rc.d/pf start" does is to flush the states with pfctl -F all. This flush looks to stop the bulk sync. # rcorder /etc/rc.d/* | grep pf /etc/rc.d/pfsync /etc/rc.d/pflog /etc/rc.d/pf For me this is a nonsense to start pf after pfsync for two reasons: - It flushes the states (may be acquired via the bluk sync). - the size of the pf's states table is not yet set (we have more than 800 000 states here, the default size is not enough and the easiest way to set the size is to load pf.conf). Anyway when starting pfsync after pf, the bulk sync works. There are other strange behaviors (by example when using service pfsync restart, the bulk sync does not work. Looks like it works only one time). I will investigate later and fill a PR. Regards. From owner-freebsd-net@freebsd.org Wed Jul 20 10:04:42 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CEE74B9C7B0 for ; Wed, 20 Jul 2016 10:04:42 +0000 (UTC) (envelope-from bra@fsn.hu) Received: from dg.fsn.hu (dg.fsn.hu [84.2.225.196]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "dg.fsn.hu", Issuer "dg.fsn.hu" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 8D3E01992 for ; Wed, 20 Jul 2016 10:04:41 +0000 (UTC) (envelope-from bra@fsn.hu) Received: by dg.fsn.hu (Postfix, from userid 1003) id 5C9D22A3A; Wed, 20 Jul 2016 11:56:07 +0200 (CEST) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MF-ACE0E1EA [pR: 8.0843] X-CRM114-CacheID: sfid-20160720_11560_D0C39C43 X-CRM114-Status: Good ( pR: 8.0843 ) X-DSPAM-Result: Whitelisted X-DSPAM-Processed: Wed Jul 20 11:56:07 2016 X-DSPAM-Confidence: 0.9899 X-DSPAM-Probability: 0.0000 X-DSPAM-Signature: 578f4ab7864118492410646 X-DSPAM-Factors: 27, User-Agent*Thunderbird/38.8.0, 0.01000, could, 0.01000, the+IP, 0.01000, the+IP, 0.01000, netbooted, 0.01000, Subject*IP+from, 0.01000, ideas+about, 0.01000, Received*online.co.hu+[195.228.243.99]), 0.01000, an, 0.01000, an, 0.01000, change+link, 0.01000, 10, 0.01000, statically, 0.01000, machine+boots, 0.01000, interface, 0.01000, interface, 0.01000, from, 0.01000, from, 0.01000, of, 0.01000, From*"Nagy, Attila" , 0.01000, User-Agent*Mozilla/5.0, 0.01000, Subject*IP, 0.01000, running+from, 0.01000, which+deletes, 0.01000, DHCP), 0.01000, NFS+root, 0.01000, X-Spambayes-Classification: ham; 0.00 Received: from [IPv6:::1] (japan.t-online.co.hu [195.228.243.99]) by dg.fsn.hu (Postfix) with ESMTPSA id 00BC22A38 for ; Wed, 20 Jul 2016 11:56:06 +0200 (CEST) To: freebsd-net@freebsd.org From: "Nagy, Attila" Subject: Interface cannot change link addresses - switching IP from one IF to another on a netbooted machine Message-ID: <578F4AB6.2090200@fsn.hu> Date: Wed, 20 Jul 2016 11:56:06 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.8.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 Jul 2016 10:04:42 -0000 Hi, I have several netbooted machines, which operate in this way: - the machine boots up from one interface (PXE, DHCP), for example igb0 - during the boot, an rc script runs* which deletes the IP from igb0 and creates the lagg interface with igb0, igb1 etc and sets the original IP on that interface (with a statically linked ifconfig) - normal booting continues from the lagg interface This has worked for years and broke somewhere in the lifecycle of 10-STABLE. Now I can't remove the IP address from the interface, ifconfig igb0 delete gives: interface igb0 cannot change link addresses! Any ideas about how could I work around this? Basically I would like to switch the IP from one interface to another while running from an NFS root. (and this has worked before svn rev xxx) Thanks, From owner-freebsd-net@freebsd.org Thu Jul 21 00:39:14 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4F5C3B834F5 for ; Thu, 21 Jul 2016 00:39:14 +0000 (UTC) (envelope-from chris@dunbar.net) Received: from 006.lax.mailroute.net (006.lax.mailroute.net [199.89.1.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.mailroute.net", Issuer "AlphaSSL CA - SHA256 - G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2FFC41ED3 for ; Thu, 21 Jul 2016 00:39:13 +0000 (UTC) (envelope-from chris@dunbar.net) Received: from localhost (localhost.localdomain [127.0.0.1]) by 006.lax.mailroute.net (Postfix) with ESMTP id 3rvvwV71fTz13L77 for ; Thu, 21 Jul 2016 00:36:26 +0000 (UTC) X-Virus-Scanned: by MailRoute Received: from 006.lax.mailroute.net ([199.89.1.9]) by localhost (006.lax.mailroute.net [127.0.0.1]) (mroute_mailscanner, port 10026) with LMTP id 9t-3oAfM_8RJ for ; Thu, 21 Jul 2016 00:36:24 +0000 (UTC) Received: from zimbra.earthside.net (unknown [50.251.189.243]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by 006.lax.mailroute.net (Postfix) with ESMTPS id 3rvvwS49Knz13L6m for ; Thu, 21 Jul 2016 00:36:24 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by zimbra.earthside.net (Postfix) with ESMTP id 96A0650267B for ; Wed, 20 Jul 2016 20:36:23 -0400 (EDT) Received: from zimbra.earthside.net ([127.0.0.1]) by localhost (zimbra.earthside.net [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id rQqOrOmUBhon for ; Wed, 20 Jul 2016 20:36:22 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by zimbra.earthside.net (Postfix) with ESMTP id B76D350267C for ; Wed, 20 Jul 2016 20:36:22 -0400 (EDT) Received: from zimbra.earthside.net ([127.0.0.1]) by localhost (zimbra.earthside.net [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id aKGHndoOY6S8 for ; Wed, 20 Jul 2016 20:36:22 -0400 (EDT) Received: from zimbra.earthside.net (zimbra.earthside.net [10.11.12.148]) by zimbra.earthside.net (Postfix) with ESMTP id 8DC2150267B for ; Wed, 20 Jul 2016 20:36:22 -0400 (EDT) Date: Wed, 20 Jul 2016 20:36:22 -0400 (EDT) From: Chris Dunbar To: freebsd-net@freebsd.org Message-ID: <1244557023.708807.1469061382192.JavaMail.zimbra@dunbar.net> Subject: Slow performance with Intel X540-T2 10Gb NIC MIME-Version: 1.0 X-Originating-IP: [10.11.12.148] X-Mailer: Zimbra 8.6.0_GA_1153 (ZimbraWebClient - GC51 (Mac)/8.6.0_GA_1153) Thread-Topic: Slow performance with Intel X540-T2 10Gb NIC Thread-Index: C+DfYhHIN+CVKUXHyyOv7qhR5CwXaw== Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Jul 2016 00:39:14 -0000 Hello, I am new to FreeBSD and recently built a file server out of new components running FreeBSD 10.3. I installed an Intel X540-T2 10 Gb NIC and am experiencing what I consider to be slow transfer speeds. I am using iperf3 to measure the speed and test the results of modifications. So far nothing I have done has made a noticeable difference. If I run iperf3 -s on the FreeBSD server, I see transfer speeds of approximately 1.6 Gb/s. If I run iperf3 in client mode, the speed improves to ~2.75 Gb/s. However, if I replace FreeBSD with CentOS 7 on the same hardware, I see iperf3 speeds surpassing 8 GB/s. The other end of my iperf3 test is a Windows 10 box that also has an Intel X540-T2 installed. I did notice that FreeBSD 10.3 (and 11.0 alpha 6 for that matter) includes a slightly older Intel driver (v3.1.13-k). I managed to build a custom kernel that removed the Intel PRO/10GbE PCIE NIC drivers. That allowed me to manually load the latest 3.1.14 driver downloaded from Intel's web site. Unfortunately that did not produce any improvements. I am working my way through man tuning() and some other articles on network performance. So far nothing I tweak makes a noticeable difference. I'm increasingly skeptical that I am going to find a setting or two that more than doubles the speed I am currently experiencing. I am open to any and all suggestions at this point. Thank you! Chris From owner-freebsd-net@freebsd.org Thu Jul 21 07:54:37 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8511ABA056E for ; Thu, 21 Jul 2016 07:54:37 +0000 (UTC) (envelope-from julien.charbon@gmail.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 5B9CF1E65 for ; Thu, 21 Jul 2016 07:54:37 +0000 (UTC) (envelope-from julien.charbon@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id 56A86BA056B; Thu, 21 Jul 2016 07:54:37 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 55EB5BA0569; Thu, 21 Jul 2016 07:54:37 +0000 (UTC) (envelope-from julien.charbon@gmail.com) Received: from mail-wm0-f53.google.com (mail-wm0-f53.google.com [74.125.82.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 09E861E63; Thu, 21 Jul 2016 07:54:36 +0000 (UTC) (envelope-from julien.charbon@gmail.com) Received: by mail-wm0-f53.google.com with SMTP id q128so1983354wma.1; Thu, 21 Jul 2016 00:54:36 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to; bh=yWSNo71n1kyYDNe2DcqDQSOI8NYHIuJdJ64TxMgsglE=; b=NrH5WHgEeqjKfgmKJFYhYAM4CjhRS9W6gWJzmSRlxr5vxkngjfA55XXmRM1YqWiu24 zA9A0zuCtKd8nWWd1Kf1cR134nhOtSpdLVwJ3EouqIcVShUgl8wr4H8MtCfFsvHnNeOJ xujXfzQhcLVkZGYj4HM7+28s9eTXwgpfI6TvNDuEVRHIVW1Ue7zqHapDAcikw2w80pDx 1xve5X5eEh0eCwOqmakz+uFXXOPWFFMZOJ5WpKYuDaYsP8hgmqjC19FuL0My39iYN9+5 1OGD6l1MVEr9hJKkL0nA5hx73MHA6NMGnaE8koed4NyNMXex+vpZrdi9IbH/ZxO48lt3 GNhQ== X-Gm-Message-State: ALyK8tIJhtChClXw/FbucNTJcDM794oQSyL9x+SzAsyMEZH/3TYsKhVRfOw4OJiCvuOmlA== X-Received: by 10.194.89.68 with SMTP id bm4mr5515631wjb.164.1469087668756; Thu, 21 Jul 2016 00:54:28 -0700 (PDT) Received: from [172.20.10.4] (47.236.197.178.dynamic.wless.lssmb00p-cgnat.res.cust.swisscom.ch. [178.197.236.47]) by smtp.gmail.com with ESMTPSA id b130sm2001088wmg.3.2016.07.21.00.54.26 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 21 Jul 2016 00:54:27 -0700 (PDT) Subject: Re: panic with tcp timers To: Larry Rosenman References: <20160617045319.GE1076@FreeBSD.org> <1f28844b-b4ea-b544-3892-811f2be327b9@freebsd.org> <20160620073917.GI1076@FreeBSD.org> <1d18d0e2-3e42-cb26-928c-2989d0751884@freebsd.org> Cc: Gleb Smirnoff , rrs@freebsd.org, hselasky@freebsd.org, net@freebsd.org, current@freebsd.org, owner-freebsd-current@freebsd.org From: Julien Charbon Message-ID: <548bf673-580d-350a-9f91-88553f3c82f1@freebsd.org> Date: Thu, 21 Jul 2016 09:54:20 +0200 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:45.0) Gecko/20100101 Thunderbird/45.2.0 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="RltEhHq6WEsTOuEtcjO8preCMk3cvFPf5" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Jul 2016 07:54:37 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --RltEhHq6WEsTOuEtcjO8preCMk3cvFPf5 Content-Type: multipart/mixed; boundary="9FJ5nB7dX6eQcKe0ImWav0rWKa4Vtkule" From: Julien Charbon To: Larry Rosenman Cc: Gleb Smirnoff , rrs@freebsd.org, hselasky@freebsd.org, net@freebsd.org, current@freebsd.org, owner-freebsd-current@freebsd.org Message-ID: <548bf673-580d-350a-9f91-88553f3c82f1@freebsd.org> Subject: Re: panic with tcp timers References: <20160617045319.GE1076@FreeBSD.org> <1f28844b-b4ea-b544-3892-811f2be327b9@freebsd.org> <20160620073917.GI1076@FreeBSD.org> <1d18d0e2-3e42-cb26-928c-2989d0751884@freebsd.org> In-Reply-To: --9FJ5nB7dX6eQcKe0ImWav0rWKa4Vtkule Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Hi, On 7/14/16 11:02 PM, Larry Rosenman wrote: > On 2016-07-14 12:01, Julien Charbon wrote: >> On 6/20/16 11:55 AM, Julien Charbon wrote: >>> On 6/20/16 9:39 AM, Gleb Smirnoff wrote: >>>> On Fri, Jun 17, 2016 at 11:27:39AM +0200, Julien Charbon wrote: >>>> J> > Comparing stable/10 and head, I see two changes that could >>>> J> > affect that: >>>> J> > >>>> J> > - callout_async_drain >>>> J> > - switch to READ lock for inp info in tcp timers >>>> J> > >>>> J> > That's why you are in To, Julien and Hans :) >>>> J> > >>>> J> > We continue investigating, and I will keep you updated. >>>> J> > However, any help is welcome. I can share cores. >>>> >>>> Now, spending some time with cores and adding a bunch of >>>> extra CTRs, I have a sequence of events that lead to the >>>> panic. In short, the bug is in the callout system. It seems >>>> to be not relevant to the callout_async_drain, at least for >>>> now. The transition to READ lock unmasked the problem, that's >>>> why NetflixBSD 10 doesn't panic. >>>> >>>> The panic requires heavy contention on the TCP info lock. >>>> >>>> [CPU 1] the callout fires, tcp_timer_keep entered >>>> [CPU 1] blocks on INP_INFO_RLOCK(&V_tcbinfo); >>>> [CPU 2] schedules the callout >>>> [CPU 2] tcp_discardcb called >>>> [CPU 2] callout successfully canceled >>>> [CPU 2] tcpcb freed >>>> [CPU 1] unblocks... panic >>>> >>>> When the lock was WLOCK, all contenders were resumed in a >>>> sequence they came to the lock. Now, that they are readers, >>>> once the lock is released, readers are resumed in a "random" >>>> order, and this allows tcp_discardcb to go before the old >>>> running callout, and this unmasks the panic. >>> >>> Highly interesting. I should be able to reproduce that (will be use= ful >>> for testing the corresponding fix). >> >> Finally, I was able to reproduce it (without glebius fix). The tric= k >> was to really lower TCP keep timer expiration: >> >> $ sysctl -a | grep tcp.keep >> net.inet.tcp.keepidle: 7200000 >> net.inet.tcp.keepintvl: 75000 >> net.inet.tcp.keepinit: 75000 >> net.inet.tcp.keepcnt: 8 >> $ sudo bash -c "sysctl net.inet.tcp.keepidle=3D10 && sysctl >> net.inet.tcp.keepintvl=3D50 && sysctl net.inet.tcp.keepinit=3D10" >> Password: >> net.inet.tcp.keepidle: 7200000 -> 10 >> net.inet.tcp.keepintvl: 75000 -> 50 >> net.inet.tcp.keepinit: 75000 -> 10 >> >> Note: It will certainly close all your ssh connections to the tested >> server. >> >> Now I will test in order: >> >> #1. glebius fix >> https://svnweb.freebsd.org/base?view=3Drevision&revision=3D302350 >> >> #2. rss extra fix >> https://reviews.freebsd.org/D7135 >> >> #3. rrs TCP Timer cleanup >> https://reviews.freebsd.org/D7136 >=20 > please see also https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D210= 884 My tests result so far: #1. r302350: First glebius TCP timer fix: No more TCP timer kernel panic during 48h under 200k TCP query per second load. Sadly I was unable to reproduce the issue described here: panic: bogus refcnt 0 on lle 0xfffff80004608c00 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D210884 #2. r303098: Got all kernel callout changes since r302350, (updates on callout code are indeed always full of surprises): https://svnweb.freebsd.org/base/head/sys/kern/kern_timeout.c?view=3Dlog&p= athrev=3D303098 No kernel panic either. Still to test: #3. rss extra fix (if still relevant now) https://reviews.freebsd.org/D7135 #4. rrs TCP Timer cleanup: https://reviews.freebsd.org/D7136 My 2 cents. -- Julien --9FJ5nB7dX6eQcKe0ImWav0rWKa4Vtkule-- --RltEhHq6WEsTOuEtcjO8preCMk3cvFPf5 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQEcBAEBCgAGBQJXkH+yAAoJEKVlQ5Je6dhxGe0H+gJyAT5R0hpGgjBBTICN3h+q aGvIgBPC3HgVDJhU1ZKhU0xjNZirq2icxgh/0UV+iuZvOUZCTteT4IsVl8WoZDUQ 0VODwVSj748EJdftA5GqDR464nY+6McIj1FrWtmbVgqtYkKP2oAuOQzy0w2lRYeK c3m8gb9JP0bN8M9zFRee2IzaIikzQJtaapMX77XzBR5umxuzAnp4tbSuAmJdE3Ln +ddBH/4DcTLQEKSBboqQwM/VLYzoWl33e5IQhrYyUzJe1dfXLZHBS6sm2eHdug+0 NIOEuBcYRJZqp4TwYyjIGauIALAfqo6zDQCSUZvhkgqNmkriogBVtjz92pxmQPg= =5jrc -----END PGP SIGNATURE----- --RltEhHq6WEsTOuEtcjO8preCMk3cvFPf5-- From owner-freebsd-net@freebsd.org Thu Jul 21 08:01:51 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 28704BA0B0C for ; Thu, 21 Jul 2016 08:01:51 +0000 (UTC) (envelope-from hps@selasky.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 1262F15BE for ; Thu, 21 Jul 2016 08:01:51 +0000 (UTC) (envelope-from hps@selasky.org) Received: by mailman.ysv.freebsd.org (Postfix) id 0E423BA0B09; Thu, 21 Jul 2016 08:01:51 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0D94BBA0B07; Thu, 21 Jul 2016 08:01:51 +0000 (UTC) (envelope-from hps@selasky.org) Received: from mail.turbocat.net (mail.turbocat.net [IPv6:2a01:4f8:d16:4514::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9C9A215BB; Thu, 21 Jul 2016 08:01:50 +0000 (UTC) (envelope-from hps@selasky.org) Received: from laptop015.home.selasky.org (unknown [62.141.129.119]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.turbocat.net (Postfix) with ESMTPSA id B844E1FE024; Thu, 21 Jul 2016 10:01:46 +0200 (CEST) Subject: Re: panic with tcp timers To: Julien Charbon , Larry Rosenman References: <20160617045319.GE1076@FreeBSD.org> <1f28844b-b4ea-b544-3892-811f2be327b9@freebsd.org> <20160620073917.GI1076@FreeBSD.org> <1d18d0e2-3e42-cb26-928c-2989d0751884@freebsd.org> <548bf673-580d-350a-9f91-88553f3c82f1@freebsd.org> Cc: Gleb Smirnoff , rrs@freebsd.org, net@freebsd.org, current@freebsd.org, owner-freebsd-current@freebsd.org From: Hans Petter Selasky Message-ID: <3c38de84-1e69-4321-f8b8-3c259d689b6d@selasky.org> Date: Thu, 21 Jul 2016 10:05:48 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:45.0) Gecko/20100101 Thunderbird/45.0 MIME-Version: 1.0 In-Reply-To: <548bf673-580d-350a-9f91-88553f3c82f1@freebsd.org> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Jul 2016 08:01:51 -0000 On 07/21/16 09:54, Julien Charbon wrote: > > Hi, > > On 7/14/16 11:02 PM, Larry Rosenman wrote: >> On 2016-07-14 12:01, Julien Charbon wrote: >>> On 6/20/16 11:55 AM, Julien Charbon wrote: >>>> On 6/20/16 9:39 AM, Gleb Smirnoff wrote: >>>>> On Fri, Jun 17, 2016 at 11:27:39AM +0200, Julien Charbon wrote: >>>>> J> > Comparing stable/10 and head, I see two changes that could >>>>> J> > affect that: >>>>> J> > >>>>> J> > - callout_async_drain >>>>> J> > - switch to READ lock for inp info in tcp timers >>>>> J> > >>>>> J> > That's why you are in To, Julien and Hans :) >>>>> J> > >>>>> J> > We continue investigating, and I will keep you updated. >>>>> J> > However, any help is welcome. I can share cores. >>>>> >>>>> Now, spending some time with cores and adding a bunch of >>>>> extra CTRs, I have a sequence of events that lead to the >>>>> panic. In short, the bug is in the callout system. It seems >>>>> to be not relevant to the callout_async_drain, at least for >>>>> now. The transition to READ lock unmasked the problem, that's >>>>> why NetflixBSD 10 doesn't panic. >>>>> >>>>> The panic requires heavy contention on the TCP info lock. >>>>> >>>>> [CPU 1] the callout fires, tcp_timer_keep entered >>>>> [CPU 1] blocks on INP_INFO_RLOCK(&V_tcbinfo); >>>>> [CPU 2] schedules the callout >>>>> [CPU 2] tcp_discardcb called >>>>> [CPU 2] callout successfully canceled >>>>> [CPU 2] tcpcb freed >>>>> [CPU 1] unblocks... panic >>>>> >>>>> When the lock was WLOCK, all contenders were resumed in a >>>>> sequence they came to the lock. Now, that they are readers, >>>>> once the lock is released, readers are resumed in a "random" >>>>> order, and this allows tcp_discardcb to go before the old >>>>> running callout, and this unmasks the panic. >>>> >>>> Highly interesting. I should be able to reproduce that (will be useful >>>> for testing the corresponding fix). >>> >>> Finally, I was able to reproduce it (without glebius fix). The trick >>> was to really lower TCP keep timer expiration: >>> >>> $ sysctl -a | grep tcp.keep >>> net.inet.tcp.keepidle: 7200000 >>> net.inet.tcp.keepintvl: 75000 >>> net.inet.tcp.keepinit: 75000 >>> net.inet.tcp.keepcnt: 8 >>> $ sudo bash -c "sysctl net.inet.tcp.keepidle=10 && sysctl >>> net.inet.tcp.keepintvl=50 && sysctl net.inet.tcp.keepinit=10" >>> Password: >>> net.inet.tcp.keepidle: 7200000 -> 10 >>> net.inet.tcp.keepintvl: 75000 -> 50 >>> net.inet.tcp.keepinit: 75000 -> 10 >>> >>> Note: It will certainly close all your ssh connections to the tested >>> server. >>> >>> Now I will test in order: >>> >>> #1. glebius fix >>> https://svnweb.freebsd.org/base?view=revision&revision=302350 >>> >>> #2. rss extra fix >>> https://reviews.freebsd.org/D7135 >>> >>> #3. rrs TCP Timer cleanup >>> https://reviews.freebsd.org/D7136 >> >> please see also https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=210884 > > My tests result so far: > > #1. r302350: First glebius TCP timer fix: No more TCP timer kernel > panic during 48h under 200k TCP query per second load. > > Sadly I was unable to reproduce the issue described here: > > panic: bogus refcnt 0 on lle 0xfffff80004608c00 > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=210884 > > #2. r303098: Got all kernel callout changes since r302350, (updates on > callout code are indeed always full of surprises): > https://svnweb.freebsd.org/base/head/sys/kern/kern_timeout.c?view=log&pathrev=303098 > > No kernel panic either. > > Still to test: > > #3. rss extra fix (if still relevant now) > https://reviews.freebsd.org/D7135 > > #4. rrs TCP Timer cleanup: > https://reviews.freebsd.org/D7136 > > My 2 cents. > Hi, You should also check for memory leaks using "vmstat -m" . --HPS From owner-freebsd-net@freebsd.org Thu Jul 21 17:27:27 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AB567BA0EBE for ; Thu, 21 Jul 2016 17:27:27 +0000 (UTC) (envelope-from ricera10@gmail.com) Received: from mail-yw0-f176.google.com (mail-yw0-f176.google.com [209.85.161.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 72D4F19CD for ; Thu, 21 Jul 2016 17:27:26 +0000 (UTC) (envelope-from ricera10@gmail.com) Received: by mail-yw0-f176.google.com with SMTP id u134so80322978ywg.3 for ; Thu, 21 Jul 2016 10:27:26 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=S+Ah1h2oWuG+iPrP4X0V2iBpfBxyobhjeU2LEAjIv+A=; b=EaLdzf5i7SPnu04D3xTVr5SI+VvS5YzfAXI8ijrqlvKNI5o9z2YI+wIjjI3GBkBpK3 /h4SxK8QXSJjkmjf17BrForuhqQal7nKf8Jy6c6W0ZFxT0tADP/6otSkPCo9kTSK1ZuK L4CGPYPuUGXL3qveQhjo/wO7YRIVFTYYrsev4C9TVocGDV9h2/CjFjBO5JK0dNW95A7u c6Eg4sqceVv15qBt7IdIpyZk/RBDHnkssqpDMw0yOtJedaGouyblFcR3MIZRN7sVVFk/ uSTECGzO8ILUR2MwdFuGzww88LQX3io8frxdb10xaJTKTGDnAuEfmNHuQSY4wyNupb1d /itA== X-Gm-Message-State: ALyK8tKhSpT3TeVw2qdcfUPFhzZwt4UM8xaI+6EVf3Jo/Iwy+DhHTkyHNOTFswI9i47YQQ== X-Received: by 10.129.48.144 with SMTP id w138mr33430793yww.204.1469122040142; Thu, 21 Jul 2016 10:27:20 -0700 (PDT) Received: from mail-yw0-f180.google.com (mail-yw0-f180.google.com. [209.85.161.180]) by smtp.gmail.com with ESMTPSA id u187sm3651914ywu.27.2016.07.21.10.27.19 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 21 Jul 2016 10:27:20 -0700 (PDT) Received: by mail-yw0-f180.google.com with SMTP id r9so81131127ywg.0 for ; Thu, 21 Jul 2016 10:27:19 -0700 (PDT) X-Received: by 10.129.77.4 with SMTP id a4mr35769372ywb.67.1469122039719; Thu, 21 Jul 2016 10:27:19 -0700 (PDT) MIME-Version: 1.0 References: <1244557023.708807.1469061382192.JavaMail.zimbra@dunbar.net> In-Reply-To: <1244557023.708807.1469061382192.JavaMail.zimbra@dunbar.net> From: Eric Joyner Date: Thu, 21 Jul 2016 17:27:10 +0000 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: Slow performance with Intel X540-T2 10Gb NIC To: Chris Dunbar , freebsd-net@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Jul 2016 17:27:27 -0000 (Replying-all this time) Did you try to set these settings that ESnet recommends? https://fasterdata.es.net/host-tuning/freebsd/ We don't use iperf3 here at Intel (we use netperf instead), so I'm not sure I can be much help diagnosing what's wrong. On Wed, Jul 20, 2016 at 5:39 PM Chris Dunbar wrote: > Hello, > > I am new to FreeBSD and recently built a file server out of new components > running FreeBSD 10.3. I installed an Intel X540-T2 10 Gb NIC and am > experiencing what I consider to be slow transfer speeds. I am using iperf3 > to measure the speed and test the results of modifications. So far nothing > I have done has made a noticeable difference. If I run iperf3 -s on the > FreeBSD server, I see transfer speeds of approximately 1.6 Gb/s. If I run > iperf3 in client mode, the speed improves to ~2.75 Gb/s. However, if I > replace FreeBSD with CentOS 7 on the same hardware, I see iperf3 speeds > surpassing 8 GB/s. The other end of my iperf3 test is a Windows 10 box that > also has an Intel X540-T2 installed. > > I did notice that FreeBSD 10.3 (and 11.0 alpha 6 for that matter) includes > a slightly older Intel driver (v3.1.13-k). I managed to build a custom > kernel that removed the Intel PRO/10GbE PCIE NIC drivers. That allowed me > to manually load the latest 3.1.14 driver downloaded from Intel's web site. > Unfortunately that did not produce any improvements. I am working my way > through man tuning() and some other articles on network performance. So far > nothing I tweak makes a noticeable difference. I'm increasingly skeptical > that I am going to find a setting or two that more than doubles the speed I > am currently experiencing. > > I am open to any and all suggestions at this point. > > Thank you! > Chris > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > From owner-freebsd-net@freebsd.org Thu Jul 21 19:39:41 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 536FBBA12C3 for ; Thu, 21 Jul 2016 19:39:41 +0000 (UTC) (envelope-from gallatin@cs.duke.edu) Received: from duke.cs.duke.edu (duke.cs.duke.edu [152.3.140.1]) (using TLSv1.1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2AF9F1D0A; Thu, 21 Jul 2016 19:39:41 +0000 (UTC) (envelope-from gallatin@cs.duke.edu) Received: from [192.168.200.3] (c-73-147-115-187.hsd1.va.comcast.net [73.147.115.187]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: gallatin) by duke.cs.duke.edu (Postfix) with ESMTPSA id 0BBB0F80530; Thu, 21 Jul 2016 15:31:48 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=cs.duke.edu; s=mail; t=1469129509; bh=NHyvsN2G6ymlfY0Yw6oDeD9rtITCbHUH10LjDpR7Ohs=; h=Subject:To:From:Date; b=fQ64nyPiF2ERbAZcJquTXkUM6b7tSBOkOFYqIImH7SUyQVkG8XFnruL9t5R5O0FpM vDJF0rq1gmznu36M+IK0y27ZIRjEkbkj4C6tyNEaKs/2JlIcvsoRFKhKOHrkI+qk3y F6VglJ/mdlyzlN48JNjqcbyZIMyVAf1nBi9pbs++OGA7fnnhtQuhZZBXbRNXQa/2Zs Oh5PWlpd5rtbXboMDmSegyCOCshXap1pO277kDdlrNJeU9n0Cr6QmVjzhDrOw0adaR HgiIQm5xeWKaFwQ5wO8UuCtlv9yA+gFuuswn1UPu05qddzZ4W9yJ1bfikZvzFQxnkL e/0ZweZnR9X+A== Subject: Re: proposal: splitting NIC RSS up from stack RSS To: Adrian Chadd , FreeBSD Net References: From: Andrew Gallatin Message-ID: <306af514-70ff-f3bf-5b4f-da7ac1ec6580@cs.duke.edu> Date: Thu, 21 Jul 2016 15:31:46 -0400 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:45.0) Gecko/20100101 Thunderbird/45.1.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Jul 2016 19:39:41 -0000 On 07/14/2016 16:06, Adrian Chadd wrote: > I'd appreciate any other feedback/comments/suggestions. If you're > using RSS and you haven't told me then please let me know! Hi Adrian, I'm a huge fan of your RSS work. In fact, I did a backport of RSS to Netflix's stable-10 about 6 months ago. I was really interesting in breaking up the global network hashtables, where we see a lot of contention. PCBGROUP didn't help much (just spread contention around), so I was hoping that RSS would be the magic bullet. Things may have progressed since then, but the real deficiencies that I saw were: o RSS (at the time) would only use a power-of-two number of cores. Sadly, Intel and AMD are building lots of chips with oddball core counts. So in a workload like ours where most work is initiated via the NIC rx ithread, having a 14-core machine meant leaving almost 1/2 the machine mostly idle, while 8 cores were maxed out. o There is (or was at the time) no library interface for RSS, and no patches for popular web servers (like nginx) to use RSS. The only example RSS users I could find were a few things from your blog. These 2 things lead me to abandon the backport, as I didn't have time to address them on top of other work I was doing. I especially think getting a real API and a real example consumer would help a lot. Best regards, Drew From owner-freebsd-net@freebsd.org Thu Jul 21 20:55:55 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 14D3FBA03E6 for ; Thu, 21 Jul 2016 20:55:55 +0000 (UTC) (envelope-from chris@dunbar.net) Received: from 002.las.mailroute.net (002.las.mailroute.net [199.89.4.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.mailroute.net", Issuer "AlphaSSL CA - SHA256 - G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E5B39198A for ; Thu, 21 Jul 2016 20:55:54 +0000 (UTC) (envelope-from chris@dunbar.net) Received: from localhost (localhost.localdomain [127.0.0.1]) by 002.las.mailroute.net (Postfix) with ESMTP id 3rwQx52q5kz1y8m for ; Thu, 21 Jul 2016 20:53:45 +0000 (UTC) X-Virus-Scanned: by MailRoute Received: from 002.las.mailroute.net ([199.89.4.5]) by localhost (002.las.mailroute.net. [127.0.0.1]) (mroute_mailscanner, port 10026) with LMTP id CKPEg6XBX1hp for ; Thu, 21 Jul 2016 20:53:43 +0000 (UTC) Received: from zimbra.earthside.net (unknown [50.251.189.243]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by 002.las.mailroute.net (Postfix) with ESMTPS id 3rwQx26XZSz1y88 for ; Thu, 21 Jul 2016 20:53:42 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by zimbra.earthside.net (Postfix) with ESMTP id B401D50267B for ; Thu, 21 Jul 2016 16:53:41 -0400 (EDT) Received: from zimbra.earthside.net ([127.0.0.1]) by localhost (zimbra.earthside.net [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id fZvdDxXumRwq for ; Thu, 21 Jul 2016 16:53:40 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by zimbra.earthside.net (Postfix) with ESMTP id 7721750267D for ; Thu, 21 Jul 2016 16:53:40 -0400 (EDT) Received: from zimbra.earthside.net ([127.0.0.1]) by localhost (zimbra.earthside.net [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id FuTTrZZmA0JE for ; Thu, 21 Jul 2016 16:53:40 -0400 (EDT) Received: from zimbra.earthside.net (zimbra.earthside.net [10.11.12.148]) by zimbra.earthside.net (Postfix) with ESMTP id 531BB50267B for ; Thu, 21 Jul 2016 16:53:40 -0400 (EDT) Date: Thu, 21 Jul 2016 16:53:40 -0400 (EDT) From: Chris Dunbar To: freebsd-net Message-ID: <1441424852.712842.1469134420198.JavaMail.zimbra@dunbar.net> In-Reply-To: References: <1244557023.708807.1469061382192.JavaMail.zimbra@dunbar.net> Subject: Re: Slow performance with Intel X540-T2 10Gb NIC MIME-Version: 1.0 X-Originating-IP: [10.11.12.148] X-Mailer: Zimbra 8.6.0_GA_1153 (ZimbraWebClient - GC51 (Mac)/8.6.0_GA_1153) Thread-Topic: Slow performance with Intel X540-T2 10Gb NIC Thread-Index: bkXeUbMihgtikNh5aEfKledchPk+cA== Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Jul 2016 20:55:55 -0000 Eric, et al: I haven't tried netperf yet, but I do have some new information to share. I have two systems that I am using for testing: the new server and an older (not too old) desktop PC. I installed CentOS on the new server again because I know it can achieve >9 GB/s with the X540. I replaced Windows on the desktop PC with FreeBSD 10.3 (it also has an X540) and ran iperf3 again. I was able to achieve >9 GB/s so I know the problem isn't the X540 and I know the problem isn't anything with the default installation of FreeBSD 10.3. So, what in the world might be nutty in my BIOS settings (or elsewhere) that would cause the new server + FreeBSD 10.3 + X540 to equal slow performance? Regards, Chris From: "Eric Joyner" To: "chris" , "freebsd-net" Sent: Thursday, July 21, 2016 1:27:10 PM Subject: Re: Slow performance with Intel X540-T2 10Gb NIC (Replying-all this time) Did you try to set these settings that ESnet recommends? https://fasterdata.es.net/host-tuning/freebsd/ We don't use iperf3 here at Intel (we use netperf instead), so I'm not sure I can be much help diagnosing what's wrong. On Wed, Jul 20, 2016 at 5:39 PM Chris Dunbar < chris@dunbar.net > wrote: Hello, I am new to FreeBSD and recently built a file server out of new components running FreeBSD 10.3. I installed an Intel X540-T2 10 Gb NIC and am experiencing what I consider to be slow transfer speeds. I am using iperf3 to measure the speed and test the results of modifications. So far nothing I have done has made a noticeable difference. If I run iperf3 -s on the FreeBSD server, I see transfer speeds of approximately 1.6 Gb/s. If I run iperf3 in client mode, the speed improves to ~2.75 Gb/s. However, if I replace FreeBSD with CentOS 7 on the same hardware, I see iperf3 speeds surpassing 8 GB/s. The other end of my iperf3 test is a Windows 10 box that also has an Intel X540-T2 installed. I did notice that FreeBSD 10.3 (and 11.0 alpha 6 for that matter) includes a slightly older Intel driver (v3.1.13-k). I managed to build a custom kernel that removed the Intel PRO/10GbE PCIE NIC drivers. That allowed me to manually load the latest 3.1.14 driver downloaded from Intel's web site. Unfortunately that did not produce any improvements. I am working my way through man tuning() and some other articles on network performance. So far nothing I tweak makes a noticeable difference. I'm increasingly skeptical that I am going to find a setting or two that more than doubles the speed I am currently experiencing. I am open to any and all suggestions at this point. Thank you! Chris _______________________________________________ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to " freebsd-net-unsubscribe@freebsd.org " From owner-freebsd-net@freebsd.org Thu Jul 21 21:07:17 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 413D2BA0872 for ; Thu, 21 Jul 2016 21:07:17 +0000 (UTC) (envelope-from jfvogel@gmail.com) Received: from mail-vk0-x22e.google.com (mail-vk0-x22e.google.com [IPv6:2607:f8b0:400c:c05::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id ED41E1289 for ; Thu, 21 Jul 2016 21:07:16 +0000 (UTC) (envelope-from jfvogel@gmail.com) Received: by mail-vk0-x22e.google.com with SMTP id s189so130981877vkh.1 for ; Thu, 21 Jul 2016 14:07:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=JAQBn2X1ZcKgduuiTghYh6/KYgXa8+wH1O0TPEDxV8I=; b=HnMu0iMN7o48dEqYuZYSz8GKZc1xGdAnqzUPpTxjoA8rw4k2iOdqLLWRXv3xIPrDXh +5mMsQRIUVkotWRwedOMW9Y5ZAASdNrS1Wo1w6Lu7kL728Mgu3c1Z4bHtmUqy2KT0zfi oJl2fbMLya7TuaWZ0vLdPw+iHqPA1Hq7vZVI6tYYlzgCJGmoFoy4eN57rpsmfN/pKuWs OjBt1Ah0cbKTqIwSNsYkDWxV6TkIOKVSTjDAhiwFo4FposzaQGWnZS4lWa915hiEeGkf DKOi8oiL4TGhYIcnrHx+m53pyb2yyHPHin+ASHlIY+nLPoLUnsI1s0tn/HsCfcregNkW Js9w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=JAQBn2X1ZcKgduuiTghYh6/KYgXa8+wH1O0TPEDxV8I=; b=hhj5zyhtam578WWqmJQ3bE3zfS1ZPho++4cHeP4qIdMsUh02XhrB7y5UyiVFZa6L88 6ZRr+Ia+ZhBYNcADCFyZKEqxrY1ZzbSDVwEmiynqKwp9xI6FGExUoNz22jbTcSEc5Id6 QekMQvNLA79SibacjJj+5wJy1QBUvJKQebS3SMKlc7QhJQuYN+iFOSe2C5G5wqrgbPAY TXOxR7DhWCX3yfwTRn0SYXKGwiZeVn1JjrpNSR+hHIhZedXf8kJR3A7dgDh1aLHleNAN vV0X0V6VoZyhnGp8TBmmdhfLPZjSOkHQiEp7c+S2b1pgxKiKZnVETKZisj3TdGwoPv4m C+Cg== X-Gm-Message-State: AEkoousOKPdKOBCxYzpHWcQb7dceshBPqyJrhSDDlgd8boW72WIYhnSDeCdh/y9F7f/uWs+4pll75Loo0g/jcw== X-Received: by 10.159.41.163 with SMTP id s32mr148342uas.149.1469135236149; Thu, 21 Jul 2016 14:07:16 -0700 (PDT) MIME-Version: 1.0 Received: by 10.31.76.132 with HTTP; Thu, 21 Jul 2016 14:07:15 -0700 (PDT) In-Reply-To: <1441424852.712842.1469134420198.JavaMail.zimbra@dunbar.net> References: <1244557023.708807.1469061382192.JavaMail.zimbra@dunbar.net> <1441424852.712842.1469134420198.JavaMail.zimbra@dunbar.net> From: Jack Vogel Date: Thu, 21 Jul 2016 14:07:15 -0700 Message-ID: Subject: Re: Slow performance with Intel X540-T2 10Gb NIC To: Chris Dunbar Cc: freebsd-net Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Jul 2016 21:07:17 -0000 NUMA issues maybe? They have been a problem on some recent system architectures. On Thu, Jul 21, 2016 at 1:53 PM, Chris Dunbar wrote: > Eric, et al: > > I haven't tried netperf yet, but I do have some new information to share. > I have two systems that I am using for testing: the new server and an older > (not too old) desktop PC. I installed CentOS on the new server again > because I know it can achieve >9 GB/s with the X540. I replaced Windows on > the desktop PC with FreeBSD 10.3 (it also has an X540) and ran iperf3 > again. I was able to achieve >9 GB/s so I know the problem isn't the X540 > and I know the problem isn't anything with the default installation of > FreeBSD 10.3. So, what in the world might be nutty in my BIOS settings (or > elsewhere) that would cause the new server + FreeBSD 10.3 + X540 to equal > slow performance? > > Regards, > Chris > > > From: "Eric Joyner" > To: "chris" , "freebsd-net" > Sent: Thursday, July 21, 2016 1:27:10 PM > Subject: Re: Slow performance with Intel X540-T2 10Gb NIC > > (Replying-all this time) > > Did you try to set these settings that ESnet recommends? > https://fasterdata.es.net/host-tuning/freebsd/ > > We don't use iperf3 here at Intel (we use netperf instead), so I'm not > sure I can be much help diagnosing what's wrong. > > On Wed, Jul 20, 2016 at 5:39 PM Chris Dunbar < chris@dunbar.net > wrote: > > > Hello, > > I am new to FreeBSD and recently built a file server out of new components > running FreeBSD 10.3. I installed an Intel X540-T2 10 Gb NIC and am > experiencing what I consider to be slow transfer speeds. I am using iperf3 > to measure the speed and test the results of modifications. So far nothing > I have done has made a noticeable difference. If I run iperf3 -s on the > FreeBSD server, I see transfer speeds of approximately 1.6 Gb/s. If I run > iperf3 in client mode, the speed improves to ~2.75 Gb/s. However, if I > replace FreeBSD with CentOS 7 on the same hardware, I see iperf3 speeds > surpassing 8 GB/s. The other end of my iperf3 test is a Windows 10 box that > also has an Intel X540-T2 installed. > > I did notice that FreeBSD 10.3 (and 11.0 alpha 6 for that matter) includes > a slightly older Intel driver (v3.1.13-k). I managed to build a custom > kernel that removed the Intel PRO/10GbE PCIE NIC drivers. That allowed me > to manually load the latest 3.1.14 driver downloaded from Intel's web site. > Unfortunately that did not produce any improvements. I am working my way > through man tuning() and some other articles on network performance. So far > nothing I tweak makes a noticeable difference. I'm increasingly skeptical > that I am going to find a setting or two that more than doubles the speed I > am currently experiencing. > > I am open to any and all suggestions at this point. > > Thank you! > Chris > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to " freebsd-net-unsubscribe@freebsd.org " > > > > > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > From owner-freebsd-net@freebsd.org Thu Jul 21 21:33:43 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9B47FBA0F78 for ; Thu, 21 Jul 2016 21:33:43 +0000 (UTC) (envelope-from chris@dunbar.net) Received: from 005.lax.mailroute.net (005.lax.mailroute.net [199.89.1.8]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.mailroute.net", Issuer "AlphaSSL CA - SHA256 - G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6D72112D3 for ; Thu, 21 Jul 2016 21:33:43 +0000 (UTC) (envelope-from chris@dunbar.net) Received: from localhost (localhost.localdomain [127.0.0.1]) by 005.lax.mailroute.net (Postfix) with ESMTP id 3rwRlB36YLz1V45K for ; Thu, 21 Jul 2016 21:30:14 +0000 (UTC) X-Virus-Scanned: by MailRoute Received: from 005.lax.mailroute.net ([199.89.1.8]) by localhost (005.lax.mailroute.net [127.0.0.1]) (mroute_mailscanner, port 10026) with LMTP id lCNW2CFTlse4 for ; Thu, 21 Jul 2016 21:30:14 +0000 (UTC) Received: from zimbra.earthside.net (unknown [50.251.189.243]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by 005.lax.mailroute.net (Postfix) with ESMTPS id 3rwRlB0HgWz1V3vt for ; Thu, 21 Jul 2016 21:30:14 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by zimbra.earthside.net (Postfix) with ESMTP id 19CBA50267B for ; Thu, 21 Jul 2016 17:30:13 -0400 (EDT) Received: from zimbra.earthside.net ([127.0.0.1]) by localhost (zimbra.earthside.net [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id Q_tuU0HMfus1 for ; Thu, 21 Jul 2016 17:30:12 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by zimbra.earthside.net (Postfix) with ESMTP id 2587250267D for ; Thu, 21 Jul 2016 17:30:12 -0400 (EDT) Received: from zimbra.earthside.net ([127.0.0.1]) by localhost (zimbra.earthside.net [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id gZtZTvUwTgbV for ; Thu, 21 Jul 2016 17:30:12 -0400 (EDT) Received: from zimbra.earthside.net (zimbra.earthside.net [10.11.12.148]) by zimbra.earthside.net (Postfix) with ESMTP id F33D850267B for ; Thu, 21 Jul 2016 17:30:11 -0400 (EDT) Date: Thu, 21 Jul 2016 17:30:11 -0400 (EDT) From: Chris Dunbar To: freebsd-net Message-ID: <183608784.713013.1469136611853.JavaMail.zimbra@dunbar.net> In-Reply-To: <1441424852.712842.1469134420198.JavaMail.zimbra@dunbar.net> References: <1244557023.708807.1469061382192.JavaMail.zimbra@dunbar.net> <1441424852.712842.1469134420198.JavaMail.zimbra@dunbar.net> Subject: Re: Slow performance with Intel X540-T2 10Gb NIC MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [10.11.12.148] X-Mailer: Zimbra 8.6.0_GA_1153 (ZimbraWebClient - GC51 (Mac)/8.6.0_GA_1153) Thread-Topic: Slow performance with Intel X540-T2 10Gb NIC Thread-Index: bkXeUbMihgtikNh5aEfKledchPk+cClMnpsk X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Jul 2016 21:33:43 -0000 Hello again, I have good news and bad news: The bad news first: I am an idiot and I have wasted some of your time for which I apologize. The good news: Testing now between two FreeBSD 10.3 systems, I am achieving blistering speeds with iperf3. I apparently fell into the trap of assuming the new thing (FreeBSD is new to me) was broken. Now I see that I was assuming Windows was working fine and focusing all my attention on FreeBSD. Looking back over everything I have done to troubleshoot this situation I must conclude that the performance issue was on the Windows side and not the FreeBSD side. I am less concerned about that because my ultimate goal is to install my three X540s into one FreeBSD server and two VMware ESXi hosts. I am now fairly confident performance will be great. Many thanks for your collective attention and the suggestions I received from Eric and others. Regards, Chris ----- Original Message ----- From: "chris" To: "freebsd-net" Sent: Thursday, July 21, 2016 4:53:40 PM Subject: Re: Slow performance with Intel X540-T2 10Gb NIC Eric, et al: I haven't tried netperf yet, but I do have some new information to share. I have two systems that I am using for testing: the new server and an older (not too old) desktop PC. I installed CentOS on the new server again because I know it can achieve >9 GB/s with the X540. I replaced Windows on the desktop PC with FreeBSD 10.3 (it also has an X540) and ran iperf3 again. I was able to achieve >9 GB/s so I know the problem isn't the X540 and I know the problem isn't anything with the default installation of FreeBSD 10.3. So, what in the world might be nutty in my BIOS settings (or elsewhere) that would cause the new server + FreeBSD 10.3 + X540 to equal slow performance? Regards, Chris From: "Eric Joyner" To: "chris" , "freebsd-net" Sent: Thursday, July 21, 2016 1:27:10 PM Subject: Re: Slow performance with Intel X540-T2 10Gb NIC (Replying-all this time) Did you try to set these settings that ESnet recommends? https://fasterdata.es.net/host-tuning/freebsd/ We don't use iperf3 here at Intel (we use netperf instead), so I'm not sure I can be much help diagnosing what's wrong. On Wed, Jul 20, 2016 at 5:39 PM Chris Dunbar < chris@dunbar.net > wrote: Hello, I am new to FreeBSD and recently built a file server out of new components running FreeBSD 10.3. I installed an Intel X540-T2 10 Gb NIC and am experiencing what I consider to be slow transfer speeds. I am using iperf3 to measure the speed and test the results of modifications. So far nothing I have done has made a noticeable difference. If I run iperf3 -s on the FreeBSD server, I see transfer speeds of approximately 1.6 Gb/s. If I run iperf3 in client mode, the speed improves to ~2.75 Gb/s. However, if I replace FreeBSD with CentOS 7 on the same hardware, I see iperf3 speeds surpassing 8 GB/s. The other end of my iperf3 test is a Windows 10 box that also has an Intel X540-T2 installed. I did notice that FreeBSD 10.3 (and 11.0 alpha 6 for that matter) includes a slightly older Intel driver (v3.1.13-k). I managed to build a custom kernel that removed the Intel PRO/10GbE PCIE NIC drivers. That allowed me to manually load the latest 3.1.14 driver downloaded from Intel's web site. Unfortunately that did not produce any improvements. I am working my way through man tuning() and some other articles on network performance. So far nothing I tweak makes a noticeable difference. I'm increasingly skeptical that I am going to find a setting or two that more than doubles the speed I am currently experiencing. I am open to any and all suggestions at this point. Thank you! Chris _______________________________________________ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to " freebsd-net-unsubscribe@freebsd.org " _______________________________________________ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" From owner-freebsd-net@freebsd.org Thu Jul 21 22:39:35 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C33C1BA1D41 for ; Thu, 21 Jul 2016 22:39:35 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-io0-x232.google.com (mail-io0-x232.google.com [IPv6:2607:f8b0:4001:c06::232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8D7A8157F for ; Thu, 21 Jul 2016 22:39:35 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: by mail-io0-x232.google.com with SMTP id q83so89644083iod.1 for ; Thu, 21 Jul 2016 15:39:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc; bh=ydDguXEC/sQsfxSeci/K7XnieiH5tlvocfdDCZdFYYo=; b=ooZq4sXxRZHsie1W0ADwtIIw1HFlepfYDPNvKUXNHXE+856+NA79WvU6CZ6gK6X0Xy KBNG7z0dg+gBhbY0YNfjVNTsRFasLrOYf8O5XWieqkPjC2w/bTzPKXItQUnygi2JtIMT lA4dO2dTv0RKlIXMPX1o7vcysOT60+Y3y2xopi72eIjnIEUZm5MRqEvGpC6zaACw38pl LSWMaTBe0kbLS6RWRy85GGRlr2MrX5zGcr5FeqRMldsh3TM3nMekqmV5KClA5qTOIfbJ Qvp8k5wTruJJdh+7azyYBfYRxSA/GKOVadpESBZ5IPUWVybtcUTLcfgrGCEXjiGdbArO TYKw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc; bh=ydDguXEC/sQsfxSeci/K7XnieiH5tlvocfdDCZdFYYo=; b=OfZFArCMsi1QLsG2rCmF9u2ulNoXQdp2BzdFj8SOOGXkoLHABsjlMmJktsRyvt2yha 9RzQIAiF0w3vRji68184fljCTptcZNTvBA3oLtM6bZqAaUO/Cr91NJE0eG0EF/QCm0KN u+0sUlM5u8fNZdo6WXs0BQ1ll1PUpqcJHl+RzrLaR0OS/4eGUo9T3oaAUCuCD/PhC3kd RwuodhNEDQ63skIsswqrtX8cuxHfVxRFsCVBxs7SwqU2d8M5twYIy0MJHRUlCgKqz7yX /iS/1zWYOyiPLMaagYjvJUi8eYrRWQQz+NZWoQhP3TKjZYCZr6XX3L1dPwb2Dq/AhrPG jKDA== X-Gm-Message-State: AEkooutCP0uSXPC3BkjsClqT3LpV20aG8IxQsvSobt9Q0fZfd8m9lPKYk9heso+pft30e/zHapuWpWChWtX7fg== X-Received: by 10.107.144.10 with SMTP id s10mr811684iod.165.1469140774760; Thu, 21 Jul 2016 15:39:34 -0700 (PDT) MIME-Version: 1.0 Sender: adrian.chadd@gmail.com Received: by 10.36.141.129 with HTTP; Thu, 21 Jul 2016 15:39:33 -0700 (PDT) In-Reply-To: <306af514-70ff-f3bf-5b4f-da7ac1ec6580@cs.duke.edu> References: <306af514-70ff-f3bf-5b4f-da7ac1ec6580@cs.duke.edu> From: Adrian Chadd Date: Thu, 21 Jul 2016 15:39:33 -0700 X-Google-Sender-Auth: rN2n3JNUJkPLDEnRqtQs1l7-PnA Message-ID: Subject: Re: proposal: splitting NIC RSS up from stack RSS To: Andrew Gallatin Cc: FreeBSD Net Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Jul 2016 22:39:35 -0000 hi, Cool! Yeah, the RSS bits thing can be removed, as it's just doing a bitmask instead of a % operator to do mapping. I think we can just go to % and if people need the extra speed from a power-of-two operation, they can reintroduce it. I'll add that to the list. There's a librss, I don't think I committed it to -HEAD. I'll go dig it out and throw it into freebsd-head soon. -adrian From owner-freebsd-net@freebsd.org Fri Jul 22 01:54:10 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8E4A2BA0B2D for ; Fri, 22 Jul 2016 01:54:10 +0000 (UTC) (envelope-from sepherosa@gmail.com) Received: from mail-vk0-x22e.google.com (mail-vk0-x22e.google.com [IPv6:2607:f8b0:400c:c05::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 51AB51171; Fri, 22 Jul 2016 01:54:10 +0000 (UTC) (envelope-from sepherosa@gmail.com) Received: by mail-vk0-x22e.google.com with SMTP id n129so93072428vke.3; Thu, 21 Jul 2016 18:54:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=GZAhGKgtlep3n5lAGiN/Cp6MCbRPS7NI5LzoVvreNO0=; b=PqI06/yzdHROwGAIsyMPxL5KJKBlsgV2r6rg72N1KwvkO61jodVl1B/9EoTxkoX7rE be4v6LIzE5zjnGbr0IV5WUtEH5Pf0fCSiflHi1rwBCiWudC2u+G4nFIcti7n2UYiUyh+ 4Ry9v+CNYRh6Wu38Z/vhozMMr0drssJMezJptdvrBhDtkD2UGW0fpRaV50qyZ744xZv7 tB4DvkMco3otJ1/EVhjbf+8zTzgtK03MvBK0lN/gWyAcMiLv2hVyTQFjsJrWWudpC2Ko I8FcaUN52u8BdaxFHXFGmeS3GQFR5WNRfmfiDHyj9qGJWtWIRAPqCNS2/QU1hQqWMNHj Zq6g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=GZAhGKgtlep3n5lAGiN/Cp6MCbRPS7NI5LzoVvreNO0=; b=Ogmds4trac38GvsxgEoyLs+ocCkPsC4iZCYHPryqDGFI5/HmnSOlsY5SwWwRsiQYM/ cfHOFM5j7l27Y190sq3R+hVYgTWrdCyJs/WrNfhljcwuKdHHaZGJY+hwgNMJ6RWhjPKx Qutf2BsX7+eqf5elJDM+dA/F/1bIqDiM92x9kfjLOpwoHJsgvoJB/yymPelcikY0gjwx 9aa/pKPefhFHGxs4HVtFDPdB6UsQlt0QKQzn9piFUz21jfAiosCd3wAeoleJy4KGsYs+ PrIgyw0MhORULaXcZlUodlHppFjRug5ft0GcNog5ULTD6NNSwlodLddM/cVAW81wCxJM mKBQ== X-Gm-Message-State: AEkoousIRphMTCCCvZpaQtpifOOqmVsjWsSaPrjUeten0eJWKgrAs7Mn7N0NEsNvY8dS7n/L5pxragdYa3/Uew== X-Received: by 10.159.40.167 with SMTP id d36mr608648uad.60.1469152449033; Thu, 21 Jul 2016 18:54:09 -0700 (PDT) MIME-Version: 1.0 Received: by 10.176.83.101 with HTTP; Thu, 21 Jul 2016 18:54:08 -0700 (PDT) In-Reply-To: References: <306af514-70ff-f3bf-5b4f-da7ac1ec6580@cs.duke.edu> From: Sepherosa Ziehau Date: Fri, 22 Jul 2016 09:54:08 +0800 Message-ID: Subject: Re: proposal: splitting NIC RSS up from stack RSS To: Adrian Chadd Cc: Andrew Gallatin , FreeBSD Net Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Jul 2016 01:54:10 -0000 On Fri, Jul 22, 2016 at 6:39 AM, Adrian Chadd wrote: > hi, > > Cool! Yeah, the RSS bits thing can be removed, as it's just doing a > bitmask instead of a % operator to do mapping. I think we can just go > to % and if people need the extra speed from a power-of-two operation, > they can reintroduce it. I thought about it a while ago (the most popular E5-2560v{1,2,3} only has 6 cores, but E5-2560v4 has 8 cores! :). Since the raw RSS hash value is '& 0x1f' (I believe most of the NICs use 128 entry indirect table as defined by MS RSS) to select an entry in the indirect table, simply '%' on the raw RSS hash value probably will not work properly; you will need (hash&0x1f)%mp_ncpus at least. And well, since the indirect table's size if 128, you still will get some uneven CPU workload for non-power-of-2 cpus. And if you take cpu affinity into consideration, the situation will be even more complex ... Thanks, sephe From owner-freebsd-net@freebsd.org Fri Jul 22 03:17:01 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EA944BA1BE8 for ; Fri, 22 Jul 2016 03:17:01 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id DA5381E71 for ; Fri, 22 Jul 2016 03:17:01 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u6M3H1P6027987 for ; Fri, 22 Jul 2016 03:17:01 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-net@FreeBSD.org Subject: [Bug 208409] [PATCH] igb and ALTQ Date: Fri, 22 Jul 2016 03:17:02 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.3-BETA2 X-Bugzilla-Keywords: IntelNetworking, patch X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: commit-hook@freebsd.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Jul 2016 03:17:02 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D208409 --- Comment #7 from commit-hook@freebsd.org --- A commit references this bug: Author: sbruno Date: Fri Jul 22 03:16:38 UTC 2016 New revision: 303174 URL: https://svnweb.freebsd.org/changeset/base/303174 Log: MFC r299182 If ALTQ is defined in the kern conf, switch to Legacy Mode. PR: 208409 Changes: stable/10/sys/dev/e1000/if_igb.h --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-net@freebsd.org Fri Jul 22 12:34:32 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B7958BA1C81 for ; Fri, 22 Jul 2016 12:34:32 +0000 (UTC) (envelope-from sodynet1@gmail.com) Received: from mail-lf0-x22b.google.com (mail-lf0-x22b.google.com [IPv6:2a00:1450:4010:c07::22b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2402816EE for ; Fri, 22 Jul 2016 12:34:32 +0000 (UTC) (envelope-from sodynet1@gmail.com) Received: by mail-lf0-x22b.google.com with SMTP id b199so84700238lfe.0 for ; Fri, 22 Jul 2016 05:34:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc; bh=x62ruNJw2Jtxrghu6NhWGEf/0ObSOCuiZZa9LFfITcI=; b=tiaU3N9KhIxCdwEI4BGrxeDvL5OzthIyk3Q5WRUhBqu4NZVoEkDXtQEvM8RBKVIITI 1ZJDKHAzSaj2WfBQDIaGGt9I9oAHhgOtCBj0nbebJYp0W8BRZj4NXl6FWZxKDoMm1/Qs ekjwg5fRUVz/GhI8nkbkFZQViXEOQIiLZrBFlVeDQ8VxX9T1Msz181dnEqCasTlscLrK spRGao/ntJ9Fmngds6Z7uE7mLMPFOHklyxWv26gCxrSfbUMDsJp4daUXtv9qa0J04e44 M9yT/i17pwMV4H4JPyWkVJ/AKRM9iZTaB/EFt/HAQPuMBT9yzONFqMSD7AZlt9Z7Ei+t PaVg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc; bh=x62ruNJw2Jtxrghu6NhWGEf/0ObSOCuiZZa9LFfITcI=; b=N4Pc71IlL/Sq36OPKliZgo6lTZdctPQU4jP9u0uplwwzkjh3kd9eogaCBJtvlIJySi /5WmL6oWnGBRvYeksXVnUdmmj3/Rj66xcpZV+iH20WSfH0/l5EGQZVE9dVI/DHnWbVm9 u76K0q+Z7gY55nAdEK7hh5JrMPjN69FXgwrN512rsVdJTTRShYh7KPLJfNQY2+gyPIjZ NPByLCfWvaAmeyoKoRMTgc1JdeerPygWu2v8A/v8itNEIhl3jwzeMB3gnks2V491DK2R sUH6u42Ln8ip9h/b4FypqvXEHeA6U1jPFsTh1A43QaZdMnR0wlco7GKCy0tekEsAVb+w Q73w== X-Gm-Message-State: AEkoouv2fkSYRGw7YOY5izkY1OUJa/d7IaZQlVG0Z+5v3DvybClV6Mf8uVvmtTOA7JLsvlhRF8CA3m7tSSyJPg== MIME-Version: 1.0 X-Received: by 10.46.1.26 with SMTP id 26mr2540436ljb.26.1469190870170; Fri, 22 Jul 2016 05:34:30 -0700 (PDT) Received: by 10.25.196.17 with HTTP; Fri, 22 Jul 2016 05:34:30 -0700 (PDT) Received: by 10.25.196.17 with HTTP; Fri, 22 Jul 2016 05:34:30 -0700 (PDT) In-Reply-To: <183608784.713013.1469136611853.JavaMail.zimbra@dunbar.net> References: <1244557023.708807.1469061382192.JavaMail.zimbra@dunbar.net> <1441424852.712842.1469134420198.JavaMail.zimbra@dunbar.net> <183608784.713013.1469136611853.JavaMail.zimbra@dunbar.net> Date: Fri, 22 Jul 2016 15:34:30 +0300 Message-ID: Subject: Re: Slow performance with Intel X540-T2 10Gb NIC From: Sami Halabi To: Chris Dunbar Cc: freebsd-net Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Jul 2016 12:34:32 -0000 hi, would you share what was wrong in the windows side and how you solved it? Sami =D7=91=D7=AA=D7=90=D7=A8=D7=99=D7=9A 22 =D7=91=D7=99=D7=95=D7=9C=D7=99 2016= 12:33 AM,=E2=80=8F "Chris Dunbar" =D7=9B=D7=AA=D7=91: > Hello again, > > I have good news and bad news: > > The bad news first: I am an idiot and I have wasted some of your time for > which I apologize. > > The good news: Testing now between two FreeBSD 10.3 systems, I am > achieving blistering speeds with iperf3. I apparently fell into the trap = of > assuming the new thing (FreeBSD is new to me) was broken. Now I see that = I > was assuming Windows was working fine and focusing all my attention on > FreeBSD. Looking back over everything I have done to troubleshoot this > situation I must conclude that the performance issue was on the Windows > side and not the FreeBSD side. I am less concerned about that because my > ultimate goal is to install my three X540s into one FreeBSD server and tw= o > VMware ESXi hosts. I am now fairly confident performance will be great. > > Many thanks for your collective attention and the suggestions I received > from Eric and others. > > Regards, > Chris > > ----- Original Message ----- > From: "chris" > To: "freebsd-net" > Sent: Thursday, July 21, 2016 4:53:40 PM > Subject: Re: Slow performance with Intel X540-T2 10Gb NIC > > Eric, et al: > > I haven't tried netperf yet, but I do have some new information to share. > I have two systems that I am using for testing: the new server and an old= er > (not too old) desktop PC. I installed CentOS on the new server again > because I know it can achieve >9 GB/s with the X540. I replaced Windows o= n > the desktop PC with FreeBSD 10.3 (it also has an X540) and ran iperf3 > again. I was able to achieve >9 GB/s so I know the problem isn't the X540 > and I know the problem isn't anything with the default installation of > FreeBSD 10.3. So, what in the world might be nutty in my BIOS settings (o= r > elsewhere) that would cause the new server + FreeBSD 10.3 + X540 to equal > slow performance? > > Regards, > Chris > > > From: "Eric Joyner" > To: "chris" , "freebsd-net" > Sent: Thursday, July 21, 2016 1:27:10 PM > Subject: Re: Slow performance with Intel X540-T2 10Gb NIC > > (Replying-all this time) > > Did you try to set these settings that ESnet recommends? > https://fasterdata.es.net/host-tuning/freebsd/ > > We don't use iperf3 here at Intel (we use netperf instead), so I'm not > sure I can be much help diagnosing what's wrong. > > On Wed, Jul 20, 2016 at 5:39 PM Chris Dunbar < chris@dunbar.net > wrote: > > > Hello, > > I am new to FreeBSD and recently built a file server out of new component= s > running FreeBSD 10.3. I installed an Intel X540-T2 10 Gb NIC and am > experiencing what I consider to be slow transfer speeds. I am using iperf= 3 > to measure the speed and test the results of modifications. So far nothin= g > I have done has made a noticeable difference. If I run iperf3 -s on the > FreeBSD server, I see transfer speeds of approximately 1.6 Gb/s. If I run > iperf3 in client mode, the speed improves to ~2.75 Gb/s. However, if I > replace FreeBSD with CentOS 7 on the same hardware, I see iperf3 speeds > surpassing 8 GB/s. The other end of my iperf3 test is a Windows 10 box th= at > also has an Intel X540-T2 installed. > > I did notice that FreeBSD 10.3 (and 11.0 alpha 6 for that matter) include= s > a slightly older Intel driver (v3.1.13-k). I managed to build a custom > kernel that removed the Intel PRO/10GbE PCIE NIC drivers. That allowed me > to manually load the latest 3.1.14 driver downloaded from Intel's web sit= e. > Unfortunately that did not produce any improvements. I am working my way > through man tuning() and some other articles on network performance. So f= ar > nothing I tweak makes a noticeable difference. I'm increasingly skeptical > that I am going to find a setting or two that more than doubles the speed= I > am currently experiencing. > > I am open to any and all suggestions at this point. > > Thank you! > Chris > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to " freebsd-net-unsubscribe@freebsd.org " > > > > > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > From owner-freebsd-net@freebsd.org Fri Jul 22 13:52:23 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 45A8ABA0792 for ; Fri, 22 Jul 2016 13:52:23 +0000 (UTC) (envelope-from chris@dunbar.net) Received: from 004.lax.mailroute.net (004.lax.mailroute.net [199.89.1.7]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.mailroute.net", Issuer "AlphaSSL CA - SHA256 - G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1539A162D for ; Fri, 22 Jul 2016 13:52:22 +0000 (UTC) (envelope-from chris@dunbar.net) Received: from localhost (localhost.localdomain [127.0.0.1]) by 004.lax.mailroute.net (Postfix) with ESMTP id 3rwsSF50tbz15KC6 for ; Fri, 22 Jul 2016 13:48:45 +0000 (UTC) X-Virus-Scanned: by MailRoute Received: from 004.lax.mailroute.net ([199.89.1.7]) by localhost (004.lax.mailroute.net [127.0.0.1]) (mroute_mailscanner, port 10026) with LMTP id HLCn-0y-bFdM for ; Fri, 22 Jul 2016 13:48:44 +0000 (UTC) Received: from zimbra.earthside.net (unknown [50.251.189.243]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by 004.lax.mailroute.net (Postfix) with ESMTPS id 3rwsSC6nr7z15KBq for ; Fri, 22 Jul 2016 13:48:43 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by zimbra.earthside.net (Postfix) with ESMTP id B1A2F50267B for ; Fri, 22 Jul 2016 09:48:42 -0400 (EDT) Received: from zimbra.earthside.net ([127.0.0.1]) by localhost (zimbra.earthside.net [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id QENgP6oNx-ss for ; Fri, 22 Jul 2016 09:48:40 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by zimbra.earthside.net (Postfix) with ESMTP id CDADE50267C for ; Fri, 22 Jul 2016 09:48:40 -0400 (EDT) Received: from zimbra.earthside.net ([127.0.0.1]) by localhost (zimbra.earthside.net [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id sxnSN9G02ob7 for ; Fri, 22 Jul 2016 09:48:40 -0400 (EDT) Received: from zimbra.earthside.net (zimbra.earthside.net [10.11.12.148]) by zimbra.earthside.net (Postfix) with ESMTP id 598CC50267B for ; Fri, 22 Jul 2016 09:48:40 -0400 (EDT) Date: Fri, 22 Jul 2016 09:48:40 -0400 (EDT) From: Chris Dunbar To: freebsd-net Message-ID: <144391790.714645.1469195320062.JavaMail.zimbra@dunbar.net> In-Reply-To: References: <1244557023.708807.1469061382192.JavaMail.zimbra@dunbar.net> <1441424852.712842.1469134420198.JavaMail.zimbra@dunbar.net> <183608784.713013.1469136611853.JavaMail.zimbra@dunbar.net> Subject: Re: Slow performance with Intel X540-T2 10Gb NIC MIME-Version: 1.0 X-Originating-IP: [10.11.12.148] X-Mailer: Zimbra 8.6.0_GA_1153 (ZimbraWebClient - GC51 (Mac)/8.6.0_GA_1153) Thread-Topic: Slow performance with Intel X540-T2 10Gb NIC Thread-Index: AEXdseC/lXu8ZlsAGuMNm0T6Ia8d1w== Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Jul 2016 13:52:23 -0000 Hi Sami,=20 I haven't actually fixed anything yet. I have only demonstrated that the po= or performance does not appear to happen between two FreeBSD boxes and poss= ibly between a Linux and FreeBSD, but I am going to confirm that now. I hav= e also seen good performance between the Windows box and Linux so that does= n't quite add up either. I may have to break out Wireshark and make some pa= cket captures to see if I can tell what's going on. If I find anything, I w= ill be sure to share it.=20 Regards,=20 Chris=20 From: "Sami Halabi" =20 To: "chris" =20 Cc: "freebsd-net" =20 Sent: Friday, July 22, 2016 8:34:30 AM=20 Subject: Re: Slow performance with Intel X540-T2 10Gb NIC=20 hi,=20 would you share what was wrong in the windows side and how you solved it?= =20 Sami=20 =D7=91=D7=AA=D7=90=D7=A8=D7=99=D7=9A 22 =D7=91=D7=99=D7=95=D7=9C=D7=99 2016= 12:33 AM,=E2=80=8F "Chris Dunbar" < chris@dunbar.net > =D7=9B=D7=AA=D7=91:= =20 Hello again,=20 I have good news and bad news:=20 The bad news first: I am an idiot and I have wasted some of your time for w= hich I apologize.=20 The good news: Testing now between two FreeBSD 10.3 systems, I am achieving= blistering speeds with iperf3. I apparently fell into the trap of assuming= the new thing (FreeBSD is new to me) was broken. Now I see that I was assu= ming Windows was working fine and focusing all my attention on FreeBSD. Loo= king back over everything I have done to troubleshoot this situation I must= conclude that the performance issue was on the Windows side and not the Fr= eeBSD side. I am less concerned about that because my ultimate goal is to i= nstall my three X540s into one FreeBSD server and two VMware ESXi hosts. I = am now fairly confident performance will be great.=20 Many thanks for your collective attention and the suggestions I received fr= om Eric and others.=20 Regards,=20 Chris=20 ----- Original Message -----=20 From: "chris" < chris@dunbar.net >=20 To: "freebsd-net" < freebsd-net@freebsd.org >=20 Sent: Thursday, July 21, 2016 4:53:40 PM=20 Subject: Re: Slow performance with Intel X540-T2 10Gb NIC=20 Eric, et al:=20 I haven't tried netperf yet, but I do have some new information to share. I= have two systems that I am using for testing: the new server and an older = (not too old) desktop PC. I installed CentOS on the new server again becaus= e I know it can achieve >9 GB/s with the X540. I replaced Windows on the de= sktop PC with FreeBSD 10.3 (it also has an X540) and ran iperf3 again. I wa= s able to achieve >9 GB/s so I know the problem isn't the X540 and I know t= he problem isn't anything with the default installation of FreeBSD 10.3. So= , what in the world might be nutty in my BIOS settings (or elsewhere) that = would cause the new server + FreeBSD 10.3 + X540 to equal slow performance?= =20 Regards,=20 Chris=20 From: "Eric Joyner" < erj@freebsd.org >=20 To: "chris" < chris@dunbar.net >, "freebsd-net" < freebsd-net@freebsd.org >= =20 Sent: Thursday, July 21, 2016 1:27:10 PM=20 Subject: Re: Slow performance with Intel X540-T2 10Gb NIC=20 (Replying-all this time)=20 Did you try to set these settings that ESnet recommends? https://fasterdata= .es.net/host-tuning/freebsd/=20 We don't use iperf3 here at Intel (we use netperf instead), so I'm not sure= I can be much help diagnosing what's wrong.=20 On Wed, Jul 20, 2016 at 5:39 PM Chris Dunbar < chris@dunbar.net > wrote:=20 Hello,=20 I am new to FreeBSD and recently built a file server out of new components = running FreeBSD 10.3. I installed an Intel X540-T2 10 Gb NIC and am experie= ncing what I consider to be slow transfer speeds. I am using iperf3 to meas= ure the speed and test the results of modifications. So far nothing I have = done has made a noticeable difference. If I run iperf3 -s on the FreeBSD se= rver, I see transfer speeds of approximately 1.6 Gb/s. If I run iperf3 in c= lient mode, the speed improves to ~2.75 Gb/s. However, if I replace FreeBSD= with CentOS 7 on the same hardware, I see iperf3 speeds surpassing 8 GB/s.= The other end of my iperf3 test is a Windows 10 box that also has an Intel= X540-T2 installed.=20 I did notice that FreeBSD 10.3 (and 11.0 alpha 6 for that matter) includes = a slightly older Intel driver (v3.1.13-k). I managed to build a custom kern= el that removed the Intel PRO/10GbE PCIE NIC drivers. That allowed me to ma= nually load the latest 3.1.14 driver downloaded from Intel's web site. Unfo= rtunately that did not produce any improvements. I am working my way throug= h man tuning() and some other articles on network performance. So far nothi= ng I tweak makes a noticeable difference. I'm increasingly skeptical that I= am going to find a setting or two that more than doubles the speed I am cu= rrently experiencing.=20 I am open to any and all suggestions at this point.=20 Thank you!=20 Chris=20 _______________________________________________=20 freebsd-net@freebsd.org mailing list=20 https://lists.freebsd.org/mailman/listinfo/freebsd-net=20 To unsubscribe, send any mail to " freebsd-net-unsubscribe@freebsd.org "=20 _______________________________________________=20 freebsd-net@freebsd.org mailing list=20 https://lists.freebsd.org/mailman/listinfo/freebsd-net=20 To unsubscribe, send any mail to " freebsd-net-unsubscribe@freebsd.org "=20 _______________________________________________=20 freebsd-net@freebsd.org mailing list=20 https://lists.freebsd.org/mailman/listinfo/freebsd-net=20 To unsubscribe, send any mail to " freebsd-net-unsubscribe@freebsd.org "=20 From owner-freebsd-net@freebsd.org Fri Jul 22 18:23:27 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 972DFBA1D08 for ; Fri, 22 Jul 2016 18:23:27 +0000 (UTC) (envelope-from kob6558@gmail.com) Received: from mail-it0-x230.google.com (mail-it0-x230.google.com [IPv6:2607:f8b0:4001:c0b::230]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4ECDA130D for ; Fri, 22 Jul 2016 18:23:27 +0000 (UTC) (envelope-from kob6558@gmail.com) Received: by mail-it0-x230.google.com with SMTP id f6so42918589ith.1 for ; Fri, 22 Jul 2016 11:23:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc; bh=O2V1+NCw7m9pB2rJkiUAyK2MVZnVqtGjivxdvoBoBMk=; b=uHAV3TMnpb4+gCiXAn3SBKjztnXprArxGJsGueIPhZhzbTJoz6TRLXyJOLrEtTf2eX qnWIu4vTTZYKZZzeMIgJ1sIodW4ZWEBOr8oyDbsGCM98PEOqvfTCfKrN7wgZx2zE0DbE o2YBZhGwBfss5Shs6hhXNQQSqLZfIDjeQAXXm1is5bER5oWrgw28J6InuAcNqM7r80+c bdi3CEeh+OgV3oc9EA4pRbLQRu5nifGJ+VsdLziwzKBU2KN8ABmALVY2fVls8QOOnnmW S5MoOrozPujGEwjDa9bmtck+bKb3T6ts6YNnQlBlyBT3aGQcVOLe+ZynslGcZ3TCteK6 vTWA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc; bh=O2V1+NCw7m9pB2rJkiUAyK2MVZnVqtGjivxdvoBoBMk=; b=h2OYzdzxcr0/P2aMU3nUA6DJmjsgCbvKR6XDPbPDiuOKAOcYcPBAy3M1unBmEKEgsU hpjlB7noEzxdD62emKf+sVB3YcvY6ZCRcgB5WdPzmpr2lRCmljFK5o30xptgniXCRq17 h+x5qIk4LOhXw+Hr6q9g3P7NIt+bqJJ+Fb0Zk5oCGy+zCmw3MfAonEovnYjH0tkM/G5Z c1x/SYWbb414h/dVdY33gQR+0uOO+imvU+Fej9Kc81TwFn8U7KD52zpoh9/hrOJDX2z9 2Oivi3W/uhZnUyT9cOZ3rrDpRsqPqd5mDOgn7VbUiHU6hdfOaqDowERcvEM8hth9CmN8 cCTQ== X-Gm-Message-State: AEkoousB+JORXwjWIGKa2J77rIW8GjbXUwkiwACCLsfv1zBtZ8xd8D3TPLdUIy5ArqTRhfY86RyURgte3uQ8YQ== X-Received: by 10.36.120.150 with SMTP id p144mr5819156itc.44.1469211806676; Fri, 22 Jul 2016 11:23:26 -0700 (PDT) MIME-Version: 1.0 Sender: kob6558@gmail.com Received: by 10.79.78.213 with HTTP; Fri, 22 Jul 2016 11:23:26 -0700 (PDT) In-Reply-To: <144391790.714645.1469195320062.JavaMail.zimbra@dunbar.net> References: <1244557023.708807.1469061382192.JavaMail.zimbra@dunbar.net> <1441424852.712842.1469134420198.JavaMail.zimbra@dunbar.net> <183608784.713013.1469136611853.JavaMail.zimbra@dunbar.net> <144391790.714645.1469195320062.JavaMail.zimbra@dunbar.net> From: Kevin Oberman Date: Fri, 22 Jul 2016 11:23:26 -0700 X-Google-Sender-Auth: 3l9u0UIuh2eLyCqhrfuqh0YNgos Message-ID: Subject: Re: Slow performance with Intel X540-T2 10Gb NIC To: Chris Dunbar Cc: freebsd-net Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Jul 2016 18:23:27 -0000 On Fri, Jul 22, 2016 at 6:48 AM, Chris Dunbar wrote: > Hi Sami, > > I haven't actually fixed anything yet. I have only demonstrated that the > poor performance does not appear to happen between two FreeBSD boxes and > possibly between a Linux and FreeBSD, but I am going to confirm that now.= I > have also seen good performance between the Windows box and Linux so that > doesn't quite add up either. I may have to break out Wireshark and make > some packet captures to see if I can tell what's going on. If I find > anything, I will be sure to share it. > > Regards, > Chris > > > From: "Sami Halabi" > To: "chris" > Cc: "freebsd-net" > Sent: Friday, July 22, 2016 8:34:30 AM > Subject: Re: Slow performance with Intel X540-T2 10Gb NIC > > > > hi, > would you share what was wrong in the windows side and how you solved it? > > Sami > > =D7=91=D7=AA=D7=90=D7=A8=D7=99=D7=9A 22 =D7=91=D7=99=D7=95=D7=9C=D7=99 20= 16 12:33 AM,=E2=80=8F "Chris Dunbar" < chris@dunbar.net > =D7=9B=D7=AA=D7= =91: > > > Hello again, > > I have good news and bad news: > > The bad news first: I am an idiot and I have wasted some of your time for > which I apologize. > > The good news: Testing now between two FreeBSD 10.3 systems, I am > achieving blistering speeds with iperf3. I apparently fell into the trap = of > assuming the new thing (FreeBSD is new to me) was broken. Now I see that = I > was assuming Windows was working fine and focusing all my attention on > FreeBSD. Looking back over everything I have done to troubleshoot this > situation I must conclude that the performance issue was on the Windows > side and not the FreeBSD side. I am less concerned about that because my > ultimate goal is to install my three X540s into one FreeBSD server and tw= o > VMware ESXi hosts. I am now fairly confident performance will be great. > > Many thanks for your collective attention and the suggestions I received > from Eric and others. > > Regards, > Chris > > ----- Original Message ----- > From: "chris" < chris@dunbar.net > > To: "freebsd-net" < freebsd-net@freebsd.org > > Sent: Thursday, July 21, 2016 4:53:40 PM > Subject: Re: Slow performance with Intel X540-T2 10Gb NIC > > Eric, et al: > > I haven't tried netperf yet, but I do have some new information to share. > I have two systems that I am using for testing: the new server and an old= er > (not too old) desktop PC. I installed CentOS on the new server again > because I know it can achieve >9 GB/s with the X540. I replaced Windows o= n > the desktop PC with FreeBSD 10.3 (it also has an X540) and ran iperf3 > again. I was able to achieve >9 GB/s so I know the problem isn't the X540 > and I know the problem isn't anything with the default installation of > FreeBSD 10.3. So, what in the world might be nutty in my BIOS settings (o= r > elsewhere) that would cause the new server + FreeBSD 10.3 + X540 to equal > slow performance? > > Regards, > Chris > > > From: "Eric Joyner" < erj@freebsd.org > > To: "chris" < chris@dunbar.net >, "freebsd-net" < freebsd-net@freebsd.org > > > Sent: Thursday, July 21, 2016 1:27:10 PM > Subject: Re: Slow performance with Intel X540-T2 10Gb NIC > > (Replying-all this time) > > Did you try to set these settings that ESnet recommends? > https://fasterdata.es.net/host-tuning/freebsd/ > > We don't use iperf3 here at Intel (we use netperf instead), so I'm not > sure I can be much help diagnosing what's wrong. > > On Wed, Jul 20, 2016 at 5:39 PM Chris Dunbar < chris@dunbar.net > wrote: > > > Hello, > > I am new to FreeBSD and recently built a file server out of new component= s > running FreeBSD 10.3. I installed an Intel X540-T2 10 Gb NIC and am > experiencing what I consider to be slow transfer speeds. I am using iperf= 3 > to measure the speed and test the results of modifications. So far nothin= g > I have done has made a noticeable difference. If I run iperf3 -s on the > FreeBSD server, I see transfer speeds of approximately 1.6 Gb/s. If I run > iperf3 in client mode, the speed improves to ~2.75 Gb/s. However, if I > replace FreeBSD with CentOS 7 on the same hardware, I see iperf3 speeds > surpassing 8 GB/s. The other end of my iperf3 test is a Windows 10 box th= at > also has an Intel X540-T2 installed. > > I did notice that FreeBSD 10.3 (and 11.0 alpha 6 for that matter) include= s > a slightly older Intel driver (v3.1.13-k). I managed to build a custom > kernel that removed the Intel PRO/10GbE PCIE NIC drivers. That allowed me > to manually load the latest 3.1.14 driver downloaded from Intel's web sit= e. > Unfortunately that did not produce any improvements. I am working my way > through man tuning() and some other articles on network performance. So f= ar > nothing I tweak makes a noticeable difference. I'm increasingly skeptical > that I am going to find a setting or two that more than doubles the speed= I > am currently experiencing. > > I am open to any and all suggestions at this point. > > Thank you! > Chris > This sort of problem can be very tricky to diagnose. I'd like to suggest that one of the tool you use should be SIFTR. It does kernel level collection of network statistics and is a loadable module. By default it i IPv4 only. It will have to be re-built with "CFLAGS+=3D-DSIFTR_IPV6" uncommented in /sys/modules/siftr/Makefile for IPv6 support. It starts, stops, amd manages collection under the control of 4 sysctls. I have found it invaluable for analysis of netork performance issues, but seems to not be widely known. -- Kevin Oberman, Part time kid herder and retired Network Engineer E-mail: rkoberman@gmail.com PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683 From owner-freebsd-net@freebsd.org Fri Jul 22 18:36:14 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D4B4EBA106A for ; Fri, 22 Jul 2016 18:36:14 +0000 (UTC) (envelope-from chris@dunbar.net) Received: from 001.las.mailroute.net (001.las.mailroute.net [199.89.4.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.mailroute.net", Issuer "AlphaSSL CA - SHA256 - G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id AE33B189A for ; Fri, 22 Jul 2016 18:36:14 +0000 (UTC) (envelope-from chris@dunbar.net) Received: from localhost (localhost.localdomain [127.0.0.1]) by 001.las.mailroute.net (Postfix) with ESMTP id 3rwzqr4yzxz8swx for ; Fri, 22 Jul 2016 18:36:08 +0000 (UTC) X-Virus-Scanned: by MailRoute Received: from 001.las.mailroute.net ([199.89.4.4]) by localhost (001.las.mailroute.net. [127.0.0.1]) (mroute_mailscanner, port 10026) with LMTP id 9aTPEVnPN_VJ for ; Fri, 22 Jul 2016 18:36:05 +0000 (UTC) Received: from zimbra.earthside.net (unknown [50.251.189.243]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by 001.las.mailroute.net (Postfix) with ESMTPS id 3rwzqn2XNxz8swv for ; Fri, 22 Jul 2016 18:36:05 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by zimbra.earthside.net (Postfix) with ESMTP id 2E4BD50267B for ; Fri, 22 Jul 2016 14:36:04 -0400 (EDT) Received: from zimbra.earthside.net ([127.0.0.1]) by localhost (zimbra.earthside.net [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id mCzKBVlQN6TP for ; Fri, 22 Jul 2016 14:36:02 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by zimbra.earthside.net (Postfix) with ESMTP id 76AC150267C for ; Fri, 22 Jul 2016 14:36:02 -0400 (EDT) Received: from zimbra.earthside.net ([127.0.0.1]) by localhost (zimbra.earthside.net [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id Ijy7VpTXjLup for ; Fri, 22 Jul 2016 14:36:02 -0400 (EDT) Received: from zimbra.earthside.net (zimbra.earthside.net [10.11.12.148]) by zimbra.earthside.net (Postfix) with ESMTP id 36B3B50267B for ; Fri, 22 Jul 2016 14:36:02 -0400 (EDT) Date: Fri, 22 Jul 2016 14:36:01 -0400 (EDT) From: Chris Dunbar To: freebsd-net Message-ID: <1366880114.715041.1469212561939.JavaMail.zimbra@dunbar.net> In-Reply-To: References: <1244557023.708807.1469061382192.JavaMail.zimbra@dunbar.net> <1441424852.712842.1469134420198.JavaMail.zimbra@dunbar.net> <183608784.713013.1469136611853.JavaMail.zimbra@dunbar.net> <144391790.714645.1469195320062.JavaMail.zimbra@dunbar.net> Subject: Re: Slow performance with Intel X540-T2 10Gb NIC MIME-Version: 1.0 X-Originating-IP: [10.11.12.148] X-Mailer: Zimbra 8.6.0_GA_1153 (ZimbraWebClient - GC51 (Mac)/8.6.0_GA_1153) Thread-Topic: Slow performance with Intel X540-T2 10Gb NIC Thread-Index: 3f3VpvMUTp9/e8FXS7bEPsFIQ3i+OA== Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Jul 2016 18:36:14 -0000 Thank you - I will check that out.=20 From: "Kevin Oberman" =20 To: "chris" =20 Cc: "freebsd-net" =20 Sent: Friday, July 22, 2016 2:23:26 PM=20 Subject: Re: Slow performance with Intel X540-T2 10Gb NIC=20 On Fri, Jul 22, 2016 at 6:48 AM, Chris Dunbar < chris@dunbar.net > wrote:= =20 Hi Sami,=20 I haven't actually fixed anything yet. I have only demonstrated that the po= or performance does not appear to happen between two FreeBSD boxes and poss= ibly between a Linux and FreeBSD, but I am going to confirm that now. I hav= e also seen good performance between the Windows box and Linux so that does= n't quite add up either. I may have to break out Wireshark and make some pa= cket captures to see if I can tell what's going on. If I find anything, I w= ill be sure to share it.=20 Regards,=20 Chris=20 From: "Sami Halabi" < sodynet1@gmail.com >=20 To: "chris" < chris@dunbar.net >=20 Cc: "freebsd-net" < freebsd-net@freebsd.org >=20 Sent: Friday, July 22, 2016 8:34:30 AM=20 Subject: Re: Slow performance with Intel X540-T2 10Gb NIC=20 hi,=20 would you share what was wrong in the windows side and how you solved it?= =20 Sami=20 =D7=91=D7=AA=D7=90=D7=A8=D7=99=D7=9A 22 =D7=91=D7=99=D7=95=D7=9C=D7=99 2016= 12:33 AM,=E2=80=8F "Chris Dunbar" < chris@dunbar.net > =D7=9B=D7=AA=D7=91:= =20 Hello again,=20 I have good news and bad news:=20 The bad news first: I am an idiot and I have wasted some of your time for w= hich I apologize.=20 The good news: Testing now between two FreeBSD 10.3 systems, I am achieving= blistering speeds with iperf3. I apparently fell into the trap of assuming= the new thing (FreeBSD is new to me) was broken. Now I see that I was assu= ming Windows was working fine and focusing all my attention on FreeBSD. Loo= king back over everything I have done to troubleshoot this situation I must= conclude that the performance issue was on the Windows side and not the Fr= eeBSD side. I am less concerned about that because my ultimate goal is to i= nstall my three X540s into one FreeBSD server and two VMware ESXi hosts. I = am now fairly confident performance will be great.=20 Many thanks for your collective attention and the suggestions I received fr= om Eric and others.=20 Regards,=20 Chris=20 ----- Original Message -----=20 From: "chris" < chris@dunbar.net >=20 To: "freebsd-net" < freebsd-net@freebsd.org >=20 Sent: Thursday, July 21, 2016 4:53:40 PM=20 Subject: Re: Slow performance with Intel X540-T2 10Gb NIC=20 Eric, et al:=20 I haven't tried netperf yet, but I do have some new information to share. I= have two systems that I am using for testing: the new server and an older = (not too old) desktop PC. I installed CentOS on the new server again becaus= e I know it can achieve >9 GB/s with the X540. I replaced Windows on the de= sktop PC with FreeBSD 10.3 (it also has an X540) and ran iperf3 again. I wa= s able to achieve >9 GB/s so I know the problem isn't the X540 and I know t= he problem isn't anything with the default installation of FreeBSD 10.3. So= , what in the world might be nutty in my BIOS settings (or elsewhere) that = would cause the new server + FreeBSD 10.3 + X540 to equal slow performance?= =20 Regards,=20 Chris=20 From: "Eric Joyner" < erj@freebsd.org >=20 To: "chris" < chris@dunbar.net >, "freebsd-net" < freebsd-net@freebsd.org >= =20 Sent: Thursday, July 21, 2016 1:27:10 PM=20 Subject: Re: Slow performance with Intel X540-T2 10Gb NIC=20 (Replying-all this time)=20 Did you try to set these settings that ESnet recommends? https://fasterdata= .es.net/host-tuning/freebsd/=20 We don't use iperf3 here at Intel (we use netperf instead), so I'm not sure= I can be much help diagnosing what's wrong.=20 On Wed, Jul 20, 2016 at 5:39 PM Chris Dunbar < chris@dunbar.net > wrote:=20 Hello,=20 I am new to FreeBSD and recently built a file server out of new components = running FreeBSD 10.3. I installed an Intel X540-T2 10 Gb NIC and am experie= ncing what I consider to be slow transfer speeds. I am using iperf3 to meas= ure the speed and test the results of modifications. So far nothing I have = done has made a noticeable difference. If I run iperf3 -s on the FreeBSD se= rver, I see transfer speeds of approximately 1.6 Gb/s. If I run iperf3 in c= lient mode, the speed improves to ~2.75 Gb/s. However, if I replace FreeBSD= with CentOS 7 on the same hardware, I see iperf3 speeds surpassing 8 GB/s.= The other end of my iperf3 test is a Windows 10 box that also has an Intel= X540-T2 installed.=20 I did notice that FreeBSD 10.3 (and 11.0 alpha 6 for that matter) includes = a slightly older Intel driver (v3.1.13-k). I managed to build a custom kern= el that removed the Intel PRO/10GbE PCIE NIC drivers. That allowed me to ma= nually load the latest 3.1.14 driver downloaded from Intel's web site. Unfo= rtunately that did not produce any improvements. I am working my way throug= h man tuning() and some other articles on network performance. So far nothi= ng I tweak makes a noticeable difference. I'm increasingly skeptical that I= am going to find a setting or two that more than doubles the speed I am cu= rrently experiencing.=20 I am open to any and all suggestions at this point.=20 Thank you!=20 Chris=20 This sort of problem can be very tricky to diagnose. I'd like to suggest th= at one of the tool you use should be SIFTR. It does kernel level collection= of network statistics and is a loadable module. By default it i IPv4 only.= It will have to be re-built with "CFLAGS+=3D-DSIFTR_IPV6" uncommented in /= sys/modules/siftr/Makefile for IPv6 support. It starts, stops, amd manages = collection under the control of 4 sysctls.=20 I have found it invaluable for analysis of netork performance issues, but s= eems to not be widely known.=20 --=20 Kevin Oberman, Part time kid herder and retired Network Engineer=20 E-mail: rkoberman@gmail.com=20 PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683=20 From owner-freebsd-net@freebsd.org Fri Jul 22 19:23:37 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 80F44BA1C9B for ; Fri, 22 Jul 2016 19:23:37 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-io0-x22f.google.com (mail-io0-x22f.google.com [IPv6:2607:f8b0:4001:c06::22f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4ECBD1E4A for ; Fri, 22 Jul 2016 19:23:37 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: by mail-io0-x22f.google.com with SMTP id b62so113617831iod.3 for ; Fri, 22 Jul 2016 12:23:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc; bh=nL76gfArXyNS5cIl4Is1mwrmdVjBY+lEVdIlEJuopwg=; b=xMxAcJAW22pnMURVXHJo7OzThYn1OWVuEvWYjKkuhBMAiPW+QZWyiOXdVTjB1l8jFo lwP0wM+35LRSWjBnclKBo31fFzWVs7A64U4gL5WS2DXR7BZPap1x5WHXsIRZDpxmykuz j8MllznwA4TkN8rApoSREAvMwkiP3yh3G5HEZm7HkyctrIdx6FSgED0VoN9oiuor4/2M GhX9E/aPmtBp8Zknaf39Ks6GkbfXf+EmAm+zJ20pS+EOtog7XorZlqDaRfQBm5p0mjOo GQsA662EykaGjnNnoTRmWZ6Yh5+NRLr03fRcplSxSJuyCgVBdnskLnZayR/0mxh5wu+x PfDA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc; bh=nL76gfArXyNS5cIl4Is1mwrmdVjBY+lEVdIlEJuopwg=; b=bCg0uaEEArzbVXPExyBYxRS3yfo+ROy+zH+luVX53amm7BSLNOu34ipjg6Yx5V9lg0 6dIyPz67kJaboT/0VrwrVYhPQZ55mmlXKLIGH1w6mC6p6WeQFaq1QZ32VdJZmzJqZQCt bav23E6qhwpXmx2cwy3B5XPTyjbwb5b7dEiwRwwdRTfJBzU/aS6lOlABTBTzEkYdM2T0 RbDaXYCN7F4rXom3jIhbk9v/4z/Q7ZGNrUlw76XfQ2Rb1uSOuUP+t7yWLkP+ZTTTTH9Y DqinP2naZtICPl7wyTduQwuriOwMYlKl+lRhB55194d6dkbYh6wMJ57rEgfD91cMYeIG CgoA== X-Gm-Message-State: AEkoouuI+1zGBOMcXil35FCY11WN28ORlbpwKaYxM2UxuStPTJvI4G+nj17eEXiszATn+ppOlGfEmUT0U18T+Q== X-Received: by 10.107.13.70 with SMTP id 67mr6519163ion.75.1469215416621; Fri, 22 Jul 2016 12:23:36 -0700 (PDT) MIME-Version: 1.0 Sender: adrian.chadd@gmail.com Received: by 10.36.141.129 with HTTP; Fri, 22 Jul 2016 12:23:35 -0700 (PDT) In-Reply-To: References: <306af514-70ff-f3bf-5b4f-da7ac1ec6580@cs.duke.edu> From: Adrian Chadd Date: Fri, 22 Jul 2016 12:23:35 -0700 X-Google-Sender-Auth: M4J2YYsd2MsUcoW01BbDvmSLtIA Message-ID: Subject: Re: proposal: splitting NIC RSS up from stack RSS To: Sepherosa Ziehau Cc: Andrew Gallatin , FreeBSD Net Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Jul 2016 19:23:37 -0000 On 21 July 2016 at 18:54, Sepherosa Ziehau wrote: > On Fri, Jul 22, 2016 at 6:39 AM, Adrian Chadd wrote: >> hi, >> >> Cool! Yeah, the RSS bits thing can be removed, as it's just doing a >> bitmask instead of a % operator to do mapping. I think we can just go >> to % and if people need the extra speed from a power-of-two operation, >> they can reintroduce it. > > I thought about it a while ago (the most popular E5-2560v{1,2,3} only > has 6 cores, but E5-2560v4 has 8 cores! :). Since the raw RSS hash > value is '& 0x1f' (I believe most of the NICs use 128 entry indirect > table as defined by MS RSS) to select an entry in the indirect table, > simply '%' on the raw RSS hash value probably will not work properly; > you will need (hash&0x1f)%mp_ncpus at least. And well, since the > indirect table's size if 128, you still will get some uneven CPU > workload for non-power-of-2 cpus. And if you take cpu affinity into > consideration, the situation will be even more complex ... Hi, Sure. The biggest annoying part is that a lot of the kernel infrastructure for queueing packets (netisr) and scheduling stack work (callouts) are indexed on CPU, not on "thing". If it was indexed on "thing" then we could do a two stage work redistribution method that'd scale O(1): * packets get plonked into "thing" via some mapping table - eg, map 128 or 256 buckets to queues that do work / schedule call outs / netisr; and * the queues aren't tied to a CPU at this point, and it can get shuffled around by using cpumasks. It'd be really, really nice IMHO if we had netisr and callouts be "thing" based rather than "cpu" based, so we could just shift work by changing the CPU mask - then we don't have to worry about rescheduling packets or work onto the new CPU when we want to move load around. That doesn't risk out of order packet handling behaviour and it means we can (in theory!) put a given RSS bucket into more than one CPU, for things like TCP processing. Trouble is, this is somewhat contentious. I could do the netisr change without upsetting people, but the callout code honestly makes me want to set everything (in sys/kern) on fire and start again. After all of the current issues with the callout subsystem I kind of just want to see hps finish his work and land it into head, complete with more sensible lock semantics, before I look at breaking it out to not be per-CPU based but instead allow subsystems to create their own worker pools for callouts. I'm sure NFS and CAM would like this kind of thing too. Since people have asked me about this in the past, the side effect of support dynamic hash mapping (even in software) is that for any given flow, once you change the hash mapping you will have some packets in said flow in the old queue and some packets in the new queue. For things like stack TCP/UDP where it's using pcbgroups it can vary from being slow to (eventually, when the global list goes away) plainly not making it to the right pcb/socket, which is okay for some workloads and not for others. That may be a fun project to work on once the general stack / driver tidyups are done, but I'm going to resist doing it myself for a while because it'll introduce the above uncertainties which will cause out-of-order behaviour that'll likely generate more problem reports than I want to handle. (Read: since I'm doing this for free, I'm not going to do anything risky, as I'm not getting paid to wade through the repercussions just right now.) FWIW, we had this same problem in ye olde past with squid and WCCP with its hash based system. Squid's WCCP implementation was simple and static. The commercial solutions (read: cisco, etc) implemented handling the cache set changing / hash traffic map changing by having the caches redirect traffic to the /old/ cache whenever the hash or cache set changed. Squid didn't do this out of the box, so if the cache topology changed it would send traffic to the wrong box and the existing connections would break. -adrian From owner-freebsd-net@freebsd.org Sat Jul 23 19:21:59 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id ABBB3BA2518 for ; Sat, 23 Jul 2016 19:21:59 +0000 (UTC) (envelope-from hiren@strugglingcoder.info) Received: from mail.strugglingcoder.info (strugglingcoder.info [104.236.146.68]) by mx1.freebsd.org (Postfix) with ESMTP id 9A05A1E00 for ; Sat, 23 Jul 2016 19:21:59 +0000 (UTC) (envelope-from hiren@strugglingcoder.info) Received: from localhost (unknown [10.1.1.3]) (Authenticated sender: hiren@strugglingcoder.info) by mail.strugglingcoder.info (Postfix) with ESMTPA id 501D517B37; Sat, 23 Jul 2016 12:21:53 -0700 (PDT) Date: Sat, 23 Jul 2016 12:21:53 -0700 From: hiren panchasara To: Kevin Oberman Cc: Chris Dunbar , freebsd-net Subject: Re: Slow performance with Intel X540-T2 10Gb NIC Message-ID: <20160723192153.GB94850@strugglingcoder.info> References: <1244557023.708807.1469061382192.JavaMail.zimbra@dunbar.net> <1441424852.712842.1469134420198.JavaMail.zimbra@dunbar.net> <183608784.713013.1469136611853.JavaMail.zimbra@dunbar.net> <144391790.714645.1469195320062.JavaMail.zimbra@dunbar.net> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="0eh6TmSyL6TZE2Uz" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.23 (2014-03-12) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 23 Jul 2016 19:21:59 -0000 --0eh6TmSyL6TZE2Uz Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 07/22/16 at 11:23P, Kevin Oberman wrote: [skip] >=20 > This sort of problem can be very tricky to diagnose. I'd like to suggest > that one of the tool you use should be SIFTR. It does kernel level > collection of network statistics and is a loadable module. By default it i > IPv4 only. It will have to be re-built with "CFLAGS+=3D-DSIFTR_IPV6" > uncommented in /sys/modules/siftr/Makefile for IPv6 support. It starts, > stops, amd manages collection under the control of 4 sysctls. >=20 > I have found it invaluable for analysis of netork performance issues, but > seems to not be widely known. Another such tool which I (personally) find more powerful and less known is dtrace in this context. For example, in output direction, right when tcp is about to send a packet to ip, there is a dtrace trace point which you can use to get a ton of useful information: # dtrace -n 'tcp:::send / args[2]->ip_saddr =3D=3D "192.168.0.1" / {printf = ("%8u", args[3]->tcps_mss)}' This can let me see MSS for that out going packet. You can see all of tcp control block data this way. The mapping is in /usr/lib/dtrace/tcp.d file. You can always add whatever you want in there and use it. https://github.com/brendangregg/DTrace-book-scripts/blob/master/Chap6/tcpio= =2Ed is one such awesome script that you can modify to match your needs and look at whatever bidirectional traffic in detail. With dtrace predicates, you can filter out what you want to log which is not possible yet with siftr. This makes running siftr a little annoying on a busy box as it logs all the things and you have to do post processing to get what you want to see. Just my 2 rupees. Cheers, Hiren --0eh6TmSyL6TZE2Uz Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQF8BAABCgBmBQJXk8PNXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRBNEUyMEZBMUQ4Nzg4RjNGMTdFNjZGMDI4 QjkyNTBFMTU2M0VERkU1AAoJEIuSUOFWPt/lNzIIAKTiYKKySQkVvqZcR08OF/Ei 4f+x+CCiUbCCyG5VK410AYiVw8MB1gHJbPiKj7qNq+34Tf2BIJo0tIJZ8+nEnRbV EsmEFaYv8hTmwldQm60TPKMHyeJOpuY4L1s+bJ8MEmshJCC5tgruHglMmM1gXCgB +v/zmGa2LaVloiLD1JAXBqpkHH2/ue/Or8GgL0vUTham0APqZBXfRSTaUF6GKPo/ ibrXSvLOJwuChFDMzBptuiwdo1fviwthQtnH13d+n8jxKb7uEAiddyh7tOqhh3qS yS/We8RPRjAdWFV17dMEpvTmcorspMn6y5aYtV9E389mdYCqmFQlCM8heacaFBM= =cfTd -----END PGP SIGNATURE----- --0eh6TmSyL6TZE2Uz--