From owner-freebsd-net@freebsd.org Sun Jul 31 19:37:35 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B9EE7BA94ED for ; Sun, 31 Jul 2016 19:37:35 +0000 (UTC) (envelope-from kob6558@gmail.com) Received: from mailman.ysv.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 9B7221EE6 for ; Sun, 31 Jul 2016 19:37:35 +0000 (UTC) (envelope-from kob6558@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id 9751FBA94EC; Sun, 31 Jul 2016 19:37:35 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 94B16BA94EB for ; Sun, 31 Jul 2016 19:37:35 +0000 (UTC) (envelope-from kob6558@gmail.com) Received: from mail-it0-x22f.google.com (mail-it0-x22f.google.com [IPv6:2607:f8b0:4001:c0b::22f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5A0C41EE5 for ; Sun, 31 Jul 2016 19:37:35 +0000 (UTC) (envelope-from kob6558@gmail.com) Received: by mail-it0-x22f.google.com with SMTP id f6so150281352ith.0 for ; Sun, 31 Jul 2016 12:37:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:from:date:message-id:subject:to; bh=LVkvajKdgbQSIBNZhXL3QiitgBxujGYamXPmbskrXzo=; b=cfmje3EW+P5fUl0GrMejd6wSGR9dwAGt8foiF0tHNk1HBrlQdDwWIzD7+q8IL7n/Z/ 3R6SOfKRQ0VMfDEQr80hktFtObG8DqSLTrOTRB9lILnoJNrB8mh9tom7kuS+S6t+RDTs sRiGT1UMbhzRX8m+OLHZlLUCxyxLGYtdiMkxxtQp2rbxRRdq22ZRX6szcW+a8Ma5cAN2 AUUSew40UiasirMIPtdIxwg/mldKDj7Qzw5pPEInqpXk6ayF7mPM+D3HKwGkJdhz6xSK Xwmeu6bo4pssJMHsqf1Grf7mdmzNs822uzdqDgiuqiL5xjOEHk6jkcu4q6TxRpXj7jju 19NA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:from:date:message-id:subject :to; bh=LVkvajKdgbQSIBNZhXL3QiitgBxujGYamXPmbskrXzo=; b=AqUV3Bs6UzIv74MrQC3UWNouOkFwKwh1t0R9+eRcztrUqFuv5mrWv4G/CAUAwwG3yq +JCqxqpzAd3AVBuqjI5d+JFWLssIUU3Dmod/rSjYwzKKQ/owBsqOWz5TYRBQkWTPRtTU luH/UUVOqWUHwZccoPYlrPjn5zwiZYI2z82Ud5YSHRJ/nwxkteixeRzpwnSHKtk5dxuR lasJF9YQRRbmhAtfkycp95AMKFqdIfOBILL7aYYHlpOuHrxeMEOv+tAWPyrdC9i3klrf 2edurwQsGXcS5xMfLxfNiG8zixWXQXYtg9TiLz3rZ9pb0hYXnzZxEHm4M9prxCn7hErT AWrQ== X-Gm-Message-State: AEkooutmai3YGK3yBaGGqo1AaW1hgPTrNzldI5qNmeY5B7GeYQZ2/7kP0VLJkoiDWhiVCDlejBuEuEbd6zPHAg== X-Received: by 10.36.120.150 with SMTP id p144mr52115938itc.44.1469993854566; Sun, 31 Jul 2016 12:37:34 -0700 (PDT) MIME-Version: 1.0 Sender: kob6558@gmail.com Received: by 10.79.119.144 with HTTP; Sun, 31 Jul 2016 12:37:34 -0700 (PDT) From: Kevin Oberman Date: Sun, 31 Jul 2016 12:37:34 -0700 X-Google-Sender-Auth: LQeGXSIre_ssDJi8qjg1PuwyX9w Message-ID: Subject: IWN issue in 11-BETA3 To: FreeBSD Net Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 31 Jul 2016 19:37:35 -0000 This morning I updated my laptop system to 11-BETA3. The WiFi card uses iwn: iwn0@pci0:3:0:0: class=0x028000 card=0x13118086 chip=0x00858086 rev=0x34$ vendor = 'Intel Corporation' device = 'Centrino Advanced-N 6205 [Taylor Peak]' class = network At boot time the interface takes a very long time to associate and dhclient fails to get an address for it. I have 'ifconfig_wlan0="WPA SYNCDHCP"' in /etc/rc.conf, but that did nto help. After the startup, the interface was associated, but had no address other than link-local and no routes. I did "service dhclient restart wlan0" which assigned the address and that resolved hte issue until I reboot. Is this a known issue? Should I open a bug report? This would leave me in a bad position should I ever need to reboot the system remotely or should it crash when I am not near the system. -- Kevin Oberman, Part time kid herder and retired Network Engineer E-mail: rkoberman@gmail.com PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683 From owner-freebsd-net@freebsd.org Mon Aug 1 09:18:45 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A274FBAA589 for ; Mon, 1 Aug 2016 09:18:45 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9269D1372 for ; Mon, 1 Aug 2016 09:18:45 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u719IjiW051256 for ; Mon, 1 Aug 2016 09:18:45 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-net@FreeBSD.org Subject: [Bug 208205] re0 watchdog timeout Date: Mon, 01 Aug 2016 09:18:45 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.3-BETA2 X-Bugzilla-Keywords: needs-qa X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: koobs@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: keywords see_also cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 01 Aug 2016 09:18:45 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D208205 Kubilay Kocak changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |needs-qa See Also| |https://bugs.freebsd.org/bu | |gzilla/show_bug.cgi?id=3D1= 667 | |24 CC| |yongari@FreeBSD.org --- Comment #3 from Kubilay Kocak --- @Nick, is this still reproducible in the latest release? --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-net@freebsd.org Mon Aug 1 11:34:46 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7B024BA36A2 for ; Mon, 1 Aug 2016 11:34:46 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6AD77119B for ; Mon, 1 Aug 2016 11:34:46 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u71BYjUd091374 for ; Mon, 1 Aug 2016 11:34:46 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-net@FreeBSD.org Subject: [Bug 206185] [netgraph][patch] refactoring ng_patch Date: Mon, 01 Aug 2016 11:34:46 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.2-STABLE X-Bugzilla-Keywords: patch X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: commit-hook@freebsd.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 01 Aug 2016 11:34:46 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D206185 --- Comment #1 from commit-hook@freebsd.org --- A commit references this bug: Author: julian Date: Mon Aug 1 11:34:13 UTC 2016 New revision: 303611 URL: https://svnweb.freebsd.org/changeset/base/303611 Log: slite style changes. There is an incoming patch that rewrites a lot of this module and I want to get the style and whitespace changes in a separate commit (or maybe more). PR: 206185 Submitted by: Dmitry Vagin MFC after: 1 month Changes: head/sys/netgraph/ng_patch.c head/sys/netgraph/ng_patch.h --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-net@freebsd.org Mon Aug 1 12:14:51 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EEFF2BAAA13 for ; Mon, 1 Aug 2016 12:14:51 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id DE8DB192D for ; Mon, 1 Aug 2016 12:14:51 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u71CEpBI046628 for ; Mon, 1 Aug 2016 12:14:51 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-net@FreeBSD.org Subject: [Bug 206186] [patch][netgraph] New netgraph node for calculate IP IP6 TCP UDP checksums Date: Mon, 01 Aug 2016 12:14:52 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.3-STABLE X-Bugzilla-Keywords: patch X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: commit-hook@freebsd.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 01 Aug 2016 12:14:52 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D206186 --- Comment #3 from commit-hook@freebsd.org --- A commit references this bug: Author: julian Date: Mon Aug 1 12:14:22 UTC 2016 New revision: 303613 URL: https://svnweb.freebsd.org/changeset/base/303613 Log: Man page for the new checksum netgraph module. PR: 206186 Submitted by: Dmitry Vagin MFC after: 1 month Changes: head/share/man/man4/ng_checksum.4 --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-net@freebsd.org Mon Aug 1 14:16:36 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 70DB9BAAFD3 for ; Mon, 1 Aug 2016 14:16:36 +0000 (UTC) (envelope-from robbelics@gmail.com) Received: from mail-yw0-x230.google.com (mail-yw0-x230.google.com [IPv6:2607:f8b0:4002:c05::230]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3C66C1D55 for ; Mon, 1 Aug 2016 14:16:36 +0000 (UTC) (envelope-from robbelics@gmail.com) Received: by mail-yw0-x230.google.com with SMTP id u134so173546933ywg.3 for ; Mon, 01 Aug 2016 07:16:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:from:date:message-id:subject:to; bh=nIY0H9b+QSZAQ2ZXi0FMKjvln+QY5lZn2+E063kg4NU=; b=dMzafd9kA0u6kluea35SyZZJYpf0qfR9WcUgd1BewaqPhUNaaqx3udrJwcW/DA8s+p qLjvNq6zeVEDolQXXc9EUKnlWvWqqDELGaspfEHMttp7UqTThvFC9zniCooMMKPfpETv dGWHf8vPwOafuax6XJjbMS4ODmN8CjiBbq2yBUJvNPVdI/pzWOeQRAaq+2Lf7vmInWUc SkCvzklWYvhHxD6umGbrYhh3KWN9WaY4zbrlw4rd9Zne4gU3U4J108OeKmfdUgV7NndR hM2QHf6X1RY3falqcjfwEczk3GLRxRtRLrnsYzVfknGDgKvv3rxpyL7/jI//AbZC3Pyb 4F1Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=nIY0H9b+QSZAQ2ZXi0FMKjvln+QY5lZn2+E063kg4NU=; b=PriRIcJc7/Ylre5gM3cTXhnhRjmNIU/3N9gVocEZbze4Tl2A8rXMsVXngwo/8xTsam wXXSasp0n4fRgvES+NftMl3LtGGBxZBo24L86PE0PYOCla6hoLYG+tILZ0PCSKaJOc4x u4cUI7qHY20OOmA++4Sbj6q1SxrnfgVFky+X5A0xZvUvlyhy5eQ1ncSjggtv7h/NIRht Z/BaBxOfMT0dUcg5DMrUbmKzua2wlr+rY0IgKj5mU6plfnfLAN3fjTyfDLt4Wu+L97xx wiyrKoTH7tYv1c0EnZL3j7xNIG9h1ONXVQzR+4lhg196YpdA744u+VuVV2ciimjwOFQp V5oA== X-Gm-Message-State: AEkoouucTWi52EMwCX+Er9QjhCZzKVfJrg7r31Z8m96OoNyPUBCKM5V2MLctqQj0nXP9H1Rj4RXzk+eAmp9T/g== X-Received: by 10.13.213.3 with SMTP id x3mr45644493ywd.253.1470060995162; Mon, 01 Aug 2016 07:16:35 -0700 (PDT) MIME-Version: 1.0 Received: by 10.37.220.17 with HTTP; Mon, 1 Aug 2016 07:16:34 -0700 (PDT) From: Rob Belics Date: Mon, 1 Aug 2016 09:16:34 -0500 Message-ID: Subject: Troubleshooting Idrops To: freebsd-net@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 01 Aug 2016 14:16:36 -0000 I have a VPS server running FreeBSD 10.3-RELEASE-p4 and nginx. It contains three very low volume web sites that have been up for about three years. I was tinkering with TLS and SSL ciphers by eliminating TLSv1 and TLSv1.1 with different ciphers when I noticed my daily "Network interface status" report one morning saying I was getting Idrops of 48665. Network interface status: Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs Coll Drop em0 1500 xx:xx:3c:cd:7e:c7 225569570 0 48665 923463 0 0 0 em0 - xxxx::xxx:3cf fe80::216:3cff:fe 0 - - 4 - - - em0 - 107.xxx.xx.xx mysite1.co 94833 - - 0 - - - em0 - 107.xxx.xx.0 mysite2.co 479981 - - 920067 - - - lo0 16384 783 0 0 783 0 0 0 lo0 - ::1 ::1 0 - - 0 - - - lo0 - xxxx::1%lo0 xxxx::1%lo0 0 - - 0 - - - lo0 - your-net localhost 783 - - 783 - - - I reverted my TLS/SSL changes but, the next day, that exact same number of Idrops happened and continued for a couple of days afterwards. I just don't know what I could have done to cause this and am looking for troubleshooting help since it's been so long since I've had to deal with this and forgotten nearly everything. All the sites seem to function normally and I should note that, besides the nginx server, there are also two nodejs servers listening via proxy. I do nothing with IPv6. Here is part of vmstat -z where I notice FAILs: ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP UMA Hash: 128, 0, 5, 26, 7, 0, 0 4 Bucket: 16, 0, 8, 496, 21344, 0, 0 6 Bucket: 24, 0, 0, 336, 121, 0, 0 8 Bucket: 32, 0, 2, 376, 1600, 0, 0 12 Bucket: 48, 0, 0, 0, 97831, 0, 0 16 Bucket: 64, 0, 12, 303, 9585, 8, 0 32 Bucket: 128, 0, 14, 389, 46423, 0, 0 64 Bucket: 256, 0, 20, 235, 48362, 0, 0 128 Bucket: 512, 0, 19, 101, 23133, 0, 0 mbuf_packet: 256, 30975, 256, 253,455561904,97330, 0 mbuf: 256, 30975, 2, 254, 2124523, 0, 0 mbuf_cluster: 2048, 4840, 509, 3, 17874,194660, 2 mbuf_jumbo_page: 4096, 2419, 0, 2, 10659, 0, 0 mbuf_jumbo_9k: 9216, 716, 0, 0, 0, 0, 0 mbuf_jumbo_16k: 16384, 403, 0, 0, 0, 0, 0 mbuf_ext_refcnt: 4, 0, 0, 0, 0, 0, 0 256 Bucket: 1024, 0, 27, 45, 56839,5856, 0 vmem btag: 28, 0, 6648, 1848, 68924, 58, 0 I ran netstat -s and can post that here if it's OK for something that big or if someone wants something specific from that. In addition, what can I do to see these drops without having to wait till the next day for the report? I know I can do netstat -i but that contains drops for the current day. From owner-freebsd-net@freebsd.org Mon Aug 1 20:17:34 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7F7C8BAA88C for ; Mon, 1 Aug 2016 20:17:34 +0000 (UTC) (envelope-from daemon-user@freebsd.org) Received: from reviews.nyi.freebsd.org (reviews.nyi.freebsd.org [IPv6:2610:1c1:1:607c::16:b]) by mx1.freebsd.org (Postfix) with ESMTP id 58AE1100C for ; Mon, 1 Aug 2016 20:17:34 +0000 (UTC) (envelope-from daemon-user@freebsd.org) Received: by reviews.nyi.freebsd.org (Postfix, from userid 1346) id E3D82E740; Mon, 1 Aug 2016 20:17:33 +0000 (UTC) Date: Mon, 1 Aug 2016 20:17:33 +0000 To: freebsd-net@freebsd.org From: "rpokala (Ravi Pokala)" Reply-to: D1986+325+381818416dc12ca2@reviews.freebsd.org Subject: [Differential] D1986: Teach lagg(4) to change MTU Message-ID: X-Priority: 3 X-Phabricator-Sent-This-Message: Yes X-Mail-Transport-Agent: MetaMTA X-Auto-Response-Suppress: All X-Phabricator-Mail-Tags: , Thread-Topic: D1986: Teach lagg(4) to change MTU X-Herald-Rules: none, <28> X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-Cc: X-Phabricator-Cc: X-Phabricator-Cc: X-Phabricator-Cc: X-Phabricator-Cc: X-Phabricator-Cc: X-Phabricator-Cc: X-Phabricator-Cc: X-Phabricator-Cc: X-Phabricator-Cc: Precedence: bulk In-Reply-To: References: Thread-Index: ODZhMzNlYThiYzMxOTgzYmRhMDE5M2Q2Yzk4IFefrl0= MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Type: text/plain; charset="utf-8" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 01 Aug 2016 20:17:34 -0000 cnBva2FsYSBhYmFuZG9uZWQgdGhpcyByZXZpc2lvbi4KcnBva2FsYSBhZGRlZCBhIGNvbW1lbnQu CgoKICBUaGlzIHR1cm5lZCBvdXQgdG8gYmUgd2F5IG1vcmUgY29tcGxpY2F0ZWQgdGhhbiBhbnlv bmUgZXhwZWN0ZWQsIHRoZSBwZW9wbGUgd2hvIHdlcmUgd29ya2luZyBvbiBpdCBmb3IgUGFuYXNh cyBhcmUgbm8gbG9uZ2VyIHdpdGggdGhlIGNvbXBhbnksIHRoZSBjaGFuZ2VzIGhhdmUgYml0cm90 dGVkIGR1ZSB0byBzaXggbW9udGhzIG9mIG5lZ2xlY3QgZnJvbSBtZSwgYW5kIHdlIGNhbWUgdXAg d2l0aCBhIHJlYXNvbmFibGUgd29ya2Fyb3VuZCBpbiB0aGUgc2NyaXB0aW5nIGxheWVyLiBGb3Ig YWxsIHRob3NlIHJlYXNvbnMsIEknbSBhYmFuZG9uaW5nIHRoaXMgZGlmZi4KClJFUE9TSVRPUlkK ICByUyBGcmVlQlNEIHNyYyByZXBvc2l0b3J5CgpSRVZJU0lPTiBERVRBSUwKICBodHRwczovL3Jl dmlld3MuZnJlZWJzZC5vcmcvRDE5ODYKCkVNQUlMIFBSRUZFUkVOQ0VTCiAgaHR0cHM6Ly9yZXZp ZXdzLmZyZWVic2Qub3JnL3NldHRpbmdzL3BhbmVsL2VtYWlscHJlZmVyZW5jZXMvCgpUbzogcnBv a2FsYSwgcnN0b25lLCBycG9rYWxhLXBhbmFzYXMuY29tLCBzbWgsIG1lbGlmYXJvCkNjOiBycG9r YWxhLXBhbmFzYXMuY29tLCBzbWgsIGltcCwgbWVsaWZhcm8sIGhycywgc2JydW5vLCBsYWtzaG1p Lm5fbXN5c3RlY2hub2xvZ2llcy5jb20sIGVtYXN0ZSwgYWUsIGZyZWVic2QtbmV0LWxpc3QK From owner-freebsd-net@freebsd.org Tue Aug 2 06:37:12 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B2E40BAA758 for ; Tue, 2 Aug 2016 06:37:12 +0000 (UTC) (envelope-from daemon-user@freebsd.org) Received: from reviews.nyi.freebsd.org (reviews.nyi.freebsd.org [IPv6:2610:1c1:1:607c::16:b]) by mx1.freebsd.org (Postfix) with ESMTP id 8D2B51A27 for ; Tue, 2 Aug 2016 06:37:12 +0000 (UTC) (envelope-from daemon-user@freebsd.org) Received: by reviews.nyi.freebsd.org (Postfix, from userid 1346) id 2A7AAC53B; Tue, 2 Aug 2016 06:37:12 +0000 (UTC) Date: Tue, 2 Aug 2016 06:37:12 +0000 To: freebsd-net@freebsd.org From: Phabricator Reply-to: D6689+325+6c89ed8b7a9bc66d@reviews.freebsd.org Subject: [Differential] D6689: tcp/lro: Implement hash table for LRO entries. Message-ID: <55c05f816cbae37b80111aff14f946cb@localhost.localdomain> X-Priority: 3 X-Phabricator-Sent-This-Message: Yes X-Mail-Transport-Agent: MetaMTA X-Auto-Response-Suppress: All X-Phabricator-Mail-Tags: , Thread-Topic: D6689: tcp/lro: Implement hash table for LRO entries. X-Herald-Rules: <64> X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-Cc: Precedence: bulk In-Reply-To: References: Thread-Index: ZGRkYTdkZDNmZDVlODIxOWE3MGU3NDg3NmVjIFegP5g= MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="b1_55c05f816cbae37b80111aff14f946cb" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Aug 2016 06:37:12 -0000 --b1_55c05f816cbae37b80111aff14f946cb Content-Type: text/plain; charset = "utf-8" Content-Transfer-Encoding: base64 VGhpcyByZXZpc2lvbiB3YXMgYXV0b21hdGljYWxseSB1cGRhdGVkIHRvIHJlZmxlY3QgdGhlIGNv bW1pdHRlZCBjaGFuZ2VzLgpDbG9zZWQgYnkgY29tbWl0IHJTMzAzNjU2OiB0Y3AvbHJvOiBJbXBs ZW1lbnQgaGFzaCB0YWJsZSBmb3IgTFJPIGVudHJpZXMuIChhdXRob3JlZCBieSBzZXBoZSkuCgpD SEFOR0VEIFBSSU9SIFRPIENPTU1JVAogIGh0dHBzOi8vcmV2aWV3cy5mcmVlYnNkLm9yZy9ENjY4 OT92cz0xNzU2OSZpZD0xODk1MyN0b2MKClJFUE9TSVRPUlkKICByUyBGcmVlQlNEIHNyYyByZXBv c2l0b3J5CgpDSEFOR0VTIFNJTkNFIExBU1QgVVBEQVRFCiAgaHR0cHM6Ly9yZXZpZXdzLmZyZWVi c2Qub3JnL0Q2Njg5P3ZzPTE3NTY5JmlkPTE4OTUzCgpSRVZJU0lPTiBERVRBSUwKICBodHRwczov L3Jldmlld3MuZnJlZWJzZC5vcmcvRDY2ODkKCkFGRkVDVEVEIEZJTEVTCiAgaGVhZC9zeXMvbmV0 aW5ldC90Y3BfbHJvLmMKICBoZWFkL3N5cy9uZXRpbmV0L3RjcF9scm8uaAoKRU1BSUwgUFJFRkVS RU5DRVMKICBodHRwczovL3Jldmlld3MuZnJlZWJzZC5vcmcvc2V0dGluZ3MvcGFuZWwvZW1haWxw cmVmZXJlbmNlcy8KClRvOiBzZXBoZXJvc2FfZ21haWwuY29tLCBnbGViaXVzLCBnbm4sIGJ6LCBy d2F0c29uLCBnYWxsYXRpbiwgaHNlbGFza3ksICN0cmFuc3BvcnQsIHJycwpDYzogZnJlZWJzZC1u ZXQtbGlzdAo= --b1_55c05f816cbae37b80111aff14f946cb Content-Type: text/x-patch; charset=utf-8; name="D6689.18953.patch" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="D6689.18953.patch" ZGlmZiAtLWdpdCBhL2hlYWQvc3lzL25ldGluZXQvdGNwX2xyby5oIGIvaGVhZC9zeXMvbmV0aW5l dC90Y3BfbHJvLmgKLS0tIGEvaGVhZC9zeXMvbmV0aW5ldC90Y3BfbHJvLmgKKysrIGIvaGVhZC9z eXMvbmV0aW5ldC90Y3BfbHJvLmgKQEAgLTQwLDYgKzQwLDcgQEAKIAogc3RydWN0IGxyb19lbnRy eSB7CiAJTElTVF9FTlRSWShscm9fZW50cnkpCW5leHQ7CisJTElTVF9FTlRSWShscm9fZW50cnkp CWhhc2hfbmV4dDsKIAlzdHJ1Y3QgbWJ1ZgkJKm1faGVhZDsKIAlzdHJ1Y3QgbWJ1ZgkJKm1fdGFp bDsKIAl1bmlvbiB7CkBAIC05NSw2ICs5Niw4IEBACiAJdW5zaWduZWQgc2hvcnQJbHJvX2Fja2Nu dF9saW07CQkvKiBtYXggIyBvZiBhZ2dyZWdhdGVkIEFDS3MgKi8KIAl1bnNpZ25lZCAJbHJvX2xl bmd0aF9saW07CQkvKiBtYXggbGVuIG9mIGFnZ3JlZ2F0ZWQgZGF0YSAqLwogCisJdV9sb25nCQls cm9faGFzaHN6OworCXN0cnVjdCBscm9faGVhZAkqbHJvX2hhc2g7CiAJc3RydWN0IGxyb19oZWFk CWxyb19hY3RpdmU7CiAJc3RydWN0IGxyb19oZWFkCWxyb19mcmVlOwogfTsKZGlmZiAtLWdpdCBh L2hlYWQvc3lzL25ldGluZXQvdGNwX2xyby5jIGIvaGVhZC9zeXMvbmV0aW5ldC90Y3BfbHJvLmMK LS0tIGEvaGVhZC9zeXMvbmV0aW5ldC90Y3BfbHJvLmMKKysrIGIvaGVhZC9zeXMvbmV0aW5ldC90 Y3BfbHJvLmMKQEAgLTY4LDE5ICs2OCwyNCBAQAogI2VuZGlmCiAKIHN0YXRpYyB2b2lkCXRjcF9s cm9fcnhfZG9uZShzdHJ1Y3QgbHJvX2N0cmwgKmxjKTsKK3N0YXRpYyBpbnQJdGNwX2xyb19yeDIo c3RydWN0IGxyb19jdHJsICpsYywgc3RydWN0IG1idWYgKm0sCisJCSAgICB1aW50MzJfdCBjc3Vt LCBpbnQgdXNlX2hhc2gpOwogCiBzdGF0aWMgX19pbmxpbmUgdm9pZAotdGNwX2xyb19hY3RpdmVf aW5zZXJ0KHN0cnVjdCBscm9fY3RybCAqbGMsIHN0cnVjdCBscm9fZW50cnkgKmxlKQordGNwX2xy b19hY3RpdmVfaW5zZXJ0KHN0cnVjdCBscm9fY3RybCAqbGMsIHN0cnVjdCBscm9faGVhZCAqYnVj a2V0LAorICAgIHN0cnVjdCBscm9fZW50cnkgKmxlKQogewogCiAJTElTVF9JTlNFUlRfSEVBRCgm bGMtPmxyb19hY3RpdmUsIGxlLCBuZXh0KTsKKwlMSVNUX0lOU0VSVF9IRUFEKGJ1Y2tldCwgbGUs IGhhc2hfbmV4dCk7CiB9CiAKIHN0YXRpYyBfX2lubGluZSB2b2lkCiB0Y3BfbHJvX2FjdGl2ZV9y ZW1vdmUoc3RydWN0IGxyb19lbnRyeSAqbGUpCiB7CiAKLQlMSVNUX1JFTU9WRShsZSwgbmV4dCk7 CisJTElTVF9SRU1PVkUobGUsIG5leHQpOwkJLyogYWN0aXZlIGxpc3QgKi8KKwlMSVNUX1JFTU9W RShsZSwgaGFzaF9uZXh0KTsJLyogaGFzaCBidWNrZXQgKi8KIH0KIAogaW50CkBAIC05NSw3ICsx MDAsNyBAQAogewogCXN0cnVjdCBscm9fZW50cnkgKmxlOwogCXNpemVfdCBzaXplOwotCXVuc2ln bmVkIGk7CisJdW5zaWduZWQgaSwgZWxlbWVudHM7CiAKIAlsYy0+bHJvX2JhZF9jc3VtID0gMDsK IAlsYy0+bHJvX3F1ZXVlZCA9IDA7CkBAIC0xMTAsNiArMTE1LDE4IEBACiAJTElTVF9JTklUKCZs Yy0+bHJvX2ZyZWUpOwogCUxJU1RfSU5JVCgmbGMtPmxyb19hY3RpdmUpOwogCisJLyogY3JlYXRl IGhhc2ggdGFibGUgdG8gYWNjZWxlcmF0ZSBlbnRyeSBsb29rdXAgKi8KKwlpZiAobHJvX2VudHJp ZXMgPiBscm9fbWJ1ZnMpCisJCWVsZW1lbnRzID0gbHJvX2VudHJpZXM7CisJZWxzZQorCQllbGVt ZW50cyA9IGxyb19tYnVmczsKKwlsYy0+bHJvX2hhc2ggPSBwaGFzaGluaXRfZmxhZ3MoZWxlbWVu dHMsIE1fTFJPLCAmbGMtPmxyb19oYXNoc3osCisJICAgIEhBU0hfTk9XQUlUKTsKKwlpZiAobGMt Pmxyb19oYXNoID09IE5VTEwpIHsKKwkJbWVtc2V0KGxjLCAwLCBzaXplb2YoKmxjKSk7CisJCXJl dHVybiAoRU5PTUVNKTsKKwl9CisKIAkvKiBjb21wdXRlIHNpemUgdG8gYWxsb2NhdGUgKi8KIAlz aXplID0gKGxyb19tYnVmcyAqIHNpemVvZihzdHJ1Y3QgbHJvX21idWZfc29ydCkpICsKIAkgICAg KGxyb19lbnRyaWVzICogc2l6ZW9mKCpsZSkpOwpAQCAtMTQ3LDYgKzE2NCwxMyBAQAogCQltX2Zy ZWVtKGxlLT5tX2hlYWQpOwogCX0KIAorCS8qIGZyZWUgaGFzaCB0YWJsZSAqLworCWlmIChsYy0+ bHJvX2hhc2ggIT0gTlVMTCkgeworCQlmcmVlKGxjLT5scm9faGFzaCwgTV9MUk8pOworCQlsYy0+ bHJvX2hhc2ggPSBOVUxMOworCX0KKwlsYy0+bHJvX2hhc2hzeiA9IDA7CisKIAkvKiBmcmVlIG1i dWYgYXJyYXksIGlmIGFueSAqLwogCWZvciAoeCA9IDA7IHggIT0gbGMtPmxyb19tYnVmX2NvdW50 OyB4KyspCiAJCW1fZnJlZW0obGMtPmxyb19tYnVmX2RhdGFbeF0ubWIpOwpAQCAtNDg3LDcgKzUx MSw3IEBACiAJCX0KIAogCQkvKiBhZGQgcGFja2V0IHRvIExSTyBlbmdpbmUgKi8KLQkJaWYgKHRj cF9scm9fcngobGMsIG1iLCAwKSAhPSAwKSB7CisJCWlmICh0Y3BfbHJvX3J4MihsYywgbWIsIDAs IDApICE9IDApIHsKIAkJCS8qIGlucHV0IHBhY2tldCB0byBuZXR3b3JrIGxheWVyICovCiAJCQko KmxjLT5pZnAtPmlmX2lucHV0KShsYy0+aWZwLCBtYik7CiAJCQlsYy0+bHJvX3F1ZXVlZCsrOwpA QCAtNTYxLDggKzU4NSw4IEBACiB9CiAjZW5kaWYKIAotaW50Ci10Y3BfbHJvX3J4KHN0cnVjdCBs cm9fY3RybCAqbGMsIHN0cnVjdCBtYnVmICptLCB1aW50MzJfdCBjc3VtKQorc3RhdGljIGludAor dGNwX2xyb19yeDIoc3RydWN0IGxyb19jdHJsICpsYywgc3RydWN0IG1idWYgKm0sIHVpbnQzMl90 IGNzdW0sIGludCB1c2VfaGFzaCkKIHsKIAlzdHJ1Y3QgbHJvX2VudHJ5ICpsZTsKIAlzdHJ1Y3Qg ZXRoZXJfaGVhZGVyICplaDsKQEAgLTU3OCw2ICs2MDIsNyBAQAogCXRjcF9zZXEgc2VxOwogCWlu dCBlcnJvciwgaXBfbGVuLCBsOwogCXVpbnQxNl90IGVoX3R5cGUsIHRjcF9kYXRhX2xlbjsKKwlz dHJ1Y3QgbHJvX2hlYWQgKmJ1Y2tldDsKIAogCS8qIFdlIGV4cGVjdCBhIGNvbnRpZ3VvdXMgaGVh ZGVyIFtlaCwgaXAsIHRjcF0uICovCiAKQEAgLTY3MCw4ICs2OTUsNDEgQEAKIAogCXNlcSA9IG50 b2hsKHRoLT50aF9zZXEpOwogCisJaWYgKCF1c2VfaGFzaCkgeworCQlidWNrZXQgPSAmbGMtPmxy b19oYXNoWzBdOworCX0gZWxzZSBpZiAoTV9IQVNIVFlQRV9JU0hBU0gobSkpIHsKKwkJYnVja2V0 ID0gJmxjLT5scm9faGFzaFttLT5tX3BrdGhkci5mbG93aWQgJSBsYy0+bHJvX2hhc2hzel07CisJ fSBlbHNlIHsKKwkJdWludDMyX3QgaGFzaDsKKworCQlzd2l0Y2ggKGVoX3R5cGUpIHsKKyNpZmRl ZiBJTkVUCisJCWNhc2UgRVRIRVJUWVBFX0lQOgorCQkJaGFzaCA9IGlwNC0+aXBfc3JjLnNfYWRk ciArIGlwNC0+aXBfZHN0LnNfYWRkcjsKKwkJCWJyZWFrOworI2VuZGlmCisjaWZkZWYgSU5FVDYK KwkJY2FzZSBFVEhFUlRZUEVfSVBWNjoKKwkJCWhhc2ggPSBpcDYtPmlwNl9zcmMuczZfYWRkcjMy WzBdICsKKwkJCSAgICBpcDYtPmlwNl9kc3QuczZfYWRkcjMyWzBdOworCQkJaGFzaCArPSBpcDYt PmlwNl9zcmMuczZfYWRkcjMyWzFdICsKKwkJCSAgICBpcDYtPmlwNl9kc3QuczZfYWRkcjMyWzFd OworCQkJaGFzaCArPSBpcDYtPmlwNl9zcmMuczZfYWRkcjMyWzJdICsKKwkJCSAgICBpcDYtPmlw Nl9kc3QuczZfYWRkcjMyWzJdOworCQkJaGFzaCArPSBpcDYtPmlwNl9zcmMuczZfYWRkcjMyWzNd ICsKKwkJCSAgICBpcDYtPmlwNl9kc3QuczZfYWRkcjMyWzNdOworCQkJYnJlYWs7CisjZW5kaWYK KwkJZGVmYXVsdDoKKwkJCWhhc2ggPSAwOworCQkJYnJlYWs7CisJCX0KKwkJaGFzaCArPSB0aC0+ dGhfc3BvcnQgKyB0aC0+dGhfZHBvcnQ7CisJCWJ1Y2tldCA9ICZsYy0+bHJvX2hhc2hbaGFzaCAl IGxjLT5scm9faGFzaHN6XTsKKwl9CisKIAkvKiBUcnkgdG8gZmluZCBhIG1hdGNoaW5nIHByZXZp b3VzIHNlZ21lbnQuICovCi0JTElTVF9GT1JFQUNIKGxlLCAmbGMtPmxyb19hY3RpdmUsIG5leHQp IHsKKwlMSVNUX0ZPUkVBQ0gobGUsIGJ1Y2tldCwgaGFzaF9uZXh0KSB7CiAJCWlmIChsZS0+ZWhf dHlwZSAhPSBlaF90eXBlKQogCQkJY29udGludWU7CiAJCWlmIChsZS0+c291cmNlX3BvcnQgIT0g dGgtPnRoX3Nwb3J0IHx8CkBAIC03NzksNyArODM3LDcgQEAKIAkvKiBTdGFydCBhIG5ldyBzZWdt ZW50IGNoYWluLiAqLwogCWxlID0gTElTVF9GSVJTVCgmbGMtPmxyb19mcmVlKTsKIAlMSVNUX1JF TU9WRShsZSwgbmV4dCk7Ci0JdGNwX2xyb19hY3RpdmVfaW5zZXJ0KGxjLCBsZSk7CisJdGNwX2xy b19hY3RpdmVfaW5zZXJ0KGxjLCBidWNrZXQsIGxlKTsKIAlnZXRtaWNyb3RpbWUoJmxlLT5tdGlt ZSk7CiAKIAkvKiBTdGFydCBmaWxsaW5nIGluIGRldGFpbHMuICovCkBAIC04MzcsNiArODk1LDEz IEBACiAJcmV0dXJuICgwKTsKIH0KIAoraW50Cit0Y3BfbHJvX3J4KHN0cnVjdCBscm9fY3RybCAq bGMsIHN0cnVjdCBtYnVmICptLCB1aW50MzJfdCBjc3VtKQoreworCisJcmV0dXJuIHRjcF9scm9f cngyKGxjLCBtLCBjc3VtLCAxKTsKK30KKwogdm9pZAogdGNwX2xyb19xdWV1ZV9tYnVmKHN0cnVj dCBscm9fY3RybCAqbGMsIHN0cnVjdCBtYnVmICptYikKIHsKCg== --b1_55c05f816cbae37b80111aff14f946cb-- From owner-freebsd-net@freebsd.org Tue Aug 2 08:55:09 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 85715BACC7E for ; Tue, 2 Aug 2016 08:55:09 +0000 (UTC) (envelope-from gerrit.kuehn@aei.mpg.de) Received: from umail.aei.mpg.de (umail.aei.mpg.de [194.94.224.6]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4143A183C for ; Tue, 2 Aug 2016 08:55:08 +0000 (UTC) (envelope-from gerrit.kuehn@aei.mpg.de) Received: from mailgate.aei.mpg.de (mailgate.aei.mpg.de [194.94.224.5]) by umail.aei.mpg.de (Postfix) with ESMTP id B1B462001C3 for ; Tue, 2 Aug 2016 10:49:30 +0200 (CEST) Received: from mailgate.aei.mpg.de (localhost [127.0.0.1]) by localhost (Postfix) with SMTP id A1BCB406ADE for ; Tue, 2 Aug 2016 10:49:30 +0200 (CEST) Received: from intranet.aei.uni-hannover.de (ahin1.aei.uni-hannover.de [130.75.117.40]) by mailgate.aei.mpg.de (Postfix) with ESMTP id 86CC9406ADB for ; Tue, 2 Aug 2016 10:49:29 +0200 (CEST) Received: from arc.aei.uni-hannover.de ([130.75.117.1]) by intranet.aei.uni-hannover.de (IBM Domino Release 9.0.1FP5) with ESMTP id 2016080210492920-62483 ; Tue, 2 Aug 2016 10:49:29 +0200 Date: Tue, 2 Aug 2016 10:49:29 +0200 From: Gerrit =?ISO-8859-1?Q?K=FChn?= To: freebsd-net@freebsd.org Subject: NFS on 10G interfaces still painfully slow Message-Id: <20160802104929.a10602a4786c68b4547a45b9@aei.mpg.de> Organization: Max Planck Gesellschaft X-Mailer: Sylpheed 3.4.2 (GTK+ 2.24.22; amd64-portbld-freebsd10.0) Mime-Version: 1.0 X-MIMETrack: Itemize by SMTP Server on intranet/aei-hannover(Release 9.0.1FP5|November 22, 2015) at 02/08/2016 10:49:29, Serialize by Router on intranet/aei-hannover(Release 9.0.1FP5|November 22, 2015) at 02/08/2016 10:49:29, Serialize complete at 02/08/2016 10:49:29 X-TNEFEvaluated: 1 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII X-PMX-Version: 6.0.2.2308539, Antispam-Engine: 2.7.2.2107409, Antispam-Data: 2016.8.2.83917 X-PerlMx-Spam: Gauge=IIIIIIII, Probability=8%, Report=' HTML_00_01 0.05, HTML_00_10 0.05, MIME_LOWER_CASE 0.05, BODYTEXTP_SIZE_3000_LESS 0, BODY_SIZE_1000_LESS 0, BODY_SIZE_2000_LESS 0, BODY_SIZE_5000_LESS 0, BODY_SIZE_7000_LESS 0, BODY_SIZE_700_799 0, NO_CTA_URI_FOUND 0, NO_URI_FOUND 0, NO_URI_HTTPS 0, __CT 0, __CTE 0, __CT_TEXT_PLAIN 0, __HAS_FROM 0, __HAS_MSGID 0, __HAS_X_MAILER 0, __MIME_TEXT_ONLY 0, __MIME_VERSION 0, __SANE_MSGID 0, __SUBJ_ALPHA_END 0, __TO_MALFORMED_2 0, __TO_NO_NAME 0' X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Aug 2016 08:55:09 -0000 Hi all, I already reported this issue here a year ago and unfortunately was not able to fix it back then. Now I had another run at it, using two recent 10.3-machines with a direct 10G link. I still see nfs is painfully slow (around 20-80MB/s). I tried both nfsv3 and nfsv4, with almost the same results. Everything I tried so far (mtu size, wcommitsize, readahead...) only makes things worse or at least not much better. Moving data in different ways (scp, ggate) is much faster, so plain network speed should not be an issue. Is there anyone around here who can confirm that nfs can go faster over 10G links? Any hints for further tuning/debugging are greatly appreciated. cu Gerrit From owner-freebsd-net@freebsd.org Tue Aug 2 09:00:35 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3B1FCBACEDC for ; Tue, 2 Aug 2016 09:00:35 +0000 (UTC) (envelope-from borjam@sarenet.es) Received: from cu1176c.smtpx.saremail.com (cu1176c.smtpx.saremail.com [195.16.148.151]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 047D8196E for ; Tue, 2 Aug 2016 09:00:34 +0000 (UTC) (envelope-from borjam@sarenet.es) Received: from [172.16.8.36] (izaro.sarenet.es [192.148.167.11]) by proxypop02.sare.net (Postfix) with ESMTPSA id D27499DC72E; Tue, 2 Aug 2016 11:00:25 +0200 (CEST) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: NFS on 10G interfaces still painfully slow From: Borja Marcos In-Reply-To: <20160802104929.a10602a4786c68b4547a45b9@aei.mpg.de> Date: Tue, 2 Aug 2016 11:00:24 +0200 Cc: freebsd-net@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <4D7BC972-D3EF-4BDE-86AD-C88DCFE99362@sarenet.es> References: <20160802104929.a10602a4786c68b4547a45b9@aei.mpg.de> To: =?utf-8?Q?Gerrit_K=C3=BChn?= X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Aug 2016 09:00:35 -0000 > On 02 Aug 2016, at 10:49, Gerrit K=C3=BChn = wrote: >=20 > Is there anyone around here who can confirm that nfs can go faster = over > 10G links? > Any hints for further tuning/debugging are greatly appreciated. Can you show us ifconfig output, please? Borja. From owner-freebsd-net@freebsd.org Tue Aug 2 14:45:09 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3D9FBBACB78 for ; Tue, 2 Aug 2016 14:45:09 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-oi0-x22b.google.com (mail-oi0-x22b.google.com [IPv6:2607:f8b0:4003:c06::22b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id EF0CE1383 for ; Tue, 2 Aug 2016 14:45:08 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by mail-oi0-x22b.google.com with SMTP id w18so239614483oiw.3 for ; Tue, 02 Aug 2016 07:45:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc:content-transfer-encoding; bh=Zs19BeAik0lxWCbo/045u+0halLZ2k23niWRGsqZD6M=; b=A42v3dfMuLjFAOJlG6CbdxsthvDDW5zEyqsOYEGzU8H8z6djwjDbC855GYMt/5nlbF o5YZF7qfERTVsNKj6usCZrNaP7uE9c1OPc0eUnB/iIcSDFX9wkDnfTCGfTQ0OU5YJx4D xd2YdTzr6xUaEVyeuS0KeM5Rh5jfwItqNUK1qZrL77OhbRRzbz37aIWiNNzN4Dy+lTrz jWFu8E7MINt9Zl0EkORnHzOrMS4emqMsfGhkuBPPVtmrkfM+8Sp8Gal2RKVUhjAUj7xE xLSyQBKG3R/T/etaenGeGuvUREL6zwrNBEjra5i2nh9yKu2tzPniaLUa8vY2G/eScfRp jLPw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc:content-transfer-encoding; bh=Zs19BeAik0lxWCbo/045u+0halLZ2k23niWRGsqZD6M=; b=Bi6iCrBA7JS9wspBhtJSe67iGRjecXj+Y5wb8NmYyc8F+/bX+KDsbRKtM0IJUBbe+g YYb11aaUIKTfOUF0LaHNSaWrwOaz/91xaGR5UnzoXrivyABDHKEc32TFHrze7A22Pfu7 xEbAPFUtZNOsC3Pg5pfubDw/OLPVqEigRj09456lMS8dUQFL/xXxhQ+GAFIwaAfPwx/7 L7UWn6jhGTbwYmHcBrypfMY7Vsmiu0dKGbgjybXp63lB8EZjXoqqkzB1frBIGAe1jEZN pjS/9FPy9rBvAZGr90PzDiR6Ze05gLSRQNi0WoQWj4seoxJxMsGSn1OcMPI9FBaF9nYs Sotw== X-Gm-Message-State: AEkoouvIAwNgEZHG9Z5Os9ItZia/9wLT2ZPMP9XD1iUkA2HSV+xXCB3RgSNYrrL2MNynW+cHzhv2p/sw8fwL8g== X-Received: by 10.202.230.141 with SMTP id d135mr37910728oih.182.1470149107993; Tue, 02 Aug 2016 07:45:07 -0700 (PDT) MIME-Version: 1.0 Sender: asomers@gmail.com Received: by 10.202.196.149 with HTTP; Tue, 2 Aug 2016 07:45:07 -0700 (PDT) In-Reply-To: <20160802104929.a10602a4786c68b4547a45b9@aei.mpg.de> References: <20160802104929.a10602a4786c68b4547a45b9@aei.mpg.de> From: Alan Somers Date: Tue, 2 Aug 2016 08:45:07 -0600 X-Google-Sender-Auth: PskSLoEnl5UZ1dvDAMG43IyLgnI Message-ID: Subject: Re: NFS on 10G interfaces still painfully slow To: =?UTF-8?B?R2Vycml0IEvDvGhu?= Cc: FreeBSD Net Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Aug 2016 14:45:09 -0000 On Tue, Aug 2, 2016 at 2:49 AM, Gerrit K=C3=BChn = wrote: > Hi all, > > I already reported this issue here a year ago and unfortunately was not > able to fix it back then. Now I had another run at it, using two recent > 10.3-machines with a direct 10G link. I still see nfs is painfully > slow (around 20-80MB/s). I tried both nfsv3 and nfsv4, with almost the sa= me > results. Everything I tried so far (mtu size, wcommitsize, readahead...) > only makes things worse or at least not much better. > Moving data in different ways (scp, ggate) is much faster, so plain > network speed should not be an issue. > > Is there anyone around here who can confirm that nfs can go faster over > 10G links? > Any hints for further tuning/debugging are greatly appreciated. > > > cu > Gerrit I can get 1GB/s over NFS on a 10G link, so it's not always slow. There's probably something about your setup that's slowing it down. What is your NFS client? If Linux, make sure that you're using the "async" mount option instead of "sync". What benchmark are you using to measure that speed? Did you remember to start lockd and statd? If you post your /etc/exports and the client's /etc/fstab, that might reveal something. From owner-freebsd-net@freebsd.org Tue Aug 2 18:43:05 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 61B66BACA04 for ; Tue, 2 Aug 2016 18:43:05 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: from mail-wm0-x242.google.com (mail-wm0-x242.google.com [IPv6:2a00:1450:400c:c09::242]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id EC2E6117C for ; Tue, 2 Aug 2016 18:43:04 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: by mail-wm0-x242.google.com with SMTP id x83so32328128wma.3 for ; Tue, 02 Aug 2016 11:43:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:content-transfer-encoding:subject:message-id:date:to :mime-version; bh=tKUYGJqy+sEEV+BFGH+3WzdR2s5vrMd3D7d/YVqWfHA=; b=m5uW/DE/WNMc9i/aVGfrP67bF/ci1/HeJ1DrywpAoW7Fted7i6veR5b/1JpvJ1394u ltnoHYjlMgtkBlY2afwF5f5c6dcGhTLpl4ELGueDVA6MLjyEvnFDmobGTn4L1kLTGdMM eQRfue36CgwvF/zQz8QSw1Ii/LQtBQ2HMvEuN6LTFLmRdgkfxDSj+24uPf4qrFh+anwW FEc2v0YmMcW9Z9y5xGagm3lbXdpklOCa1F5ViIhYEN3KlW8QpWSvfz8Cb2RPT/6Us3ji YO8oKEmzH6hMfY8r1Hse2nI5ezpbpTwOdFCLq21EefRhbUSbJWTHGRFELvuAq3VgSsrq /JzA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:content-transfer-encoding:subject :message-id:date:to:mime-version; bh=tKUYGJqy+sEEV+BFGH+3WzdR2s5vrMd3D7d/YVqWfHA=; b=OPwfHGoWrSqqkSDMrE/ddYk2o24sQ/hgbqLgAXnih5TKTzVrc0olHKhhR9GyCGTo1z ONmymgGt+0B/4k3g+Cvx7G8H3aWEDWXfqcWwyE8zBHkZJ95dK1Azx+8wAU2w9yf2SnAB WsFA6nryInlBCGJOZM2YP6I6X6DBPoYsm0gSZ3X9v1i6aDfZ/ggs4KQO9NMq0bvoVntg 7BIAhQw1aWRiaQ4BWNFOq+8QU8WU1EqF3hIvpZA5rFb0yoc/EszEGc8eKzNMGT4l2snd vrhVWj5G2Qvta5NdLIk9cdCQQfVOSEaHWWr3iUNuGmGi/IJKBPiO1b1vjpghy76Fs0jL YlHw== X-Gm-Message-State: AEkoouu+H+DPHfEDxxy3RTHoTVx6sMyp6ZJ1mGPyAy9nU+0aAs7UL9WJsirPpGtGEnjTyw== X-Received: by 10.28.229.1 with SMTP id c1mr67186225wmh.0.1470163382815; Tue, 02 Aug 2016 11:43:02 -0700 (PDT) Received: from macbook-air-de-benjamin-1.home (LFbn-1-7077-85.w90-116.abo.wanadoo.fr. [90.116.246.85]) by smtp.gmail.com with ESMTPSA id x203sm23274966wmg.0.2016.08.02.11.43.01 for (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Tue, 02 Aug 2016 11:43:02 -0700 (PDT) From: Ben RUBSON Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Subject: Unstable local network throughput Message-Id: <3C0D892F-2BE8-4650-B9FC-93C8EE0443E1@gmail.com> Date: Tue, 2 Aug 2016 20:43:01 +0200 To: freebsd-net@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Aug 2016 18:43:05 -0000 Hello, I'm trying to reach the 40Gb/s max throughtput between 2 hosts running a = ConnectX-3 Mellanox network adapter. FreeBSD 10.3 just installed, last updates performed. Network adapters running last firmwares / last drivers. No workload at all, just iPerf as the benchmark tool. ### Step 1 : I never achieved to go beyond around 30Gb/s. I did the usual tuning (MTU, kern.ipc.maxsockbuf, = net.inet.tcp.sendbuf_max, net.inet.tcp.recvbuf_max...). I played with adapter interrupt moderation. I played with iPerf options (window / buffer size, number of = threads...). But it did not help. Results fluctuate, throughput is not sustained, and using 2 or more = iPerf threads did not help but degraded the results "quality". ### Step 2 : Let's start Linux on these 2 physical hosts. I only had to use jumbo frames in order to achieve the 40Gb/s max = throughtput... OK, network between the 2 hosts is not the root cause, and my hardware = can run these adapters up to their max throughput. Good point. ### Step 3 : Go back to FreeBSD on these physical hosts. Let's run this simple command to test FreeBSD itself : # iperf -c 127.0.0.1 -i 1 -t 60 Strangely enough, higher results are around 35GB/s. Even more strange, from one run to another, I do not get identical = results : sometimes 17Gb/s, sometimes 20, sometimes 30... Throughput can also suddenly drop down, then increase again... Power management in BIOS is totally disabled, as well as FreeBSD powerd, = so CPU frequency is not throttled. Another strange thing, increasing the number of iPerf threads (-P 2 for = example), does not improve the results at all. iPerf3 gave the same random results. ### Step 4 : Let's start Linux again on these 2 hosts. Let's run the same simple command : # iperf -c 127.0.0.1 -i 1 -t 60 Result : 45Gb/s. With 2 threads : 90Gb/s. With 4 threads : 180Gb/s. So here we have expected results, and they stay identical over the time. ### Step 5 : Does FreeBSD suffers when sending or when receiving ? Let's start one host with Linux, the other one with FreeBSD. Results : Linux --> FreeBSD : around 30GB/s. FreeBSD --> Linux : 40Gb/s. So sounds like FreeBSD suffers when receiving. ### Step 6 : FreeBSD 11-BETA3 gave the same random results. ### Questions : I think my tests show that there is something wrong with FreeBSD (tuning = ? something else ?). Do you have the same kind of random results on your hosts ? Could you help me trying to have sustained througput @step3, as we have = @step4 (I think this is what we should expect) ? There would then be no reason not to achieve max throughput through = Mellanox adapters themselves. Thank you very much ! Best regards, Ben From owner-freebsd-net@freebsd.org Tue Aug 2 19:31:40 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1C041BACB20 for ; Tue, 2 Aug 2016 19:31:40 +0000 (UTC) (envelope-from hps@selasky.org) Received: from mail.turbocat.net (heidi.turbocat.net [88.198.202.214]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id DAFC11D66 for ; Tue, 2 Aug 2016 19:31:39 +0000 (UTC) (envelope-from hps@selasky.org) Received: from laptop015.home.selasky.org (unknown [62.141.129.119]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.turbocat.net (Postfix) with ESMTPSA id DA6B31FE022; Tue, 2 Aug 2016 21:31:30 +0200 (CEST) Subject: Re: Unstable local network throughput To: Ben RUBSON , freebsd-net@freebsd.org References: <3C0D892F-2BE8-4650-B9FC-93C8EE0443E1@gmail.com> From: Hans Petter Selasky Message-ID: Date: Tue, 2 Aug 2016 21:35:48 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:45.0) Gecko/20100101 Thunderbird/45.0 MIME-Version: 1.0 In-Reply-To: <3C0D892F-2BE8-4650-B9FC-93C8EE0443E1@gmail.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Aug 2016 19:31:40 -0000 On 08/02/16 20:43, Ben RUBSON wrote: > Hello, > > I'm trying to reach the 40Gb/s max throughtput between 2 hosts running a ConnectX-3 Mellanox network adapter. > > FreeBSD 10.3 just installed, last updates performed. > Network adapters running last firmwares / last drivers. > No workload at all, just iPerf as the benchmark tool. > Hi, The CX-3 driver doesn't bind the worker threads to specific CPU cores by default, so if your CPU has more than one so-called numa, you'll end up that the bottle-neck is the high-speed link between the CPU cores and not the card. A quick and dirty workaround is to "cpuset" iperf and the interrupt and taskqueue threads to specific CPU cores. Are you using "options RSS" and "options PCBGROUP" in your kernel config? Are you also testing CX-4 cards from Mellanox? --HPS From owner-freebsd-net@freebsd.org Tue Aug 2 20:09:07 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 65339BAD5DB for ; Tue, 2 Aug 2016 20:09:07 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 54BCA15FC for ; Tue, 2 Aug 2016 20:09:07 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u72K97Bt040098 for ; Tue, 2 Aug 2016 20:09:07 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-net@FreeBSD.org Subject: [Bug 208136] kldunload if_cxgbe causes kernel panic Date: Tue, 02 Aug 2016 20:09:07 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.3-BETA2 X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: jhb@FreeBSD.org X-Bugzilla-Status: Closed X-Bugzilla-Resolution: FIXED X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: resolution cc bug_status Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Aug 2016 20:09:07 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D208136 John Baldwin changed: What |Removed |Added ---------------------------------------------------------------------------- Resolution|--- |FIXED CC| |jhb@FreeBSD.org Status|New |Closed --- Comment #2 from John Baldwin --- I believe this was fixed by Navdeep's MFC. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-net@freebsd.org Tue Aug 2 20:11:58 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 03E01BAD7DC for ; Tue, 2 Aug 2016 20:11:58 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: from mail-wm0-x236.google.com (mail-wm0-x236.google.com [IPv6:2a00:1450:400c:c09::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8C5E419B4 for ; Tue, 2 Aug 2016 20:11:57 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: by mail-wm0-x236.google.com with SMTP id o80so306366546wme.1 for ; Tue, 02 Aug 2016 13:11:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=88JR0gGY/5mVgwoiQukv7AUiFfBfDqqSb9Ew/NXq0RQ=; b=PKohHy/BOFmIL1gnSY5p/1EjxVvJV3RSrVOJiydXrHSXu2iynf2/2gaoMPm9U9h7AH NVSrBYBfeNCMhjh96SGgp6i3RqZiaOWyZWMYGnUyDocZO2QKTrW+u8NIznp70Gr86xEv PQL211e3U+++25io5h/1Tf+paGUGFem+52vqlDzkZ8J4/TPfYG0/3aJusCWbjAHmGUP3 lJxO3K1wOi9b7jETxlG5M1c9CK16/xJhi02EuU/EOFAX7XtTc18Z/NLKFvNbV8TjGv4f rzFbvY1iQx3VHSCNTkksQ2IDgzMYEZq22cq5koz++dmsYMz1dxgZ3V64+k/YXOzVMIO5 dCag== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=88JR0gGY/5mVgwoiQukv7AUiFfBfDqqSb9Ew/NXq0RQ=; b=ExEPhNMjyJuWWfByMNqMZU7FO+KwcOzIR59Aip4+T1d88CdVyrWmgSPZudLxf4awkC vgDI3VO6sza6o7vhx9RLefPAz+M4w6E4wig4DlkkpcmBSMdGiDIwjs+Iv9tSoLbHIvRd kQJks0LskF3oCM7021qcLQuZZ9f/S49GmWNiZmmMn0mUZu0NxeJx+JsUrT5biTcALhae JVv9nTNKd06VGvS4vVBUSsv2BDvlU7fiqgoCWvH7yiqGIcXXvjDQxY1y9c8AG4ZHm7A7 V9mXtKJyt+hRgu8h67hwfdiSQOKh5xwE6LdUfJjvnsIkIepg8zrdG3T1xvAce2jwBn39 y8EA== X-Gm-Message-State: AEkoouvPMB5Pmt7t7eRl0Ny1PDqkBqJj66ozBNXT458q/Dih/svBE9WtIAawUekIzZH90w== X-Received: by 10.28.208.140 with SMTP id h134mr67275716wmg.101.1470168714991; Tue, 02 Aug 2016 13:11:54 -0700 (PDT) Received: from macbook-air-de-benjamin.home (LFbn-1-7077-85.w90-116.abo.wanadoo.fr. [90.116.246.85]) by smtp.gmail.com with ESMTPSA id i1sm4181944wjl.9.2016.08.02.13.11.54 for (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Tue, 02 Aug 2016 13:11:54 -0700 (PDT) Content-Type: text/plain; charset=windows-1252 Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: Unstable local network throughput From: Ben RUBSON In-Reply-To: Date: Tue, 2 Aug 2016 22:11:53 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: <3B164B7B-CBFB-4518-B57D-A96EABB71647@gmail.com> References: <3C0D892F-2BE8-4650-B9FC-93C8EE0443E1@gmail.com> To: freebsd-net@freebsd.org X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Aug 2016 20:11:58 -0000 > On 02 Aug 2016, at 21:35, Hans Petter Selasky wrote: >=20 > Hi, Thank you for your answer Hans Petter ! > The CX-3 driver doesn't bind the worker threads to specific CPU cores = by default, so if your CPU has more than one so-called numa, you'll end = up that the bottle-neck is the high-speed link between the CPU cores and = not the card. A quick and dirty workaround is to "cpuset" iperf and the = interrupt and taskqueue threads to specific CPU cores. My CPUs : 2x E5-2620v3 with DDR4@1866. What is strange is that even without using the card (iPerf on = localhost), as my results show, I have very low and unstable random = throughput (compared to Linux on the same host). > Are you using "options RSS" and "options PCBGROUP" in your kernel = config? I only installed FreeBSD 10.3 and updated it, so I use the GENERIC = kernel. RSS and PCBGROUP are not defined in /usr/src/sys/amd64/conf/GENERIC, so = I think I do not use them. > Are you also testing CX-4 cards from Mellanox? No, I only have CX-3 at my disposal :) Ben PS : in my previous mail I sometimes used GB/s, of course you must read = Gb/s everywhere.= From owner-freebsd-net@freebsd.org Tue Aug 2 20:57:39 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 37D3CBAD504 for ; Tue, 2 Aug 2016 20:57:39 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 273AD1F4D for ; Tue, 2 Aug 2016 20:57:39 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u72KvcK9046216 for ; Tue, 2 Aug 2016 20:57:39 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-net@FreeBSD.org Subject: [Bug 211062] [ixv] sr-iov virtual function driver fails to attach Date: Tue, 02 Aug 2016 20:57:39 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: IntelNetworking, needs-patch X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: keywords assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Aug 2016 20:57:39 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D211062 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |IntelNetworking Assignee|freebsd-bugs@FreeBSD.org |freebsd-net@FreeBSD.org --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-net@freebsd.org Tue Aug 2 21:00:22 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CC195BAD614 for ; Tue, 2 Aug 2016 21:00:22 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id BB74F1253 for ; Tue, 2 Aug 2016 21:00:22 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u72L0MsO055556 for ; Tue, 2 Aug 2016 21:00:22 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-net@FreeBSD.org Subject: [Bug 211219] NIC status does not pass into a state of "no carrier" after disconnecting the cable. Date: Tue, 02 Aug 2016 21:00:22 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 10.3-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Aug 2016 21:00:22 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D211219 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|freebsd-bugs@FreeBSD.org |freebsd-net@FreeBSD.org --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-net@freebsd.org Tue Aug 2 21:05:53 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7D3FEBADA8C for ; Tue, 2 Aug 2016 21:05:53 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6CDBC1A08 for ; Tue, 2 Aug 2016 21:05:53 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u72L5r9Y002100 for ; Tue, 2 Aug 2016 21:05:53 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-net@FreeBSD.org Subject: [Bug 211386] Kernel build failes when TCP_RFC7413 option set [regression] Date: Tue, 02 Aug 2016 21:05:53 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-STABLE X-Bugzilla-Keywords: regression X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to keywords Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Aug 2016 21:05:53 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D211386 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|freebsd-bugs@FreeBSD.org |freebsd-net@FreeBSD.org Keywords| |regression --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-net@freebsd.org Tue Aug 2 21:06:49 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 09BC8BADBE3 for ; Tue, 2 Aug 2016 21:06:49 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id ED51C1BF1 for ; Tue, 2 Aug 2016 21:06:48 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u72L6mop018582 for ; Tue, 2 Aug 2016 21:06:48 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-net@FreeBSD.org Subject: [Bug 211441] incorrect handling of TCPS_SYN_SENT and TCPS_SYN_RECEIVED in API tcp_usrclosed() in file tcp_usrreq.c Date: Tue, 02 Aug 2016 21:06:49 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to component Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Aug 2016 21:06:49 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D211441 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|freebsd-standards@FreeBSD.o |freebsd-net@FreeBSD.org |rg | Component|standards |kern --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-net@freebsd.org Tue Aug 2 21:11:48 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 35D3ABAD112 for ; Tue, 2 Aug 2016 21:11:48 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 257A8158A for ; Tue, 2 Aug 2016 21:11:48 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u72LBmos013868 for ; Tue, 2 Aug 2016 21:11:48 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-net@FreeBSD.org Subject: [Bug 211386] Kernel build fails when TCP_RFC7413 option set [regression] Date: Tue, 02 Aug 2016 21:11:48 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-STABLE X-Bugzilla-Keywords: regression X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: ngie@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: short_desc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Aug 2016 21:11:48 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D211386 Ngie Cooper changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|Kernel build failes when |Kernel build fails when |TCP_RFC7413 option set |TCP_RFC7413 option set |[regression] |[regression] --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-net@freebsd.org Tue Aug 2 21:53:16 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B3099BA8E6A for ; Tue, 2 Aug 2016 21:53:16 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A2C6C1C21 for ; Tue, 2 Aug 2016 21:53:16 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u72LrGpc091172 for ; Tue, 2 Aug 2016 21:53:16 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-net@FreeBSD.org Subject: [Bug 211386] Kernel build fails when TCP_RFC7413 option set [regression] Date: Tue, 02 Aug 2016 21:53:16 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-STABLE X-Bugzilla-Keywords: regression X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: pluknet@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Aug 2016 21:53:16 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D211386 Sergey Kandaurov changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |pluknet@FreeBSD.org --- Comment #3 from Sergey Kandaurov --- This is a probable fallout from r297225, which is confirmed by that TCP_RFC7413 is successfully built on stable/10 that doesn't have net/route.h in netinet/in_pcb.h. You can try to workaround it with inclusion from tcp_fastopen.c --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-net@freebsd.org Tue Aug 2 22:37:38 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0BC14BABB56 for ; Tue, 2 Aug 2016 22:37:38 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C77271720 for ; Tue, 2 Aug 2016 22:37:37 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from slw by zxy.spb.ru with local (Exim 4.86 (FreeBSD)) (envelope-from ) id 1bUiJU-0008M5-Pc for freebsd-net@freebsd.org; Wed, 03 Aug 2016 01:37:28 +0300 Date: Wed, 3 Aug 2016 01:37:28 +0300 From: Slawa Olhovchenkov To: freebsd-net@freebsd.org Subject: Is NFSv4 mount interruptible? Message-ID: <20160802223728.GN8192@zxy.spb.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.24 (2015-08-30) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: slw@zxy.spb.ru X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Aug 2016 22:37:38 -0000 I have autofs NFSv4 mount with /etc/nfs.map: /NFS -nfsv4,intr,soft,sec=krb5i,gssname=host storage01:/ Currently I am see NFS hung and I am can't kill shell and can't unmount /NFS. Is NFSv4 mount interruptible? Or intr support only by NFSv3? From owner-freebsd-net@freebsd.org Tue Aug 2 22:55:04 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8BD23BAC388 for ; Tue, 2 Aug 2016 22:55:04 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 51BE71383 for ; Tue, 2 Aug 2016 22:55:04 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from slw by zxy.spb.ru with local (Exim 4.86 (FreeBSD)) (envelope-from ) id 1bUiaU-0008wX-Gl; Wed, 03 Aug 2016 01:55:02 +0300 Date: Wed, 3 Aug 2016 01:55:02 +0300 From: Slawa Olhovchenkov To: Rick Macklem Cc: "freebsd-net@freebsd.org" Subject: Re: Is NFSv4 mount interruptible? Message-ID: <20160802225502.GO8192@zxy.spb.ru> References: <20160802223728.GN8192@zxy.spb.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.24 (2015-08-30) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: slw@zxy.spb.ru X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Aug 2016 22:55:04 -0000 On Tue, Aug 02, 2016 at 10:48:52PM +0000, Rick Macklem wrote: > Slawa wrote: > >I have autofs NFSv4 mount with /etc/nfs.map: > > > >/NFS -nfsv4,intr,soft,sec=krb5i,gssname=host storage01:/ > > > >Currently I am see NFS hung and I am can't kill shell and can't > >unmount /NFS. > > > >Is NFSv4 mount interruptible? > >Or intr support only by NFSv3? > Well, in the BUGS section of "man mount_nfs" it notes that soft and intr cannot be > safely used with NFSv4. The problem is that, if an RPC that updates state (Open, Close,...) > terminates, then the state is no longer usable. > > You can "umount -f" the file system, but only if no other "umount" is already in progress. > (I am working on a new umount option that will allow "umount -f" to succeed when another > umount is stuck. It requires "umount(8)" to avoid all the path checking, etc.) > > Personally, I would never use "soft" for any mount (except read-only maybe), since it can > result in a write failure when the NFS server is slow. Using "intr" should be ok for NFSv3, > since it is stateless. For me, this mount like r/o. I am don't have significant write to this mount. If some data lost I am able to create this data again. Currently NFS hang after long idle. I am try to do umount, w/o success: # ps -ax | grep mou 706 - Is 0:00.00 /usr/sbin/moused -p /dev/ums0 -t auto -I /var/run/moused.ums0.pid 1116 - Ds 1:16.50 /usr/sbin/autounmountd 86794 3- T+ 0:00.00 umount -f /NFS # procstat 86794 PID PPID PGID SID TSID THR LOGIN WCHAN EMUL COMM 86794 86791 86794 86788 86788 1 admin autofsvl FreeBSD ELF64 umount # procstat 1116 PID PPID PGID SID TSID THR LOGIN WCHAN EMUL COMM 1116 1 1116 1116 0 1 root newnfs FreeBSD ELF64 autounmountd From owner-freebsd-net@freebsd.org Tue Aug 2 23:03:15 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B7A93BAC57E for ; Tue, 2 Aug 2016 23:03:15 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from na01-bn1-obe.outbound.protection.outlook.com (mail-bn1bon0094.outbound.protection.outlook.com [157.56.111.94]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "Microsoft IT SSL SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 538B41776 for ; Tue, 2 Aug 2016 23:03:14 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from YTXPR01MB0495.CANPRD01.PROD.OUTLOOK.COM (10.165.220.17) by YTXPR01MB0496.CANPRD01.PROD.OUTLOOK.COM (10.165.220.18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.549.15; Tue, 2 Aug 2016 22:48:52 +0000 Received: from YTXPR01MB0495.CANPRD01.PROD.OUTLOOK.COM ([10.165.220.17]) by YTXPR01MB0495.CANPRD01.PROD.OUTLOOK.COM ([10.165.220.17]) with mapi id 15.01.0549.022; Tue, 2 Aug 2016 22:48:52 +0000 From: Rick Macklem To: Slawa Olhovchenkov , "freebsd-net@freebsd.org" Subject: Re: Is NFSv4 mount interruptible? Thread-Topic: Is NFSv4 mount interruptible? Thread-Index: AQHR7Q5869k2obPpPEiFFbi693at8aA2RBqg Date: Tue, 2 Aug 2016 22:48:52 +0000 Message-ID: References: <20160802223728.GN8192@zxy.spb.ru> In-Reply-To: <20160802223728.GN8192@zxy.spb.ru> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=rmacklem@uoguelph.ca; x-originating-ip: [24.57.164.61] x-ms-office365-filtering-correlation-id: 56ad765c-0912-4a52-777a-08d3bb272d1f x-microsoft-exchange-diagnostics: 1; YTXPR01MB0496; 6:JOKBlfUE5JQ6Ud6n6yyaRRwx31Qd/0zCCmPf6h0ZcZXqwfFfB7tCoAuPsfXGtCuCaIgFmcK/vpS9a7lG3VomXsX+P4MSBcrbwPAjr+8VXia/bf+/7LE50nBXn31RFVKNeWeolrdmiYmRL8l6j0T+/qGoPpV931V0gnFvmWqb5xMCvLqW4AJ8q7RLcsotnNeA0TO/RSTfaw2YAkicwA1EYOHveUD6JAJPJ0akznCxRKgYJoGjkk2SK1WlkpvYQLnoN1nVr2U7mzNKps1axWKsRVW5vtde7gJoZSDWjL4xUxNxFzkRKW4scpnUO6I2rd//; 5:zX8Z5VBOTCoFS3Z1cKMs/SNUuOKytYHEZvdyy2Av/F2umyeBXM9yB33RvYZHkiLfkw5OOfkTVso8yKyEaMMdhyPB3PALHfKBFQlZizg1Tcdx0C7pb7uYAsKZqi8wVHwXUAIz3MKnEMbWLlyrVjW0dg==; 24:HLfBYBP+P6n+cWPY8PfiZRQhWWR397q/yjM+ijok5HsBwtRlr2TEO+84H8a5GjnI4Jpwy5VxUWhwtasOngnSuCtAfZRpmEnpFct0a0SLc2E=; 7:pJjLO5BDoTaPs53T06qkEbWbnpLx8ObzE5/SUcZHENjFa2gghidWWQ7qZWNDjey5O3vfntmJ3fJ9MluYsl+dpiWA3j8AohvudP9TxNhKSTAQv5WeyT79NUlh7IHKR+TAIw1aRFILGoeBXWUhHZiBKV8O341QUqRW5rigIZY2R7OZiKDrfBHvtOOc305iupadD7DW97k0mRUM97ZAJjCVG1bxS1MHagvtX7JxfIiXqTkBROmOvJhlMT1Binl8MFHO x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:YTXPR01MB0496; x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(158342451672863)(75325880899374); x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(6040130)(2401047)(8121501046)(5005006)(10201501046)(3002001)(6041072)(6043046); SRVR:YTXPR01MB0496; BCL:0; PCL:0; RULEID:; SRVR:YTXPR01MB0496; x-forefront-prvs: 0022134A87 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(7916002)(199003)(24454002)(189002)(7906003)(68736007)(97736004)(19625215002)(77096005)(586003)(2501003)(5001770100001)(19617315012)(15975445007)(3280700002)(66066001)(33656002)(3660700001)(2906002)(19627405001)(87936001)(122556002)(10400500002)(7736002)(7696003)(74316002)(54356999)(7846002)(105586002)(11100500001)(86362001)(5002640100001)(76176999)(8936002)(92566002)(19580405001)(81156014)(81166006)(16236675004)(19580395003)(107886002)(106356001)(9686002)(106116001)(2900100001)(50986999)(2950100001)(102836003)(74482002)(101416001)(189998001)(8676002)(6116002)(3846002); DIR:OUT; SFP:1101; SCL:1; SRVR:YTXPR01MB0496; H:YTXPR01MB0495.CANPRD01.PROD.OUTLOOK.COM; FPR:; SPF:None; PTR:InfoNoRecords; A:1; MX:1; LANG:en; received-spf: None (protection.outlook.com: uoguelph.ca does not designate permitted sender hosts) spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-originalarrivaltime: 02 Aug 2016 22:48:52.2155 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-Transport-CrossTenantHeadersStamped: YTXPR01MB0496 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Aug 2016 23:03:15 -0000 Slawa wrote: >I have autofs NFSv4 mount with /etc/nfs.map: > >/NFS -nfsv4,intr,soft,sec=3Dkrb5i,gssname=3Dhost storage01:/ > >Currently I am see NFS hung and I am can't kill shell and can't >unmount /NFS. > >Is NFSv4 mount interruptible? >Or intr support only by NFSv3? Well, in the BUGS section of "man mount_nfs" it notes that soft and intr ca= nnot be safely used with NFSv4. The problem is that, if an RPC that updates state (= Open, Close,...) terminates, then the state is no longer usable. You can "umount -f" the file system, but only if no other "umount" is alrea= dy in progress. (I am working on a new umount option that will allow "umount -f" to succeed= when another umount is stuck. It requires "umount(8)" to avoid all the path checking, e= tc.) Personally, I would never use "soft" for any mount (except read-only maybe)= , since it can result in a write failure when the NFS server is slow. Using "intr" should = be ok for NFSv3, since it is stateless. rick >_______________________________________________ >freebsd-net@freebsd.org mailing list >https://lists.freebsd.org/mailman/listinfo/freebsd-net >To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" From owner-freebsd-net@freebsd.org Tue Aug 2 23:14:05 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 563C3BAC7C1 for ; Tue, 2 Aug 2016 23:14:05 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from na01-bn1-obe.outbound.protection.outlook.com (mail-bn1bon0056.outbound.protection.outlook.com [157.56.111.56]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "Microsoft IT SSL SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C0E821D85; Tue, 2 Aug 2016 23:14:04 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from YTXPR01MB0495.CANPRD01.PROD.OUTLOOK.COM (10.165.220.17) by YTXPR01MB0496.CANPRD01.PROD.OUTLOOK.COM (10.165.220.18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.549.15; Tue, 2 Aug 2016 22:41:17 +0000 Received: from YTXPR01MB0495.CANPRD01.PROD.OUTLOOK.COM ([10.165.220.17]) by YTXPR01MB0495.CANPRD01.PROD.OUTLOOK.COM ([10.165.220.17]) with mapi id 15.01.0549.022; Tue, 2 Aug 2016 22:41:17 +0000 From: Rick Macklem To: Alan Somers , =?iso-8859-1?Q?Gerrit_K=FChn?= CC: FreeBSD Net Subject: Re: NFS on 10G interfaces still painfully slow Thread-Topic: NFS on 10G interfaces still painfully slow Thread-Index: AQHR7JuWK5VNejCg2ESCcdbpFf4ey6A1v7WAgACDZOg= Date: Tue, 2 Aug 2016 22:41:17 +0000 Message-ID: References: <20160802104929.a10602a4786c68b4547a45b9@aei.mpg.de>, In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=rmacklem@uoguelph.ca; x-originating-ip: [24.57.164.61] x-ms-office365-filtering-correlation-id: f852f848-80bd-40a4-324a-08d3bb261dc7 x-microsoft-exchange-diagnostics: 1; YTXPR01MB0496; 6:xMgOdnC6poMGYHE91aEdnPjenrfeJyPRYMckD2u9+Jaxo1D5CvFY5ZnSHvv86acHYgU+F4u/lpLpT+jsclsVC//knqyxDaYmKB69G9vVJWOZZCVt/yleNBnqq+DhpyZX82pkgwP+Vv/utUk5/DicJ2Yhlo2AkA2wcvWt9PBg1QAWoolgmQ/HocXwkNXQsAGRWn1ccUqhGSKnhaU4qGfMka7J/ioAEAXN3R202vAvvYdUlgwLfm+rPR+omKRvSHSCyZ8pUxcXopyXhT/46nuFYuGs6/rn6EERylMV83bzRjrLwk6JF0wmJ9k/RKxTGQXk; 5:QMsa++/wg6ozQbPJLGUDkiC4KE7z9D9RY2hT29YxLc2Dajb89d0Ct8KKYDtlixhyLYI9xTlOoy7DEVjgBvW6AE17PrDx2icyuao4IAkKHbhU2Ioe2duxUEuEBnEuB2ySK09l8Ne9tMx4QhL78zvO5Q==; 24:VUgve7MUfi1spdyOP0ldKQBXFaEe2J1+YmVRQSh4eh1cnzU7BaYFuRoaTR7tkpXuVDxQEIXfJ5o6IPCsOmlE9qzyrBBwooLvZwHUxI1kKYo=; 7:iOF1a7alAzKveK2LardD8lieHrtFk/9u+vDlW8M/MjPpIUepTJHy1E4uePEpP4/30po9FJT3hfLzzVxqgncnA2zRGjZPX67xwdAQ0IykuUL3iLTBIMrkFVrHFXAOVJ09Kcl0bTOgvyPF0lPcRsr7HBcQ66thXlQLvQEKyytFyADBABmhg/95QTZ9Ik3hKz3hp4x7UZuo6AGbtXbKmXHTJBOJ9yYiq10rbkqXTxOe+cR6OEvFD0uhgvYLHdxmDM3R x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:YTXPR01MB0496; x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(75325880899374); x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(6040130)(2401047)(8121501046)(5005006)(10201501046)(3002001)(6041072)(6043046); SRVR:YTXPR01MB0496; BCL:0; PCL:0; RULEID:; SRVR:YTXPR01MB0496; x-forefront-prvs: 0022134A87 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(7916002)(377454003)(199003)(24454002)(53754006)(189002)(16236675004)(19580395003)(19580405001)(81156014)(81166006)(2900100001)(106356001)(9686002)(4326007)(106116001)(5002640100001)(76176999)(8936002)(92566002)(189998001)(8676002)(3846002)(6116002)(50986999)(74482002)(2950100001)(102836003)(101416001)(3660700001)(33656002)(3280700002)(66066001)(2906002)(19625215002)(68736007)(97736004)(7906003)(5001770100001)(586003)(19617315012)(15975445007)(77096005)(105586002)(54356999)(7846002)(11100500001)(86362001)(7696003)(7736002)(74316002)(87936001)(122556002)(10400500002)(19627405001); DIR:OUT; SFP:1101; SCL:1; SRVR:YTXPR01MB0496; H:YTXPR01MB0495.CANPRD01.PROD.OUTLOOK.COM; FPR:; SPF:None; PTR:InfoNoRecords; A:1; MX:1; LANG:en; received-spf: None (protection.outlook.com: uoguelph.ca does not designate permitted sender hosts) spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-originalarrivaltime: 02 Aug 2016 22:41:17.0385 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-Transport-CrossTenantHeadersStamped: YTXPR01MB0496 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Aug 2016 23:14:05 -0000 Alan Somers wrote: >On Tue, Aug 2, 2016 at 2:49 AM, Gerrit K=FChn wr= ote: >> Hi all, >> >> I already reported this issue here a year ago and unfortunately was not >> able to fix it back then. Now I had another run at it, using two recent >> 10.3-machines with a direct 10G link. I still see nfs is painfully >> slow (around 20-80MB/s). I tried both nfsv3 and nfsv4, with almost the s= ame >> results. Everything I tried so far (mtu size, wcommitsize, readahead...) >> only makes things worse or at least not much better. >> Moving data in different ways (scp, ggate) is much faster, so plain >> network speed should not be an issue. >> >> Is there anyone around here who can confirm that nfs can go faster over >> 10G links? >> Any hints for further tuning/debugging are greatly appreciated. >> I can't help much, but a couple of things you can try: - Disable TSO - Turn off/reduce interrupt moderation on the net interface. (NFS perf. depends on response time and anything that delays interrupt servicing will slow it down.) Good luck with it, rick >> >> cu >> Gerrit > > >I can get 1GB/s over NFS on a 10G link, so it's not always slow. >There's probably something about your setup that's slowing it down. "async" mount option instead of "sync". What benchmark are you using you post your /etc/exports and the client's /etc/fstab, that might >reveal something. >_______________________________________________ >freebsd-net@freebsd.org mailing list >https://lists.freebsd.org/mailman/listinfo/freebsd-net >To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" From owner-freebsd-net@freebsd.org Tue Aug 2 23:54:41 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AADC7BAD32F for ; Tue, 2 Aug 2016 23:54:41 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9A7BB159C for ; Tue, 2 Aug 2016 23:54:41 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u72NseAo015395 for ; Tue, 2 Aug 2016 23:54:41 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-net@FreeBSD.org Subject: [Bug 211386] Kernel build fails when TCP_RFC7413 option set Date: Tue, 02 Aug 2016 23:54:40 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-STABLE X-Bugzilla-Keywords: regression X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: ngie@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_severity short_desc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Aug 2016 23:54:41 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D211386 Ngie Cooper changed: What |Removed |Added ---------------------------------------------------------------------------- Severity|Affects Only Me |Affects Some People Summary|Kernel build fails when |Kernel build fails when |TCP_RFC7413 option set |TCP_RFC7413 option set |[regression] | --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-net@freebsd.org Wed Aug 3 00:26:37 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 560FFBADF04 for ; Wed, 3 Aug 2016 00:26:37 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 1B4BF1695; Wed, 3 Aug 2016 00:26:37 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from slw by zxy.spb.ru with local (Exim 4.86 (FreeBSD)) (envelope-from ) id 1bUk12-000CCe-Qg; Wed, 03 Aug 2016 03:26:32 +0300 Date: Wed, 3 Aug 2016 03:26:32 +0300 From: Slawa Olhovchenkov To: Rick Macklem , trasz@FreeBSD.org Cc: "freebsd-net@freebsd.org" Subject: Re: Is NFSv4 mount interruptible? Message-ID: <20160803002632.GP8192@zxy.spb.ru> References: <20160802223728.GN8192@zxy.spb.ru> <20160802225502.GO8192@zxy.spb.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160802225502.GO8192@zxy.spb.ru> User-Agent: Mutt/1.5.24 (2015-08-30) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: slw@zxy.spb.ru X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 03 Aug 2016 00:26:37 -0000 On Wed, Aug 03, 2016 at 01:55:02AM +0300, Slawa Olhovchenkov wrote: > On Tue, Aug 02, 2016 at 10:48:52PM +0000, Rick Macklem wrote: > > > Slawa wrote: > > >I have autofs NFSv4 mount with /etc/nfs.map: > > > > > >/NFS -nfsv4,intr,soft,sec=krb5i,gssname=host storage01:/ > > > > > >Currently I am see NFS hung and I am can't kill shell and can't > > >unmount /NFS. > > > > > >Is NFSv4 mount interruptible? > > >Or intr support only by NFSv3? > > Well, in the BUGS section of "man mount_nfs" it notes that soft and intr cannot be > > safely used with NFSv4. The problem is that, if an RPC that updates state (Open, Close,...) > > terminates, then the state is no longer usable. > > > > You can "umount -f" the file system, but only if no other "umount" is already in progress. > > (I am working on a new umount option that will allow "umount -f" to succeed when another > > umount is stuck. It requires "umount(8)" to avoid all the path checking, etc.) > > > > Personally, I would never use "soft" for any mount (except read-only maybe), since it can > > result in a write failure when the NFS server is slow. Using "intr" should be ok for NFSv3, > > since it is stateless. > > For me, this mount like r/o. > I am don't have significant write to this mount. > If some data lost I am able to create this data again. > Currently NFS hang after long idle. > > I am try to do umount, w/o success: > > # ps -ax | grep mou > 706 - Is 0:00.00 /usr/sbin/moused -p /dev/ums0 -t auto -I /var/run/moused.ums0.pid > 1116 - Ds 1:16.50 /usr/sbin/autounmountd > 86794 3- T+ 0:00.00 umount -f /NFS > > # procstat 86794 > PID PPID PGID SID TSID THR LOGIN WCHAN EMUL COMM > 86794 86791 86794 86788 86788 1 admin autofsvl FreeBSD ELF64 umount > > # procstat 1116 > PID PPID PGID SID TSID THR LOGIN WCHAN EMUL COMM > 1116 1 1116 1116 0 1 root newnfs FreeBSD ELF64 autounmountd > May be this is autofs issuse? Can be autofsvlk lock leaks? # procstat -a | grep autofsvl 23767 23759 23757 23738 0 1 root autofsvl FreeBSD ELF64 find 27588 1067 27588 27588 0 1 root autofsvl FreeBSD ELF64 sshd 27722 1067 27722 27722 0 1 root autofsvl FreeBSD ELF64 sshd 28520 28519 28520 28513 28513 1 admin autofsvl FreeBSD ELF64 csh 44829 44825 44717 44717 0 1 root autofsvl FreeBSD ELF64 find 86577 1067 86577 86577 0 1 root autofsvl FreeBSD ELF64 sshd 86588 86587 86588 86582 86582 1 admin autofsvl FreeBSD ELF64 csh 86794 86791 86794 86788 86788 1 admin autofsvl FreeBSD ELF64 umount # procstat -k 23767 27588 27722 28520 44829 86577 86588 86794 PID TID COMM TDNAME KSTACK 23767 101917 find - mi_switch sleepq_wait _sx_xlock_hard _sx_xlock autofs_node_vn lookup namei kern_statat_vnhook sys_fstatat amd64_syscall Xfast_syscall 27588 102087 sshd - mi_switch sleepq_wait _sx_xlock_hard _sx_xlock autofs_node_vn lookup namei vn_open_cred kern_openat amd64_syscall Xfast_syscall 27722 101649 sshd - mi_switch sleepq_wait _sx_xlock_hard _sx_xlock autofs_node_vn lookup namei vn_open_cred kern_openat amd64_syscall Xfast_syscall 28520 101961 csh - mi_switch sleepq_wait _sx_xlock_hard _sx_xlock autofs_node_vn lookup namei kern_statat_vnhook sys_stat amd64_syscall Xfast_syscall 44829 102258 find - mi_switch sleepq_wait _sx_xlock_hard _sx_xlock autofs_node_vn lookup namei kern_statat_vnhook sys_fstatat amd64_syscall Xfast_syscall 86577 101721 sshd - mi_switch sleepq_wait _sx_xlock_hard _sx_xlock autofs_node_vn lookup namei vn_open_cred kern_openat amd64_syscall Xfast_syscall 86588 101765 csh - mi_switch sleepq_wait _sx_xlock_hard _sx_xlock autofs_node_vn lookup namei kern_statat_vnhook sys_stat amd64_syscall Xfast_syscall 86794 101690 umount - mi_switch sleepq_wait _sx_xlock_hard _sx_xlock autofs_node_vn lookup namei kern_statfs sys_statfs amd64_syscall Xfast_syscall From owner-freebsd-net@freebsd.org Wed Aug 3 02:33:18 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D9FA2BACF22 for ; Wed, 3 Aug 2016 02:33:18 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from hz.grosbein.net (hz.grosbein.net [78.47.246.247]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "hz.grosbein.net", Issuer "hz.grosbein.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 740AB14C3 for ; Wed, 3 Aug 2016 02:33:17 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from eg.sd.rdtc.ru (root@eg.sd.rdtc.ru [62.231.161.221]) by hz.grosbein.net (8.14.9/8.14.9) with ESMTP id u732X6hD025487 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT); Wed, 3 Aug 2016 04:33:07 +0200 (CEST) (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: ben.rubson@gmail.com Received: from [10.58.0.10] (dadvw [10.58.0.10]) by eg.sd.rdtc.ru (8.15.2/8.15.2) with ESMTPS id u732X3kT063800 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Wed, 3 Aug 2016 09:33:03 +0700 (KRAT) (envelope-from eugen@grosbein.net) Subject: Re: Unstable local network throughput To: Ben RUBSON , freebsd-net@freebsd.org References: <3C0D892F-2BE8-4650-B9FC-93C8EE0443E1@gmail.com> From: Eugene Grosbein Message-ID: <57A157D9.3040707@grosbein.net> Date: Wed, 3 Aug 2016 09:32:57 +0700 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.7.2 MIME-Version: 1.0 In-Reply-To: <3C0D892F-2BE8-4650-B9FC-93C8EE0443E1@gmail.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=0.3 required=5.0 tests=BAYES_00,LOCAL_FROM autolearn=no version=3.3.2 X-Spam-Report: * -2.3 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * 2.6 LOCAL_FROM From my domains X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on hz.grosbein.net X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 03 Aug 2016 02:33:18 -0000 03.08.2016 1:43, Ben RUBSON пишет: > Hello, > > I'm trying to reach the 40Gb/s max throughtput between 2 hosts running a ConnectX-3 Mellanox network adapter. If you have gateway_enable="YES" (sysctl net.inet.ip.forwarding=1) then try to disable this forwarding setting and rerun your tests to compare results. From owner-freebsd-net@freebsd.org Wed Aug 3 05:59:58 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id ECEFCBADA8C for ; Wed, 3 Aug 2016 05:59:58 +0000 (UTC) (envelope-from gerrit.kuehn@aei.mpg.de) Received: from umail.aei.mpg.de (umail.aei.mpg.de [194.94.224.6]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7CF6318EB; Wed, 3 Aug 2016 05:59:58 +0000 (UTC) (envelope-from gerrit.kuehn@aei.mpg.de) Received: from mailgate.aei.mpg.de (mailgate.aei.mpg.de [194.94.224.5]) by umail.aei.mpg.de (Postfix) with ESMTP id 254E52000BA; Wed, 3 Aug 2016 07:59:55 +0200 (CEST) Received: from mailgate.aei.mpg.de (localhost [127.0.0.1]) by localhost (Postfix) with SMTP id 0D5E2406ADE; Wed, 3 Aug 2016 07:59:55 +0200 (CEST) Received: from intranet.aei.uni-hannover.de (ahin1.aei.uni-hannover.de [130.75.117.40]) by mailgate.aei.mpg.de (Postfix) with ESMTP id BB747406ADB; Wed, 3 Aug 2016 07:59:54 +0200 (CEST) Received: from arc.aei.uni-hannover.de ([130.75.117.1]) by intranet.aei.uni-hannover.de (IBM Domino Release 9.0.1FP5) with ESMTP id 2016080307595428-63260 ; Wed, 3 Aug 2016 07:59:54 +0200 Date: Wed, 3 Aug 2016 07:59:54 +0200 From: Gerrit =?ISO-8859-1?Q?K=FChn?= To: Alan Somers Cc: FreeBSD Net Subject: Re: NFS on 10G interfaces still painfully slow Message-Id: <20160803075954.35914021394d6c3f9b3802f3@aei.mpg.de> In-Reply-To: References: <20160802104929.a10602a4786c68b4547a45b9@aei.mpg.de> Organization: Max Planck Gesellschaft X-Mailer: Sylpheed 3.4.2 (GTK+ 2.24.22; amd64-portbld-freebsd10.0) Mime-Version: 1.0 X-MIMETrack: Itemize by SMTP Server on intranet/aei-hannover(Release 9.0.1FP5|November 22, 2015) at 03/08/2016 07:59:54, Serialize by Router on intranet/aei-hannover(Release 9.0.1FP5|November 22, 2015) at 03/08/2016 07:59:54, Serialize complete at 03/08/2016 07:59:54 X-TNEFEvaluated: 1 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII X-PMX-Version: 6.0.2.2308539, Antispam-Engine: 2.7.2.2107409, Antispam-Data: 2016.8.3.55116 X-PerlMx-Spam: Gauge=IIIIIIII, Probability=8%, Report=' HTML_00_01 0.05, HTML_00_10 0.05, MIME_LOWER_CASE 0.05, SUPERLONG_LINE 0.05, BODYTEXTP_SIZE_3000_LESS 0, BODY_SIZE_2000_2999 0, BODY_SIZE_5000_LESS 0, BODY_SIZE_7000_LESS 0, IN_REP_TO 0, LEGITIMATE_NEGATE 0, LEGITIMATE_SIGNS 0, MSG_THREAD 0, MULTIPLE_REAL_RCPTS 0, NO_URI_HTTPS 0, REFERENCES 0, __ANY_URI 0, __BOUNCE_CHALLENGE_SUBJ 0, __BOUNCE_NDR_SUBJ_EXEMPT 0, __C230066_P5 0, __CC_NAME 0, __CC_NAME_DIFF_FROM_ACC 0, __CC_REAL_NAMES 0, __CT 0, __CTE 0, __CT_TEXT_PLAIN 0, __HAS_CC_HDR 0, __HAS_FROM 0, __HAS_MSGID 0, __HAS_X_MAILER 0, __IN_REP_TO 0, __MIME_TEXT_ONLY 0, __MIME_VERSION 0, __REFERENCES 0, __SANE_MSGID 0, __SUBJ_ALPHA_END 0, __SUBJ_ALPHA_NEGATE 0, __TO_MALFORMED_2 0, __TO_NAME 0, __TO_NAME_DIFF_FROM_ACC 0, __TO_REAL_NAMES 0, __URI_NO_WWW 0, __URI_NS ' X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 03 Aug 2016 05:59:59 -0000 On Tue, 2 Aug 2016 08:45:07 -0600 Alan Somers wrote about Re: NFS on 10G interfaces still painfully slow: > > Is there anyone around here who can confirm that nfs can go faster > > over 10G links? > > Any hints for further tuning/debugging are greatly appreciated. AS> I can get 1GB/s over NFS on a 10G link, so it's not always slow. AS> There's probably something about your setup that's slowing it down. AS> What is your NFS client? This time, FreeBSD 10.3 on both client and server (to make debugging easier). AS> What benchmark are you using to measure that speed? Right now only very simple things like using dd from /dev/zero or copying large files. In my experience it is useless to go for more sophisticated benchmarks, if these simple things already don't work as expected. AS> Did you remember to start lockd and statd? Yes. AS> If you post your /etc/exports and the client's /etc/fstab, that might AS> reveal something. exports on the server side: V4: /mt-rear -sec=sys 192.168.1.11 /mt-right 192.168.1.11 -maproot=root /mt-rear -maproot=root 192.168.1.11 /mt-left 192.168.1.11 -maproot=root fstab on the client does not tell you anything, I still use commandline mounts during testing. This is what nfsstat -m will tell (V4 is not mounted right now): tom:/mt-rear on /net/mt-rear nfsv3,tcp,resvport,hard,cto,lockd,rdirplus,sec=sys,acdirmin=3,acdirmax=60,acregmin=5,acregmax=60,nametimeo=60,negnametimeo=60,rsize=65536,wsize=65536,readdirsize=65536,readahead=4,wcommitsize=50000000,timeout=120,retrans=2 tom:/mt-right on /net/mt-right nfsv3,tcp,resvport,hard,cto,lockd,rdirplus,sec=sys,acdirmin=3,acdirmax=60,acregmin=5,acregmax=60,nametimeo=60,negnametimeo=60,rsize=65536,wsize=65536,readdirsize=65536,readahead=1,wcommitsize=16777216,timeout=120,retrans=2 tom:/mt-left on /net/mt-left nfsv3,tcp,resvport,hard,cto,lockd,rdirplus,sec=sys,acdirmin=3,acdirmax=60,acregmin=5,acregmax=60,nametimeo=60,negnametimeo=60,rsize=65536,wsize=65536,readdirsize=65536,readahead=1,wcommitsize=16777216,timeout=120,retrans=2 This is what, e.g., dd gives: root@crest:~ # dd if=/dev/zero of=/net/mt-rear/Z bs=1024k count=1000 1000+0 records in 1000+0 records out 1048576000 bytes transferred in 1.403620 secs (747051194 bytes/sec) root@crest:~ # dd if=/dev/zero of=/net/mt-right/Z bs=1024k count=1000 1000+0 records in 1000+0 records out 1048576000 bytes transferred in 1.380546 secs (759537249 bytes/sec) And yes (before that question pops up :-), I'm using zfs on the server side, but I disabled syncing for testing purposes. cu Gerrit From owner-freebsd-net@freebsd.org Wed Aug 3 06:09:20 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B9455BADDEC for ; Wed, 3 Aug 2016 06:09:20 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: from mail-wm0-x236.google.com (mail-wm0-x236.google.com [IPv6:2a00:1450:400c:c09::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4D7261E08 for ; Wed, 3 Aug 2016 06:09:20 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: by mail-wm0-x236.google.com with SMTP id i5so319626414wmg.0 for ; Tue, 02 Aug 2016 23:09:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=ETZxRgMAgJFMVlwtfUHg2y8gkGjpuM69LqrF/+Aoe+E=; b=OH/StFitz9KUJTGxq+H6UdgFMIoRnjpqht1b8FLJ0PByALGfleS6NizKHz6qdiw0Jm LSGdI3Q78NvHS/mSgl8BpUXk/QX91v/66TuKZmwG4szz7a1ShffaZNQGzbPOBqhq13s0 cdwlR5uIkIFzeuc5UAcOHEalcBBHSmbPFrI9zhQEDn9kMHrpnpy0DsUeudmqCfLJfKMS 3UiqcG9xP3pX0MmIUvXqB3RFJ+duanPBzSRrJMiej4ZzGinJe/T9rAqHpeD20j+Uu8EA nvT47xQb+4tmuIxFsCYZy2NOCWONdt+pguGHWOrP9X9qcvJhelfj3QCe+LbksssAElhl PH0g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=ETZxRgMAgJFMVlwtfUHg2y8gkGjpuM69LqrF/+Aoe+E=; b=BYLKWhDxJ9wbhw4MAsODA89DrpqMuyouHfoEr1UKSiyXzu6g7S4QW9aCnNhH2tAyfs 2RmpxQ+bQKj6Fr5aL8pB8culVRryDM+qdp1MvFL6jpqORO3iIE7wTJGUoUW88y05Muqx kKfNH6dB48nMJQlxd4zPQFvz6ltMr6U46BfV84m9Ril0ok7O/gLr4fmlfMD58XITgPk6 GExBZcpLdDgRU2GCWztL4qa9er825C3XhDSbEJX392y1AHnDipZ/Hht7wjJ/w1+9oiqS RUo1TbJrNAyiFUcuZ6vZNKIPETgL2X7Ok0W5KcSDIPd+qOsLEuyqnueZmQpyTQrDXq9R m6lg== X-Gm-Message-State: AEkoout8RDDz/6FPUnC2lQDXsAATTryMhByCw5CVz3ZMTFKoKIbzMAEgLjWuls6N0AZImw== X-Received: by 10.28.223.139 with SMTP id w133mr23032943wmg.90.1470204558639; Tue, 02 Aug 2016 23:09:18 -0700 (PDT) Received: from macbook-air-de-benjamin-1.home (LFbn-1-7077-85.w90-116.abo.wanadoo.fr. [90.116.246.85]) by smtp.gmail.com with ESMTPSA id uo4sm5907581wjc.36.2016.08.02.23.09.17 for (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Tue, 02 Aug 2016 23:09:18 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: Unstable local network throughput From: Ben RUBSON In-Reply-To: <57A157D9.3040707@grosbein.net> Date: Wed, 3 Aug 2016 08:09:18 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: References: <3C0D892F-2BE8-4650-B9FC-93C8EE0443E1@gmail.com> <57A157D9.3040707@grosbein.net> To: freebsd-net@freebsd.org X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 03 Aug 2016 06:09:20 -0000 > On 03 Aug 2016, at 04:32, Eugene Grosbein wrote: >=20 > If you have gateway_enable=3D"YES" (sysctl net.inet.ip.forwarding=3D1) > then try to disable this forwarding setting and rerun your tests to = compare results. Thank you Eugene for this, but net.inet.ip.forwarding is disabled by = default and I did not enabled it. # sysctl net.inet.ip.forwarding net.inet.ip.forwarding: 0= From owner-freebsd-net@freebsd.org Wed Aug 3 06:55:02 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C64DCBABA0F for ; Wed, 3 Aug 2016 06:55:02 +0000 (UTC) (envelope-from gerrit.kuehn@aei.mpg.de) Received: from umail.aei.mpg.de (umail.aei.mpg.de [194.94.224.6]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7082617F2; Wed, 3 Aug 2016 06:55:01 +0000 (UTC) (envelope-from gerrit.kuehn@aei.mpg.de) Received: from mailgate.aei.mpg.de (mailgate.aei.mpg.de [194.94.224.5]) by umail.aei.mpg.de (Postfix) with ESMTP id 2C82020005B; Wed, 3 Aug 2016 08:55:00 +0200 (CEST) Received: from mailgate.aei.mpg.de (localhost [127.0.0.1]) by localhost (Postfix) with SMTP id 1DD5C406ADE; Wed, 3 Aug 2016 08:55:00 +0200 (CEST) Received: from intranet.aei.uni-hannover.de (ahin1.aei.uni-hannover.de [130.75.117.40]) by mailgate.aei.mpg.de (Postfix) with ESMTP id ECBA5406ADB; Wed, 3 Aug 2016 08:54:59 +0200 (CEST) Received: from arc.aei.uni-hannover.de ([130.75.117.1]) by intranet.aei.uni-hannover.de (IBM Domino Release 9.0.1FP5) with ESMTP id 2016080308545917-63351 ; Wed, 3 Aug 2016 08:54:59 +0200 Date: Wed, 3 Aug 2016 08:54:59 +0200 From: Gerrit =?ISO-8859-1?Q?K=FChn?= To: freebsd-net@freebsd.org Cc: Alan Somers Subject: Re: NFS on 10G interfaces still painfully slow Message-Id: <20160803085459.273a26404c4d37ee6b7e2b6f@aei.mpg.de> In-Reply-To: <31939_1470204015_57A1886F_31939_1565_1_20160803075954.35914021394d6c3f9b3802f3@aei.mpg.de> References: <20160802104929.a10602a4786c68b4547a45b9@aei.mpg.de> <31939_1470204015_57A1886F_31939_1565_1_20160803075954.35914021394d6c3f9b3802f3@aei.mpg.de> Organization: Max Planck Gesellschaft X-Mailer: Sylpheed 3.4.2 (GTK+ 2.24.22; amd64-portbld-freebsd10.0) Mime-Version: 1.0 X-MIMETrack: Itemize by SMTP Server on intranet/aei-hannover(Release 9.0.1FP5|November 22, 2015) at 03/08/2016 08:54:59, Serialize by Router on intranet/aei-hannover(Release 9.0.1FP5|November 22, 2015) at 03/08/2016 08:54:59, Serialize complete at 03/08/2016 08:54:59 X-TNEFEvaluated: 1 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=ISO-8859-1 X-PMX-Version: 6.0.2.2308539, Antispam-Engine: 2.7.2.2107409, Antispam-Data: 2016.8.3.64517 X-PerlMx-Spam: Gauge=IIIIIIII, Probability=8%, Report=' HTML_00_01 0.05, HTML_00_10 0.05, MIME_LOWER_CASE 0.05, BODYTEXTP_SIZE_3000_LESS 0, BODY_SIZE_1000_LESS 0, BODY_SIZE_2000_LESS 0, BODY_SIZE_5000_LESS 0, BODY_SIZE_7000_LESS 0, BODY_SIZE_700_799 0, IN_REP_TO 0, LEGITIMATE_NEGATE 0, LEGITIMATE_SIGNS 0, MSG_THREAD 0, MULTIPLE_REAL_RCPTS 0, NO_URI_HTTPS 0, REFERENCES 0, __ANY_URI 0, __BOUNCE_CHALLENGE_SUBJ 0, __BOUNCE_NDR_SUBJ_EXEMPT 0, __C230066_P5 0, __CC_NAME 0, __CC_NAME_DIFF_FROM_ACC 0, __CC_REAL_NAMES 0, __CT 0, __CTE 0, __CT_TEXT_PLAIN 0, __HAS_CC_HDR 0, __HAS_FROM 0, __HAS_MSGID 0, __HAS_X_MAILER 0, __IN_REP_TO 0, __MIME_TEXT_ONLY 0, __MIME_VERSION 0, __REFERENCES 0, __SANE_MSGID 0, __SUBJ_ALPHA_END 0, __SUBJ_ALPHA_NEGATE 0, __TO_MALFORMED_2 0, __TO_NO_NAME 0, __URI_NO_WWW 0, __URI_NS ' X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 03 Aug 2016 06:55:02 -0000 On Wed, 3 Aug 2016 07:59:54 +0200 Gerrit K=FChn wrote about Re: NFS on 10G interfaces still painfully slow: GK> 1048576000 bytes transferred in 1.403620 secs (747051194 bytes/sec) GK> 1048576000 bytes transferred in 1.380546 secs (759537249 bytes/sec) Argh! ;-) Obviously, I cannot read and miscounted the number of digits. I should have read the seconds instead. 750MB/s is still not the maximum, but totally fine for my purposes here. Reverting step-by-step all the things I tried yesterday, it looks like disabling the sync on the zfs volume gave the biggest performance boost, and I simply overlooked the extra digit when it went from 25MB/s or so to 750MB/s. Sorry for the noise... cu Gerrit From owner-freebsd-net@freebsd.org Wed Aug 3 14:21:26 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EF35BBAD987 for ; Wed, 3 Aug 2016 14:21:26 +0000 (UTC) (envelope-from galtsev@kicp.uchicago.edu) Received: from cosmo.uchicago.edu (cosmo.uchicago.edu [128.135.70.90]) by mx1.freebsd.org (Postfix) with ESMTP id AF08C1432; Wed, 3 Aug 2016 14:21:26 +0000 (UTC) (envelope-from galtsev@kicp.uchicago.edu) Received: by cosmo.uchicago.edu (Postfix, from userid 48) id A467DCB8CB5; Wed, 3 Aug 2016 08:49:57 -0500 (CDT) Received: from 128.135.52.6 (SquirrelMail authenticated user valeri) by cosmo.uchicago.edu with HTTP; Wed, 3 Aug 2016 08:49:57 -0500 (CDT) Message-ID: <38638.128.135.52.6.1470232197.squirrel@cosmo.uchicago.edu> In-Reply-To: <20160803075954.35914021394d6c3f9b3802f3@aei.mpg.de> References: <20160802104929.a10602a4786c68b4547a45b9@aei.mpg.de> <20160803075954.35914021394d6c3f9b3802f3@aei.mpg.de> Date: Wed, 3 Aug 2016 08:49:57 -0500 (CDT) Subject: Re: NFS on 10G interfaces still painfully slow From: "Valeri Galtsev" To: Gerrit =?iso-8859-1?Q?K=FChn?= Cc: "Alan Somers" , "FreeBSD Net" Reply-To: galtsev@kicp.uchicago.edu User-Agent: SquirrelMail/1.4.8-5.el5.centos.7 MIME-Version: 1.0 Content-Type: text/plain;charset=iso-8859-1 Content-Transfer-Encoding: 8bit X-Priority: 3 (Normal) Importance: Normal X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 03 Aug 2016 14:21:27 -0000 On Wed, August 3, 2016 12:59 am, Gerrit Kühn wrote: > On Tue, 2 Aug 2016 08:45:07 -0600 Alan Somers wrote > about Re: NFS on 10G interfaces still painfully slow: > >> > Is there anyone around here who can confirm that nfs can go faster >> > over 10G links? >> > Any hints for further tuning/debugging are greatly appreciated. > > AS> I can get 1GB/s over NFS on a 10G link, so it's not always slow. > AS> There's probably something about your setup that's slowing it down. > AS> What is your NFS client? > > This time, FreeBSD 10.3 on both client and server (to make debugging > easier). > > AS> What benchmark are you using to measure that speed? > > Right now only very simple things like using dd from /dev/zero or copying > large files. When I had trouble (too slow data throughput, never mind on what medium) using dd - it was: not specifying bs=[big number] in dd command. It turned out without that dd sends stuff down the pipe in very small chunks and wouldn't send next chunk till acknowledgement on previous is received. Imagine hard drive wrighting 4 kb at a time send 1 byte at a time, or RAID expected to write 8 x 64 kB striped in parallel sent 1 byte at a time. In your case this [big number] ideally should be equal or slightly smaller than size of jumbo packet of you net connection. Just speculating (remembered really trivial thing _I_ hit myself ;-) Thanks. Valeri > In my experience it is useless to go for more sophisticated > benchmarks, if these simple things already don't work as expected. > > AS> Did you remember to start lockd and statd? > > Yes. > > AS> If you post your /etc/exports and the client's /etc/fstab, that might > AS> reveal something. > > exports on the server side: > > V4: /mt-rear -sec=sys 192.168.1.11 > /mt-right 192.168.1.11 -maproot=root > /mt-rear -maproot=root 192.168.1.11 > /mt-left 192.168.1.11 -maproot=root > > > fstab on the client does not tell you anything, I still use commandline > mounts during testing. This is what nfsstat -m will tell (V4 is not > mounted right now): > > tom:/mt-rear on /net/mt-rear > nfsv3,tcp,resvport,hard,cto,lockd,rdirplus,sec=sys,acdirmin=3,acdirmax=60,acregmin=5,acregmax=60,nametimeo=60,negnametimeo=60,rsize=65536,wsize=65536,readdirsize=65536,readahead=4,wcommitsize=50000000,timeout=120,retrans=2 > tom:/mt-right on /net/mt-right > nfsv3,tcp,resvport,hard,cto,lockd,rdirplus,sec=sys,acdirmin=3,acdirmax=60,acregmin=5,acregmax=60,nametimeo=60,negnametimeo=60,rsize=65536,wsize=65536,readdirsize=65536,readahead=1,wcommitsize=16777216,timeout=120,retrans=2 > tom:/mt-left on /net/mt-left > nfsv3,tcp,resvport,hard,cto,lockd,rdirplus,sec=sys,acdirmin=3,acdirmax=60,acregmin=5,acregmax=60,nametimeo=60,negnametimeo=60,rsize=65536,wsize=65536,readdirsize=65536,readahead=1,wcommitsize=16777216,timeout=120,retrans=2 > > > This is what, e.g., dd gives: > > root@crest:~ # dd if=/dev/zero of=/net/mt-rear/Z bs=1024k count=1000 > 1000+0 records in > 1000+0 records out > 1048576000 bytes transferred in 1.403620 secs (747051194 bytes/sec) > > root@crest:~ # dd if=/dev/zero of=/net/mt-right/Z bs=1024k > count=1000 1000+0 records in > 1000+0 records out > 1048576000 bytes transferred in 1.380546 secs (759537249 bytes/sec) > > > And yes (before that question pops up :-), I'm using zfs on the server > side, but I disabled syncing for testing purposes. > > > cu > Gerrit > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > ++++++++++++++++++++++++++++++++++++++++ Valeri Galtsev Sr System Administrator Department of Astronomy and Astrophysics Kavli Institute for Cosmological Physics University of Chicago Phone: 773-702-4247 ++++++++++++++++++++++++++++++++++++++++ From owner-freebsd-net@freebsd.org Wed Aug 3 16:57:33 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D19B5BAE811 for ; Wed, 3 Aug 2016 16:57:33 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: from mail-wm0-x22e.google.com (mail-wm0-x22e.google.com [IPv6:2a00:1450:400c:c09::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 65C6D1E8B for ; Wed, 3 Aug 2016 16:57:33 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: by mail-wm0-x22e.google.com with SMTP id f65so455653832wmi.0 for ; Wed, 03 Aug 2016 09:57:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=BNSdKKh+nEfRACYOTUr9EcrEmfFWi9OcYhB4vODZwho=; b=lkC5q7NMpp9ZbqHAOWdcN8YTL212NDycmTZP+nmhwoAEzwoRrUAxt5Z7ZlourmcvI4 lGYHFBAIZkp5DEuLgh4SBtoOSQ69wOjhB/gPqv6VtLz/bxXv4gdsdTLeEiVJsRrPO/P3 mJKJ54FYomIewOLYicgJ5y8brWLsGtU33zBW5HrG1melfb93IEJtUqys8kfdiUfgdjBW z4OnMUuA7Jukkc+MbfxmMq3tRLyQkcji62EJoKjxLShHTAEH88C3p/CcLFnBakcSKGPM 8iDsdbpko72NQW0K19fnxAOcnKBgJ5QMV6va4PiN2mawN6j1wAnr/b/RskJ44z/wuRz4 PBPw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=BNSdKKh+nEfRACYOTUr9EcrEmfFWi9OcYhB4vODZwho=; b=BWpDMMHA1xTjXNDHDMntnXclEA68wGG0iqyXoEnUlZimIRXgnYXZlfv4AT6fR0B6yq gyNyc4Rcl+ZwUZyVKp6o6j1EUOimJ1Kpx/L5NivTCtE92pFWAXRwpaK8AsTBiI88X7Uk VZ5oibK1+lHvCknjL/R7OVm7GYAYTIB9cI6rpBaznR3necjIO3RiZ2lMqON5Y1SgieYL uPkkvYyWKtfY4lekkXm4dFXtaYjZov2cJn12jfKhzb3e87DiBG1dSxo4l+g19w5/AgJ2 LPKUtfcjQv3wzulbMGhE15vt41NAfH8LuXjYcrpPLurRzUssKyhZhO7abwL2DTXsL+aZ cyrg== X-Gm-Message-State: AEkoousUMl6FtsoSFMlvRJx9QIACONtA5n5DIuzIpzsW99ADW+7/zzbQC04Vl0jNqFGeMg== X-Received: by 10.194.81.137 with SMTP id a9mr62610534wjy.106.1470243451430; Wed, 03 Aug 2016 09:57:31 -0700 (PDT) Received: from macbook-air-de-benjamin-1.home (LFbn-1-7077-85.w90-116.abo.wanadoo.fr. [90.116.246.85]) by smtp.gmail.com with ESMTPSA id yz6sm8587043wjb.35.2016.08.03.09.57.29 for (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 03 Aug 2016 09:57:30 -0700 (PDT) Content-Type: text/plain; charset=windows-1252 Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: Unstable local network throughput From: Ben RUBSON In-Reply-To: Date: Wed, 3 Aug 2016 18:57:28 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: <465615D0-C1CB-42F0-8964-AFAAB5C05E26@gmail.com> References: <3C0D892F-2BE8-4650-B9FC-93C8EE0443E1@gmail.com> To: freebsd-net@freebsd.org X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 03 Aug 2016 16:57:33 -0000 > On 02 Aug 2016, at 21:35, Hans Petter Selasky wrote: >=20 > The CX-3 driver doesn't bind the worker threads to specific CPU cores = by default, so if your CPU has more than one so-called numa, you'll end = up that the bottle-neck is the high-speed link between the CPU cores and = not the card. A quick and dirty workaround is to "cpuset" iperf and the = interrupt and taskqueue threads to specific CPU cores. Hans Petter, I'm testing "cpuset" and sometimes get better results, I'm still trying = to get the best. I've cpuset iperf & Mellanox interrupts, but what do you mean by = taskqueue threads ? Thank U ! Ben= From owner-freebsd-net@freebsd.org Wed Aug 3 17:57:51 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1FFB2BADBB3 for ; Wed, 3 Aug 2016 17:57:51 +0000 (UTC) (envelope-from hps@selasky.org) Received: from mail.turbocat.net (heidi.turbocat.net [88.198.202.214]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id DDDA01BC1 for ; Wed, 3 Aug 2016 17:57:50 +0000 (UTC) (envelope-from hps@selasky.org) Received: from laptop015.home.selasky.org (unknown [62.141.129.119]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.turbocat.net (Postfix) with ESMTPSA id 1B39C1FE022; Wed, 3 Aug 2016 19:57:48 +0200 (CEST) Subject: Re: Unstable local network throughput To: Ben RUBSON , freebsd-net@freebsd.org References: <3C0D892F-2BE8-4650-B9FC-93C8EE0443E1@gmail.com> <465615D0-C1CB-42F0-8964-AFAAB5C05E26@gmail.com> From: Hans Petter Selasky Message-ID: <8dc4bad9-a50a-42a5-dbb4-9f362c4ffbb8@selasky.org> Date: Wed, 3 Aug 2016 20:02:06 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:45.0) Gecko/20100101 Thunderbird/45.0 MIME-Version: 1.0 In-Reply-To: <465615D0-C1CB-42F0-8964-AFAAB5C05E26@gmail.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 03 Aug 2016 17:57:51 -0000 On 08/03/16 18:57, Ben RUBSON wrote: > taskqueue threads ? The mlx4 send and receive queues have each their set of taskqueues. Look in output from "ps auxww". --HPS From owner-freebsd-net@freebsd.org Wed Aug 3 18:41:53 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DB1BABAE88F for ; Wed, 3 Aug 2016 18:41:53 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: from mail-wm0-x235.google.com (mail-wm0-x235.google.com [IPv6:2a00:1450:400c:c09::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6A08819D4 for ; Wed, 3 Aug 2016 18:41:53 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: by mail-wm0-x235.google.com with SMTP id f65so458804591wmi.0 for ; Wed, 03 Aug 2016 11:41:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=tLUkS6TYyTT1fIw5V2QiSZ2m5ohl6mTbaxiIrL+Baek=; b=j7Fi8vtV4pwYY5zXQD9RKgAmcR/10Uxp+oUq8mTZhQ+V3V/L5LAlpEojiDmyYTA+4e SLlYQXeFApwSOVvK8mcsFxnSAb4QrcbnRuJKUtWTWuqfcL/zlWdvAHPa3GVj8fVGRiQ9 M2nMCiPSZ1gWTE0iHLjmUAwlry3KK7cGs7PddXIqr8s7+nlhemPhzLsIaSZ57N3AolOD v7m906wVOPMDQFC24slwCsAkjaNlQ2a8zMRKSA5cACbg7FcY3OITpFpF203YTD5nRTYi J7taKn83IfV3yj4n3shNoyqPP8BOjoQLO5VhXcPYuPnbdwk5nteM0EG+XyerT/Cc0qj1 1CBA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=tLUkS6TYyTT1fIw5V2QiSZ2m5ohl6mTbaxiIrL+Baek=; b=c8Y+XaJmZfeHRjak8SUWPQkLlWEsUCzYWH/XlwKHYptbmVDdS6q2TKI7geYcg7HvYG NR0N7w4BBbYy7eAOh2wiImmQrQboOGGjuxjo2YlE8EPV5pODKWwym/sZHCoIByQiIWrW hdaZHlq80tcfIiPkNp/8cQd5QHab8PU67zYrzDZu67zuozlhcoQ1YtYEap6WCins//nB Jloh7MI1qxQBkI/JBhcBbSjTlu1DNH/NlCeoF62NYrkP0uVXVwzQ1Zugsjj4MgtAUFP4 150ZARPIDLbtmPej+BhnfcBur7N+raQ2+XAHhBDE/CqgU0dmm5Kk+lW1GZTRM/NOjHtf cKbA== X-Gm-Message-State: AEkoouvCIO/eOambFyUC2KbRjGKEl1XG1sK4NkdchhDEYR5SKHZEJ2i+D38ubuEkqXmIGA== X-Received: by 10.194.185.202 with SMTP id fe10mr60117321wjc.93.1470249711320; Wed, 03 Aug 2016 11:41:51 -0700 (PDT) Received: from macbook-air-de-benjamin-1.home (LFbn-1-7077-85.w90-116.abo.wanadoo.fr. [90.116.246.85]) by smtp.gmail.com with ESMTPSA id r13sm9375291wmf.12.2016.08.03.11.41.50 for (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 03 Aug 2016 11:41:50 -0700 (PDT) Content-Type: text/plain; charset=windows-1252 Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: Unstable local network throughput From: Ben RUBSON In-Reply-To: <8dc4bad9-a50a-42a5-dbb4-9f362c4ffbb8@selasky.org> Date: Wed, 3 Aug 2016 20:41:49 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: References: <3C0D892F-2BE8-4650-B9FC-93C8EE0443E1@gmail.com> <465615D0-C1CB-42F0-8964-AFAAB5C05E26@gmail.com> <8dc4bad9-a50a-42a5-dbb4-9f362c4ffbb8@selasky.org> To: freebsd-net@freebsd.org X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 03 Aug 2016 18:41:54 -0000 > On 03 Aug 2016, at 20:02, Hans Petter Selasky wrote: >=20 > The mlx4 send and receive queues have each their set of taskqueues. = Look in output from "ps auxww". I can't find them, I even unloaded/reloaded the driver in order to catch = the differences, but I did not found any relevant process. Here are the process I have when driver is loaded (I removed my own = process lines) : # ps auxxw USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME = COMMAND root 11 2398.1 0.0 0 384 - RL Mon10pm 65969:09.19 [idle] root 0 0.0 0.0 0 8288 - DLs Mon10pm 4:59.31 = [kernel] root 1 0.0 0.0 9492 872 - ILs Mon10pm 0:00.04 = /sbin/init -- root 2 0.0 0.0 0 96 - DL Mon10pm 0:00.82 [cam] root 3 0.0 0.0 0 176 - DL Mon10pm 0:17.88 = [zfskern] root 4 0.0 0.0 0 16 - DL Mon10pm 0:00.00 = [sctp_iterator] root 5 0.0 0.0 0 16 - DL Mon10pm 0:00.75 = [enc_daemon0] root 6 0.0 0.0 0 16 - DL Mon10pm 0:00.50 = [enc_daemon1] root 7 0.0 0.0 0 16 - DL Mon10pm 0:00.05 = [enc_daemon2] root 8 0.0 0.0 0 16 - DL Mon10pm 0:00.05 = [enc_daemon3] root 9 0.0 0.0 0 16 - DL Mon10pm 0:00.00 = [g_mirror swap] root 10 0.0 0.0 0 16 - DL Mon10pm 0:00.00 = [audit] root 12 0.0 0.0 0 1408 - WL Mon10pm 186:01.05 [intr] root 13 0.0 0.0 0 48 - DL Mon10pm 0:05.24 [geom] root 14 0.0 0.0 0 16 - DL Mon10pm 1:07.19 = [rand_harvestq] root 15 0.0 0.0 0 160 - DL Mon10pm 0:08.22 [usb] root 16 0.0 0.0 0 32 - DL Mon10pm 0:00.23 = [pagedaemon] root 17 0.0 0.0 0 16 - DL Mon10pm 0:00.00 = [vmdaemon] root 18 0.0 0.0 0 16 - DL Mon10pm 0:00.00 = [pagezero] root 19 0.0 0.0 0 16 - DL Mon10pm 0:00.12 = [bufdaemon] root 20 0.0 0.0 0 16 - DL Mon10pm 0:00.13 = [vnlru] root 21 0.0 0.0 0 16 - DL Mon10pm 2:13.02 = [syncer] root 124 0.0 0.0 12360 1736 - Is Mon10pm 0:00.00 = adjkerntz -i root 618 0.0 0.0 13628 4868 - Ss Mon10pm 0:00.03 = /sbin/devd From owner-freebsd-net@freebsd.org Thu Aug 4 07:55:44 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 13446BAE6D2 for ; Thu, 4 Aug 2016 07:55:44 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-io0-x22e.google.com (mail-io0-x22e.google.com [IPv6:2607:f8b0:4001:c06::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id CF7751E99 for ; Thu, 4 Aug 2016 07:55:43 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: by mail-io0-x22e.google.com with SMTP id b62so264871431iod.3 for ; Thu, 04 Aug 2016 00:55:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc; bh=YsnkyOdyj4Tu6CsuyWJepsTsMQ/6neNGMiufv8XzkFw=; b=DXOdTfDbhVGV8Iczly7wwqMw/iKBG1OLLR01zSkw1QRWwkGZlZGq5kZiDO1xrepqFq fs8w3XeZkQlv7gzmeQ3uVDWKpza/kKKFxCVxzcqLVCwelsGLz+8OByKtSKbU4X/eynEV Za9zZSi1vX7sb3G0ltuV4sgkePk1/LqyAL/ucyUMbOA+D43SxXLMY7OG+TksW/YZ4jPx 7RR9qfzvZQvjMirHY0EymPsQwDyMQsP1S7mN39dWtR6bf5JIs4a1aAw+i7U7rQOvMhMU c7nuvKVKXG9wTU/4j3wL3eMRhmhaVoYPm7MqPbmnTrmFg2wuaXXJN6HWyzyxZ0gh81vj GLig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc; bh=YsnkyOdyj4Tu6CsuyWJepsTsMQ/6neNGMiufv8XzkFw=; b=KUgb1HFUOjqFnVWZEA6edjyqsQa35wjad6pwjy34cS7hMhDhgVPU8eqO2eSYe1E7kf Dr1iWcvXknfzir8oxY26X5CF7I6hNe965hyJxhjXsPM8CUfcT0v5JMZ3RJ4yHioUw6Fk ep/+Sch+GUDRZuKyW4I5IgR0i4ck0/rWWXhgoXA7C42QSKL5zIDDFjMqiFF5Lx38q2fj dTnFvyq8N/3FHMr1YqFBqIh5GbjcSjEJzTOdunbMc2mGZxRkW6pqr3h9cwRDpNqLXHao naXgXOni68lY95db/qSJbnPZ769Mj1zMoIQWusd/roZMaKSygNsQvcfdgWkrJyZD98hY 838w== X-Gm-Message-State: AEkoouvGB83KXm5pdP+MBTld7ekjHdZnL426DfjE6/d/YrWRVAPS76HN/GQdujMdcGLUpcXdMzMQjtUOeHltOw== X-Received: by 10.107.15.229 with SMTP id 98mr27323125iop.123.1470297343290; Thu, 04 Aug 2016 00:55:43 -0700 (PDT) MIME-Version: 1.0 Sender: adrian.chadd@gmail.com Received: by 10.36.141.129 with HTTP; Thu, 4 Aug 2016 00:55:42 -0700 (PDT) In-Reply-To: References: <306af514-70ff-f3bf-5b4f-da7ac1ec6580@cs.duke.edu> From: Adrian Chadd Date: Thu, 4 Aug 2016 00:55:42 -0700 X-Google-Sender-Auth: i2cdOmHcSClTl_-qm4hP9DLTv_A Message-ID: Subject: Re: proposal: splitting NIC RSS up from stack RSS To: Sepherosa Ziehau Cc: Andrew Gallatin , FreeBSD Net Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Aug 2016 07:55:44 -0000 [snip] I've updated the wiki with this TODO list. How's it look? https://wiki.freebsd.org/NetworkRSS -adrian From owner-freebsd-net@freebsd.org Thu Aug 4 07:56:53 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2B1C2BAE774; Thu, 4 Aug 2016 07:56:53 +0000 (UTC) (envelope-from kmacybsd@gmail.com) Received: from mail-it0-x236.google.com (mail-it0-x236.google.com [IPv6:2607:f8b0:4001:c0b::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id EC1FB1F77; Thu, 4 Aug 2016 07:56:52 +0000 (UTC) (envelope-from kmacybsd@gmail.com) Received: by mail-it0-x236.google.com with SMTP id u186so316976897ita.0; Thu, 04 Aug 2016 00:56:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:from:date:message-id:subject:to; bh=d5/uh7CGZrBIoHGOke4fIFhM+qp9Zgzigl1Ijpw2tUk=; b=KtpZyNkII2Q2VojdCe4bFDjQOIuAl2cLn/XtKKMQL/zX4pQv0Uxej1EXz541Ru4RHR /NtAodMzmQnyVzgXTJKNKbfdo9sMKRa/BO84S4WHJVTvpT35h3UHu7KCAq5xiC30yKsk Bkvm79uL8uJiBjTBCCEs/u450ophJlyGthtZMZ0owFE4q3EDaFlLq3EWFmSl3a4w1gHl cLVN9EB6hFJBxGYHIB1g7cxpXMuRzApUsnzDe60tXKHkluoi5EZg8D08No6LMvvSDCZe HL146XHcmOlCv77UjWY47cjnyt/+H67lBCl6ElO3IoDad6Kln++Ej4+CpplWKTmQGDai sRqQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:from:date:message-id:subject :to; bh=d5/uh7CGZrBIoHGOke4fIFhM+qp9Zgzigl1Ijpw2tUk=; b=AtC9zpR7lrH9wCCpU/Jlqj/eY7rAIgOILWYZjMcFT6bPEUWUTtHYQuT+XhSw6SJ2Yz 8p3Kz/v/Ptag18oXEc0LIxLDxEyLKPY9//dWTCaAChxP5236oC7Z9Q2z1+exV0Md12QG MA9fruE+E4N4QOaP9jxn2mJ9/gvTRFzUpoF7sn+ygoanFMDkx+ti17Dbf5DlB19uH33C Hriq8iT+V0lv5b2xQ2Y3RhwV5MvGizGpssO5k0Igu2uYwpNN+aOolZ7Zv6SUcwZkKeWJ XMQ/vl4LwUVwZ6xzql1jdKz3aPX5fTYtgCQabba2kcC4LttFmRAFUlN9m6n6r+autINk tHkQ== X-Gm-Message-State: AEkoouvaRs0mfs8EWmPyr/1Yv6r8AJPnJgJSgv7dHvACrznFH73YI6lId0saKjdPBPgX8VcXX9kOSGq25LND0g== X-Received: by 10.36.214.209 with SMTP id o200mr29059783itg.56.1470297411987; Thu, 04 Aug 2016 00:56:51 -0700 (PDT) MIME-Version: 1.0 Sender: kmacybsd@gmail.com Received: by 10.107.143.11 with HTTP; Thu, 4 Aug 2016 00:56:51 -0700 (PDT) From: "K. Macy" Date: Thu, 4 Aug 2016 00:56:51 -0700 X-Google-Sender-Auth: 23JC8GXRsyTBAiDrNqJifBG3vxw Message-ID: Subject: iwm load panic To: FreeBSD Current , "freebsd-net@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Aug 2016 07:56:53 -0000 I get this panic periodically at iwm load time: (kgdb) p ic->ic_tq value has been optimized out (kgdb) down #12 taskqueue_drain (queue=0x0, task=0xfffffe004fc17150) at /usr/home/mmacy/drm-next-4.6/sys/kern/subr_taskqueue.c:554 554 TQ_LOCK(queue); (kgdb) bt #0 __curthread () at ./machine/pcpu.h:221 #1 doadump (textdump=0) at /usr/home/mmacy/drm-next-4.6/sys/kern/kern_shutdown.c:298 #2 0xffffffff82703ac9 in vt_kms_postswitch (arg=) at /usr/home/mmacy/drm-next-4.6/sys/modules/drm2/drm2/../../../dev/drm2/linux_fb.c:82 #3 0xffffffff8093fb55 in vt_window_switch (vw=0xffffffff817e87d0 ) at /usr/home/mmacy/drm-next-4.6/sys/dev/vt/vt_core.c:540 #4 0xffffffff8093c9b0 in vtterm_cngrab (tm=) at /usr/home/mmacy/drm-next-4.6/sys/dev/vt/vt_core.c:1465 #5 0xffffffff80a78f12 in cngrab () at /usr/home/mmacy/drm-next-4.6/sys/kern/kern_cons.c:368 #6 0xffffffff80ae88c6 in vpanic (fmt=0xffffffff810f771b "%s", ap=0xfffffe0461098e70) at /usr/home/mmacy/drm-next-4.6/sys/kern/kern_shutdown.c:745 #7 0xffffffff80ae87b3 in panic (fmt=) at /usr/home/mmacy/drm-next-4.6/sys/kern/kern_shutdown.c:690 #8 0xffffffff80fc0361 in trap_fatal (frame=0xfffffe0461099170, eva=100) at /usr/home/mmacy/drm-next-4.6/sys/amd64/amd64/trap.c:841 #9 0xffffffff80fc0553 in trap_pfault (frame=0xfffffe0461099170, usermode=0) at /usr/home/mmacy/drm-next-4.6/sys/amd64/amd64/trap.c:691 #10 0xffffffff80fbfafc in trap (frame=0xfffffe0461099170) at /usr/home/mmacy/drm-next-4.6/sys/amd64/amd64/trap.c:442 #11 #12 taskqueue_drain (queue=0x0, task=0xfffffe004fc17150) at /usr/home/mmacy/drm-next-4.6/sys/kern/subr_taskqueue.c:554 #13 0xffffffff827823bb in ieee80211_draintask (ic=, task=0xfffffe004fc17150) at /usr/home/mmacy/drm-next-4.6/sys/net80211/ieee80211_var.h:796 #14 iwm_detach_local (sc=, do_net80211=) at /usr/home/mmacy/drm-next-4.6/sys/modules/iwm/../../dev/iwm/if_iwm.c:6121 #15 0xffffffff80b21e8c in run_interrupt_driven_config_hooks () at /usr/home/mmacy/drm-next-4.6/sys/kern/subr_autoconf.c:118 #16 0xffffffff80b21ccb in config_intrhook_establish (hook=) at /usr/home/mmacy/drm-next-4.6/sys/kern/subr_autoconf.c:182 #17 0xffffffff82781d03 in iwm_attach (dev=) at /usr/home/mmacy/drm-next-4.6/sys/modules/iwm/../../dev/iwm/if_iwm.c:5825 #18 0xffffffff80b26340 in DEVICE_ATTACH (dev=0xfffff8000744f700) at ./device_if.h:180 #19 device_attach (dev=0xfffff8000744f700) at /usr/home/mmacy/drm-next-4.6/sys/kern/subr_bus.c:2900 #20 0xffffffff8071730d in pci_driver_added (dev=, driver=) at /usr/home/mmacy/drm-next-4.6/sys/dev/pci/pci.c:4330 #21 0xffffffff80b23e9d in BUS_DRIVER_ADDED (_dev=0xfffff8000744f800, _driver=0xffffffff82790c08 ) at ./bus_if.h:204 #22 devclass_driver_added (dc=, driver=) at /usr/home/mmacy/drm-next-4.6/sys/kern/subr_bus.c:1099 #23 0xffffffff80b23e02 in devclass_add_driver (dc=, driver=, pass=, dcp=) at /usr/home/mmacy/drm-next-4.6/sys/kern/subr_bus.c:1172 #24 0xffffffff80ac2300 in module_register_init (arg=) at /usr/home/mmacy/drm-next-4.6/sys/kern/kern_module.c:123 #25 0xffffffff80ab3f14 in linker_file_sysinit (lf=) at /usr/home/mmacy/drm-next-4.6/sys/kern/kern_linker.c:234 #26 linker_load_file (filename=, result=) at /usr/home/mmacy/drm-next-4.6/sys/kern/kern_linker.c:434 #27 linker_load_module (kldname=, modname=0xfffff8000f0a8000 "if_iwm", parent=, verinfo=, lfpp=) at /usr/home/mmacy/drm-next-4.6/sys/kern/kern_linker.c:2024 #28 0xffffffff80ab5d48 in kern_kldload (td=, file=, fileid=0xfffffe0461099874) at /usr/home/mmacy/drm-next-4.6/sys/kern/kern_linker.c:1041 #29 0xffffffff80ab5eab in sys_kldload (td=0xfffff800966f1500, uap=) at /usr/home/mmacy/drm-next-4.6/sys/kern/kern_linker.c:1067 #30 0xffffffff80fc0cc8 in syscallenter (td=, sa=) at /usr/home/mmacy/drm-next-4.6/sys/amd64/amd64/../../kern/subr_syscall.c:135 #31 amd64_syscall (td=, traced=0) at /usr/home/mmacy/drm-next-4.6/sys/amd64/amd64/trap.c:942 #32 #33 0x000000080086d38a in ?? () From owner-freebsd-net@freebsd.org Thu Aug 4 07:59:54 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 47F62BAE9C7; Thu, 4 Aug 2016 07:59:54 +0000 (UTC) (envelope-from hps@selasky.org) Received: from mail.turbocat.net (mail.turbocat.net [IPv6:2a01:4f8:d16:4514::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 14B0F14A9; Thu, 4 Aug 2016 07:59:54 +0000 (UTC) (envelope-from hps@selasky.org) Received: from laptop015.home.selasky.org (unknown [62.141.129.119]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.turbocat.net (Postfix) with ESMTPSA id 35ED41FE022; Thu, 4 Aug 2016 09:59:52 +0200 (CEST) Subject: Re: iwm load panic To: "K. Macy" , FreeBSD Current , "freebsd-net@freebsd.org" References: From: Hans Petter Selasky Message-ID: Date: Thu, 4 Aug 2016 10:04:10 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:45.0) Gecko/20100101 Thunderbird/45.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Aug 2016 07:59:54 -0000 On 08/04/16 09:56, K. Macy wrote: > #12 taskqueue_drain (queue=0x0, task=0xfffffe004fc17150) at Hi, Looks like a NULL pointer, queue=NULL --HPS From owner-freebsd-net@freebsd.org Thu Aug 4 08:10:50 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 37CCCBAC089; Thu, 4 Aug 2016 08:10:50 +0000 (UTC) (envelope-from kmacybsd@gmail.com) Received: from mail-it0-x22b.google.com (mail-it0-x22b.google.com [IPv6:2607:f8b0:4001:c0b::22b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 024541C8C; Thu, 4 Aug 2016 08:10:50 +0000 (UTC) (envelope-from kmacybsd@gmail.com) Received: by mail-it0-x22b.google.com with SMTP id j124so256633711ith.1; Thu, 04 Aug 2016 01:10:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc; bh=Fe7bJYFQ4S0gwbi/c6mBmabAM7a7dCE3VRhM7RA66Eg=; b=HoWTpm4ReE7CEL44h3AXVG0J9rz4FnHMc4PnUayovHG5qQxDLNRDGEkQkCxUd4GJ7O Qmvrincq/LP/bF9NA4ENO4Mtu/1OwLB2zI5GKLBa8/J/UYUPeL4ZbSVcGJHB/cKjNo8A wsC4fOS3om6iPBgU8KncQCtJE2T5etBSX6eaVQBve1x/5sXLRK+h8p1G+O/jMtXgbrf1 ouTtSINk70A31vFeDuUTwYFzXIT036wwuUpRZ6CpYyqz+xavEYkKVgYuy71yjVqmUf4g DGwoTO56X+bQ0DEWb8F83giplXCe3qXyf2aLpV3yGwlGYnGASBQYAtfjWHlbB8Z9sNb7 8X9g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc; bh=Fe7bJYFQ4S0gwbi/c6mBmabAM7a7dCE3VRhM7RA66Eg=; b=EGjpSvYMwJ09968z2jKGeu++fC9+MLBL2m8orMSWdo8nS5O48RPnuvMAgmlrbQwLF5 lJsT1XdsiS0KYHKKpJkLULrJ0HqQOMFp6rwPwQgUFMJXYsJ6pnHWvkHU1Ety+m+8ZL1T 6laom0hirwIO1E6Mvn7NZ5Sxqku4KUgsThULl3ZzheXlJBhAyJxNra6XpFlh7D1KKShJ RCQUxNz7/R4Zo4NdoM/fomCrtgJLSXLqYaFoIo/o5Tj7K8lLn1tXgTwhNX4o9p0qHew7 oAdKcTzmsaRuV6dgcj9guNXJwGaRu4cNRo84xlktq4mh0Q77TMpYi86x4cm08yn2nJMp Ni9w== X-Gm-Message-State: AEkoouuUixaM0Yma7mPE/csT/6ZRWQCtzwMZnDz1zTzxxLc/6NPd8Bky67JwGH9GDURYvvyC2e//I34igB19gQ== X-Received: by 10.36.14.68 with SMTP id 65mr34277334ite.99.1470298249497; Thu, 04 Aug 2016 01:10:49 -0700 (PDT) MIME-Version: 1.0 Sender: kmacybsd@gmail.com Received: by 10.107.143.11 with HTTP; Thu, 4 Aug 2016 01:10:49 -0700 (PDT) In-Reply-To: References: From: "K. Macy" Date: Thu, 4 Aug 2016 01:10:49 -0700 X-Google-Sender-Auth: Biqd4xiepADdGr0OGr6WpKh-6WA Message-ID: Subject: Re: iwm load panic To: Hans Petter Selasky Cc: FreeBSD Current , "freebsd-net@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Aug 2016 08:10:50 -0000 Uhm, yes. I can read that too. I'm suggesting that someone working on the iwm driver can fix it. On the boot immediately prior to this my system panicked with an assert in idr - which is much more my bailiwick. -M On Thu, Aug 4, 2016 at 1:04 AM, Hans Petter Selasky wrote: > On 08/04/16 09:56, K. Macy wrote: >> >> #12 taskqueue_drain (queue=0x0, task=0xfffffe004fc17150) at > > > Hi, > > Looks like a NULL pointer, queue=NULL > > --HPS From owner-freebsd-net@freebsd.org Thu Aug 4 08:51:03 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7B3A0BACEEE for ; Thu, 4 Aug 2016 08:51:03 +0000 (UTC) (envelope-from daemon-user@freebsd.org) Received: from reviews.nyi.freebsd.org (reviews.nyi.freebsd.org [IPv6:2610:1c1:1:607c::16:b]) by mx1.freebsd.org (Postfix) with ESMTP id 3EA84114D for ; Thu, 4 Aug 2016 08:51:03 +0000 (UTC) (envelope-from daemon-user@freebsd.org) Received: by reviews.nyi.freebsd.org (Postfix, from userid 1346) id A7C26E0FE; Thu, 4 Aug 2016 08:51:02 +0000 (UTC) Date: Thu, 4 Aug 2016 08:51:02 +0000 To: freebsd-net@freebsd.org From: "sepherosa_gmail.com (Sepherosa Ziehau)" Reply-to: D7415+325+6ee7a792a2d68e99@reviews.freebsd.org Subject: [Differential] D7415: tcp/lro: If timestamps mismatch or it's a FIN, force flush. Message-ID: X-Priority: 3 X-Phabricator-Sent-This-Message: Yes X-Mail-Transport-Agent: MetaMTA X-Auto-Response-Suppress: All X-Phabricator-Mail-Tags: , , , Thread-Topic: D7415: tcp/lro: If timestamps mismatch or it's a FIN, force flush. X-Herald-Rules: <64> X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-Cc: Precedence: bulk Thread-Index: NzYyZWZjYzdkNzk4OWQzYmY5NTlkNDcxZWJh MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="b1_4a0d337d0c1cfd6748e8fe024593770b" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Aug 2016 08:51:03 -0000 --b1_4a0d337d0c1cfd6748e8fe024593770b Content-Type: text/plain; charset = "utf-8" Content-Transfer-Encoding: base64 c2VwaGVyb3NhX2dtYWlsLmNvbSBjcmVhdGVkIHRoaXMgcmV2aXNpb24uCnNlcGhlcm9zYV9nbWFp bC5jb20gYWRkZWQgcmV2aWV3ZXJzOiBycnMsIGdhbGxhdGluLCBoc2VsYXNreSwgbnAsIGdsZWJp dXMuCnNlcGhlcm9zYV9nbWFpbC5jb20gYWRkZWQgYSBzdWJzY3JpYmVyOiBmcmVlYnNkLW5ldC1s aXN0LgpIZXJhbGQgYWRkZWQgYSByZXZpZXdlcjogdHJhbnNwb3J0LgoKUkVWSVNJT04gU1VNTUFS WQogICAgVGhpcyBrZWVwcyB0aGUgc2VnbWVudHMvQUNLL0ZJTiBkZWxpdmVyeSBvcmRlci4KICAg IAogICAgQmVmb3JlIHRoaXMgcGF0Y2gsIGl0IHdhcyBvYnNlcnZlZDogaWYgQSBzZW50IEZJTiBp bW1lZGlhdGVseSBhZnRlcgogICAgYW4gQUNLLCBCIHdvdWxkIGRlbGl2ZXIgRklOIGZpcnN0IHRv IHRoZSBUQ1Agc3RhY2ssIHRoZW4gdGhlIEFDSy4KICAgIFRoaXMgb3V0LW9mLW9yZGVyIGRlbGl2 ZXJ5IGNhdXNlcyBvbmUgdW5uZWNlc3NhcnkgQUNLIHNlbnQgZnJvbSBCLgogICAgCiAgICBPYnRh aW5lZCBmcm9tOiAgZ2FsbGF0aW4sIHJycwoKUkVWSVNJT04gREVUQUlMCiAgaHR0cHM6Ly9yZXZp ZXdzLmZyZWVic2Qub3JnL0Q3NDE1CgpBRkZFQ1RFRCBGSUxFUwogIHN5cy9uZXRpbmV0L3RjcF9s cm8uYwoKRU1BSUwgUFJFRkVSRU5DRVMKICBodHRwczovL3Jldmlld3MuZnJlZWJzZC5vcmcvc2V0 dGluZ3MvcGFuZWwvZW1haWxwcmVmZXJlbmNlcy8KClRvOiBzZXBoZXJvc2FfZ21haWwuY29tLCAj dHJhbnNwb3J0LCBycnMsIGdhbGxhdGluLCBoc2VsYXNreSwgbnAsIGdsZWJpdXMKQ2M6IGZyZWVi c2QtbmV0LWxpc3QK --b1_4a0d337d0c1cfd6748e8fe024593770b Content-Type: text/x-patch; charset=utf-8; name="D7415.19031.patch" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="D7415.19031.patch" ZGlmZiAtLWdpdCBhL3N5cy9uZXRpbmV0L3RjcF9scm8uYyBiL3N5cy9uZXRpbmV0L3RjcF9scm8u YwotLS0gYS9zeXMvbmV0aW5ldC90Y3BfbHJvLmMKKysrIGIvc3lzL25ldGluZXQvdGNwX2xyby5j CkBAIC02MDMsNiArNjAzLDcgQEAKIAlpbnQgZXJyb3IsIGlwX2xlbiwgbDsKIAl1aW50MTZfdCBl aF90eXBlLCB0Y3BfZGF0YV9sZW47CiAJc3RydWN0IGxyb19oZWFkICpidWNrZXQ7CisJaW50IGZv cmNlX2ZsdXNoID0gMDsKIAogCS8qIFdlIGV4cGVjdCBhIGNvbnRpZ3VvdXMgaGVhZGVyIFtlaCwg aXAsIHRjcF0uICovCiAKQEAgLTY2OSw4ICs2NzAsMTUgQEAKIAkgKiBDaGVjayBUQ1AgaGVhZGVy IGNvbnN0cmFpbnRzLgogCSAqLwogCS8qIEVuc3VyZSBubyBiaXRzIHNldCBiZXNpZGVzIEFDSyBv ciBQU0guICovCi0JaWYgKCh0aC0+dGhfZmxhZ3MgJiB+KFRIX0FDSyB8IFRIX1BVU0gpKSAhPSAw KQotCQlyZXR1cm4gKFRDUF9MUk9fQ0FOTk9UKTsKKwlpZiAoKHRoLT50aF9mbGFncyAmIH4oVEhf QUNLIHwgVEhfUFVTSCkpICE9IDApIHsKKwkJaWYgKHRoLT50aF9mbGFncyAmIFRIX1NZTikKKwkJ CXJldHVybiAoVENQX0xST19DQU5OT1QpOworCQkvKgorCQkgKiBNYWtlIHN1cmUgdGhhdCBwcmV2 aW91c2x5IHNlZW4gc2VnZW1lbnRzL0FDS3MgYXJlIGRlbGl2ZXJlZAorCQkgKiBiZWZvcmUgdGhp cyBzZWdlbWVudCwgZS5nLiBGSU4uCisJCSAqLworCQlmb3JjZV9mbHVzaCA9IDE7CisJfQogCiAJ LyogWFhYLUJaIFdlIGxvc2UgYSBBQ0t8UFVTSCBmbGFnIGNvbmNhdGVuYXRpbmcgbXVsdGlwbGUg c2VnbWVudHMuICovCiAJLyogWFhYLUJaIElkZWFsbHkgd2UnZCBmbHVzaCBvbiBQVVNIPyAqLwpA QCAtNjg2LDggKzY5NCwxMyBAQAogCXRzX3B0ciA9ICh1aW50MzJfdCAqKSh0aCArIDEpOwogCWlm IChsICE9IDAgJiYgKF9fcHJlZGljdF9mYWxzZShsICE9IFRDUE9MRU5fVFNUQU1QX0FQUEEpIHx8 CiAJICAgICgqdHNfcHRyICE9IG50b2hsKFRDUE9QVF9OT1A8PDI0fFRDUE9QVF9OT1A8PDE2fAot CSAgICBUQ1BPUFRfVElNRVNUQU1QPDw4fFRDUE9MRU5fVElNRVNUQU1QKSkpKQotCQlyZXR1cm4g KFRDUF9MUk9fQ0FOTk9UKTsKKwkgICAgVENQT1BUX1RJTUVTVEFNUDw8OHxUQ1BPTEVOX1RJTUVT VEFNUCkpKSkgeworCQkvKgorCQkgKiBNYWtlIHN1cmUgdGhhdCBwcmV2aW91c2x5IHNlZW4gc2Vn ZW1lbnRzL0FDS3MgYXJlIGRlbGl2ZXJlZAorCQkgKiBiZWZvcmUgdGhpcyBzZWdlbWVudC4KKwkJ ICovCisJCWZvcmNlX2ZsdXNoID0gMTsKKwl9CiAKIAkvKiBJZiB0aGUgZHJpdmVyIGRpZCBub3Qg cGFzcyBpbiB0aGUgY2hlY2tzdW0sIHNldCBpdCBub3cuICovCiAJaWYgKGNzdW0gPT0gMHgwMDAw KQpAQCAtNzU0LDYgKzc2NywxMyBAQAogI2VuZGlmCiAJCX0KIAorCQlpZiAoZm9yY2VfZmx1c2gp IHsKKwkJCS8qIFRpbWVzdGFtcHMgbWlzbWF0Y2g7IHRoaXMgaXMgYSBGSU4sIGV0YyAqLworCQkJ dGNwX2xyb19hY3RpdmVfcmVtb3ZlKGxlKTsKKwkJCXRjcF9scm9fZmx1c2gobGMsIGxlKTsKKwkJ CXJldHVybiAoVENQX0xST19DQU5OT1QpOworCQl9CisKIAkJLyogRmx1c2ggbm93IGlmIGFwcGVu ZGluZyB3aWxsIHJlc3VsdCBpbiBvdmVyZmxvdy4gKi8KIAkJaWYgKGxlLT5wX2xlbiA+IChsYy0+ bHJvX2xlbmd0aF9saW0gLSB0Y3BfZGF0YV9sZW4pKSB7CiAJCQl0Y3BfbHJvX2FjdGl2ZV9yZW1v dmUobGUpOwoK --b1_4a0d337d0c1cfd6748e8fe024593770b-- From owner-freebsd-net@freebsd.org Thu Aug 4 08:54:35 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A6F8BBAD152 for ; Thu, 4 Aug 2016 08:54:35 +0000 (UTC) (envelope-from daemon-user@freebsd.org) Received: from reviews.nyi.freebsd.org (reviews.nyi.freebsd.org [IPv6:2610:1c1:1:607c::16:b]) by mx1.freebsd.org (Postfix) with ESMTP id 82FD914B4 for ; Thu, 4 Aug 2016 08:54:35 +0000 (UTC) (envelope-from daemon-user@freebsd.org) Received: by reviews.nyi.freebsd.org (Postfix, from userid 1346) id 27C7CEA68; Thu, 4 Aug 2016 08:54:35 +0000 (UTC) Date: Thu, 4 Aug 2016 08:54:35 +0000 To: freebsd-net@freebsd.org From: "sepherosa_gmail.com (Sepherosa Ziehau)" Reply-to: D7415+325+6ee7a792a2d68e99@reviews.freebsd.org Subject: [Differential] D7415: tcp/lro: If timestamps mismatch or it's a FIN, force flush. Message-ID: <2b8e3c348e6493acafdf8b33b754b7dc@localhost.localdomain> X-Priority: 3 X-Phabricator-Sent-This-Message: Yes X-Mail-Transport-Agent: MetaMTA X-Auto-Response-Suppress: All X-Phabricator-Mail-Tags: Thread-Topic: D7415: tcp/lro: If timestamps mismatch or it's a FIN, force flush. X-Herald-Rules: <64> X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-Cc: Precedence: bulk In-Reply-To: References: Thread-Index: NzYyZWZjYzdkNzk4OWQzYmY5NTlkNDcxZWJhIFejAss= MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Type: text/plain; charset="utf-8" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Aug 2016 08:54:35 -0000 c2VwaGVyb3NhX2dtYWlsLmNvbSBhZGRlZCByZXZpZXdlcnM6IGhpcmVuLCBiei4KClJFVklTSU9O IERFVEFJTAogIGh0dHBzOi8vcmV2aWV3cy5mcmVlYnNkLm9yZy9ENzQxNQoKRU1BSUwgUFJFRkVS RU5DRVMKICBodHRwczovL3Jldmlld3MuZnJlZWJzZC5vcmcvc2V0dGluZ3MvcGFuZWwvZW1haWxw cmVmZXJlbmNlcy8KClRvOiBzZXBoZXJvc2FfZ21haWwuY29tLCBycnMsIGdhbGxhdGluLCBoc2Vs YXNreSwgbnAsIGdsZWJpdXMsICN0cmFuc3BvcnQsIGhpcmVuLCBiegpDYzogZnJlZWJzZC1uZXQt bGlzdAo= From owner-freebsd-net@freebsd.org Thu Aug 4 09:07:17 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 20A9ABAD869 for ; Thu, 4 Aug 2016 09:07:17 +0000 (UTC) (envelope-from daemon-user@freebsd.org) Received: from reviews.nyi.freebsd.org (reviews.nyi.freebsd.org [IPv6:2610:1c1:1:607c::16:b]) by mx1.freebsd.org (Postfix) with ESMTP id E2FC41CE7 for ; Thu, 4 Aug 2016 09:07:16 +0000 (UTC) (envelope-from daemon-user@freebsd.org) Received: by reviews.nyi.freebsd.org (Postfix, from userid 1346) id 3F268C52C; Thu, 4 Aug 2016 09:07:16 +0000 (UTC) Date: Thu, 4 Aug 2016 09:07:16 +0000 To: freebsd-net@freebsd.org From: "hselasky (Hans Petter Selasky)" Reply-to: D7415+325+6ee7a792a2d68e99@reviews.freebsd.org Subject: [Differential] D7415: tcp/lro: If timestamps mismatch or it's a FIN, force flush. Message-ID: <85ff8022e56c17d14d6297f3da1d6d55@localhost.localdomain> X-Priority: 3 X-Phabricator-Sent-This-Message: Yes X-Mail-Transport-Agent: MetaMTA X-Auto-Response-Suppress: All X-Phabricator-Mail-Tags: Thread-Topic: D7415: tcp/lro: If timestamps mismatch or it's a FIN, force flush. X-Herald-Rules: <64> X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-Cc: Precedence: bulk In-Reply-To: References: Thread-Index: NzYyZWZjYzdkNzk4OWQzYmY5NTlkNDcxZWJhIFejBcQ= MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Type: text/plain; charset="utf-8" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Aug 2016 09:07:17 -0000 aHNlbGFza3kgYWRkZWQgaW5saW5lIGNvbW1lbnRzLgoKSU5MSU5FIENPTU1FTlRTCgo+IHRjcF9s cm8uYzo4NTIKPiAgCX0KPiAgCj4gIAkvKiBUcnkgdG8gZmluZCBhbiBlbXB0eSBzbG90LiAqLwoK U2hvdWxkIHRoZXJlIGJlIGFkZGVkOgoKaWYgKGZvcmNlX2ZsdXNoKQoKICByZXR1cm4gKFRDUF9M Uk9fQ0FOTk9UKTsKCklmIG5vIG1hdGNoIGlzIGZvdW5kIGluIHRoZSBsaXN0PwoKUkVWSVNJT04g REVUQUlMCiAgaHR0cHM6Ly9yZXZpZXdzLmZyZWVic2Qub3JnL0Q3NDE1CgpFTUFJTCBQUkVGRVJF TkNFUwogIGh0dHBzOi8vcmV2aWV3cy5mcmVlYnNkLm9yZy9zZXR0aW5ncy9wYW5lbC9lbWFpbHBy ZWZlcmVuY2VzLwoKVG86IHNlcGhlcm9zYV9nbWFpbC5jb20sIHJycywgZ2FsbGF0aW4sIGhzZWxh c2t5LCBucCwgZ2xlYml1cywgI3RyYW5zcG9ydCwgaGlyZW4sIGJ6CkNjOiBmcmVlYnNkLW5ldC1s aXN0Cg== From owner-freebsd-net@freebsd.org Thu Aug 4 09:19:32 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 46FC4BADA82 for ; Thu, 4 Aug 2016 09:19:32 +0000 (UTC) (envelope-from daemon-user@freebsd.org) Received: from reviews.nyi.freebsd.org (reviews.nyi.freebsd.org [IPv6:2610:1c1:1:607c::16:b]) by mx1.freebsd.org (Postfix) with ESMTP id 1E8F3128A for ; Thu, 4 Aug 2016 09:19:32 +0000 (UTC) (envelope-from daemon-user@freebsd.org) Received: by reviews.nyi.freebsd.org (Postfix, from userid 1346) id B9483D2D3; Thu, 4 Aug 2016 09:19:31 +0000 (UTC) Date: Thu, 4 Aug 2016 09:19:31 +0000 To: freebsd-net@freebsd.org From: "sepherosa_gmail.com (Sepherosa Ziehau)" Reply-to: D7415+325+6ee7a792a2d68e99@reviews.freebsd.org Subject: [Differential] D7415: tcp/lro: If timestamps mismatch or it's a FIN, force flush. Message-ID: X-Priority: 3 X-Phabricator-Sent-This-Message: Yes X-Mail-Transport-Agent: MetaMTA X-Auto-Response-Suppress: All X-Phabricator-Mail-Tags: Thread-Topic: D7415: tcp/lro: If timestamps mismatch or it's a FIN, force flush. X-Herald-Rules: <64> X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-Cc: Precedence: bulk In-Reply-To: References: Thread-Index: NzYyZWZjYzdkNzk4OWQzYmY5NTlkNDcxZWJhIFejCKM= MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Type: text/plain; charset="utf-8" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Aug 2016 09:19:32 -0000 c2VwaGVyb3NhX2dtYWlsLmNvbSBhZGRlZCBpbmxpbmUgY29tbWVudHMuCgpJTkxJTkUgQ09NTUVO VFMKCj4gaHNlbGFza3kgd3JvdGUgaW4gdGNwX2xyby5jOjg1Mgo+IFNob3VsZCB0aGVyZSBiZSBh ZGRlZDoKPiAKPiBpZiAoZm9yY2VfZmx1c2gpCj4gCj4gICByZXR1cm4gKFRDUF9MUk9fQ0FOTk9U KTsKPiAKPiBJZiBubyBtYXRjaCBpcyBmb3VuZCBpbiB0aGUgbGlzdD8KCkFoLCB5ZWFoLCBpdCB3 YXMgaW4gdGhlIG9yaWdpbmFsIHBhdGNoIGZyb20gcnJzLCBnb3QgbG9zdCBieSBtZSA6UAoKUkVW SVNJT04gREVUQUlMCiAgaHR0cHM6Ly9yZXZpZXdzLmZyZWVic2Qub3JnL0Q3NDE1CgpFTUFJTCBQ UkVGRVJFTkNFUwogIGh0dHBzOi8vcmV2aWV3cy5mcmVlYnNkLm9yZy9zZXR0aW5ncy9wYW5lbC9l bWFpbHByZWZlcmVuY2VzLwoKVG86IHNlcGhlcm9zYV9nbWFpbC5jb20sIHJycywgZ2FsbGF0aW4s IGhzZWxhc2t5LCBucCwgZ2xlYml1cywgI3RyYW5zcG9ydCwgaGlyZW4sIGJ6CkNjOiBmcmVlYnNk LW5ldC1saXN0Cg== From owner-freebsd-net@freebsd.org Thu Aug 4 09:26:16 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1B8ADBADE11 for ; Thu, 4 Aug 2016 09:26:16 +0000 (UTC) (envelope-from daemon-user@freebsd.org) Received: from reviews.nyi.freebsd.org (reviews.nyi.freebsd.org [IPv6:2610:1c1:1:607c::16:b]) by mx1.freebsd.org (Postfix) with ESMTP id D635D17AD for ; Thu, 4 Aug 2016 09:26:15 +0000 (UTC) (envelope-from daemon-user@freebsd.org) Received: by reviews.nyi.freebsd.org (Postfix, from userid 1346) id 4F5C7E467; Thu, 4 Aug 2016 09:26:15 +0000 (UTC) Date: Thu, 4 Aug 2016 09:26:15 +0000 To: freebsd-net@freebsd.org From: "sepherosa_gmail.com (Sepherosa Ziehau)" Reply-to: D7415+325+6ee7a792a2d68e99@reviews.freebsd.org Subject: [Differential] D7415: tcp/lro: If timestamps mismatch or it's a FIN, force flush. Message-ID: X-Priority: 3 X-Phabricator-Sent-This-Message: Yes X-Mail-Transport-Agent: MetaMTA X-Auto-Response-Suppress: All X-Phabricator-Mail-Tags: , , Thread-Topic: D7415: tcp/lro: If timestamps mismatch or it's a FIN, force flush. X-Herald-Rules: <64> X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-Cc: Precedence: bulk In-Reply-To: References: Thread-Index: NzYyZWZjYzdkNzk4OWQzYmY5NTlkNDcxZWJhIFejCjc= MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="b1_b3b8064c155f5f5ec9704d8b57ce9e8c" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Aug 2016 09:26:16 -0000 --b1_b3b8064c155f5f5ec9704d8b57ce9e8c Content-Type: text/plain; charset = "utf-8" Content-Transfer-Encoding: base64 c2VwaGVyb3NhX2dtYWlsLmNvbSB1cGRhdGVkIHRoaXMgcmV2aXNpb24gdG8gRGlmZiAxOTAzMi4K c2VwaGVyb3NhX2dtYWlsLmNvbSBhZGRlZCBhIGNvbW1lbnQuCgoKICBEb24ndCBjcmVhdGUgZW50 cnkgZm9yIHNlZ21lbnRzIGNhbnQgYmUgZnVydGhlciBhZ2dyZWdhdGVkL2RlbGF5ZWQuICBOb3Rp Y2VkLWJ5OiBocHMKCkNIQU5HRVMgU0lOQ0UgTEFTVCBVUERBVEUKICBodHRwczovL3Jldmlld3Mu ZnJlZWJzZC5vcmcvRDc0MTU/dnM9MTkwMzEmaWQ9MTkwMzIKClJFVklTSU9OIERFVEFJTAogIGh0 dHBzOi8vcmV2aWV3cy5mcmVlYnNkLm9yZy9ENzQxNQoKQUZGRUNURUQgRklMRVMKICBzeXMvbmV0 aW5ldC90Y3BfbHJvLmMKCkVNQUlMIFBSRUZFUkVOQ0VTCiAgaHR0cHM6Ly9yZXZpZXdzLmZyZWVi c2Qub3JnL3NldHRpbmdzL3BhbmVsL2VtYWlscHJlZmVyZW5jZXMvCgpUbzogc2VwaGVyb3NhX2dt YWlsLmNvbSwgcnJzLCBnYWxsYXRpbiwgaHNlbGFza3ksIG5wLCBnbGViaXVzLCBoaXJlbiwgYnos ICN0cmFuc3BvcnQKQ2M6IGZyZWVic2QtbmV0LWxpc3QK --b1_b3b8064c155f5f5ec9704d8b57ce9e8c Content-Type: text/x-patch; charset=utf-8; name="D7415.19032.patch" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="D7415.19032.patch" ZGlmZiAtLWdpdCBhL3N5cy9uZXRpbmV0L3RjcF9scm8uYyBiL3N5cy9uZXRpbmV0L3RjcF9scm8u YwotLS0gYS9zeXMvbmV0aW5ldC90Y3BfbHJvLmMKKysrIGIvc3lzL25ldGluZXQvdGNwX2xyby5j CkBAIC02MDMsNiArNjAzLDcgQEAKIAlpbnQgZXJyb3IsIGlwX2xlbiwgbDsKIAl1aW50MTZfdCBl aF90eXBlLCB0Y3BfZGF0YV9sZW47CiAJc3RydWN0IGxyb19oZWFkICpidWNrZXQ7CisJaW50IGZv cmNlX2ZsdXNoID0gMDsKIAogCS8qIFdlIGV4cGVjdCBhIGNvbnRpZ3VvdXMgaGVhZGVyIFtlaCwg aXAsIHRjcF0uICovCiAKQEAgLTY2OSw4ICs2NzAsMTUgQEAKIAkgKiBDaGVjayBUQ1AgaGVhZGVy IGNvbnN0cmFpbnRzLgogCSAqLwogCS8qIEVuc3VyZSBubyBiaXRzIHNldCBiZXNpZGVzIEFDSyBv ciBQU0guICovCi0JaWYgKCh0aC0+dGhfZmxhZ3MgJiB+KFRIX0FDSyB8IFRIX1BVU0gpKSAhPSAw KQotCQlyZXR1cm4gKFRDUF9MUk9fQ0FOTk9UKTsKKwlpZiAoKHRoLT50aF9mbGFncyAmIH4oVEhf QUNLIHwgVEhfUFVTSCkpICE9IDApIHsKKwkJaWYgKHRoLT50aF9mbGFncyAmIFRIX1NZTikKKwkJ CXJldHVybiAoVENQX0xST19DQU5OT1QpOworCQkvKgorCQkgKiBNYWtlIHN1cmUgdGhhdCBwcmV2 aW91c2x5IHNlZW4gc2VnZW1lbnRzL0FDS3MgYXJlIGRlbGl2ZXJlZAorCQkgKiBiZWZvcmUgdGhp cyBzZWdlbWVudCwgZS5nLiBGSU4uCisJCSAqLworCQlmb3JjZV9mbHVzaCA9IDE7CisJfQogCiAJ LyogWFhYLUJaIFdlIGxvc2UgYSBBQ0t8UFVTSCBmbGFnIGNvbmNhdGVuYXRpbmcgbXVsdGlwbGUg c2VnbWVudHMuICovCiAJLyogWFhYLUJaIElkZWFsbHkgd2UnZCBmbHVzaCBvbiBQVVNIPyAqLwpA QCAtNjg2LDggKzY5NCwxMyBAQAogCXRzX3B0ciA9ICh1aW50MzJfdCAqKSh0aCArIDEpOwogCWlm IChsICE9IDAgJiYgKF9fcHJlZGljdF9mYWxzZShsICE9IFRDUE9MRU5fVFNUQU1QX0FQUEEpIHx8 CiAJICAgICgqdHNfcHRyICE9IG50b2hsKFRDUE9QVF9OT1A8PDI0fFRDUE9QVF9OT1A8PDE2fAot CSAgICBUQ1BPUFRfVElNRVNUQU1QPDw4fFRDUE9MRU5fVElNRVNUQU1QKSkpKQotCQlyZXR1cm4g KFRDUF9MUk9fQ0FOTk9UKTsKKwkgICAgVENQT1BUX1RJTUVTVEFNUDw8OHxUQ1BPTEVOX1RJTUVT VEFNUCkpKSkgeworCQkvKgorCQkgKiBNYWtlIHN1cmUgdGhhdCBwcmV2aW91c2x5IHNlZW4gc2Vn ZW1lbnRzL0FDS3MgYXJlIGRlbGl2ZXJlZAorCQkgKiBiZWZvcmUgdGhpcyBzZWdlbWVudC4KKwkJ ICovCisJCWZvcmNlX2ZsdXNoID0gMTsKKwl9CiAKIAkvKiBJZiB0aGUgZHJpdmVyIGRpZCBub3Qg cGFzcyBpbiB0aGUgY2hlY2tzdW0sIHNldCBpdCBub3cuICovCiAJaWYgKGNzdW0gPT0gMHgwMDAw KQpAQCAtNzU0LDYgKzc2NywxMyBAQAogI2VuZGlmCiAJCX0KIAorCQlpZiAoZm9yY2VfZmx1c2gp IHsKKwkJCS8qIFRpbWVzdGFtcHMgbWlzbWF0Y2g7IHRoaXMgaXMgYSBGSU4sIGV0YyAqLworCQkJ dGNwX2xyb19hY3RpdmVfcmVtb3ZlKGxlKTsKKwkJCXRjcF9scm9fZmx1c2gobGMsIGxlKTsKKwkJ CXJldHVybiAoVENQX0xST19DQU5OT1QpOworCQl9CisKIAkJLyogRmx1c2ggbm93IGlmIGFwcGVu ZGluZyB3aWxsIHJlc3VsdCBpbiBvdmVyZmxvdy4gKi8KIAkJaWYgKGxlLT5wX2xlbiA+IChsYy0+ bHJvX2xlbmd0aF9saW0gLSB0Y3BfZGF0YV9sZW4pKSB7CiAJCQl0Y3BfbHJvX2FjdGl2ZV9yZW1v dmUobGUpOwpAQCAtODMwLDYgKzg1MCwxNCBAQAogCQlyZXR1cm4gKDApOwogCX0KIAorCWlmIChm b3JjZV9mbHVzaCkgeworCQkvKgorCQkgKiBOb3RoaW5nIHRvIGZsdXNoLCBidXQgdGhpcyBzZWdt ZW50IGNhbiBub3QgYmUgZnVydGhlcgorCQkgKiBhZ2dyZWdhdGVkL2RlbGF5ZWQuCisJCSAqLwor CQlyZXR1cm4gKFRDUF9MUk9fQ0FOTk9UKTsKKwl9CisKIAkvKiBUcnkgdG8gZmluZCBhbiBlbXB0 eSBzbG90LiAqLwogCWlmIChMSVNUX0VNUFRZKCZsYy0+bHJvX2ZyZWUpKQogCQlyZXR1cm4gKFRD UF9MUk9fTk9fRU5UUklFUyk7Cgo= --b1_b3b8064c155f5f5ec9704d8b57ce9e8c-- From owner-freebsd-net@freebsd.org Thu Aug 4 09:32:25 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BF699BAE135 for ; Thu, 4 Aug 2016 09:32:25 +0000 (UTC) (envelope-from daemon-user@freebsd.org) Received: from reviews.nyi.freebsd.org (reviews.nyi.freebsd.org [IPv6:2610:1c1:1:607c::16:b]) by mx1.freebsd.org (Postfix) with ESMTP id 99F051D85 for ; Thu, 4 Aug 2016 09:32:25 +0000 (UTC) (envelope-from daemon-user@freebsd.org) Received: by reviews.nyi.freebsd.org (Postfix, from userid 1346) id 4A169EAFF; Thu, 4 Aug 2016 09:32:25 +0000 (UTC) Date: Thu, 4 Aug 2016 09:32:25 +0000 To: freebsd-net@freebsd.org From: "hselasky (Hans Petter Selasky)" Reply-to: D7415+325+6ee7a792a2d68e99@reviews.freebsd.org Subject: [Differential] D7415: tcp/lro: If timestamps mismatch or it's a FIN, force flush. Message-ID: <8052bba4e6d96ac170e6a20bfc625c51@localhost.localdomain> X-Priority: 3 X-Phabricator-Sent-This-Message: Yes X-Mail-Transport-Agent: MetaMTA X-Auto-Response-Suppress: All X-Phabricator-Mail-Tags: , , Thread-Topic: D7415: tcp/lro: If timestamps mismatch or it's a FIN, force flush. X-Herald-Rules: <64> X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-Cc: Precedence: bulk In-Reply-To: References: Thread-Index: NzYyZWZjYzdkNzk4OWQzYmY5NTlkNDcxZWJhIFejC6k= MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Type: text/plain; charset="utf-8" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Aug 2016 09:32:25 -0000 aHNlbGFza3kgYWNjZXB0ZWQgdGhpcyByZXZpc2lvbi4KaHNlbGFza3kgYWRkZWQgYSBjb21tZW50 LgoKCiAgTG9va3MgZ29vZCB0byBtZS4KICAKICAtLUhQUwoKUkVWSVNJT04gREVUQUlMCiAgaHR0 cHM6Ly9yZXZpZXdzLmZyZWVic2Qub3JnL0Q3NDE1CgpFTUFJTCBQUkVGRVJFTkNFUwogIGh0dHBz Oi8vcmV2aWV3cy5mcmVlYnNkLm9yZy9zZXR0aW5ncy9wYW5lbC9lbWFpbHByZWZlcmVuY2VzLwoK VG86IHNlcGhlcm9zYV9nbWFpbC5jb20sIHJycywgZ2FsbGF0aW4sIG5wLCBnbGViaXVzLCBoaXJl biwgYnosICN0cmFuc3BvcnQsIGhzZWxhc2t5CkNjOiBmcmVlYnNkLW5ldC1saXN0Cg== From owner-freebsd-net@freebsd.org Thu Aug 4 09:40:14 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9C361BAE28D for ; Thu, 4 Aug 2016 09:40:14 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: from mail-wm0-x22d.google.com (mail-wm0-x22d.google.com [IPv6:2a00:1450:400c:c09::22d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 22B9B1157 for ; Thu, 4 Aug 2016 09:40:14 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: by mail-wm0-x22d.google.com with SMTP id o80so370722516wme.1 for ; Thu, 04 Aug 2016 02:40:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=WRhF4iTDC1VxLB9ZrqVCa6l5JqB26hdUDttESrAHejU=; b=gweKI3JEP+OahUlozZ16PZuqrtAIlOPBD77Rlzfkn+mWn2JReue7KQ0B1Ig6tSJQfT a7ltdUTAWnVaeEhqXJIzBV46WWwkc6rjWKoxMxMRUhGY7QuuFQaSjq7BkKzCcF5cb4gX wtx2YdfDsh8IuyDOUweo/28Wj9F1RyHBvYxDj2nNxtq8RJvEPNpOL9yHS78MQHGdGtMs bMtSH02fldfJMXNyKEfZyRgDsmzcGiVmffZWKkrWQkB4FFtYmzwE22ol9ag9/ufnPOR2 LUEvfqrtSUSr2Ql8J3uGlL20IXCl9InL+o+0YLmg4mXnoSoMtz/s+nAvaz1r9UsXWI9j MKvA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=WRhF4iTDC1VxLB9ZrqVCa6l5JqB26hdUDttESrAHejU=; b=bjNBIXpeHPWec6eku/TYA9nWPDSZaynbgYK/uon3PBXzTvhGmQRQKAKQ/t+Aggfucz /JwurJuSVFZvBfWWrBormzh3VFnfV4KfKpQ+IgdIWNdWsvBuiI+ud99Bk2dDPKD+owZH 44pxmnrpYqXqH2AHnO2m/5SMZjrCRC51QLMa4uwCsyJ7DWtzrRnSBwzuB2+9SOYkOq4y usgjrMf/DKbk7mOz8XeNupUYWPIShqXTTO6ezwp4cLSGSMJ9jJhqSgbrwmSj4KywEuXU uzdd1zGLnKonbOVun/r0hVjN5XP0waD/fmVcLsjsdTLydm9ffYZdEAofAEbAOl9sUJbs EtrA== X-Gm-Message-State: AEkoous0DYpjMW0HkFkI5tJFVJLFijEvRPFuURTzrMm2OasWMi/RyJ5/jVGzrTSK7XUzqA== X-Received: by 10.194.135.205 with SMTP id pu13mr74007856wjb.37.1470303612276; Thu, 04 Aug 2016 02:40:12 -0700 (PDT) Received: from macbook-air-de-benjamin-1.home (LFbn-1-7077-85.w90-116.abo.wanadoo.fr. [90.116.246.85]) by smtp.gmail.com with ESMTPSA id qe2sm11930816wjc.28.2016.08.04.02.40.11 for (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Thu, 04 Aug 2016 02:40:11 -0700 (PDT) Content-Type: text/plain; charset=windows-1252 Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: Unstable local network throughput From: Ben RUBSON In-Reply-To: <3B164B7B-CBFB-4518-B57D-A96EABB71647@gmail.com> Date: Thu, 4 Aug 2016 11:40:10 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: <5D6DF8EA-D9AA-4617-8561-2D7E22A738C3@gmail.com> References: <3C0D892F-2BE8-4650-B9FC-93C8EE0443E1@gmail.com> <3B164B7B-CBFB-4518-B57D-A96EABB71647@gmail.com> To: freebsd-net@freebsd.org X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Aug 2016 09:40:14 -0000 > On 02 Aug 2016, at 22:11, Ben RUBSON wrote: >=20 >> On 02 Aug 2016, at 21:35, Hans Petter Selasky = wrote: >>=20 >> The CX-3 driver doesn't bind the worker threads to specific CPU cores = by default, so if your CPU has more than one so-called numa, you'll end = up that the bottle-neck is the high-speed link between the CPU cores and = not the card. A quick and dirty workaround is to "cpuset" iperf and the = interrupt and taskqueue threads to specific CPU cores. >=20 > My CPUs : 2x E5-2620v3 with DDR4@1866. OK, so I cpuset all Mellanox interrupts to one NUMA, as well as the = iPerf processes, and I'm able to reach max bandwidth. Choosing the wrong NUMA (or both, or one for interrupts, the other one = for iPerf, etc...) totally kills throughput. However, full-duplex throughput is still limited, I can't manage to = reach 2x40Gb/s, throttle is at about 45Gb/s. I tried many different cpuset layouts, but I never went above 45Gb/s. (Linux allowed me to reach 2x40Gb/s so hardware is not a bottleneck) >> Are you using "options RSS" and "options PCBGROUP" in your kernel = config? I will then give RSS a try. Any other clue perhaps regarding the full-duplex limitation ? Many thanks ! Ben From owner-freebsd-net@freebsd.org Thu Aug 4 12:20:45 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2C549BAF58C for ; Thu, 4 Aug 2016 12:20:45 +0000 (UTC) (envelope-from daemon-user@freebsd.org) Received: from reviews.nyi.freebsd.org (reviews.nyi.freebsd.org [IPv6:2610:1c1:1:607c::16:b]) by mx1.freebsd.org (Postfix) with ESMTP id 06FDE1B09 for ; Thu, 4 Aug 2016 12:20:45 +0000 (UTC) (envelope-from daemon-user@freebsd.org) Received: by reviews.nyi.freebsd.org (Postfix, from userid 1346) id 94CBDCC8D; Thu, 4 Aug 2016 12:20:44 +0000 (UTC) Date: Thu, 4 Aug 2016 12:20:44 +0000 To: freebsd-net@freebsd.org From: "gallatin (Andrew Gallatin)" Reply-to: D7415+325+6ee7a792a2d68e99@reviews.freebsd.org Subject: [Differential] D7415: tcp/lro: If timestamps mismatch or it's a FIN, force flush. Message-ID: X-Priority: 3 X-Phabricator-Sent-This-Message: Yes X-Mail-Transport-Agent: MetaMTA X-Auto-Response-Suppress: All X-Phabricator-Mail-Tags: , Thread-Topic: D7415: tcp/lro: If timestamps mismatch or it's a FIN, force flush. X-Herald-Rules: <64> X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-Cc: Precedence: bulk In-Reply-To: References: Thread-Index: NzYyZWZjYzdkNzk4OWQzYmY5NTlkNDcxZWJhIFejMxw= MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Type: text/plain; charset="utf-8" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Aug 2016 12:20:45 -0000 Z2FsbGF0aW4gYWNjZXB0ZWQgdGhpcyByZXZpc2lvbi4KClJFVklTSU9OIERFVEFJTAogIGh0dHBz Oi8vcmV2aWV3cy5mcmVlYnNkLm9yZy9ENzQxNQoKRU1BSUwgUFJFRkVSRU5DRVMKICBodHRwczov L3Jldmlld3MuZnJlZWJzZC5vcmcvc2V0dGluZ3MvcGFuZWwvZW1haWxwcmVmZXJlbmNlcy8KClRv OiBzZXBoZXJvc2FfZ21haWwuY29tLCBycnMsIG5wLCBnbGViaXVzLCBoaXJlbiwgYnosICN0cmFu c3BvcnQsIGhzZWxhc2t5LCBnYWxsYXRpbgpDYzogZnJlZWJzZC1uZXQtbGlzdAo= From owner-freebsd-net@freebsd.org Thu Aug 4 15:24:23 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1283CBAF02C for ; Thu, 4 Aug 2016 15:24:23 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: from mail-wm0-x235.google.com (mail-wm0-x235.google.com [IPv6:2a00:1450:400c:c09::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9A11D1533 for ; Thu, 4 Aug 2016 15:24:22 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: by mail-wm0-x235.google.com with SMTP id o80so382933530wme.1 for ; Thu, 04 Aug 2016 08:24:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=OpDoBMORj5YXGM8FkrfH6PgwpXtmLLAAIHB7R1/BlxM=; b=ksCFdk1VEyc1jRmd0EtO1hZ7gy2iwriY2TnzW0cRVT+2gtitFgG6q013Sq9j+kFuJx hX2HDXenzAwL4fxAYl8E5Vf8z1zW+mNWKs72IdEP5Qof6eB2cZaOW8VXGya6Dytxgjdb EH8GFX1C+E9xln4QzJrdXiuIJgmKk2zfHVH1sMKVF38XpWEZhDN68llbbl/1aAMY35KY i2LIMnvANXvWcfrsTB+lu21wk/WRJug++24oN3sIry75sCZyEXkN0VMUsw6FKc42AH4U wCuh8s4vLOl6lgPA/JH+3CXROHtXsj15q6rTz2F7LtdXaczfIkAVh++AZU9TF5ZVzFrK 1ffg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=OpDoBMORj5YXGM8FkrfH6PgwpXtmLLAAIHB7R1/BlxM=; b=LCSEoorMLAE7Ii4d1MWS9KJ+L2sLYjGkvOXJL0s6LNj0d2BUbQg3fbTclprbGZ9A6G 8lcHD4/V3ZgbXy79R4PvF17Bi8Q7nWJbtqQDLCp4jF4Ucbh/+DKRoQfqWPQhkuEjCqHC DrVUYrZ7kURAFwXC7NPpc/Kywr9hSALgNpmC8Un15Pi6qJTWMPGSyjSoXJN2coPrZRvz bBHmz/sAwXLYwDi0C4fiuCGV2KhF6b2wTMhdEPZMMf7AV/bfKzLj2KBeZKIH1QH2FbXS S3uXRv4ytyS0e1hBhxJRvh9WluKUZCozVycY3ZWcIzAjplgEL73dV/rbkXh9aUW6EcHA JDzw== X-Gm-Message-State: AEkoouvgY2Pxn904tLah+YsibXtPxnVOJGgXxjSGJ1lpoDWKmr4LkYd1vfz2z9PLVopmhQ== X-Received: by 10.28.24.5 with SMTP id 5mr30908416wmy.6.1470324259307; Thu, 04 Aug 2016 08:24:19 -0700 (PDT) Received: from macbook-air-de-benjamin-1.home (LFbn-1-7077-85.w90-116.abo.wanadoo.fr. [90.116.246.85]) by smtp.gmail.com with ESMTPSA id p23sm4160634wme.8.2016.08.04.08.24.18 for (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Thu, 04 Aug 2016 08:24:18 -0700 (PDT) Content-Type: text/plain; charset=windows-1252 Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: Unstable local network throughput From: Ben RUBSON In-Reply-To: <5D6DF8EA-D9AA-4617-8561-2D7E22A738C3@gmail.com> Date: Thu, 4 Aug 2016 17:24:17 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: <06E414D5-9CDA-46D1-A26F-0B07E76FDB34@gmail.com> References: <3C0D892F-2BE8-4650-B9FC-93C8EE0443E1@gmail.com> <3B164B7B-CBFB-4518-B57D-A96EABB71647@gmail.com> <5D6DF8EA-D9AA-4617-8561-2D7E22A738C3@gmail.com> To: freebsd-net@freebsd.org X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Aug 2016 15:24:23 -0000 > On 04 Aug 2016, at 11:40, Ben RUBSON wrote: >=20 >> On 02 Aug 2016, at 22:11, Ben RUBSON wrote: >>=20 >>> On 02 Aug 2016, at 21:35, Hans Petter Selasky = wrote: >>>=20 >>> The CX-3 driver doesn't bind the worker threads to specific CPU = cores by default, so if your CPU has more than one so-called numa, = you'll end up that the bottle-neck is the high-speed link between the = CPU cores and not the card. A quick and dirty workaround is to "cpuset" = iperf and the interrupt and taskqueue threads to specific CPU cores. >>=20 >> My CPUs : 2x E5-2620v3 with DDR4@1866. >=20 > OK, so I cpuset all Mellanox interrupts to one NUMA, as well as the = iPerf processes, and I'm able to reach max bandwidth. > Choosing the wrong NUMA (or both, or one for interrupts, the other one = for iPerf, etc...) totally kills throughput. >=20 > However, full-duplex throughput is still limited, I can't manage to = reach 2x40Gb/s, throttle is at about 45Gb/s. > I tried many different cpuset layouts, but I never went above 45Gb/s. > (Linux allowed me to reach 2x40Gb/s so hardware is not a bottleneck) >=20 >>> Are you using "options RSS" and "options PCBGROUP" in your kernel = config? >=20 > I will then give RSS a try. Without RSS : A ---> B : 40Gbps (unidirectional) A <--> B : 45Gbps (bidirectional) With RSS : A ---> B : 28Gbps (unidirectional) A <--> B : 28Gbps (bidirectional) Sounds like RSS does not help :/ Why, without RSS, do I have difficulties to reach 2x40Gbps (full-duplex) = ? Thank U != From owner-freebsd-net@freebsd.org Thu Aug 4 15:28:54 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7DC2FBAF2CC for ; Thu, 4 Aug 2016 15:28:54 +0000 (UTC) (envelope-from hps@selasky.org) Received: from mail.turbocat.net (mail.turbocat.net [IPv6:2a01:4f8:d16:4514::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48E3B1B8F for ; Thu, 4 Aug 2016 15:28:54 +0000 (UTC) (envelope-from hps@selasky.org) Received: from laptop015.home.selasky.org (unknown [62.141.129.119]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.turbocat.net (Postfix) with ESMTPSA id 5AEA61FE022; Thu, 4 Aug 2016 17:28:52 +0200 (CEST) Subject: Re: Unstable local network throughput To: Ben RUBSON , freebsd-net@freebsd.org References: <3C0D892F-2BE8-4650-B9FC-93C8EE0443E1@gmail.com> <3B164B7B-CBFB-4518-B57D-A96EABB71647@gmail.com> <5D6DF8EA-D9AA-4617-8561-2D7E22A738C3@gmail.com> <06E414D5-9CDA-46D1-A26F-0B07E76FDB34@gmail.com> From: Hans Petter Selasky Message-ID: <0b14bf39-ed71-b9fb-1998-bd9676466df6@selasky.org> Date: Thu, 4 Aug 2016 17:33:10 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:45.0) Gecko/20100101 Thunderbird/45.0 MIME-Version: 1.0 In-Reply-To: <06E414D5-9CDA-46D1-A26F-0B07E76FDB34@gmail.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Aug 2016 15:28:54 -0000 On 08/04/16 17:24, Ben RUBSON wrote: > >> On 04 Aug 2016, at 11:40, Ben RUBSON wrote: >> >>> On 02 Aug 2016, at 22:11, Ben RUBSON wrote: >>> >>>> On 02 Aug 2016, at 21:35, Hans Petter Selasky wrote: >>>> >>>> The CX-3 driver doesn't bind the worker threads to specific CPU cores by default, so if your CPU has more than one so-called numa, you'll end up that the bottle-neck is the high-speed link between the CPU cores and not the card. A quick and dirty workaround is to "cpuset" iperf and the interrupt and taskqueue threads to specific CPU cores. >>> >>> My CPUs : 2x E5-2620v3 with DDR4@1866. >> >> OK, so I cpuset all Mellanox interrupts to one NUMA, as well as the iPerf processes, and I'm able to reach max bandwidth. >> Choosing the wrong NUMA (or both, or one for interrupts, the other one for iPerf, etc...) totally kills throughput. >> >> However, full-duplex throughput is still limited, I can't manage to reach 2x40Gb/s, throttle is at about 45Gb/s. >> I tried many different cpuset layouts, but I never went above 45Gb/s. >> (Linux allowed me to reach 2x40Gb/s so hardware is not a bottleneck) >> >>>> Are you using "options RSS" and "options PCBGROUP" in your kernel config? >> >> I will then give RSS a try. > > Without RSS : > A ---> B : 40Gbps (unidirectional) > A <--> B : 45Gbps (bidirectional) > > With RSS : > A ---> B : 28Gbps (unidirectional) > A <--> B : 28Gbps (bidirectional) > > Sounds like RSS does not help :/ > > Why, without RSS, do I have difficulties to reach 2x40Gbps (full-duplex) ? > Hi, Possibly because the packets are arriving at the wrong CPU compared to what RSS expects. Then RSS will invoke a taskqueue to process the packets on the correct CPU, if I'm not mistaken. The mlx4 driver does not fully support RSS. Then mlx5 does. --HPS From owner-freebsd-net@freebsd.org Thu Aug 4 15:33:22 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C9602BAF42E for ; Thu, 4 Aug 2016 15:33:22 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: from mail-wm0-x22d.google.com (mail-wm0-x22d.google.com [IPv6:2a00:1450:400c:c09::22d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 699491EAF for ; Thu, 4 Aug 2016 15:33:22 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: by mail-wm0-x22d.google.com with SMTP id f65so488257720wmi.0 for ; Thu, 04 Aug 2016 08:33:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=mQhP+eJi71vJ9hSLp4uJCL8OHK/lff5dYWrQnKXiRcU=; b=CtaCOa1p13fNjIbTta/Yl+tkR2l+emxHs1vJkj3HQYWQNcWfZbA/RkXof3GidLvvOa b64DGI+EgJM01ESQ5rFsryTyP/co1J/YL0k79drSkG3W6hdYLN1qkvLyCpfufKxl5pry NZATJhn4vlTUMtKtNQBujDmq7SLbBI+OK0or1ca70879ed/AI4XWZB88u5iS8nasVI+J jyAqTCnlnjE87bYzUOxEnn1nJ0ObGGz6qg2XHLId7m1xGdmoIOD0sFRNLPYvcV+fIoXt mRH4AouIsxEINoyjA147xYoV2lnxOZBAdyhap/W6b2hmkLfWPIQJKGFXETgcfalhgZNd kpRA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=mQhP+eJi71vJ9hSLp4uJCL8OHK/lff5dYWrQnKXiRcU=; b=m2UwrjlOZLwYaCTaA4lq+puTAWwfjeK7pS6RvthYaMHstRL1XdTCkde8Ya5SnG9jNW QhQDhOTQ8YWVV7Pn2ahUYytQfzFWMoJfQNHXWXiCApTkNntFriIMx26O+dsse6H2GLfz 52d7glkpHiQWkF8lEVwQwPWk0hbjE/xPfmuDch1qJz6sBiAbs/9BmaXaMYyaIEsLcLGq b6zmTrhvF8j+2DDkdeYwj4RHXp8KcUT0Hd3hmOtEtilzseK0Dh8K1N1Uh2ySiF3K0KtQ zSW+8DWRs97+BEfJmo3NMk5t8yuZ2U9MpGFp9c32V8lsNKUSUK8DFmjH+XS5QwoVPt39 VUSQ== X-Gm-Message-State: AEkooutFzUD6VeWjbnHy9DukLHVTH80tB9PifDYhDwk+UqOwUyrAkUDHEVrkBhvzJ9akhA== X-Received: by 10.194.238.42 with SMTP id vh10mr66507672wjc.111.1470324800607; Thu, 04 Aug 2016 08:33:20 -0700 (PDT) Received: from macbook-air-de-benjamin-1.home (LFbn-1-7077-85.w90-116.abo.wanadoo.fr. [90.116.246.85]) by smtp.gmail.com with ESMTPSA id e65sm4194752wmg.3.2016.08.04.08.33.19 for (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Thu, 04 Aug 2016 08:33:19 -0700 (PDT) Content-Type: text/plain; charset=windows-1252 Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: Unstable local network throughput From: Ben RUBSON In-Reply-To: <0b14bf39-ed71-b9fb-1998-bd9676466df6@selasky.org> Date: Thu, 4 Aug 2016 17:33:19 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: References: <3C0D892F-2BE8-4650-B9FC-93C8EE0443E1@gmail.com> <3B164B7B-CBFB-4518-B57D-A96EABB71647@gmail.com> <5D6DF8EA-D9AA-4617-8561-2D7E22A738C3@gmail.com> <06E414D5-9CDA-46D1-A26F-0B07E76FDB34@gmail.com> <0b14bf39-ed71-b9fb-1998-bd9676466df6@selasky.org> To: freebsd-net@freebsd.org X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Aug 2016 15:33:22 -0000 > On 04 Aug 2016, at 17:33, Hans Petter Selasky wrote: >=20 > On 08/04/16 17:24, Ben RUBSON wrote: >>=20 >>> On 04 Aug 2016, at 11:40, Ben RUBSON wrote: >>>=20 >>>> On 02 Aug 2016, at 22:11, Ben RUBSON wrote: >>>>=20 >>>>> On 02 Aug 2016, at 21:35, Hans Petter Selasky = wrote: >>>>>=20 >>>>> The CX-3 driver doesn't bind the worker threads to specific CPU = cores by default, so if your CPU has more than one so-called numa, = you'll end up that the bottle-neck is the high-speed link between the = CPU cores and not the card. A quick and dirty workaround is to "cpuset" = iperf and the interrupt and taskqueue threads to specific CPU cores. >>>>=20 >>>> My CPUs : 2x E5-2620v3 with DDR4@1866. >>>=20 >>> OK, so I cpuset all Mellanox interrupts to one NUMA, as well as the = iPerf processes, and I'm able to reach max bandwidth. >>> Choosing the wrong NUMA (or both, or one for interrupts, the other = one for iPerf, etc...) totally kills throughput. >>>=20 >>> However, full-duplex throughput is still limited, I can't manage to = reach 2x40Gb/s, throttle is at about 45Gb/s. >>> I tried many different cpuset layouts, but I never went above = 45Gb/s. >>> (Linux allowed me to reach 2x40Gb/s so hardware is not a bottleneck) >>>=20 >>>>> Are you using "options RSS" and "options PCBGROUP" in your kernel = config? >>>=20 >>> I will then give RSS a try. >>=20 >> Without RSS : >> A ---> B : 40Gbps (unidirectional) >> A <--> B : 45Gbps (bidirectional) >>=20 >> With RSS : >> A ---> B : 28Gbps (unidirectional) >> A <--> B : 28Gbps (bidirectional) >>=20 >> Sounds like RSS does not help :/ >>=20 >> Why, without RSS, do I have difficulties to reach 2x40Gbps = (full-duplex) ? >>=20 >=20 > Hi, >=20 > Possibly because the packets are arriving at the wrong CPU compared to = what RSS expects. Then RSS will invoke a taskqueue to process the = packets on the correct CPU, if I'm not mistaken. But even without RSS, I should be able to go up to 2x40Gbps, don't you = think so ? Nobody already did this ? From owner-freebsd-net@freebsd.org Thu Aug 4 18:15:11 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 18CE4BAE344 for ; Thu, 4 Aug 2016 18:15:11 +0000 (UTC) (envelope-from rysto32@gmail.com) Received: from mail-it0-x234.google.com (mail-it0-x234.google.com [IPv6:2607:f8b0:4001:c0b::234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D481E15C8 for ; Thu, 4 Aug 2016 18:15:10 +0000 (UTC) (envelope-from rysto32@gmail.com) Received: by mail-it0-x234.google.com with SMTP id x130so2583030ite.1 for ; Thu, 04 Aug 2016 11:15:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=4sj6bEZDURPsVGrsUl9iDp3DeWb2z6fnYLBPCcRbY1E=; b=ARkMJu3YVLKUdTlbEibklE6q46cqEYmdVf2BRK1FsFTHiTZzdagFbFVtCX7XsnroQv Q0w9rnG8fIid/Yp9hR//Xi/dtCJsl9qEXfwaPCa8I11brx7LvbTzUA8iyaUsPo2Norkm uLYrU9BgBe7ijFEazsS3hdXDfkIFt5QqchPHootAT0+ViSaBdZW3dDlv02uNvLQRNmFw rkwGLCOyy/I3fBEcwF4t966Tm8Rogs7Wp6nEltpEpz72Fp1yXqxS/KgPEhL9U7RgxEnV 56hx6XY/6ZRq3OSNa+72en2wSJIqaGtpcAtJl51oIfeVO0G+MZzLhDxnGSGwWWsVGl5R l/mQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=4sj6bEZDURPsVGrsUl9iDp3DeWb2z6fnYLBPCcRbY1E=; b=PaUTyEbgNhH0gZpvycKYsT3eBseGr2FpIbFanl5iBgy4p3MdELfWEU2xgmusR7b0p9 4SZMOUesaasHA649kYaZsftANkx9RdrWWKit33yrxyIgPB66JZXT4yoN8RYmn8jE8GjC z17Z/E9isvvrh16TvfGf+TXDG5t15jxWyH5DxeR3ErNxMpH9cbbBRODWB7ICdf1yEUed s4K64nakCeT3f2TtfHI3djMrZYFpzRD79d7ucnAI6MwS2sbuWyL86c4QmB/v02v6bqNs cr0yLfkUm2vGSsfUNJ3Hyf3sUbGLreDoTt2b0V3XII1S4frrWwJAC2Qg4ryG13RmJueE R7Ug== X-Gm-Message-State: AEkoouu+WGwk6+M85ZetUsxmrp6MFWbGCT3BjpoHZkEZ+Awl9qjsBPaO608Tgu7bm97sIQK8Ik/kjNr6RXtJKA== X-Received: by 10.36.217.9 with SMTP id p9mr9528210itg.46.1470334510231; Thu, 04 Aug 2016 11:15:10 -0700 (PDT) MIME-Version: 1.0 Received: by 10.107.200.71 with HTTP; Thu, 4 Aug 2016 11:15:09 -0700 (PDT) In-Reply-To: References: <3C0D892F-2BE8-4650-B9FC-93C8EE0443E1@gmail.com> <3B164B7B-CBFB-4518-B57D-A96EABB71647@gmail.com> <5D6DF8EA-D9AA-4617-8561-2D7E22A738C3@gmail.com> <06E414D5-9CDA-46D1-A26F-0B07E76FDB34@gmail.com> <0b14bf39-ed71-b9fb-1998-bd9676466df6@selasky.org> From: Ryan Stone Date: Thu, 4 Aug 2016 14:15:09 -0400 Message-ID: Subject: Re: Unstable local network throughput To: Ben RUBSON Cc: freebsd-net Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Aug 2016 18:15:11 -0000 On Thu, Aug 4, 2016 at 11:33 AM, Ben RUBSON wrote: > But even without RSS, I should be able to go up to 2x40Gbps, don't you > think so ? > Nobody already did this ? > Try this patch, which should improve performance when multiple TCP streams are running in parallel over an mlx4_en port: https://people.freebsd.org/~rstone/patches/mlxen_counters.diff From owner-freebsd-net@freebsd.org Thu Aug 4 18:53:36 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 44B68BAF00E for ; Thu, 4 Aug 2016 18:53:36 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: from mail-wm0-x241.google.com (mail-wm0-x241.google.com [IPv6:2a00:1450:400c:c09::241]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C99611EA9 for ; Thu, 4 Aug 2016 18:53:35 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: by mail-wm0-x241.google.com with SMTP id q128so503936wma.1 for ; Thu, 04 Aug 2016 11:53:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=wwBbxqm3R9WWiSA2icsLeR3zXkVdYUIdatTF/V53FWU=; b=QSpdtLEqgKYlBqV4QM/bpeUEeo+hmN3GnFopZRz0/+RCrXri5ZkeTZXY3OXR2AyPR/ GnJrPKSOrR7s5QDRdjeTlYI0099v7XtsriucWK4UGWcgr+Lxom7m85aoJlARhGUXk8aw ooLiDNjCKdlq/Y48XncjKsKemIknTLEsffgxxXIUSZK+d6EAvjn1ya+QaCZLtD0027Vo A+y4H0mi4CJUEHHPU3GHoSosXzD51OKNu4O8onbePYHHggmKbizK7nmn4SyWjeJrxcRJ wguv1CPsLqtL70bziF1iK9nNKn0R4uay8ibkFW/My1Grc1CXoO3N3nbPHlljbEFrTciZ t37A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=wwBbxqm3R9WWiSA2icsLeR3zXkVdYUIdatTF/V53FWU=; b=GHZHYBvO8e1AL6cH1rhwrxl8NQMAwnRhuf9mDZab/a3IpAj6VD1HGP259jlTY140tT pOnOKwGuANaMpRuuTD2rAwfjRnSswlcq8+LKL8VXT8XTPPTCGmp91q05NqqaZ/rMIOVl T6WiAVXqF2mA3HMJkyqQXypuqnbQMDUc//kT0Po2X7Ee1FESJfJbEsbrfai5gBssY4y3 voLRnLs75jEEnZvRDmhjm9zBqVG+/X7DrFrPEIy67gmJzzcLNb/bLN56y1AIXqdH7ZbI 6VNsT5l6F7psxXy67qgQ1G7SFK2eM9DUIDpRjsMZ6k5Iq34So9riHGy222KsSFbX0Ogj I6/A== X-Gm-Message-State: AEkoouufWEbMYMlfyC+CY/EC8lL4G5rp3i+vr+3CKw1FZZrQbl5vRMXnmiCSS+2/kHeI2g== X-Received: by 10.28.168.150 with SMTP id r144mr70314841wme.66.1470336814053; Thu, 04 Aug 2016 11:53:34 -0700 (PDT) Received: from macbook-air-de-benjamin.home (LFbn-1-7077-85.w90-116.abo.wanadoo.fr. [90.116.246.85]) by smtp.gmail.com with ESMTPSA id a9sm14144138wjf.16.2016.08.04.11.53.33 for (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Thu, 04 Aug 2016 11:53:33 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: Unstable local network throughput From: Ben RUBSON In-Reply-To: Date: Thu, 4 Aug 2016 20:53:34 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: <27223F4B-6DE8-49D0-98C0-9F734C73C5EC@gmail.com> References: <3C0D892F-2BE8-4650-B9FC-93C8EE0443E1@gmail.com> <3B164B7B-CBFB-4518-B57D-A96EABB71647@gmail.com> <5D6DF8EA-D9AA-4617-8561-2D7E22A738C3@gmail.com> <06E414D5-9CDA-46D1-A26F-0B07E76FDB34@gmail.com> <0b14bf39-ed71-b9fb-1998-bd9676466df6@selasky.org> To: freebsd-net X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Aug 2016 18:53:36 -0000 > On 04 Aug 2016, at 20:15, Ryan Stone wrote: >=20 > On Thu, Aug 4, 2016 at 11:33 AM, Ben RUBSON = wrote: > But even without RSS, I should be able to go up to 2x40Gbps, don't you = think so ? > Nobody already did this ? >=20 > Try this patch, which should improve performance when multiple TCP = streams are running in parallel over an mlx4_en port: >=20 > https://people.freebsd.org/~rstone/patches/mlxen_counters.diff Thank you very much Ryan. I just tried it, but it does not help :/ Below is the cpuload during bidirectional trafic. We clearly see the 4 CPUs allocated to Mellanox IRQs, the others to = iPerf processes. No improvement if IRQs are spread over the 12 NUMA CPUs, but slightly = less throughput. Note that I get the same results if I only use 2 CPUs for IRQs. 27 processes: 1 running, 26 sleeping CPU 0: 1.1% user, 0.0% nice, 16.7% system, 0.0% interrupt, 82.2% = idle CPU 1: 1.1% user, 0.0% nice, 18.9% system, 0.0% interrupt, 80.0% = idle CPU 2: 1.9% user, 0.0% nice, 17.8% system, 0.0% interrupt, 80.4% = idle CPU 3: 1.1% user, 0.0% nice, 15.2% system, 0.0% interrupt, 83.7% = idle CPU 4: 0.4% user, 0.0% nice, 16.3% system, 0.0% interrupt, 83.3% = idle CPU 5: 1.1% user, 0.0% nice, 14.4% system, 0.0% interrupt, 84.4% = idle CPU 6: 2.6% user, 0.0% nice, 17.4% system, 0.0% interrupt, 80.0% = idle CPU 7: 2.2% user, 0.0% nice, 15.2% system, 0.0% interrupt, 82.6% = idle CPU 8: 1.1% user, 0.0% nice, 3.0% system, 15.9% interrupt, 80.0% = idle CPU 9: 0.0% user, 0.0% nice, 3.0% system, 32.2% interrupt, 64.8% = idle CPU 10: 0.0% user, 0.0% nice, 0.4% system, 58.9% interrupt, 40.7% = idle CPU 11: 0.0% user, 0.0% nice, 0.4% system, 77.4% interrupt, 22.2% = idle CPU 12: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% = idle CPU 13: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% = idle CPU 14: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% = idle CPU 15: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% = idle CPU 16: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% = idle CPU 17: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% = idle CPU 18: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% = idle CPU 19: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% = idle CPU 20: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% = idle CPU 21: 0.0% user, 0.0% nice, 0.0% system, 0.4% interrupt, 99.6% = idle CPU 22: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% = idle CPU 23: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% = idle= From owner-freebsd-net@freebsd.org Thu Aug 4 21:05:54 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3FAB9BAFAF8 for ; Thu, 4 Aug 2016 21:05:54 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2F89E1ECB for ; Thu, 4 Aug 2016 21:05:54 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u74L5rb4046813 for ; Thu, 4 Aug 2016 21:05:54 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-net@FreeBSD.org Subject: [Bug 211386] Kernel build fails when TCP_RFC7413 option set Date: Thu, 04 Aug 2016 21:05:54 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-STABLE X-Bugzilla-Keywords: regression X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: iron@mail.ua X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Aug 2016 21:05:54 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D211386 --- Comment #4 from iron@mail.ua --- (In reply to Sergey Kandaurov from comment #3) Thank you for workaround. I just tested it and it works fine. Here is patch: --- sys/netinet/tcp_fastopen.c.orig 2016-08-04 15:04:22.000000000 -0600 +++ sys/netinet/tcp_fastopen.c 2016-08-04 15:03:57.000000000 -0600 @@ -111,6 +111,7 @@ #include #include #include +#include #include --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-net@freebsd.org Thu Aug 4 21:47:33 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DB996BAF458 for ; Thu, 4 Aug 2016 21:47:33 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: from mail-wm0-x22c.google.com (mail-wm0-x22c.google.com [IPv6:2a00:1450:400c:c09::22c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6D0A51952 for ; Thu, 4 Aug 2016 21:47:33 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: by mail-wm0-x22c.google.com with SMTP id q128so9190998wma.1 for ; Thu, 04 Aug 2016 14:47:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=1WhsTSbD+yw2rVOrV9x6pCeUp97cNqp0dOhfPE7fJKA=; b=Rotrq0cRuzrSQ+8TKMO6s83yu7Gcn6AWXZn3JC0MToBlOy+TKN5H1c0YKUn2XfNxUF 8h1L8t9IlnzbjszGXPYF6SiD0+OWjCvE8WtS2QKXiFOHVgJB8z50IjSOPzLWRiz9nQvP rPK1WGMu+P9H5v8bFWjAaVzkQ8L5ZlL4hIQPC0wG8VUKAN58mSPfZG1cevdLdxmoZx7X h2WYt1ipo1rgNRWsyDWWyiqNjk4wMmeRqTz3nEEiyFMpbUpClVHW01Wk3upGQRoh4kJU r3iaRjpumNoK58NnECEeGYXTCRDa2iri1CgqqRL5PLT3uykb6CbQNiPLIswTGbkX72q9 PDOA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=1WhsTSbD+yw2rVOrV9x6pCeUp97cNqp0dOhfPE7fJKA=; b=T7W0wpjunHI9uOtxRa9LdCuo363eLR67mI0Sy1epIP9FBVEfaYO47X2adqra2Ni3NQ lqFWWO8VMJhfz8Rk/14PFsqYTXGqERYKmkvlC+Q7HBER9+gxSZ2IpMMBeJIN6co9q8RM MLDuh7pxu55BiiDz9620R0IJaK4lOI14TmA0XJYquen4TVcYA3c+u/MmPh5Fq8Zaf235 YUP8kgHu7U99mfmsu5BpxqOGa3iPKh8lepOzRMFbX2Mw+c980eOip7wFnBAbgOQg0gYc 3eO+7G8FKuuk7JXvWunPdOuRYI3MOWKeaeuRpGCVAjtwTwMkx/uhGPATx9idH4LakmuU SBig== X-Gm-Message-State: AEkoousiOwZcd7KwlAxdtHNdQe6tHftcOHCZ+1qAgxdKc7cSkwbUiXpqxeuVIAwkd5hchg== X-Received: by 10.28.225.4 with SMTP id y4mr123142wmg.98.1470347251303; Thu, 04 Aug 2016 14:47:31 -0700 (PDT) Received: from macbook-air-de-benjamin.home (LFbn-1-7077-85.w90-116.abo.wanadoo.fr. [90.116.246.85]) by smtp.gmail.com with ESMTPSA id i80sm5496697wmf.11.2016.08.04.14.47.30 for (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Thu, 04 Aug 2016 14:47:30 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: Unstable local network throughput From: Ben RUBSON In-Reply-To: Date: Thu, 4 Aug 2016 23:47:29 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: <2B910F84-B7BE-444B-B43A-9C8F2A61CB1E@gmail.com> References: <3C0D892F-2BE8-4650-B9FC-93C8EE0443E1@gmail.com> <3B164B7B-CBFB-4518-B57D-A96EABB71647@gmail.com> <5D6DF8EA-D9AA-4617-8561-2D7E22A738C3@gmail.com> <06E414D5-9CDA-46D1-A26F-0B07E76FDB34@gmail.com> <0b14bf39-ed71-b9fb-1998-bd9676466df6@selasky.org> To: freebsd-net X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Aug 2016 21:47:34 -0000 > On 04 Aug 2016, at 20:15, Ryan Stone wrote: >=20 > On Thu, Aug 4, 2016 at 11:33 AM, Ben RUBSON = wrote: > But even without RSS, I should be able to go up to 2x40Gbps, don't you = think so ? > Nobody already did this ? >=20 > Try this patch > (...) I also just tested the NODEBUG kernel but I did not help. From owner-freebsd-net@freebsd.org Thu Aug 4 21:49:19 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 61A62BAF50C for ; Thu, 4 Aug 2016 21:49:19 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: from mail-wm0-x22d.google.com (mail-wm0-x22d.google.com [IPv6:2a00:1450:400c:c09::22d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E720A1B68 for ; Thu, 4 Aug 2016 21:49:18 +0000 (UTC) (envelope-from ben.rubson@gmail.com) Received: by mail-wm0-x22d.google.com with SMTP id q128so9233208wma.1 for ; Thu, 04 Aug 2016 14:49:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=fY4IJzAyUVChmwA73cXi9oZLHnkRPb8t33nSyWXOPPo=; b=SDLfKt0pvUO7dcoXLxgxCz894yUwKjOI/c9uh1CMzVf+Bh4SXhQqicWHgdVuNXaKcE pzYPLBK9ZIOZyzad8J8uE7my/CaSOF/JZENWMNZ6HDV0NR8Fn3K8eFSdK1XAP7SFB6xJ K55Z7CDK/1GPRCzwdNxHVPIvmIUHy8+VPL8kw8SuoCC3X4H+Yw+s3dSwgvJccuxBbY6V z91QUOn2wb3TF3MxJgRetCRz594d0bTrA4v0YPsGBl7uXHdveerEyoYwzqdWM2DBkedr 34fSq9df6QpzKPFYRePsn4EXJTi5agKvfnBQBkp1zw/6hHXZGX17wOpUmquKhNdBpp5w eCpQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to; bh=fY4IJzAyUVChmwA73cXi9oZLHnkRPb8t33nSyWXOPPo=; b=gR6qXwyrWRWJR/jIJEvX8q7VmbKFMEIGl5wgWNfSzoIBQkg5jPY92TGjM7UfWRF3/c WkFIhH1Ot9nwxW17Vtx+ifcf3ESmg5lH0QGAYjRmmHYWJULzXr6Un1Vpgp/lPtHUomU4 FoTwHRHpYxJ7Eo1+KOy0fD7SSleEFjPEO0vNNpIZtTMLFJURJZVkbkeFZmA2eCqPLhDN IBOKWwsD8lPnKxsUn/aVPvXjGja0xhPvs3riJR99ZeA3T+H0dT7kKak2fRIHFZ2yOOyF YGRZFlzYQ9F+SIsvxQi9wWb9bRzSC/211tmiVmn+ZN40bdTEkLjLnNNx7S/mebmcWtq4 O9KA== X-Gm-Message-State: AEkoouvmF5DjvO0MiYuw8z02T7Hg175mMi14kWyuOTEmdX7+O4BqNMi24B/o+oBZxQH1bg== X-Received: by 10.194.164.229 with SMTP id yt5mr68322544wjb.39.1470347357294; Thu, 04 Aug 2016 14:49:17 -0700 (PDT) Received: from macbook-air-de-benjamin.home (LFbn-1-7077-85.w90-116.abo.wanadoo.fr. [90.116.246.85]) by smtp.gmail.com with ESMTPSA id yz6sm14712943wjb.35.2016.08.04.14.49.16 for (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Thu, 04 Aug 2016 14:49:16 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: Unstable local network throughput From: Ben RUBSON In-Reply-To: Date: Thu, 4 Aug 2016 23:49:16 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: References: <3C0D892F-2BE8-4650-B9FC-93C8EE0443E1@gmail.com> <3B164B7B-CBFB-4518-B57D-A96EABB71647@gmail.com> <5D6DF8EA-D9AA-4617-8561-2D7E22A738C3@gmail.com> <06E414D5-9CDA-46D1-A26F-0B07E76FDB34@gmail.com> <0b14bf39-ed71-b9fb-1998-bd9676466df6@selasky.org> To: freebsd-net X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Aug 2016 21:49:19 -0000 >=20 > On 04 Aug 2016, at 20:15, Ryan Stone wrote: >=20 > On Thu, Aug 4, 2016 at 11:33 AM, Ben RUBSON = wrote: > But even without RSS, I should be able to go up to 2x40Gbps, don't you = think so ? > Nobody already did this ? >=20 > Try this patch > (...) I also just tested the NODEBUG kernel but it did not help.= From owner-freebsd-net@freebsd.org Fri Aug 5 08:25:48 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 318FFBAE062 for ; Fri, 5 Aug 2016 08:25:48 +0000 (UTC) (envelope-from hps@selasky.org) Received: from mail.turbocat.net (mail.turbocat.net [IPv6:2a01:4f8:d16:4514::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id ED7AC18ED for ; Fri, 5 Aug 2016 08:25:47 +0000 (UTC) (envelope-from hps@selasky.org) Received: from laptop015.home.selasky.org (unknown [62.141.129.119]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.turbocat.net (Postfix) with ESMTPSA id EBEFB1FE022; Fri, 5 Aug 2016 10:25:45 +0200 (CEST) Subject: Re: Unstable local network throughput To: Ben RUBSON , freebsd-net References: <3C0D892F-2BE8-4650-B9FC-93C8EE0443E1@gmail.com> <3B164B7B-CBFB-4518-B57D-A96EABB71647@gmail.com> <5D6DF8EA-D9AA-4617-8561-2D7E22A738C3@gmail.com> <06E414D5-9CDA-46D1-A26F-0B07E76FDB34@gmail.com> <0b14bf39-ed71-b9fb-1998-bd9676466df6@selasky.org> From: Hans Petter Selasky Message-ID: Date: Fri, 5 Aug 2016 10:30:04 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:45.0) Gecko/20100101 Thunderbird/45.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Aug 2016 08:25:48 -0000 On 08/04/16 23:49, Ben RUBSON wrote: >> >> On 04 Aug 2016, at 20:15, Ryan Stone wrote: >> >> On Thu, Aug 4, 2016 at 11:33 AM, Ben RUBSON wrote: >> But even without RSS, I should be able to go up to 2x40Gbps, don't you think so ? >> Nobody already did this ? >> >> Try this patch >> (...) > > I also just tested the NODEBUG kernel but it did not help. Hi, When running these tests, do you see any CPUs fully utilized? Did you check the RX/TX pauseframes settings and the mlx4 sysctl statistics counters, if there is packet loss? --HPS From owner-freebsd-net@freebsd.org Fri Aug 5 09:08:18 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C210FBAEEB9 for ; Fri, 5 Aug 2016 09:08:18 +0000 (UTC) (envelope-from daemon-user@freebsd.org) Received: from reviews.nyi.freebsd.org (reviews.nyi.freebsd.org [IPv6:2610:1c1:1:607c::16:b]) by mx1.freebsd.org (Postfix) with ESMTP id 9C4501268 for ; Fri, 5 Aug 2016 09:08:18 +0000 (UTC) (envelope-from daemon-user@freebsd.org) Received: by reviews.nyi.freebsd.org (Postfix, from userid 1346) id 43F54E920; Fri, 5 Aug 2016 09:08:18 +0000 (UTC) Date: Fri, 5 Aug 2016 09:08:18 +0000 To: freebsd-net@freebsd.org From: Phabricator Reply-to: D7415+325+6ee7a792a2d68e99@reviews.freebsd.org Subject: [Differential] D7415: tcp/lro: If timestamps mismatch or it's a FIN, force flush. Message-ID: <47a9b539d4ea9c64f7cdcaca5f1993aa@localhost.localdomain> X-Priority: 3 X-Phabricator-Sent-This-Message: Yes X-Mail-Transport-Agent: MetaMTA X-Auto-Response-Suppress: All X-Phabricator-Mail-Tags: , Thread-Topic: D7415: tcp/lro: If timestamps mismatch or it's a FIN, force flush. X-Herald-Rules: <64> X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-To: X-Phabricator-Cc: Precedence: bulk In-Reply-To: References: Thread-Index: NzYyZWZjYzdkNzk4OWQzYmY5NTlkNDcxZWJhIFekV4I= MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="b1_47a9b539d4ea9c64f7cdcaca5f1993aa" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Aug 2016 09:08:18 -0000 --b1_47a9b539d4ea9c64f7cdcaca5f1993aa Content-Type: text/plain; charset = "utf-8" Content-Transfer-Encoding: base64 VGhpcyByZXZpc2lvbiB3YXMgYXV0b21hdGljYWxseSB1cGRhdGVkIHRvIHJlZmxlY3QgdGhlIGNv bW1pdHRlZCBjaGFuZ2VzLgpDbG9zZWQgYnkgY29tbWl0IHJTMzAzNzY2OiB0Y3AvbHJvOiBJZiB0 aW1lc3RhbXBzIG1pc21hdGNoIG9yIGl0J3MgYSBGSU4sIGZvcmNlIGZsdXNoLiAoYXV0aG9yZWQg Ynkgc2VwaGUpLgoKQ0hBTkdFRCBQUklPUiBUTyBDT01NSVQKICBodHRwczovL3Jldmlld3MuZnJl ZWJzZC5vcmcvRDc0MTU/dnM9MTkwMzImaWQ9MTkwNjMjdG9jCgpSRVBPU0lUT1JZCiAgclMgRnJl ZUJTRCBzcmMgcmVwb3NpdG9yeQoKQ0hBTkdFUyBTSU5DRSBMQVNUIFVQREFURQogIGh0dHBzOi8v cmV2aWV3cy5mcmVlYnNkLm9yZy9ENzQxNT92cz0xOTAzMiZpZD0xOTA2MwoKUkVWSVNJT04gREVU QUlMCiAgaHR0cHM6Ly9yZXZpZXdzLmZyZWVic2Qub3JnL0Q3NDE1CgpBRkZFQ1RFRCBGSUxFUwog IGhlYWQvc3lzL25ldGluZXQvdGNwX2xyby5jCgpFTUFJTCBQUkVGRVJFTkNFUwogIGh0dHBzOi8v cmV2aWV3cy5mcmVlYnNkLm9yZy9zZXR0aW5ncy9wYW5lbC9lbWFpbHByZWZlcmVuY2VzLwoKVG86 IHNlcGhlcm9zYV9nbWFpbC5jb20sIHJycywgbnAsIGdsZWJpdXMsIGhpcmVuLCBieiwgI3RyYW5z cG9ydCwgaHNlbGFza3ksIGdhbGxhdGluCkNjOiBmcmVlYnNkLW5ldC1saXN0Cg== --b1_47a9b539d4ea9c64f7cdcaca5f1993aa Content-Type: text/x-patch; charset=utf-8; name="D7415.19063.patch" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="D7415.19063.patch" ZGlmZiAtLWdpdCBhL2hlYWQvc3lzL25ldGluZXQvdGNwX2xyby5jIGIvaGVhZC9zeXMvbmV0aW5l dC90Y3BfbHJvLmMKLS0tIGEvaGVhZC9zeXMvbmV0aW5ldC90Y3BfbHJvLmMKKysrIGIvaGVhZC9z eXMvbmV0aW5ldC90Y3BfbHJvLmMKQEAgLTYwMyw2ICs2MDMsNyBAQAogCWludCBlcnJvciwgaXBf bGVuLCBsOwogCXVpbnQxNl90IGVoX3R5cGUsIHRjcF9kYXRhX2xlbjsKIAlzdHJ1Y3QgbHJvX2hl YWQgKmJ1Y2tldDsKKwlpbnQgZm9yY2VfZmx1c2ggPSAwOwogCiAJLyogV2UgZXhwZWN0IGEgY29u dGlndW91cyBoZWFkZXIgW2VoLCBpcCwgdGNwXS4gKi8KIApAQCAtNjY5LDggKzY3MCwxNSBAQAog CSAqIENoZWNrIFRDUCBoZWFkZXIgY29uc3RyYWludHMuCiAJICovCiAJLyogRW5zdXJlIG5vIGJp dHMgc2V0IGJlc2lkZXMgQUNLIG9yIFBTSC4gKi8KLQlpZiAoKHRoLT50aF9mbGFncyAmIH4oVEhf QUNLIHwgVEhfUFVTSCkpICE9IDApCi0JCXJldHVybiAoVENQX0xST19DQU5OT1QpOworCWlmICgo dGgtPnRoX2ZsYWdzICYgfihUSF9BQ0sgfCBUSF9QVVNIKSkgIT0gMCkgeworCQlpZiAodGgtPnRo X2ZsYWdzICYgVEhfU1lOKQorCQkJcmV0dXJuIChUQ1BfTFJPX0NBTk5PVCk7CisJCS8qCisJCSAq IE1ha2Ugc3VyZSB0aGF0IHByZXZpb3VzbHkgc2VlbiBzZWdlbWVudHMvQUNLcyBhcmUgZGVsaXZl cmVkCisJCSAqIGJlZm9yZSB0aGlzIHNlZ2VtZW50LCBlLmcuIEZJTi4KKwkJICovCisJCWZvcmNl X2ZsdXNoID0gMTsKKwl9CiAKIAkvKiBYWFgtQlogV2UgbG9zZSBhIEFDS3xQVVNIIGZsYWcgY29u Y2F0ZW5hdGluZyBtdWx0aXBsZSBzZWdtZW50cy4gKi8KIAkvKiBYWFgtQlogSWRlYWxseSB3ZSdk IGZsdXNoIG9uIFBVU0g/ICovCkBAIC02ODYsOCArNjk0LDEzIEBACiAJdHNfcHRyID0gKHVpbnQz Ml90ICopKHRoICsgMSk7CiAJaWYgKGwgIT0gMCAmJiAoX19wcmVkaWN0X2ZhbHNlKGwgIT0gVENQ T0xFTl9UU1RBTVBfQVBQQSkgfHwKIAkgICAgKCp0c19wdHIgIT0gbnRvaGwoVENQT1BUX05PUDw8 MjR8VENQT1BUX05PUDw8MTZ8Ci0JICAgIFRDUE9QVF9USU1FU1RBTVA8PDh8VENQT0xFTl9USU1F U1RBTVApKSkpCi0JCXJldHVybiAoVENQX0xST19DQU5OT1QpOworCSAgICBUQ1BPUFRfVElNRVNU QU1QPDw4fFRDUE9MRU5fVElNRVNUQU1QKSkpKSB7CisJCS8qCisJCSAqIE1ha2Ugc3VyZSB0aGF0 IHByZXZpb3VzbHkgc2VlbiBzZWdlbWVudHMvQUNLcyBhcmUgZGVsaXZlcmVkCisJCSAqIGJlZm9y ZSB0aGlzIHNlZ2VtZW50LgorCQkgKi8KKwkJZm9yY2VfZmx1c2ggPSAxOworCX0KIAogCS8qIElm IHRoZSBkcml2ZXIgZGlkIG5vdCBwYXNzIGluIHRoZSBjaGVja3N1bSwgc2V0IGl0IG5vdy4gKi8K IAlpZiAoY3N1bSA9PSAweDAwMDApCkBAIC03NTQsNiArNzY3LDEzIEBACiAjZW5kaWYKIAkJfQog CisJCWlmIChmb3JjZV9mbHVzaCkgeworCQkJLyogVGltZXN0YW1wcyBtaXNtYXRjaDsgdGhpcyBp cyBhIEZJTiwgZXRjICovCisJCQl0Y3BfbHJvX2FjdGl2ZV9yZW1vdmUobGUpOworCQkJdGNwX2xy b19mbHVzaChsYywgbGUpOworCQkJcmV0dXJuIChUQ1BfTFJPX0NBTk5PVCk7CisJCX0KKwogCQkv KiBGbHVzaCBub3cgaWYgYXBwZW5kaW5nIHdpbGwgcmVzdWx0IGluIG92ZXJmbG93LiAqLwogCQlp ZiAobGUtPnBfbGVuID4gKGxjLT5scm9fbGVuZ3RoX2xpbSAtIHRjcF9kYXRhX2xlbikpIHsKIAkJ CXRjcF9scm9fYWN0aXZlX3JlbW92ZShsZSk7CkBAIC04MzAsNiArODUwLDE0IEBACiAJCXJldHVy biAoMCk7CiAJfQogCisJaWYgKGZvcmNlX2ZsdXNoKSB7CisJCS8qCisJCSAqIE5vdGhpbmcgdG8g Zmx1c2gsIGJ1dCB0aGlzIHNlZ21lbnQgY2FuIG5vdCBiZSBmdXJ0aGVyCisJCSAqIGFnZ3JlZ2F0 ZWQvZGVsYXllZC4KKwkJICovCisJCXJldHVybiAoVENQX0xST19DQU5OT1QpOworCX0KKwogCS8q IFRyeSB0byBmaW5kIGFuIGVtcHR5IHNsb3QuICovCiAJaWYgKExJU1RfRU1QVFkoJmxjLT5scm9f ZnJlZSkpCiAJCXJldHVybiAoVENQX0xST19OT19FTlRSSUVTKTsKCg== --b1_47a9b539d4ea9c64f7cdcaca5f1993aa-- From owner-freebsd-net@freebsd.org Fri Aug 5 18:01:39 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DCFD7BAFC09 for ; Fri, 5 Aug 2016 18:01:38 +0000 (UTC) (envelope-from rupavath@juniper.net) Received: from NAM01-BN3-obe.outbound.protection.outlook.com (mail-bn3nam01on0123.outbound.protection.outlook.com [104.47.33.123]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "Microsoft IT SSL SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6A9DB1C8D; Fri, 5 Aug 2016 18:01:37 +0000 (UTC) (envelope-from rupavath@juniper.net) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=junipernetworks.onmicrosoft.com; s=selector1-juniper-net; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=PKkGX017Ah3nmJLDIRxsy21CNeFNmqaHzr3zwFgJmYQ=; b=Ln+jzVVg7KaP2bIAb+PoUMOnIy5VpkB8NCdjwEuC48+hRm4g9ksv1r1k8eZXMIMzmYGH3XN3+yEJQilCDKy5hrFGp+EeksvTOL8sB46MjQaDbu09q3U9knomzGPohaDlvTROHqKkXhY3gdRwe/hlTuasIIvPwqTumSBTZX7pQ1w= Received: from CY4PR05MB2824.namprd05.prod.outlook.com (10.169.182.146) by CY4PR05MB2823.namprd05.prod.outlook.com (10.169.182.145) with Microsoft SMTP Server (version=TLS1_0, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA_P384) id 15.1.557.8; Fri, 5 Aug 2016 18:01:31 +0000 Received: from CY4PR05MB2824.namprd05.prod.outlook.com ([10.169.182.146]) by CY4PR05MB2824.namprd05.prod.outlook.com ([10.169.182.146]) with mapi id 15.01.0557.009; Fri, 5 Aug 2016 18:01:30 +0000 From: Sreekanth Rupavatharam To: "freebsd-net@freebsd.org" CC: "sbruno@freebsd.org" , Jack Vogel Subject: Question about em_irq_fast Thread-Topic: Question about em_irq_fast Thread-Index: AQHR70NkOM2CalMxpEqDaCmo5kMDAg== Date: Fri, 5 Aug 2016 18:01:29 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: user-agent: Microsoft-MacOutlook/f.18.0.160709 authentication-results: spf=none (sender IP is ) smtp.mailfrom=rupavath@juniper.net; x-ms-exchange-messagesentrepresentingtype: 1 x-originating-ip: [66.129.239.14] x-ms-office365-filtering-correlation-id: a3891357-ef99-4534-4a61-08d3bd5a8721 x-microsoft-exchange-diagnostics: 1; CY4PR05MB2823; 6:LkqkeErAyeKIfGqsjzasqqZGIhwHRfCUSkXe6Rh8wMInI+mSy4Ux5cAnTJ7k9EKN7cohSjvYMxOgnxWMpLLY7p40v1mzhA9hLb5t42dk2XYJOLOHFfpaCVrOkhVUn/d1ML24tT4EnllPMlSNlr1J2mvaljICZdS5sTq1+TwUfrraXJBViTKZfzoQ+tuheEa5LybSmE2hGkMjmYd5Zvgb9oKpQpXfg0fmzpVIXu3he1no6fTMY8ROTlqfzP1s7hT18PTTrmzyHJl3vAxaCzs1G5TPyHDoyE649rNjC6OL4Gvh6+jtE9XygsPgN2ecUtzboqUBGFSB0+IgmRW9HZnP+A==; 5:ysPpzhSWN2A1vzcWORRUwGQipa0MBGq6iJ3pbGdGG6ewy6F4Mmolqs3C4w3rSQQ5Pggp8o76xiHlzqRJUVEP57wBmdm0KX4gHDM+a7MXn3H/M15Ank84hLBH8K/4U/x6olqW87wHV145L3CLzdyzNw==; 24:RfAsdcwQr7A8fPGzoPo8rd/94XkaXYslz5ccFA4k1UGkOuncqCUhge2xHuE8Zmcgn3PGRXRWcRotP3IxtXf95iiWhLpseRggQGEqpsOtOOg=; 7:IpQIzTdFKJ4V+pJZil6D3Kw2YC0GYE8kEdXBxwFAtCSMQwUvXNWhZPQf0W3lKAzyDOfHFtKNfc5At7D8X/5Kd9TxQ3/LjPpu00FvPr1nueyoXcpBflHot6QwEjFaNCy6iaAJk+Ff58KiCUdzUYhfcnXOogPG9D1xiybYEMDWDsrFWR/h0GigNCXy2di1ksRk6Epmu1ICTFGwgdOEGIHrB9CmF5Nk0u2b889KVPGHHxGk1e/GB4XlnKhkZsnXXX1F x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:CY4PR05MB2823; x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(21748063052155); x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(601004)(2401047)(8121501046)(5005006)(10201501046)(3002001)(6055026); SRVR:CY4PR05MB2823; BCL:0; PCL:0; RULEID:; SRVR:CY4PR05MB2823; x-forefront-prvs: 0025434D2D x-forefront-antispam-report: SFV:NSPM; SFS:(10019020)(6009001)(7916002)(199003)(189002)(106356001)(7736002)(8936002)(5640700001)(3280700002)(77096005)(4326007)(81166006)(101416001)(106116001)(7846002)(189998001)(68736007)(3660700001)(83506001)(2351001)(86362001)(2900100001)(97736004)(110136002)(82746002)(8676002)(229853001)(10400500002)(19580395003)(19625215002)(50986999)(122556002)(81156014)(19300405004)(2501003)(102836003)(3846002)(36756003)(54356999)(105586002)(87936001)(5002640100001)(99286002)(4001350100001)(6116002)(83716003)(16236675004)(92566002)(2906002)(33656002)(15975445007)(586003)(66066001)(104396002); DIR:OUT; SFP:1102; SCL:1; SRVR:CY4PR05MB2823; H:CY4PR05MB2824.namprd05.prod.outlook.com; FPR:; SPF:None; PTR:InfoNoRecords; A:1; MX:1; LANG:en; received-spf: None (protection.outlook.com: juniper.net does not designate permitted sender hosts) spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM MIME-Version: 1.0 X-OriginatorOrg: juniper.net X-MS-Exchange-CrossTenant-originalarrivaltime: 05 Aug 2016 18:01:30.0023 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: bea78b3c-4cdb-4130-854a-1d193232e5f4 X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY4PR05MB2823 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Aug 2016 18:01:39 -0000 V2UgaGF2ZSB0aGlzIGNvZGUgc25pcHBldCBpbiBlbV9pcnFfZmFzdA0KDQogICAgICAgIGlmcCA9 IGFkYXB0ZXItPmlmcDsNCg0KDQoNCiAgICAgICAgcmVnX2ljciA9IEUxMDAwX1JFQURfUkVHKCZh ZGFwdGVyLT5odywgRTEwMDBfSUNSKTsNCg0KDQoNCiAgICAgICAgLyogSG90IGVqZWN0PyAgKi8N Cg0KICAgICAgICBpZiAocmVnX2ljciA9PSAweGZmZmZmZmZmKQ0KDQogICAgICAgICAgICAgICAg cmV0dXJuIEZJTFRFUl9TVFJBWTsNCg0KDQoNCiAgICAgICAgLyogRGVmaW5pdGVseSBub3Qgb3Vy IGludGVycnVwdC4gICovDQoNCiAgICAgICAgaWYgKHJlZ19pY3IgPT0gMHgwKQ0KDQpyZXR1cm4g RklMVEVSX1NUUkFZOw0KDQpJIGRvbuKAmXQgdW5kZXJzdGFuZCB3aHkgdGhlIGZ1bmN0aW9uIHJl dHVybnMgaWYgdGhlIHJlYWQgdmFsdWUgaXMgMC4gRnJvbSB0aGUgcHJvZ3JhbW1lcuKAmXMgbWFu dWFsIG9mIEludGVsIE5JQywgSSBzZWUgdGhlIGZvbGxvd2luZyBkZWZpbml0aW9uLg0KDQoNCg0K DQoNCg0KDQoNCg0KDQpUWERXDQoNCjANCg0KMGINCg0KVHJhbnNtaXQgRGVzY3JpcHRvciBXcml0 dGVuIEJhY2vigKhTZXQgd2hlbiBoYXJkd2FyZSBwcm9jZXNzZXMgYSB0cmFuc21pdCBkZXNjcmlw dG9yIHdpdGggdGhlIFJTIGJpdCBzZXQgKGFuZCBwb3NzaWJseSBJREUgc2V0KS4gSWYgdXNpbmcg ZGVsYXllZCBpbnRlcnJ1cHRzIChJREUgc2V0KSwgdGhlIGludGVycnVwdCBvY2N1cnMgYWZ0ZXIg dGhlIHRpbWVyIGV4cGlyZXMuDQoNClRYUUUNCg0KMQ0KDQowYg0KDQpUcmFuc21pdCBRdWV1ZSBF bXB0eQ0KU2V0IHdoZW4gdGhlIGxhc3QgZGVzY3JpcHRvciBibG9jayBmb3IgYSB0cmFuc21pdCBx dWV1ZSBoYXMgYmVlbiB1c2VkLg0KDQpMU0MNCg0KMg0KDQowYg0KDQpMaW5rIFN0YXR1cyBDaGFu Z2XigKhUaGlzIGJpdCBpcyBzZXQgZWFjaCB0aW1lIHRoZSBsaW5rIHN0YXR1cyBjaGFuZ2VzIChl aXRoZXIgZnJvbSB1cCB0bw0KZG93biwgb3IgZnJvbSBkb3duIHRvIHVwKS4gVGhpcyBiaXQgaXMg YWZmZWN0ZWQgYnkgdGhlIGludGVybmFsIGxpbmsgaW5kaWNhdGlvbiB3aGVuIGNvbmZpZ3VyZWQg Zm9yIGludGVybmFsIFBIWSBtb2RlLg0KDQpSWFNFUQ0KDQozDQoNCjBiDQoNClJlY2VpdmUgU2Vx dWVuY2UgRXJyb3LigKhJbiBUQkkgbW9kZS9pbnRlcm5hbCBTZXJEZXMxLCBpbmNvbWluZyBwYWNr ZXRzIHdpdGggYSBiYWQgZGVsaW1pdGVyIHNlcXVlbmNlIHNldCB0aGlzIGJpdC4gSW4gb3RoZXIg ODAyLjMgaW1wbGVtZW50YXRpb25zLCB0aGlzIHdvdWxkIGJlIGNsYXNzaWZpZWQgYXMgYSBmcmFt aW5nIGVycm9yLiBBIHZhbGlkIHNlcXVlbmNlIGNvbnNpc3RzIG9mOuKAqGlkbGUg74KuIFNPRiDv gq4gZGF0YSDvgq4gcGFkIChvcHQpIEVPRiDvgq4gZmlsbCAob3B0KSDvgq4gaWRsZS7igKhUaGlz IGlzIGEgcmVzZXJ2ZWQgYml0IGZvciB0aGUgODI1NDF4eCwgODI1NDdHSS9FSSwgYW5kIDgyNTQw RVAvIEVNLiBTZXQgdG8gMGIuDQoNClJYRE1UMA0KDQo0DQoNCjBiDQoNClJlY2VpdmUgRGVzY3Jp cHRvciBNaW5pbXVtIFRocmVzaG9sZCBSZWFjaGVkDQpJbmRpY2F0ZXMgdGhhdCB0aGUgbWluaW11 bSBudW1iZXIgb2YgcmVjZWl2ZSBkZXNjcmlwdG9ycyBhcmUgYXZhaWxhYmxlIGFuZCBzb2Z0d2Fy ZSBzaG91bGQgbG9hZCBtb3JlIHJlY2VpdmUgZGVzY3JpcHRvcnMuDQoNClJlc2VydmVkDQoNCjUN Cg0KMGINCg0KUmVzZXJ2ZWQgUmVhZHMgYXMgMGIuDQoNClJYTw0KDQo2DQoNCjBiDQoNClJlY2Vp dmVyIE92ZXJydW7igKhTZXQgb24gcmVjZWl2ZSBkYXRhIEZJRk8gb3ZlcnJ1bi4gQ291bGQgYmUg Y2F1c2VkIGVpdGhlcg0KYmVjYXVzZSB0aGVyZSBhcmUgbm8gYXZhaWxhYmxlIHJlY2VpdmUgYnVm ZmVycyBvciBiZWNhdXNlIFBDSSByZWNlaXZlIGJhbmR3aWR0aCBpcyBpbmFkZXF1YXRlLg0KDQoN Cg0KDQpJZiBhbiBpbnRlcnJ1cHQgaGFwcGVucyBkdWUgdG8gYSBub3JtYWwgcmVjZWl2ZSwgc2hv dWxkbuKAmXQgdGhlIHZhbHVlIG9mIHRoaXMgcmVnaXN0ZXIgYmUgMCBhbmQgc3RpbGwgYmUgdmFs aWQ/IEkgYW0gc2VlaW5nIHRoaXMgaXNzdWUgb24gYSBWTSBndWVzdChRRU1VIGh5cGVydmlzb3Ip IHdoZXJlIGR1cmluZyBhIGZsb29kIHRlc3QsIHRoZSBkcml2ZXIgc3RhcnRzIHJlamVjdGluZyBw YWNrZXRzIGJlY2F1c2UgdGhlIHJlZ2lzdGVyIHZhbHVlIGlzIDAuICBDYW4gYW55b25lIHRlbGwg bWUgaWYgaXTigJlzIG9rIG9yIG5vdCB0byByZW1vdmUgdGhlIGNoZWNrIGZvciAwIHZhbHVlID8N Cg0KDQoNClRoYW5rcywNCg0KDQoNCi1TcmVla2FudGgNCg0KDQo= From owner-freebsd-net@freebsd.org Sat Aug 6 18:03:38 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 26A36BAFADC for ; Sat, 6 Aug 2016 18:03:38 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from hz.grosbein.net (hz.grosbein.net [78.47.246.247]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "hz.grosbein.net", Issuer "hz.grosbein.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 9A27A1D61 for ; Sat, 6 Aug 2016 18:03:37 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from eg.sd.rdtc.ru (root@eg.sd.rdtc.ru [62.231.161.221]) by hz.grosbein.net (8.14.9/8.14.9) with ESMTP id u76I3WJs044829 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT) for ; Sat, 6 Aug 2016 20:03:33 +0200 (CEST) (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: Received: from [10.58.0.10] (dadvw [10.58.0.10]) by eg.sd.rdtc.ru (8.15.2/8.15.2) with ESMTPS id u76I3OeQ012899 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT) for ; Sun, 7 Aug 2016 01:03:24 +0700 (KRAT) (envelope-from eugen@grosbein.net) To: "freebsd-net@freebsd.org" From: Eugene Grosbein Subject: 40Gbps http client benchmark Message-ID: <57A62668.7020309@grosbein.net> Date: Sun, 7 Aug 2016 01:03:20 +0700 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.7.2 MIME-Version: 1.0 Content-Type: text/plain; charset=koi8-r; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=0.3 required=5.0 tests=BAYES_00,LOCAL_FROM autolearn=no version=3.3.2 X-Spam-Report: * -2.3 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * 2.6 LOCAL_FROM From my domains X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on hz.grosbein.net X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 06 Aug 2016 18:03:38 -0000 Hi! Is there any high performance benchmark acting as http client for outer http server capable to receive 40Gbps without overwhelming CPU with insane number of syscalls? I've tried benchmarks/wrk version 4.0.2 and it works just fine upto 20Gbps for my hardware: two 6-core (HT disabled) Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz with two dual-port ix(4) 82599ES 10-Gigabit SFI/SFP+ Network Connection combined to single lagg interface (lagghash l4). But each worker pthread of wrk generates too many kqueue() system calls polling for incoming data and eats 100% of its CPU core and cannot receive more. Or, it may be some kqueue() kernel level lock contention, I do not know. More worker threads, more overloaded CPU cores, no increase of transfer over about 20Gbps. I transfer 1MByte-sized files for the benchmark (tried 8MB-sized too). I set sysctl net.inet.tcp.recvspace=4194304 (4MB). I've even patched wrk to to use read buffer sized 4MB instead of its compiled-in default 8k, as it does not change default socket options other than TCP_NODELAY: --- src/wrk.h.orig 2016-08-06 23:20:16.205906000 +0700 +++ src/wrk.h 2016-08-06 23:20:20.460579000 +0700 @@ -18,7 +18,7 @@ #include "ae.h" #include "http_parser.h" -#define RECVBUF 8192 +#define RECVBUF 4194304 #define MAX_THREAD_RATE_S 10000000 #define SOCKET_TIMEOUT_MS 2000 Nothing helps to decrease consumed system time and ktrace(1) assures that's because of kevent() calls. From owner-freebsd-net@freebsd.org Sat Aug 6 18:22:48 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 13E36BB0084 for ; Sat, 6 Aug 2016 18:22:48 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: from mail-wm0-x22f.google.com (mail-wm0-x22f.google.com [IPv6:2a00:1450:400c:c09::22f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9D1FE16AC for ; Sat, 6 Aug 2016 18:22:47 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: by mail-wm0-x22f.google.com with SMTP id i5so77581412wmg.0 for ; Sat, 06 Aug 2016 11:22:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay-co-uk.20150623.gappssmtp.com; s=20150623; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to; bh=LufnxqGQ5BSGkkYlJu3dZPddxtlbz4WKFhae6jNDp+M=; b=d1v7lI5f0Dkcizft+RJWOLbIKKGKrLVncUuxcayNqdMDAvrOIfsYpsbpIUsIQgd1XX F/76WQJk5ET1xrRN+I9k0Nld20yqyicetN/FluKDQxneQQU2gAW9uJkGWnnyccQ/Elb6 g3VkbAzBjwUlKo0Zr5BDQEXRsNEKyH5+ZFFZ34lXNIR6zZBjp9kbXHorhyzkQXJRSqGi mB0Ixv9zIu3b5llhZPcSqdbWOMpAR3oGixu8Q5/YQF3q9LpRwPOtDI/l/43VTAkvf3YT zaHGRTdncW5PXIw0xF7DOaAw17e5Kxh4LY2OreRkDFxMxnFq4rdmoBEonU9g6nbTJ+T7 nifQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to; bh=LufnxqGQ5BSGkkYlJu3dZPddxtlbz4WKFhae6jNDp+M=; b=jLFm4NeRxR7aV5XxqAbmLrIKFsyXUC6vXGzXT3+AWuUvTtus4VM6DpmyncV4nHhHWX qT7NYX1sHCa6Z6dUukdwc80xauBIbHqyF1aOR84HeRVDxDiJosUu0nIwVWCCOIdqFjuV rimisPV0Dw9RLpcFZJwvGNtGQxpkNO12CWO8XgiXrlLPFGCj7YEhoX7XExADhNZt5lMM xJ7WZSLFwXilJ/XAUQ7Xi92XXlpSVoZzoHzLkmTs4UAgeXjC60K9+eXH/vzjuLA1WFrJ vA2a5Hbc1zgGKMTuxLvbDcx2wFAnb4yr2DTFf3kVTusaUFsw2AntfQxrS37+Lm/ws4kN l7EQ== X-Gm-Message-State: AEkoouvKnxrC5I5LPDGcQ7bLNMXMFpWAeddrUvZ521q8Omt5gMyQi3EgF1QZv+l/o++cfiyv X-Received: by 10.195.12.77 with SMTP id eo13mr70566425wjd.142.1470507763960; Sat, 06 Aug 2016 11:22:43 -0700 (PDT) Received: from [10.10.1.58] (liv3d.labs.multiplay.co.uk. [82.69.141.171]) by smtp.gmail.com with ESMTPSA id 207sm14800785wmb.7.2016.08.06.11.22.42 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 06 Aug 2016 11:22:42 -0700 (PDT) Subject: Re: 40Gbps http client benchmark To: freebsd-net@freebsd.org References: <57A62668.7020309@grosbein.net> From: Steven Hartland Message-ID: Date: Sat, 6 Aug 2016 19:22:44 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.2.0 MIME-Version: 1.0 In-Reply-To: <57A62668.7020309@grosbein.net> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 06 Aug 2016 18:22:48 -0000 I'm not sure it will help you at those levels, but have you tried boom? https://github.com/rakyll/boom On 06/08/2016 19:03, Eugene Grosbein wrote: > Hi! > > Is there any high performance benchmark acting as http client for > outer http server > capable to receive 40Gbps without overwhelming CPU with insane number > of syscalls? > > I've tried benchmarks/wrk version 4.0.2 and it works just fine upto > 20Gbps > for my hardware: two 6-core (HT disabled) Intel(R) Xeon(R) CPU E5-2620 > v3 @ 2.40GHz > with two dual-port ix(4) 82599ES 10-Gigabit SFI/SFP+ Network Connection > combined to single lagg interface (lagghash l4). > > But each worker pthread of wrk generates too many kqueue() system calls > polling for incoming data and eats 100% of its CPU core and cannot > receive more. > Or, it may be some kqueue() kernel level lock contention, I do not know. > More worker threads, more overloaded CPU cores, no increase of > transfer over about 20Gbps. > > I transfer 1MByte-sized files for the benchmark (tried 8MB-sized too). > I set sysctl net.inet.tcp.recvspace=4194304 (4MB). > I've even patched wrk to to use read buffer sized 4MB instead of its > compiled-in > default 8k, as it does not change default socket options other than > TCP_NODELAY: > > --- src/wrk.h.orig 2016-08-06 23:20:16.205906000 +0700 > +++ src/wrk.h 2016-08-06 23:20:20.460579000 +0700 > @@ -18,7 +18,7 @@ > #include "ae.h" > #include "http_parser.h" > > -#define RECVBUF 8192 > +#define RECVBUF 4194304 > > #define MAX_THREAD_RATE_S 10000000 > #define SOCKET_TIMEOUT_MS 2000 > > Nothing helps to decrease consumed system time and ktrace(1) assures > that's because of kevent() calls. > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" From owner-freebsd-net@freebsd.org Sat Aug 6 19:05:22 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D651BBB0947 for ; Sat, 6 Aug 2016 19:05:22 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from hz.grosbein.net (hz.grosbein.net [78.47.246.247]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "hz.grosbein.net", Issuer "hz.grosbein.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 623581782 for ; Sat, 6 Aug 2016 19:05:21 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from eg.sd.rdtc.ru (root@eg.sd.rdtc.ru [62.231.161.221]) by hz.grosbein.net (8.14.9/8.14.9) with ESMTP id u76J5HTf044988 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT) for ; Sat, 6 Aug 2016 21:05:17 +0200 (CEST) (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: Received: from [10.58.0.10] (dadvw [10.58.0.10]) by eg.sd.rdtc.ru (8.15.2/8.15.2) with ESMTPS id u76J5CnX013454 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT) for ; Sun, 7 Aug 2016 02:05:12 +0700 (KRAT) (envelope-from eugen@grosbein.net) Subject: Re: 40Gbps http client benchmark To: "freebsd-net@freebsd.org" References: <57A62668.7020309@grosbein.net> From: Eugene Grosbein Message-ID: <57A634E5.9060702@grosbein.net> Date: Sun, 7 Aug 2016 02:05:09 +0700 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.7.2 MIME-Version: 1.0 In-Reply-To: <57A62668.7020309@grosbein.net> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=0.3 required=5.0 tests=BAYES_00,LOCAL_FROM autolearn=no version=3.3.2 X-Spam-Report: * -2.3 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * 2.6 LOCAL_FROM From my domains X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on hz.grosbein.net X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 06 Aug 2016 19:05:22 -0000 07.08.2016 1:03, Eugene Grosbein пишет: > Hi! > > Is there any high performance benchmark acting as http client for outer http server > capable to receive 40Gbps without overwhelming CPU with insane number of syscalls? > > I've tried benchmarks/wrk version 4.0.2 and it works just fine upto 20Gbps > for my hardware: two 6-core (HT disabled) Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz > with two dual-port ix(4) 82599ES 10-Gigabit SFI/SFP+ Network Connection > combined to single lagg interface (lagghash l4). > > But each worker pthread of wrk generates too many kqueue() system calls > polling for incoming data and eats 100% of its CPU core and cannot receive more. > Or, it may be some kqueue() kernel level lock contention, I do not know. > More worker threads, more overloaded CPU cores, no increase of transfer over about 20Gbps. Hmm, it seems that's not number of system calls that hurts me but some kernel-level problem. This is NUMA system running under 10.3-STABLE r303291. It has two NUMA domains with first physical CPU (cores 0-7) and first dual-port ix adapter belonging to one domain and second CPU (cores 8-11) and second dual-port adatper belonging to another domain: http://www.grosbein.net/img/r4.svg Just prepend of "cpuset -l 0-7" to wrk invocation boots transfer upto nearly 30Gbps without any other changes. That's strange.