From owner-freebsd-net@FreeBSD.ORG Thu Mar 27 22:44:02 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 9F85DB14; Thu, 27 Mar 2014 22:44:02 +0000 (UTC) Received: from esa-annu.net.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id 2D901F9; Thu, 27 Mar 2014 22:44:01 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AqAEABmpNFODaFve/2dsb2JhbABZhBiDCr9UgTB0giUBAQEEIwRSGw4KERkCBFUGE4d5rnyiUheOIyMZGweCb4FJBJBVmiyDSyGBLUE X-IronPort-AV: E=Sophos;i="4.97,745,1389762000"; d="scan'208";a="109674004" Received: from muskoka.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.222]) by esa-annu.net.uoguelph.ca with ESMTP; 27 Mar 2014 18:44:01 -0400 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id 0495CB3F4A; Thu, 27 Mar 2014 18:44:01 -0400 (EDT) Date: Thu, 27 Mar 2014 18:44:01 -0400 (EDT) From: Rick Macklem To: Christopher Forgeron Message-ID: <1292881633.1858906.1395960241007.JavaMail.root@uoguelph.ca> In-Reply-To: Subject: Re: 9.2 ixgbe tx queue hang MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_Part_1858903_1764427006.1395960241003" X-Originating-IP: [172.17.91.202] X-Mailer: Zimbra 7.2.1_GA_2790 (ZimbraWebClient - FF3.0 (Win)/7.2.1_GA_2790) Cc: Pyun YongHyeon , Andre Oppermann , FreeBSD Net , Markus Gebert , Jack Vogel , Garrett Wollman X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Mar 2014 22:44:02 -0000 ------=_Part_1858903_1764427006.1395960241003 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Christopher Forgeron wrote: > > > > > > > On Wed, Mar 26, 2014 at 9:35 PM, Rick Macklem < rmacklem@uoguelph.ca > > wrote: > > > > > I've suggested in the other thread what you suggested in a recent > post...ie. to change the default, at least until the propagation > of driver set values is resolved. > > rick > > > > I wonder if we need to worry about propagating values up from the > sub-if's - Setting the default in if.c means this is set for all > if's, and it's a simple 1 line code change. If a specific 'if' needs > a different value, it can be set before ether_attach() is called. > > > I'm more concerned with the equation we use to calculate if_hw_tsomax > - Are we considering the right variables? Are we thinking on the > wrong OSI layer for headers? > Well, I'm pragmatic (which means I mostly care about some fix that works), but it seems to me that: - The problem is that some TSO enabled network drivers/hardware can only handle 32 transmit segments (or 32 mbufs in the chain for the TSO packet to be transmitted, if that is clearer). --> Since the problem is in certain drivers, it seems that those drivers should be where the long term fix goes. --> Since some hardware can't handle more than 32, it seems that the driver should be able to specify that limit, which tcp_output() can then apply. I have an untested patch that does this by adding if_hw_tsomaxseg. (The attachment called tsomaxseg.patch.) Changing if_hw_tsomax or its default value is just a hack that gets tcp_output() to apply a limit that the driver can then fix to 32 mbufs in the chain via m_defrag(). Since if_hw_tsomax (and if_hw_tsomaxseg in the untested patch) aren't propagated up through lagg, that needs to be fixed. (Yet another attached untested patch called lagg.patch.) As I said before, I don't see these patches getting tested/reviewed etc in time for 9.3, so I think reducing the default value of if_hw_tsomax is a reasonable short term hack to work around the problem. (And it sounds like Pyun YongHyeon has volunteered to fix many of the drivers, where the 32 limit isn't a hardware one.) rick ------=_Part_1858903_1764427006.1395960241003 Content-Type: text/x-patch; name=tsomaxseg.patch Content-Disposition: attachment; filename=tsomaxseg.patch Content-Transfer-Encoding: base64 LS0tIGtlcm4vdWlwY19zb2NrYnVmLmMuc2F2CTIwMTQtMDEtMzAgMjA6Mjc6MTcuMDAwMDAwMDAw IC0wNTAwCisrKyBrZXJuL3VpcGNfc29ja2J1Zi5jCTIwMTQtMDEtMzAgMjI6MTI6MDguMDAwMDAw MDAwIC0wNTAwCkBAIC05NjUsNiArOTY1LDM5IEBAIHNic25kcHRyKHN0cnVjdCBzb2NrYnVmICpz YiwgdV9pbnQgb2ZmLCAKIH0KIAogLyoKKyAqIFJldHVybiB0aGUgZmlyc3QgbWJ1ZiBmb3IgdGhl IHByb3ZpZGVkIG9mZnNldC4KKyAqLworc3RydWN0IG1idWYgKgorc2JzbmRtYnVmKHN0cnVjdCBz b2NrYnVmICpzYiwgdV9pbnQgb2ZmLCBsb25nICpmaXJzdF9sZW4pCit7CisJc3RydWN0IG1idWYg Km07CisKKwlLQVNTRVJUKHNiLT5zYl9tYiAhPSBOVUxMLCAoIiVzOiBzYl9tYiBpcyBOVUxMIiwg X19mdW5jX18pKTsKKworCSpmaXJzdF9sZW4gPSAwOworCS8qCisJICogSXMgb2ZmIGJlbG93IHN0 b3JlZCBvZmZzZXQ/IEhhcHBlbnMgb24gcmV0cmFuc21pdHMuCisJICogSWYgc28sIGp1c3QgdXNl IHNiX21iLgorCSAqLworCWlmIChzYi0+c2Jfc25kcHRyID09IE5VTEwgfHwgc2ItPnNiX3NuZHB0 cm9mZiA+IG9mZikKKwkJbSA9IHNiLT5zYl9tYjsKKwllbHNlIHsKKwkJbSA9IHNiLT5zYl9zbmRw dHI7CisJCW9mZiAtPSBzYi0+c2Jfc25kcHRyb2ZmOworCX0KKwl3aGlsZSAob2ZmID4gMCAmJiBt ICE9IE5VTEwpIHsKKwkJaWYgKG9mZiA8IG0tPm1fbGVuKQorCQkJYnJlYWs7CisJCW9mZiAtPSBt LT5tX2xlbjsKKwkJbSA9IG0tPm1fbmV4dDsKKwl9CisJaWYgKG0gIT0gTlVMTCkKKwkJKmZpcnN0 X2xlbiA9IG0tPm1fbGVuIC0gb2ZmOworCisJcmV0dXJuIChtKTsKK30KKworLyoKICAqIERyb3Ag YSByZWNvcmQgb2ZmIHRoZSBmcm9udCBvZiBhIHNvY2tidWYgYW5kIG1vdmUgdGhlIG5leHQgcmVj b3JkIHRvIHRoZQogICogZnJvbnQuCiAgKi8KLS0tIHN5cy9zb2NrYnVmLmguc2F2CTIwMTQtMDEt MzAgMjA6NDI6MjguMDAwMDAwMDAwIC0wNTAwCisrKyBzeXMvc29ja2J1Zi5oCTIwMTQtMDEtMzAg MjI6MDg6NDMuMDAwMDAwMDAwIC0wNTAwCkBAIC0xNTMsNiArMTUzLDggQEAgaW50CXNicmVzZXJ2 ZV9sb2NrZWQoc3RydWN0IHNvY2tidWYgKnNiLAogCSAgICBzdHJ1Y3QgdGhyZWFkICp0ZCk7CiBz dHJ1Y3QgbWJ1ZiAqCiAJc2JzbmRwdHIoc3RydWN0IHNvY2tidWYgKnNiLCB1X2ludCBvZmYsIHVf aW50IGxlbiwgdV9pbnQgKm1vZmYpOworc3RydWN0IG1idWYgKgorCXNic25kbWJ1ZihzdHJ1Y3Qg c29ja2J1ZiAqc2IsIHVfaW50IG9mZiwgbG9uZyAqZmlyc3RfbGVuKTsKIHZvaWQJc2J0b3hzb2Nr YnVmKHN0cnVjdCBzb2NrYnVmICpzYiwgc3RydWN0IHhzb2NrYnVmICp4c2IpOwogaW50CXNid2Fp dChzdHJ1Y3Qgc29ja2J1ZiAqc2IpOwogaW50CXNibG9jayhzdHJ1Y3Qgc29ja2J1ZiAqc2IsIGlu dCBmbGFncyk7Ci0tLSBuZXRpbmV0L3RjcF9pbnB1dC5jLnNhdgkyMDE0LTAxLTMwIDE5OjM3OjUy LjAwMDAwMDAwMCAtMDUwMAorKysgbmV0aW5ldC90Y3BfaW5wdXQuYwkyMDE0LTAxLTMwIDE5OjM5 OjA3LjAwMDAwMDAwMCAtMDUwMApAQCAtMzYyNyw2ICszNjI3LDcgQEAgdGNwX21zcyhzdHJ1Y3Qg dGNwY2IgKnRwLCBpbnQgb2ZmZXIpCiAJaWYgKGNhcC5pZmNhcCAmIENTVU1fVFNPKSB7CiAJCXRw LT50X2ZsYWdzIHw9IFRGX1RTTzsKIAkJdHAtPnRfdHNvbWF4ID0gY2FwLnRzb21heDsKKwkJdHAt PnRfdHNvbWF4c2VncyA9IGNhcC50c29tYXhzZWdzOwogCX0KIH0KIAotLS0gbmV0aW5ldC90Y3Bf b3V0cHV0LmMuc2F2CTIwMTQtMDEtMzAgMTg6NTU6MTUuMDAwMDAwMDAwIC0wNTAwCisrKyBuZXRp bmV0L3RjcF9vdXRwdXQuYwkyMDE0LTAxLTMwIDIyOjE4OjU2LjAwMDAwMDAwMCAtMDUwMApAQCAt MTY2LDggKzE2Niw4IEBAIGludAogdGNwX291dHB1dChzdHJ1Y3QgdGNwY2IgKnRwKQogewogCXN0 cnVjdCBzb2NrZXQgKnNvID0gdHAtPnRfaW5wY2ItPmlucF9zb2NrZXQ7Ci0JbG9uZyBsZW4sIHJl Y3dpbiwgc2VuZHdpbjsKLQlpbnQgb2ZmLCBmbGFncywgZXJyb3IgPSAwOwkvKiBLZWVwIGNvbXBp bGVyIGhhcHB5ICovCisJbG9uZyBsZW4sIHJlY3dpbiwgc2VuZHdpbiwgdHNvX3RsZW47CisJaW50 IGNudCwgb2ZmLCBmbGFncywgZXJyb3IgPSAwOwkvKiBLZWVwIGNvbXBpbGVyIGhhcHB5ICovCiAJ c3RydWN0IG1idWYgKm07CiAJc3RydWN0IGlwICppcCA9IE5VTEw7CiAJc3RydWN0IGlwb3ZseSAq aXBvdiA9IE5VTEw7CkBAIC03ODAsNiArNzgwLDI0IEBAIHNlbmQ6CiAJCQl9CiAKIAkJCS8qCisJ CQkgKiBMaW1pdCB0aGUgbnVtYmVyIG9mIFRTTyB0cmFuc21pdCBzZWdtZW50cyAobWJ1ZnMKKwkJ CSAqIGluIG1idWYgbGlzdCkgdG8gdHAtPnRfdHNvbWF4c2Vncy4KKwkJCSAqLworCQkJY250ID0g MDsKKwkJCW0gPSBzYnNuZG1idWYoJnNvLT5zb19zbmQsIG9mZiwgJnRzb190bGVuKTsKKwkJCXdo aWxlIChtICE9IE5VTEwgJiYgY250IDwgdHAtPnRfdHNvbWF4c2VncyAmJgorCQkJICAgIHRzb190 bGVuIDwgbGVuKSB7CisJCQkJaWYgKGNudCA+IDApCisJCQkJCXRzb190bGVuICs9IG0tPm1fbGVu OworCQkJCWNudCsrOworCQkJCW0gPSBtLT5tX25leHQ7CisJCQl9CisJCQlpZiAobSAhPSBOVUxM ICYmIHRzb190bGVuIDwgbGVuKSB7CisJCQkJbGVuID0gdHNvX3RsZW47CisJCQkJc2VuZGFsb3Qg PSAxOworCQkJfQorCisJCQkvKgogCQkJICogUHJldmVudCB0aGUgbGFzdCBzZWdtZW50IGZyb20g YmVpbmcKIAkJCSAqIGZyYWN0aW9uYWwgdW5sZXNzIHRoZSBzZW5kIHNvY2tidWYgY2FuCiAJCQkg KiBiZSBlbXB0aWVkLgotLS0gbmV0aW5ldC90Y3Bfc3Vici5jLnNhdgkyMDE0LTAxLTMwIDE5OjQ0 OjM1LjAwMDAwMDAwMCAtMDUwMAorKysgbmV0aW5ldC90Y3Bfc3Vici5jCTIwMTQtMDEtMzAgMjA6 NTY6MTIuMDAwMDAwMDAwIC0wNTAwCkBAIC0xODAwLDYgKzE4MDAsMTIgQEAgdGNwX21heG10dShz dHJ1Y3QgaW5fY29ubmluZm8gKmluYywgc3RydQogCQkJICAgIGlmcC0+aWZfaHdhc3Npc3QgJiBD U1VNX1RTTykKIAkJCQljYXAtPmlmY2FwIHw9IENTVU1fVFNPOwogCQkJCWNhcC0+dHNvbWF4ID0g aWZwLT5pZl9od190c29tYXg7CisjaWZkZWYgbm90eWV0CisJCQkJY2FwLT50c29tYXhzZWdzID0g aWZwLT5pZl9od190c29tYXhzZWdzOworI2VuZGlmCisJCQkJaWYgKGNhcC0+dHNvbWF4c2VncyA9 PSAwKQorCQkJCQljYXAtPnRzb21heHNlZ3MgPQorCQkJCQkgICAgVENQVFNPX01BWF9UWF9TRUdT X0RFRkFVTFQ7CiAJCX0KIAkJUlRGUkVFKHNyby5yb19ydCk7CiAJfQotLS0gbmV0aW5ldC90Y3Bf dmFyLmguc2F2CTIwMTQtMDEtMzAgMTk6Mzk6MjIuMDAwMDAwMDAwIC0wNTAwCisrKyBuZXRpbmV0 L3RjcF92YXIuaAkyMDE0LTAxLTMwIDIwOjUyOjU3LjAwMDAwMDAwMCAtMDUwMApAQCAtMjA5LDYg KzIwOSw3IEBAIHN0cnVjdCB0Y3BjYiB7CiAJdV9pbnQJdF9rZWVwY250OwkJLyogbnVtYmVyIG9m IGtlZXBhbGl2ZXMgYmVmb3JlIGNsb3NlICovCiAKIAl1X2ludAl0X3Rzb21heDsJCS8qIHRzbyBi dXJzdCBsZW5ndGggbGltaXQgKi8KKwl1X2ludAl0X3Rzb21heHNlZ3M7CQkvKiB0c28gYnVyc3Qg c2VnbWVudCBsaW1pdCAqLwogCiAJdWludDMyX3QgdF9pc3BhcmVbOF07CQkvKiA1IFVUTywgMyBU QkQgKi8KIAl2b2lkCSp0X3BzcGFyZTJbNF07CQkvKiA0IFRCRCAqLwpAQCAtMjY4LDYgKzI2OSwx MSBAQCBzdHJ1Y3QgdGNwY2IgewogI2RlZmluZQlUQ1BPT0JfSEFWRURBVEEJMHgwMQogI2RlZmlu ZQlUQ1BPT0JfSEFEREFUQQkweDAyCiAKKy8qCisgKiBEZWZhdWx0IHZhbHVlIGZvciBUU08gbWF4 aW11bSBudW1iZXIgb2YgdHJhbnNtaXQgc2VnbWVudHMgKGNvdW50IG9mIG1idWZzKS4KKyAqLwor I2RlZmluZQlUQ1BUU09fTUFYX1RYX1NFR1NfREVGQVVMVAkzMAorCiAjaWZkZWYgVENQX1NJR05B VFVSRQogLyoKICAqIERlZmluZXMgd2hpY2ggYXJlIG5lZWRlZCBieSB0aGUgeGZvcm1fdGNwIG1v ZHVsZSBhbmQgdGNwX1tpbnxvdXRdcHV0CkBAIC0zMzMsNiArMzM5LDcgQEAgc3RydWN0IGhjX21l dHJpY3NfbGl0ZSB7CS8qIG11c3Qgc3RheSBpbgogc3RydWN0IHRjcF9pZmNhcCB7CiAJaW50CWlm Y2FwOwogCXVfaW50CXRzb21heDsKKwl1X2ludAl0c29tYXhzZWdzOwogfTsKIAogI2lmbmRlZiBf TkVUSU5FVF9JTl9QQ0JfSF8K ------=_Part_1858903_1764427006.1395960241003 Content-Type: text/x-patch; name=lagg.patch Content-Disposition: attachment; filename=lagg.patch Content-Transfer-Encoding: base64 LS0tIG5ldC9pZl9sYWdnLmMuc2F2CTIwMTQtMDMtMjYgMjE6MDQ6MzYuMDAwMDAwMDAwIC0wNDAw CisrKyBuZXQvaWZfbGFnZy5jCTIwMTQtMDMtMjYgMjE6MDc6NTcuMDAwMDAwMDAwIC0wNDAwCkBA IC00MjUsNiArNDI1LDcgQEAgbGFnZ19jYXBhYmlsaXRpZXMoc3RydWN0IGxhZ2dfc29mdGMgKnNj KQogCXN0cnVjdCBsYWdnX3BvcnQgKmxwOwogCWludCBjYXAgPSB+MCwgZW5hID0gfjA7CiAJdV9s b25nIGh3YSA9IH4wVUw7CisJdV9pbnQgaHdfdHNvbWF4ID0gSVBfTUFYUEFDS0VUOwogCiAJTEFH R19XTE9DS19BU1NFUlQoc2MpOwogCkBAIC00MzMsNiArNDM0LDggQEAgbGFnZ19jYXBhYmlsaXRp ZXMoc3RydWN0IGxhZ2dfc29mdGMgKnNjKQogCQljYXAgJj0gbHAtPmxwX2lmcC0+aWZfY2FwYWJp bGl0aWVzOwogCQllbmEgJj0gbHAtPmxwX2lmcC0+aWZfY2FwZW5hYmxlOwogCQlod2EgJj0gbHAt PmxwX2lmcC0+aWZfaHdhc3Npc3Q7CisJCWlmIChscC0+bHBfaWZwLT5pZl9od190c29tYXggPCBo d190c29tYXgpCisJCQlod190c29tYXggPSBscC0+bHBfaWZwLT5pZl9od190c29tYXg7CiAJfQog CWNhcCA9IChjYXAgPT0gfjAgPyAwIDogY2FwKTsKIAllbmEgPSAoZW5hID09IH4wID8gMCA6IGVu YSk7CkBAIC00NDAsMTAgKzQ0MywxMiBAQCBsYWdnX2NhcGFiaWxpdGllcyhzdHJ1Y3QgbGFnZ19z b2Z0YyAqc2MpCiAKIAlpZiAoc2MtPnNjX2lmcC0+aWZfY2FwYWJpbGl0aWVzICE9IGNhcCB8fAog CSAgICBzYy0+c2NfaWZwLT5pZl9jYXBlbmFibGUgIT0gZW5hIHx8Ci0JICAgIHNjLT5zY19pZnAt PmlmX2h3YXNzaXN0ICE9IGh3YSkgeworCSAgICBzYy0+c2NfaWZwLT5pZl9od2Fzc2lzdCAhPSBo d2EgfHwKKwkgICAgc2MtPnNjX2lmcC0+aWZfaHdfdHNvbWF4ICE9IGh3X3Rzb21heCkgewogCQlz Yy0+c2NfaWZwLT5pZl9jYXBhYmlsaXRpZXMgPSBjYXA7CiAJCXNjLT5zY19pZnAtPmlmX2NhcGVu YWJsZSA9IGVuYTsKIAkJc2MtPnNjX2lmcC0+aWZfaHdhc3Npc3QgPSBod2E7CisJCXNjLT5zY19p ZnAtPmlmX2h3X3Rzb21heCA9IGh3X3Rzb21heDsKIAkJZ2V0bWljcm90aW1lKCZzYy0+c2NfaWZw LT5pZl9sYXN0Y2hhbmdlKTsKIAogCQlpZiAoc2MtPnNjX2lmZmxhZ3MgJiBJRkZfREVCVUcpCg== ------=_Part_1858903_1764427006.1395960241003--