Date: Fri, 5 Sep 2014 18:09:12 -0400 (EDT) From: Rick Macklem <rmacklem@uoguelph.ca> To: Hans Petter Selasky <hps@selasky.org> Cc: freebsd-net@freebsd.org, Eric Joyner <ricera10@gmail.com>, FreeBSD Current <freebsd-current@freebsd.org>, Scott Long <scottl@freebsd.org>, Jack F Vogel <jfvogel@gmail.com> Subject: Re: [RFC] Patch to improve TSO limitation formula in general Message-ID: <1762951742.33012989.1409954952800.JavaMail.root@uoguelph.ca> In-Reply-To: <540A2ECB.8010502@selasky.org>
next in thread | previous in thread | raw e-mail | index | archive | help
------=_Part_33012987_2077484506.1409954952678 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Hans Petter Selesky wrote: > On 09/05/14 23:19, Eric Joyner wrote: > > There are some concerns if we use this with devices that ixl > > supports: > > > > - The maximum fragment size is 16KB-1, which isn't a power of 2. > > > > Hi Eric, > > Multiplying by powers of two are more fast, than non-powers of two. > So > in this case you would have to use 8KB as a maximum. > Well, I'm no architecture expert, but I really doubt the CPU delay of a non-power of 2 multiply/divide is significant related to doing smaller TSO segments. Long ago (as in 1970s) I did work on machines where shifts for power of 2 multiply/divide was preferable, but these days I doubt it is going to matter?? > > - You can't get the maximum TSO size for ixl devices by multiplying > > the > > maximum number of fragments by the maximum size. > > Instead the number of fragments is AFAIK unlimited, but a segment > > can only > > span 8 mbufs (including the [up to 3] mbufs containing the header), > > and the > > maximum TSO size is 256KB. > > > > And one question: > > > > - Is hdr_size_log2 supposed to be the length of the L2 header? We > > can fit > > 254 L2 bytes in our hardware during a TSO, so if that's the value, > > I guess > > that's fine, barring the it-not-being-a-power-of-2 issue. > > This is the ethernet / vlan headers. It is added with the > TCP/IP-header > in the end. > > > > > With all that said, the 8 mbuf limit per segment issue is a TSO > > limitation > > that we'd like to notify the stack about, so I wonder if that could > > be > > incorporated along with this. Right now, our driver checks to see > > if a > > segment in a TSO spans more than six mbufs and then m_defrag()'s > > the entire > > chain if one exists; it's less than optimal but necessary to > > prevent errors. > At this time, if there is a limit of 8 TSO segments (mbufs) in a transmit list, you will need to set: ifp->if_hw_tsomax = 8 * MCLBYTES - (ETHER_HDR_LEN + ETHER_VLAN_ENCAP_LEN); - just before the call to ether_ifattach(ifp); I do have an untested patch (attached in case anyone is interested) which adds if_hw_tsomaxseg that drivers can set to their maximum number of transmit segments (mbufs) fot TSO. This value is then used by tcp_output() to generate appropriately sized TSO segments. However, I'm just working on getting a way to test this patch, so I can't say if/when it will be in head. rick > It is not impossible to move from log2 syntax to non-log2 syntax, > hence > the logic will be exactly the same, only that the required division > and > multiplication will have a bit overhead I guess. > > Could you make a patch on top of my patch with the changes you think > are > required to fully support the ixl hardware? Or propose a new patch > which > also serves the MLX needs? > > Thank you! > > --HPS > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to > "freebsd-net-unsubscribe@freebsd.org" > ------=_Part_33012987_2077484506.1409954952678 Content-Type: text/x-patch; name=tsomaxseg.patch Content-Disposition: attachment; filename=tsomaxseg.patch Content-Transfer-Encoding: base64 LS0tIGtlcm4vdWlwY19zb2NrYnVmLmMuc2F2CTIwMTQtMDEtMzAgMjA6Mjc6MTcuMDAwMDAwMDAw IC0wNTAwCisrKyBrZXJuL3VpcGNfc29ja2J1Zi5jCTIwMTQtMDEtMzAgMjI6MTI6MDguMDAwMDAw MDAwIC0wNTAwCkBAIC05NjUsNiArOTY1LDM5IEBAIHNic25kcHRyKHN0cnVjdCBzb2NrYnVmICpz YiwgdV9pbnQgb2ZmLCAKIH0KIAogLyoKKyAqIFJldHVybiB0aGUgZmlyc3QgbWJ1ZiBmb3IgdGhl IHByb3ZpZGVkIG9mZnNldC4KKyAqLworc3RydWN0IG1idWYgKgorc2JzbmRtYnVmKHN0cnVjdCBz b2NrYnVmICpzYiwgdV9pbnQgb2ZmLCBsb25nICpmaXJzdF9sZW4pCit7CisJc3RydWN0IG1idWYg Km07CisKKwlLQVNTRVJUKHNiLT5zYl9tYiAhPSBOVUxMLCAoIiVzOiBzYl9tYiBpcyBOVUxMIiwg X19mdW5jX18pKTsKKworCSpmaXJzdF9sZW4gPSAwOworCS8qCisJICogSXMgb2ZmIGJlbG93IHN0 b3JlZCBvZmZzZXQ/IEhhcHBlbnMgb24gcmV0cmFuc21pdHMuCisJICogSWYgc28sIGp1c3QgdXNl IHNiX21iLgorCSAqLworCWlmIChzYi0+c2Jfc25kcHRyID09IE5VTEwgfHwgc2ItPnNiX3NuZHB0 cm9mZiA+IG9mZikKKwkJbSA9IHNiLT5zYl9tYjsKKwllbHNlIHsKKwkJbSA9IHNiLT5zYl9zbmRw dHI7CisJCW9mZiAtPSBzYi0+c2Jfc25kcHRyb2ZmOworCX0KKwl3aGlsZSAob2ZmID4gMCAmJiBt ICE9IE5VTEwpIHsKKwkJaWYgKG9mZiA8IG0tPm1fbGVuKQorCQkJYnJlYWs7CisJCW9mZiAtPSBt LT5tX2xlbjsKKwkJbSA9IG0tPm1fbmV4dDsKKwl9CisJaWYgKG0gIT0gTlVMTCkKKwkJKmZpcnN0 X2xlbiA9IG0tPm1fbGVuIC0gb2ZmOworCisJcmV0dXJuIChtKTsKK30KKworLyoKICAqIERyb3Ag YSByZWNvcmQgb2ZmIHRoZSBmcm9udCBvZiBhIHNvY2tidWYgYW5kIG1vdmUgdGhlIG5leHQgcmVj b3JkIHRvIHRoZQogICogZnJvbnQuCiAgKi8KLS0tIHN5cy9zb2NrYnVmLmguc2F2CTIwMTQtMDEt MzAgMjA6NDI6MjguMDAwMDAwMDAwIC0wNTAwCisrKyBzeXMvc29ja2J1Zi5oCTIwMTQtMDEtMzAg MjI6MDg6NDMuMDAwMDAwMDAwIC0wNTAwCkBAIC0xNTMsNiArMTUzLDggQEAgaW50CXNicmVzZXJ2 ZV9sb2NrZWQoc3RydWN0IHNvY2tidWYgKnNiLAogCSAgICBzdHJ1Y3QgdGhyZWFkICp0ZCk7CiBz dHJ1Y3QgbWJ1ZiAqCiAJc2JzbmRwdHIoc3RydWN0IHNvY2tidWYgKnNiLCB1X2ludCBvZmYsIHVf aW50IGxlbiwgdV9pbnQgKm1vZmYpOworc3RydWN0IG1idWYgKgorCXNic25kbWJ1ZihzdHJ1Y3Qg c29ja2J1ZiAqc2IsIHVfaW50IG9mZiwgbG9uZyAqZmlyc3RfbGVuKTsKIHZvaWQJc2J0b3hzb2Nr YnVmKHN0cnVjdCBzb2NrYnVmICpzYiwgc3RydWN0IHhzb2NrYnVmICp4c2IpOwogaW50CXNid2Fp dChzdHJ1Y3Qgc29ja2J1ZiAqc2IpOwogaW50CXNibG9jayhzdHJ1Y3Qgc29ja2J1ZiAqc2IsIGlu dCBmbGFncyk7Ci0tLSBuZXRpbmV0L3RjcF9pbnB1dC5jLnNhdgkyMDE0LTAxLTMwIDE5OjM3OjUy LjAwMDAwMDAwMCAtMDUwMAorKysgbmV0aW5ldC90Y3BfaW5wdXQuYwkyMDE0LTAxLTMwIDE5OjM5 OjA3LjAwMDAwMDAwMCAtMDUwMApAQCAtMzYyNyw2ICszNjI3LDcgQEAgdGNwX21zcyhzdHJ1Y3Qg dGNwY2IgKnRwLCBpbnQgb2ZmZXIpCiAJaWYgKGNhcC5pZmNhcCAmIENTVU1fVFNPKSB7CiAJCXRw LT50X2ZsYWdzIHw9IFRGX1RTTzsKIAkJdHAtPnRfdHNvbWF4ID0gY2FwLnRzb21heDsKKwkJdHAt PnRfdHNvbWF4c2VncyA9IGNhcC50c29tYXhzZWdzOwogCX0KIH0KIAotLS0gbmV0aW5ldC90Y3Bf b3V0cHV0LmMuc2F2CTIwMTQtMDEtMzAgMTg6NTU6MTUuMDAwMDAwMDAwIC0wNTAwCisrKyBuZXRp bmV0L3RjcF9vdXRwdXQuYwkyMDE0LTAxLTMwIDIyOjE4OjU2LjAwMDAwMDAwMCAtMDUwMApAQCAt MTY2LDggKzE2Niw4IEBAIGludAogdGNwX291dHB1dChzdHJ1Y3QgdGNwY2IgKnRwKQogewogCXN0 cnVjdCBzb2NrZXQgKnNvID0gdHAtPnRfaW5wY2ItPmlucF9zb2NrZXQ7Ci0JbG9uZyBsZW4sIHJl Y3dpbiwgc2VuZHdpbjsKLQlpbnQgb2ZmLCBmbGFncywgZXJyb3IgPSAwOwkvKiBLZWVwIGNvbXBp bGVyIGhhcHB5ICovCisJbG9uZyBsZW4sIHJlY3dpbiwgc2VuZHdpbiwgdHNvX3RsZW47CisJaW50 IGNudCwgb2ZmLCBmbGFncywgZXJyb3IgPSAwOwkvKiBLZWVwIGNvbXBpbGVyIGhhcHB5ICovCiAJ c3RydWN0IG1idWYgKm07CiAJc3RydWN0IGlwICppcCA9IE5VTEw7CiAJc3RydWN0IGlwb3ZseSAq aXBvdiA9IE5VTEw7CkBAIC03ODAsNiArNzgwLDI0IEBAIHNlbmQ6CiAJCQl9CiAKIAkJCS8qCisJ CQkgKiBMaW1pdCB0aGUgbnVtYmVyIG9mIFRTTyB0cmFuc21pdCBzZWdtZW50cyAobWJ1ZnMKKwkJ CSAqIGluIG1idWYgbGlzdCkgdG8gdHAtPnRfdHNvbWF4c2Vncy4KKwkJCSAqLworCQkJY250ID0g MDsKKwkJCW0gPSBzYnNuZG1idWYoJnNvLT5zb19zbmQsIG9mZiwgJnRzb190bGVuKTsKKwkJCXdo aWxlIChtICE9IE5VTEwgJiYgY250IDwgdHAtPnRfdHNvbWF4c2VncyAmJgorCQkJICAgIHRzb190 bGVuIDwgbGVuKSB7CisJCQkJaWYgKGNudCA+IDApCisJCQkJCXRzb190bGVuICs9IG0tPm1fbGVu OworCQkJCWNudCsrOworCQkJCW0gPSBtLT5tX25leHQ7CisJCQl9CisJCQlpZiAobSAhPSBOVUxM ICYmIHRzb190bGVuIDwgbGVuKSB7CisJCQkJbGVuID0gdHNvX3RsZW47CisJCQkJc2VuZGFsb3Qg PSAxOworCQkJfQorCisJCQkvKgogCQkJICogUHJldmVudCB0aGUgbGFzdCBzZWdtZW50IGZyb20g YmVpbmcKIAkJCSAqIGZyYWN0aW9uYWwgdW5sZXNzIHRoZSBzZW5kIHNvY2tidWYgY2FuCiAJCQkg KiBiZSBlbXB0aWVkLgotLS0gbmV0aW5ldC90Y3Bfc3Vici5jLnNhdgkyMDE0LTAxLTMwIDE5OjQ0 OjM1LjAwMDAwMDAwMCAtMDUwMAorKysgbmV0aW5ldC90Y3Bfc3Vici5jCTIwMTQtMDEtMzAgMjA6 NTY6MTIuMDAwMDAwMDAwIC0wNTAwCkBAIC0xODAwLDYgKzE4MDAsMTIgQEAgdGNwX21heG10dShz dHJ1Y3QgaW5fY29ubmluZm8gKmluYywgc3RydQogCQkJICAgIGlmcC0+aWZfaHdhc3Npc3QgJiBD U1VNX1RTTykKIAkJCQljYXAtPmlmY2FwIHw9IENTVU1fVFNPOwogCQkJCWNhcC0+dHNvbWF4ID0g aWZwLT5pZl9od190c29tYXg7CisjaWZkZWYgbm90eWV0CisJCQkJY2FwLT50c29tYXhzZWdzID0g aWZwLT5pZl9od190c29tYXhzZWdzOworI2VuZGlmCisJCQkJaWYgKGNhcC0+dHNvbWF4c2VncyA9 PSAwKQorCQkJCQljYXAtPnRzb21heHNlZ3MgPQorCQkJCQkgICAgVENQVFNPX01BWF9UWF9TRUdT X0RFRkFVTFQ7CiAJCX0KIAkJUlRGUkVFKHNyby5yb19ydCk7CiAJfQotLS0gbmV0aW5ldC90Y3Bf dmFyLmguc2F2CTIwMTQtMDEtMzAgMTk6Mzk6MjIuMDAwMDAwMDAwIC0wNTAwCisrKyBuZXRpbmV0 L3RjcF92YXIuaAkyMDE0LTAxLTMwIDIwOjUyOjU3LjAwMDAwMDAwMCAtMDUwMApAQCAtMjA5LDYg KzIwOSw3IEBAIHN0cnVjdCB0Y3BjYiB7CiAJdV9pbnQJdF9rZWVwY250OwkJLyogbnVtYmVyIG9m IGtlZXBhbGl2ZXMgYmVmb3JlIGNsb3NlICovCiAKIAl1X2ludAl0X3Rzb21heDsJCS8qIHRzbyBi dXJzdCBsZW5ndGggbGltaXQgKi8KKwl1X2ludAl0X3Rzb21heHNlZ3M7CQkvKiB0c28gYnVyc3Qg c2VnbWVudCBsaW1pdCAqLwogCiAJdWludDMyX3QgdF9pc3BhcmVbOF07CQkvKiA1IFVUTywgMyBU QkQgKi8KIAl2b2lkCSp0X3BzcGFyZTJbNF07CQkvKiA0IFRCRCAqLwpAQCAtMjY4LDYgKzI2OSwx MSBAQCBzdHJ1Y3QgdGNwY2IgewogI2RlZmluZQlUQ1BPT0JfSEFWRURBVEEJMHgwMQogI2RlZmlu ZQlUQ1BPT0JfSEFEREFUQQkweDAyCiAKKy8qCisgKiBEZWZhdWx0IHZhbHVlIGZvciBUU08gbWF4 aW11bSBudW1iZXIgb2YgdHJhbnNtaXQgc2VnbWVudHMgKGNvdW50IG9mIG1idWZzKS4KKyAqLwor I2RlZmluZQlUQ1BUU09fTUFYX1RYX1NFR1NfREVGQVVMVAkzMAorCiAjaWZkZWYgVENQX1NJR05B VFVSRQogLyoKICAqIERlZmluZXMgd2hpY2ggYXJlIG5lZWRlZCBieSB0aGUgeGZvcm1fdGNwIG1v ZHVsZSBhbmQgdGNwX1tpbnxvdXRdcHV0CkBAIC0zMzMsNiArMzM5LDcgQEAgc3RydWN0IGhjX21l dHJpY3NfbGl0ZSB7CS8qIG11c3Qgc3RheSBpbgogc3RydWN0IHRjcF9pZmNhcCB7CiAJaW50CWlm Y2FwOwogCXVfaW50CXRzb21heDsKKwl1X2ludAl0c29tYXhzZWdzOwogfTsKIAogI2lmbmRlZiBf TkVUSU5FVF9JTl9QQ0JfSF8K ------=_Part_33012987_2077484506.1409954952678--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1762951742.33012989.1409954952800.JavaMail.root>