Date: Mon, 23 Jun 2014 10:27:26 +0800 From: Marcelo Araujo <araujobsdport@gmail.com> To: FreeBSD Net <freebsd-net@freebsd.org> Subject: [patch][lagg] - Set a better granularity and distribution on roundrobin protocol. Message-ID: <CAOfEmZjmb1bdvn0gR6vD1WeP8o8g7KwXod4TE0iJfa=nicyeng@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
--001a11c37bdc896b5204fc779588 Content-Type: text/plain; charset=UTF-8 Hello guys, I made some changes on roundrobin protocol where from now you can via sysctl(8) set a better packets distribution among the interfaces that are part of the lagg(4) group. My motivation for this change was interfaces that use TSO, as example ixgbe(4), the performance is terrible, as we can't full fill the TSO buffer at once, the throughput drops expressively and we have much more sack between hosts. So, with this patch we can set the number of packets that will be send before switch to the next interface. In my testbed using ixgbe(4), I had a very good performance as you can see bellow: 1) Without patch: ------------------------------------------------------------ Client connecting to 192.168.1.2, TCP port 5001 TCP window size: 32.5 KByte (default) ------------------------------------------------------------ [ 3] local 192.168.1.1 port 32808 connected with 192.168.1.2 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0- 1.0 sec 406 MBytes 3.40 Gbits/sec [ 3] 1.0- 2.0 sec 391 MBytes 3.28 Gbits/sec [ 3] 2.0- 3.0 sec 406 MBytes 3.41 Gbits/sec [ 3] 3.0- 4.0 sec 585 MBytes 4.91 Gbits/sec [ 3] 4.0- 5.0 sec 477 MBytes 4.00 Gbits/sec [ 3] 5.0- 6.0 sec 429 MBytes 3.60 Gbits/sec [ 3] 6.0- 7.0 sec 520 MBytes 4.36 Gbits/sec [ 3] 7.0- 8.0 sec 385 MBytes 3.23 Gbits/sec [ 3] 8.0- 9.0 sec 414 MBytes 3.48 Gbits/sec [ 3] 9.0-10.0 sec 515 MBytes 4.32 Gbits/sec [ 3] 0.0-10.0 sec 4.42 GBytes 3.80 Gbits/sec 2) With patch: ------------------------------------------------------------ Client connecting to 192.168.1.2, TCP port 5001 TCP window size: 32.5 KByte (default) ------------------------------------------------------------ [ 3] local 192.168.1.1 port 10526 connected with 192.168.1.2 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0- 1.0 sec 694 MBytes 5.83 Gbits/sec [ 3] 1.0- 2.0 sec 999 MBytes 8.38 Gbits/sec [ 3] 2.0- 3.0 sec 1.17 GBytes 10.1 Gbits/sec [ 3] 3.0- 4.0 sec 1.34 GBytes 11.5 Gbits/sec [ 3] 4.0- 5.0 sec 1.15 GBytes 9.91 Gbits/sec [ 3] 5.0- 6.0 sec 1.19 GBytes 10.2 Gbits/sec [ 3] 6.0- 7.0 sec 1.08 GBytes 9.23 Gbits/sec [ 3] 7.0- 8.0 sec 1.10 GBytes 9.45 Gbits/sec [ 3] 8.0- 9.0 sec 1.27 GBytes 10.9 Gbits/sec [ 3] 9.0-10.0 sec 1.39 GBytes 12.0 Gbits/sec [ 3] 0.0-10.0 sec 11.3 GBytes 9.74 Gbits/sec So, basically we have a sysctl(8) called "net.link.lagg.rr_packets" where we can set the number of packets that will be send before the roundrobin move to the next interface. Any comment and review are very appreciated. Best Regards, -- Marcelo Araujo (__)araujo@FreeBSD.org \\\'',)http://www.FreeBSD.org <http://www.freebsd.org/>; \/ \ ^ Power To Server. .\. /_) --001a11c37bdc896b5204fc779588 Content-Type: application/octet-stream; name="if_lagg.patch" Content-Disposition: attachment; filename="if_lagg.patch" Content-Transfer-Encoding: base64 X-Attachment-Id: f_hwr5uz6s0 SW5kZXg6IGlmX2xhZ2cuYwo9PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09Ci0tLSBpZl9sYWdnLmMJKHJldmlzaW9uIDI2NzY2 NikKKysrIGlmX2xhZ2cuYwkod29ya2luZyBjb3B5KQpAQCAtMTg5LDYgKzE4OSwxMCBAQAogU1lT Q1RMX0lOVChfbmV0X2xpbmtfbGFnZywgT0lEX0FVVE8sIGRlZmF1bHRfZmxvd2lkX3NoaWZ0LCBD VExGTEFHX1JXLAogICAgICZkZWZfZmxvd2lkX3NoaWZ0LCAwLAogICAgICJEZWZhdWx0IHNldHRp bmcgZm9yIGZsb3dpZCBzaGlmdCBmb3IgbG9hZCBzaGFyaW5nIik7CitzdGF0aWMgaW50IGxhZ2df cnJfcGFja2V0cyA9IDA7IC8qIERlZmF1bHQgdmFsdWUgZm9yIHVzaW5nIHJyX3BhY2tldHMgKi8K K1NZU0NUTF9JTlQoX25ldF9saW5rX2xhZ2csIE9JRF9BVVRPLCBycl9wYWNrZXRzLCBDVExGTEFH X1JXLAorICAgICZsYWdnX3JyX3BhY2tldHMsIDAsCisgICAgIkhvdyBtYW55IHBhY2tldHMgdG8g YmUgc2VuZCBwZXIgaW50ZXJmYWNlIik7CiAKIHN0YXRpYyBpbnQKIGxhZ2dfbW9kZXZlbnQobW9k dWxlX3QgbW9kLCBpbnQgdHlwZSwgdm9pZCAqZGF0YSkKQEAgLTE2ODksMTQgKzE2OTMsNjggQEAK IHsKIAlzdHJ1Y3QgbGFnZ19wb3J0ICpscDsKIAl1aW50MzJfdCBwOworCXVpbnQzMl90IHBrdF9z eXNjdGxfY291bnQ7CisJaW50IGlmcF9jb3VudCA9IDE7CiAKIAlwID0gYXRvbWljX2ZldGNoYWRk XzMyKCZzYy0+c2Nfc2VxLCAxKTsKIAlwICU9IHNjLT5zY19jb3VudDsKIAlscCA9IFNMSVNUX0ZJ UlNUKCZzYy0+c2NfcG9ydHMpOwotCXdoaWxlIChwLS0pCi0JCWxwID0gU0xJU1RfTkVYVChscCwg bHBfZW50cmllcyk7CiAKIAkvKgorCSAqIElmIHRoZXJlIGlzIG5vIHJlZmVyZW5jZSBmb3IgdGhl IElGUCwgd2UgbXVzdAorIAkgKiBjb3B5IGl0IG5vdy4KKwkgKi8KKwlpZiAoc3RybGVuKHNjLT5z Y19yZWZfaWZwKSA9PSAwKQorCQlzdHJuY3B5KHNjLT5zY19yZWZfaWZwLCBscC0+bHBfaWZwLT5p Zl94bmFtZSwgc2l6ZW9mKHNjLT5zY19yZWZfaWZwKSk7CisgICAgICAgICAgICAgIAorCS8qCisJ ICogSWYgaWZwX2NvdW50IHdhcyBub3QgeWV0IGluaXRpYWxpemVkLCB3ZSBtdXN0CisJICogaW5p dGlhbGl6ZSBub3cuCisJICovCisJaWYgKHNjLT5zY19pZnBfY291bnQgPT0gMCkKKwkJc2MtPnNj X2lmcF9jb3VudCA9IDE7CisKKwkvKgorCSAqIElmIHRoZSBzeXNjdGwgcnJfcGFja2V0cyBpcyBz ZXQgdG8gMCwgd2UgbXVzdCB1c2UgdGhlCisJICogcm91bmRyb2JpbiBhcyBpdCBpcywgb3Igb3Ro ZXJ3aXNlLCB3ZSBtdXN0IGFwcGx5IHRoZQorCSAqIGdyYW51bGFyaXR5IGJldHdlZW4gdGhlIGlu dGVyZmFjZXMgdGhhdCBhcmUgcGFydCBvZiB0aGUgZ3JvdXAuCisJICovCisJaWYgKCFsYWdnX3Jy X3BhY2tldHMpIHsKKwkJd2hpbGUgKHAtLSkKKwkJCWxwID0gU0xJU1RfTkVYVChscCwgbHBfZW50 cmllcyk7CisJCWdvdG8gc2VuZF9tYnVmOworCX0gZWxzZSB7CisJCXBrdF9zeXNjdGxfY291bnQg PSBhdG9taWNfZmV0Y2hhZGRfMzIoJnNjLT5zY19wa3RfY291bnQsIDEpOworCQlpZiAocGt0X3N5 c2N0bF9jb3VudCA9PSBsYWdnX3JyX3BhY2tldHMpIHsKKwkJCWlmIChzYy0+c2NfaWZwX2NvdW50 IDw9IHNjLT5zY19jb3VudCkgeworCQkJCXdoaWxlIChpZnBfY291bnQgPCBzYy0+c2NfaWZwX2Nv dW50KSB7CisJCQkJCWxwID0gU0xJU1RfTkVYVChscCwgbHBfZW50cmllcyk7CisJCQkJCWlmcF9j b3VudCsrOworCQkJCX0KKwkJCQlzYy0+c2NfaWZwX2NvdW50Kys7CisJCQkJaWYgKHNjLT5zY19p ZnBfY291bnQgPiBzYy0+c2NfY291bnQpCisJCQkJCXNjLT5zY19pZnBfY291bnQgPSAwOworCQkJ fQorCQkJc3RybmNweShzYy0+c2NfcmVmX2lmcCwgbHAtPmxwX2lmcC0+aWZfeG5hbWUsIHNpemVv ZihzYy0+c2NfcmVmX2lmcCkpOworCQkJc2MtPnNjX3BrdF9jb3VudCA9IDA7CisJCX0KKwl9CisK KwkvKgorCSAqIENoZWNrIGlmIHRoZSBjdXJyZW50IGludGVyZmFjZSB0byBiZSBlbnF1ZXVlIGlz IG5vdCB0aGUKKwkgKiBzYW1lIHVzZWQgaW4gdGhlIGxhc3Qgcm91bmQuCisJICovCisJbHAgPSBT TElTVF9GSVJTVCgmc2MtPnNjX3BvcnRzKTsKKwlmb3IgKDs7KSB7CisJCWlmIChzdHJjbXAobHAt PmxwX2lmcC0+aWZfeG5hbWUsIHNjLT5zY19yZWZfaWZwKSA9PSAwKQorCQkJYnJlYWs7CisJCWVs c2UKKwkJCWxwID0gU0xJU1RfTkVYVChscCwgbHBfZW50cmllcyk7CisJfQorCWdvdG8gc2VuZF9t YnVmOworCitzZW5kX21idWY6CisJLyoKIAkgKiBDaGVjayB0aGUgcG9ydCdzIGxpbmsgc3RhdGUu IFRoaXMgd2lsbCByZXR1cm4gdGhlIG5leHQgYWN0aXZlCiAJICogcG9ydCBpZiB0aGUgbGluayBp cyBkb3duIG9yIHRoZSBwb3J0IGlzIE5VTEwuCiAJICovCkluZGV4OiBpZl9sYWdnLmgKPT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PQotLS0gaWZfbGFnZy5oCShyZXZpc2lvbiAyNjc2NjYpCisrKyBpZl9sYWdnLmgJKHdvcmtp bmcgY29weSkKQEAgLTIzMiw2ICsyMzIsOSBAQAogCXN0cnVjdCBzeXNjdGxfb2lkCQkqc2Nfb2lk OwkvKiBzeXNjdGwgdHJlZSBvaWQgKi8KIAlpbnQJCQkJdXNlX2Zsb3dpZDsJLyogdXNlIE1fRkxP V0lEICovCiAJaW50CQkJCWZsb3dpZF9zaGlmdDsJLyogc2hpZnQgdGhlIGZsb3dpZCAqLworCXVp bnQzMl90CQkJc2NfcGt0X2NvdW50OyAvKiB1c2UgZm9yIGNvdW50IHBhY2thdGVzIHBlciBpZnAg Ki8KKwlpbnQJCQkJc2NfaWZwX2NvdW50OyAvKiBjb3VudGVyIHJlZmVyZW5jZSBvZiBpbnRlcmZh Y2VzIG9uIHJyICovCisJY2hhcgkJCQlzY19yZWZfaWZwW0lGTkFNU0laXTsgLyogbmFtZSBvZiB0 aGUgaWZwICovCiB9OwogCiBzdHJ1Y3QgbGFnZ19wb3J0IHsK --001a11c37bdc896b5204fc779588--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAOfEmZjmb1bdvn0gR6vD1WeP8o8g7KwXod4TE0iJfa=nicyeng>