From owner-freebsd-net@FreeBSD.ORG Fri Jul 18 07:49:13 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 582B2812; Fri, 18 Jul 2014 07:49:13 +0000 (UTC) Received: from mail-wi0-x22e.google.com (mail-wi0-x22e.google.com [IPv6:2a00:1450:400c:c05::22e]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B34D3251D; Fri, 18 Jul 2014 07:49:12 +0000 (UTC) Received: by mail-wi0-f174.google.com with SMTP id d1so372478wiv.13 for ; Fri, 18 Jul 2014 00:49:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:reply-to:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=NCTzDB/O8dFVlltq3iVJ6EZa92sSB1CrpvKhNdHMIzY=; b=PTfTuiTc+kJmL+PXFeahnErRVAJZsgtJC46xRoN7tFyGjSJSEfNRE2xx4nqaOCK8ge LmykhJS/GVVvmeJ9UQOJBxDvYxAnZX14PEAu4kxQuKLCyRGfZANoOIRWKBcHe6By81Jq T87mmO0cxCuaUNaeI3lWsJJ6nL5t6QB6EAccYGWQqFYFwbeigLjA/8+WTrsYZpVDlnGO 7BxeFIJkTDxF26TBvBsI0wauc45UBe6dilbetDbP9YyXRh7pW/tAaQ8WX6q6+GTt2PoT HRf+B5MhtAM8/ydjArC+g4TCuV3f+t+ucYOkM+7qwNDo9Kia8GGywY7IR5ycLMJpjzd9 Vx7g== MIME-Version: 1.0 X-Received: by 10.180.91.6 with SMTP id ca6mr4463775wib.77.1405669750597; Fri, 18 Jul 2014 00:49:10 -0700 (PDT) Received: by 10.216.190.194 with HTTP; Fri, 18 Jul 2014 00:49:10 -0700 (PDT) Reply-To: araujo@FreeBSD.org In-Reply-To: References: Date: Fri, 18 Jul 2014 15:49:10 +0800 Message-ID: Subject: Re: [patch][lagg] - Set a better granularity and distribution on roundrobin protocol. From: Marcelo Araujo To: Adrian Chadd Content-Type: multipart/mixed; boundary=f46d043c7e0c28456204fe72fed0 X-Content-Filtered-By: Mailman/MimeDel 2.1.18 Cc: FreeBSD Net X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 18 Jul 2014 07:49:13 -0000 --f46d043c7e0c28456204fe72fed0 Content-Type: text/plain; charset=UTF-8 Hello guys, I made few changes on the lagg(4) patch. Also, I made tests using igb(4), ixgbe(4) and em(4); seems everything worked pretty well. I'm wondering if anyone else could make a review, and what I need to do, to see this patch committed. Best Regards, 2014-06-24 10:40 GMT+08:00 Marcelo Araujo : > > > 2014-06-24 6:54 GMT+08:00 Adrian Chadd : > > Hi, >> >> No, don't introduce out of order behaviour. Ever. > > > Yes, it has out of order behavior; with my patch much less. I upload two > pcap files and you can see by yourself, if you don't believe in what I'm > talking about. > > Test done using: "iperf -s" and "iperf -c -i 1 -t 10". > > 1) Don't change the number of packets(default round robin behavior). > http://people.freebsd.org/~araujo/lagg/lagg-nop.cap > 8 out of order packets. > Several SACKs. > > 2) Set the number of packets to 50. > http://people.freebsd.org/~araujo/lagg/lagg.cap > 0 out of order packets. > Less SACKs. > > >> You may not think >> it's a problem for TCP, but UDP things and VPN things will start >> getting very angry. There are VPN configurations out there that will >> drop the VPN if frames are out of order. >> > > I'm not thinking that will be a problem for TCP, but, in somehow it will > be, less throughput as I showed before, and less SACK. About the VPN, > please, tell me which softwares, and let me know where I can get a sample > to make a testbed. > > However to be very honest, I don't believe anyone here when change > something at network protocols will make this extensive testbed. It is > almost impossible to predict what software it will works or not, and I > don't believe anyone here has all these stuff in hands. > > >> >> The ixgbe driver is setting the flowid to the msix queue ID, rather >> than a 32 bit unique flow id hash value for the flow. That makes it >> hard to do traffic distribution where the flowid is available. >> > > Thanks for the explanation. > > >> >> There's an lagg option to re-hash the mbuf rather than rely on the >> flowid for outbound port choice - have you looked at using that? Did >> that make any difference? >> > > Yes, I set to 0 the net.link.lagg.0.use _flowid, it make a little > difference to the default round robin implementation, but yet I can't reach > more than 5 Gbit/s. With my patch and set the packets to 50, it improved a > bit too. > > So, thank you so much for all review, I don't know if you have time and a > testbed to make a real test, as I'm doing. I would be happy if you or more > people could make tests on that patch. Also, I have only ixgbe(4) to make > tests, would appreciate if this patch could be tested with other NICs too. > > Best Regards, > > -- > Marcelo Araujo (__) > araujo@FreeBSD.org \\\'',)http://www.FreeBSD.org \/ \ ^ > Power To Server. .\. /_) > > -- -- Marcelo Araujo (__)araujo@FreeBSD.org \\\'',)http://www.FreeBSD.org \/ \ ^ Power To Server. .\. /_) --f46d043c7e0c28456204fe72fed0 Content-Type: application/octet-stream; name="if_lagg-rr.patch" Content-Disposition: attachment; filename="if_lagg-rr.patch" Content-Transfer-Encoding: base64 X-Attachment-Id: f_hxr7fodu0 SW5kZXg6IGlmX2xhZ2cuYwo9PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09Ci0tLSBpZl9sYWdnLmMJKHJldmlzaW9uIDI2ODgz MikKKysrIGlmX2xhZ2cuYwkod29ya2luZyBjb3B5KQpAQCAtMTg3LDYgKzE4NywxMCBAQAogU1lT Q1RMX0lOVChfbmV0X2xpbmtfbGFnZywgT0lEX0FVVE8sIGRlZmF1bHRfZmxvd2lkX3NoaWZ0LCBD VExGTEFHX1JXVFVOLAogICAgICZkZWZfZmxvd2lkX3NoaWZ0LCAwLAogICAgICJEZWZhdWx0IHNl dHRpbmcgZm9yIGZsb3dpZCBzaGlmdCBmb3IgbG9hZCBzaGFyaW5nIik7CitzdGF0aWMgaW50IGxh Z2dfcnJfcGFja2V0cyA9IDA7IC8qIERlZmF1bHQgdmFsdWUgZm9yIHVzaW5nIHJyX3BhY2tldHMg Ki8KK1NZU0NUTF9JTlQoX25ldF9saW5rX2xhZ2csIE9JRF9BVVRPLCBycl9wYWNrZXRzLCBDVExG TEFHX1JXLAorICAgICZsYWdnX3JyX3BhY2tldHMsIDAsCisgICAgIkhvdyBtYW55IHBhY2tldHMg dG8gYmUgc2VuZCBwZXIgaW50ZXJmYWNlIik7CiAKIHN0YXRpYyBpbnQKIGxhZ2dfbW9kZXZlbnQo bW9kdWxlX3QgbW9kLCBpbnQgdHlwZSwgdm9pZCAqZGF0YSkKQEAgLTE2ODcsMTQgKzE2OTEsNzMg QEAKIHsKIAlzdHJ1Y3QgbGFnZ19wb3J0ICpscDsKIAl1aW50MzJfdCBwOworCXVpbnQzMl90IHAy OworCXVpbnQzMl90IHBrdF9zeXNjdGxfY291bnQ7CisJaW50IGlmcF9jb3VudCA9IDE7CiAKIAlw ID0gYXRvbWljX2ZldGNoYWRkXzMyKCZzYy0+c2Nfc2VxLCAxKTsKIAlwICU9IHNjLT5zY19jb3Vu dDsKKworCXAyID0gYXRvbWljX2ZldGNoYWRkXzMyKCZzYy0+c2Nfc2VxLCAxKTsKKwlwMiAlPSBz Yy0+c2NfY291bnQ7CisKIAlscCA9IFNMSVNUX0ZJUlNUKCZzYy0+c2NfcG9ydHMpOwotCXdoaWxl IChwLS0pCi0JCWxwID0gU0xJU1RfTkVYVChscCwgbHBfZW50cmllcyk7CiAKIAkvKgorCSAqIElm IHRoZXJlIGlzIG5vIHJlZmVyZW5jZSBmb3IgdGhlIElGUCwgd2UgbXVzdAorIAkgKiBjb3B5IGl0 IG5vdy4KKwkgKi8KKwlpZiAoc3RybGVuKHNjLT5zY19yZWZfaWZwKSA9PSAwKQorCQlzdHJuY3B5 KHNjLT5zY19yZWZfaWZwLCBscC0+bHBfaWZwLT5pZl94bmFtZSwgc2l6ZW9mKHNjLT5zY19yZWZf aWZwKSk7CisgICAgICAgICAgICAgIAorCS8qCisJICogSWYgaWZwX2NvdW50IHdhcyBub3QgeWV0 IGluaXRpYWxpemVkLCB3ZSBtdXN0CisJICogaW5pdGlhbGl6ZSBub3cuCisJICovCisJaWYgKHNj LT5zY19pZnBfY291bnQgPT0gMCkKKwkJc2MtPnNjX2lmcF9jb3VudCA9IDE7CisKKwkvKgorCSAq IElmIHRoZSBzeXNjdGwgcnJfcGFja2V0cyBpcyBzZXQgdG8gMCwgd2UgbXVzdCB1c2UgdGhlCisJ ICogcm91bmRyb2JpbiBhcyBpdCBpcywgb3Igb3RoZXJ3aXNlLCB3ZSBtdXN0IGFwcGx5IHRoZQor CSAqIGdyYW51bGFyaXR5IGJldHdlZW4gdGhlIGludGVyZmFjZXMgdGhhdCBhcmUgcGFydCBvZiB0 aGUgZ3JvdXAuCisJICovCisJaWYgKCFsYWdnX3JyX3BhY2tldHMpIHsKKwkJd2hpbGUgKHAtLSkK KwkJCWxwID0gU0xJU1RfTkVYVChscCwgbHBfZW50cmllcyk7CisJCWdvdG8gc2VuZF9tYnVmOwor CX0gZWxzZSB7CisJCXBrdF9zeXNjdGxfY291bnQgPSBhdG9taWNfZmV0Y2hhZGRfMzIoJnNjLT5z Y19wa3RfY291bnQsIDEpOworCQlpZiAocGt0X3N5c2N0bF9jb3VudCA9PSBsYWdnX3JyX3BhY2tl dHMpIHsKKwkJCWlmIChzYy0+c2NfaWZwX2NvdW50IDw9IHNjLT5zY19jb3VudCkgeworCQkJCXdo aWxlIChpZnBfY291bnQgPCBzYy0+c2NfaWZwX2NvdW50KSB7CisJCQkJCWxwID0gU0xJU1RfTkVY VChscCwgbHBfZW50cmllcyk7CisJCQkJCWlmcF9jb3VudCsrOworCQkJCX0KKwkJCQlzYy0+c2Nf aWZwX2NvdW50Kys7CisJCQkJaWYgKHNjLT5zY19pZnBfY291bnQgPiBzYy0+c2NfY291bnQpCisJ CQkJCXNjLT5zY19pZnBfY291bnQgPSAwOworCQkJfQorCQkJc3RybmNweShzYy0+c2NfcmVmX2lm cCwgbHAtPmxwX2lmcC0+aWZfeG5hbWUsIHNpemVvZihzYy0+c2NfcmVmX2lmcCkpOworCQkJc2Mt PnNjX3BrdF9jb3VudCA9IDA7CisJCX0KKwl9CisKKwkvKgorCSAqIENoZWNrIGlmIHRoZSBjdXJy ZW50IGludGVyZmFjZSB0byBiZSBlbnF1ZXVlIGlzIG5vdCB0aGUKKwkgKiBzYW1lIHVzZWQgaW4g dGhlIGxhc3Qgcm91bmQuCisJICovCisJbHAgPSBTTElTVF9GSVJTVCgmc2MtPnNjX3BvcnRzKTsK Kwl3aGlsZSAocDItLSkgeworCQlpZiAoc3RyY21wKGxwLT5scF9pZnAtPmlmX3huYW1lLCBzYy0+ c2NfcmVmX2lmcCkgPT0gMCkKKwkJCWJyZWFrOworCQllbHNlCisJCQlscCA9IFNMSVNUX05FWFQo bHAsIGxwX2VudHJpZXMpOworCX0KKwlnb3RvIHNlbmRfbWJ1ZjsKKworc2VuZF9tYnVmOgorCS8q CiAJICogQ2hlY2sgdGhlIHBvcnQncyBsaW5rIHN0YXRlLiBUaGlzIHdpbGwgcmV0dXJuIHRoZSBu ZXh0IGFjdGl2ZQogCSAqIHBvcnQgaWYgdGhlIGxpbmsgaXMgZG93biBvciB0aGUgcG9ydCBpcyBO VUxMLgogCSAqLwpJbmRleDogaWZfbGFnZy5oCj09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT0KLS0tIGlmX2xhZ2cuaAkocmV2 aXNpb24gMjY4ODMyKQorKysgaWZfbGFnZy5oCSh3b3JraW5nIGNvcHkpCkBAIC0yMzIsNiArMjMy LDkgQEAKIAlzdHJ1Y3Qgc3lzY3RsX29pZAkJKnNjX29pZDsJLyogc3lzY3RsIHRyZWUgb2lkICov CiAJaW50CQkJCXVzZV9mbG93aWQ7CS8qIHVzZSBNX0ZMT1dJRCAqLwogCWludAkJCQlmbG93aWRf c2hpZnQ7CS8qIHNoaWZ0IHRoZSBmbG93aWQgKi8KKwl1aW50MzJfdAkJCXNjX3BrdF9jb3VudDsg LyogdXNlIGZvciBjb3VudCBwYWNrYXRlcyBwZXIgaWZwICovCisJaW50CQkJCXNjX2lmcF9jb3Vu dDsgLyogY291bnRlciByZWZlcmVuY2Ugb2YgaW50ZXJmYWNlcyBvbiByciAqLworCWNoYXIJCQkJ c2NfcmVmX2lmcFtJRk5BTVNJWl07IC8qIG5hbWUgb2YgdGhlIGlmcCAqLwogfTsKIAogc3RydWN0 IGxhZ2dfcG9ydCB7Cg== --f46d043c7e0c28456204fe72fed0--