From owner-freebsd-net@FreeBSD.ORG Mon Feb 10 00:54:23 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 40FAD862 for ; Mon, 10 Feb 2014 00:54:23 +0000 (UTC) Received: from esa-annu.net.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id F0F3E1935 for ; Mon, 10 Feb 2014 00:54:22 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AqQEAD8i+FKDaFve/2dsb2JhbABZg0RXgwG7ek+BHXSCJQEBAQQBAQEgBCcgCxsOChEZAgQlAQkmBggHBAEcBIdkDadMoDMXjiwBARsZGweCb4FJBIlJhnaEAQeBEIQGkG+DSx4xgQQ5 X-IronPort-AV: E=Sophos;i="4.95,814,1384318800"; d="scan'208";a="94571820" Received: from muskoka.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.222]) by esa-annu.net.uoguelph.ca with ESMTP; 09 Feb 2014 19:54:15 -0500 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id 807BEB3F49; Sun, 9 Feb 2014 19:54:15 -0500 (EST) Date: Sun, 9 Feb 2014 19:54:15 -0500 (EST) From: Rick Macklem To: Christian Weisgerber Message-ID: <2059385169.3089582.1391993655512.JavaMail.root@uoguelph.ca> In-Reply-To: Subject: Re: Terrible NFS performance under 9.2-RELEASE? MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_Part_3089580_2077641633.1391993655510" X-Originating-IP: [172.17.91.201] X-Mailer: Zimbra 7.2.1_GA_2790 (ZimbraWebClient - FF3.0 (Win)/7.2.1_GA_2790) Cc: freebsd-net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 10 Feb 2014 00:54:23 -0000 ------=_Part_3089580_2077641633.1391993655510 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Christian Weisgerber wrote: > Rick Macklem wrote: > > > I have a "hunch" that might explain why 64K NFS reads/writes > > perform > > poorly for some network environments. > > A 64K NFS read reply/write request consists of a list of 34 mbufs > > when > > passed to TCP via sosend() and a total data length of around > > 65680bytes. > > Looking at a couple of drivers (virtio and ixgbe), they seem to > > expect > > no more than 32-33 mbufs in a list for a 65535 byte TSO xmit. I > > think > > (I don't have anything that does TSO to confirm this) that NFS will > > pass > > a list that is longer (34 plus a TCP/IP header). > > This may or may not be the same problem: > > When I switched my desktop box from FreeBSD 7 to 9, NFS read > performance from my media server (running OpenBSD) became extremely > poor. I couldn't even stream a movie any longer. Disabling TSO > on the nfe(4) interface had no effect. My workaround was to switch > from a TCP mount to a UDP one. The problem has persisted to FreeBSD > 10. > > I can now report that switching to [rw]size=32768 with a TCP mount > also works fine. > If it is convenient, trying a 64K TCP mount with a kernel that has the attached patch (which makes it use page size clusters and reduces the # of segments to 18) to see if it works well, would be interesting. rick ps: This is not a patch destined for head and shouldn't be applied to a production box, but I think it is ok for testing. > -- > Christian "naddy" Weisgerber > naddy@mips.inka.de > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to > "freebsd-net-unsubscribe@freebsd.org" > ------=_Part_3089580_2077641633.1391993655510 Content-Type: text/x-patch; name=4kmcl.patch Content-Disposition: attachment; filename=4kmcl.patch Content-Transfer-Encoding: base64 LS0tIGZzL25mcy9uZnNwb3J0Lmguc2F2MgkyMDE0LTAxLTI2IDE4OjQzOjQ3LjAwMDAwMDAwMCAt MDUwMAorKysgZnMvbmZzL25mc3BvcnQuaAkyMDE0LTAxLTI2IDE5OjA0OjI3LjAwMDAwMDAwMCAt MDUwMApAQCAtMTUzLDE0ICsxNTMsMjcgQEAKIAkJCU1HRVRIRFIoKG0pLCBNX1dBSVRPSywgTVRf REFUQSk7IAlcCiAJCX0gCQkJCQkJXAogCX0gd2hpbGUgKDApCi0jZGVmaW5lCU5GU01DTEdFVCht LCB3KQlkbyB7IAkJCQkJXAotCQlNR0VUKChtKSwgTV9XQUlUT0ssIE1UX0RBVEEpOyAJCQlcCi0J CXdoaWxlICgobSkgPT0gTlVMTCApIHsgCQkJCVwKLQkJCSh2b2lkKSBuZnNfY2F0bmFwKFBaRVJP LCAwLCAibmZzbWdldCIpOwlcCi0JCQlNR0VUKChtKSwgTV9XQUlUT0ssIE1UX0RBVEEpOyAJCVwK LQkJfSAJCQkJCQlcCi0JCU1DTEdFVCgobSksICh3KSk7CQkJCVwKKyNpZiBNSlVNUEFHRVNJWkUg PiBNQ0xCWVRFUworI2RlZmluZQlORlNNQ0xHRVQobSwgdykJZG8gewkgCQkJCQlcCisJCShtKSA9 IG1fZ2V0amNsKE1fV0FJVE9LLCBNVF9EQVRBLCAwLCBNSlVNUEFHRVNJWkUpOwlcCisJCXdoaWxl ICgobSkgPT0gTlVMTCkgewkgCQkJCVwKKwkJCSh2b2lkKW5mc19jYXRuYXAoUFpFUk8sIDAsICJu ZnNtZ2V0Iik7CQlcCisJCQlNR0VUKChtKSwgTV9XQUlUT0ssIE1UX0RBVEEpOwkgCQlcCisJCQlp ZiAoKG0pICE9IE5VTEwpCQkJCVwKKwkJCQlNQ0xHRVQoKG0pLCAodykpOwkJCVwKKwkJfQkgCQkJ CQkJXAogCX0gd2hpbGUgKDApCisjZWxzZQorI2RlZmluZQlORlNNQ0xHRVQobSwgdykJZG8gewkg CQkJCQlcCisJCShtKSA9IG1fZ2V0amNsKE1fV0FJVE9LLCBNVF9EQVRBLCAwLCBNQ0xCWVRFUyk7 CQlcCisJCXdoaWxlICgobSkgPT0gTlVMTCkgewkgCQkJCVwKKwkJCSh2b2lkKW5mc19jYXRuYXAo UFpFUk8sIDAsICJuZnNtZ2V0Iik7CQlcCisJCQlNR0VUKChtKSwgTV9XQUlUT0ssIE1UX0RBVEEp OwkgCQlcCisJCQlpZiAoKG0pICE9IE5VTEwpCQkJCVwKKwkJCQlNQ0xHRVQoKG0pLCAodykpOwkJ CVwKKwkJfQkgCQkJCQkJXAorCX0gd2hpbGUgKDApCisjZW5kaWYKICNkZWZpbmUJTkZTTUNMR0VU SERSKG0sIHcpIGRvIHsgCQkJCVwKIAkJTUdFVEhEUigobSksIE1fV0FJVE9LLCBNVF9EQVRBKTsJ CVwKIAkJd2hpbGUgKChtKSA9PSBOVUxMICkgeyAJCQkJXAotLS0gZnMvbmZzc2VydmVyL25mc19u ZnNkcG9ydC5jLnNhdjIJMjAxNC0wMS0yNiAxODo1NDoyOS4wMDAwMDAwMDAgLTA1MDAKKysrIGZz L25mc3NlcnZlci9uZnNfbmZzZHBvcnQuYwkyMDE0LTAxLTI2IDE4OjU2OjA4LjAwMDAwMDAwMCAt MDUwMApAQCAtNTY2LDggKzU2Niw3IEBAIG5mc3Zub19yZWFkbGluayhzdHJ1Y3Qgdm5vZGUgKnZw LCBzdHJ1Y3QKIAlsZW4gPSAwOwogCWkgPSAwOwogCXdoaWxlIChsZW4gPCBORlNfTUFYUEFUSExF TikgewotCQlORlNNR0VUKG1wKTsKLQkJTUNMR0VUKG1wLCBNX1dBSVRPSyk7CisJCU5GU01DTEdF VChtcCwgTV9XQUlUT0spOwogCQltcC0+bV9sZW4gPSBORlNNU0laKG1wKTsKIAkJaWYgKGxlbiA9 PSAwKSB7CiAJCQltcDMgPSBtcDIgPSBtcDsKQEAgLTYzNiw4ICs2MzUsNyBAQCBuZnN2bm9fcmVh ZChzdHJ1Y3Qgdm5vZGUgKnZwLCBvZmZfdCBvZmYsCiAJICovCiAJaSA9IDA7CiAJd2hpbGUgKGxl ZnQgPiAwKSB7Ci0JCU5GU01HRVQobSk7Ci0JCU1DTEdFVChtLCBNX1dBSVRPSyk7CisJCU5GU01D TEdFVChtLCBNX1dBSVRPSyk7CiAJCW0tPm1fbGVuID0gMDsKIAkJc2l6ID0gbWluKE1fVFJBSUxJ TkdTUEFDRShtKSwgbGVmdCk7CiAJCWxlZnQgLT0gc2l6Owo= ------=_Part_3089580_2077641633.1391993655510--