From owner-freebsd-net@FreeBSD.ORG Fri Apr 25 09:46:53 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 0C170508 for ; Fri, 25 Apr 2014 09:46:53 +0000 (UTC) Received: from umail.aei.mpg.de (umail.aei.mpg.de [194.94.224.6]) by mx1.freebsd.org (Postfix) with ESMTP id 913B01A1C for ; Fri, 25 Apr 2014 09:46:52 +0000 (UTC) Received: from mailgate.aei.mpg.de (mailgate.aei.mpg.de [194.94.224.5]) by umail.aei.mpg.de (Postfix) with ESMTP id 1986D200CD2; Fri, 25 Apr 2014 11:37:13 +0200 (CEST) Received: from mailgate.aei.mpg.de (localhost [127.0.0.1]) by localhost (Postfix) with SMTP id 0CD79406AF1; Fri, 25 Apr 2014 11:37:13 +0200 (CEST) Received: from intranet.aei.uni-hannover.de (ahin1.aei.uni-hannover.de [130.75.117.40]) by mailgate.aei.mpg.de (Postfix) with ESMTP id C1E4C405889; Fri, 25 Apr 2014 11:37:12 +0200 (CEST) Received: from cascade.aei.uni-hannover.de ([10.117.15.111]) by intranet.aei.uni-hannover.de (Lotus Domino Release 8.5.3FP6) with ESMTP id 2014042511371137-88095 ; Fri, 25 Apr 2014 11:37:11 +0200 Date: Fri, 25 Apr 2014 11:37:11 +0200 From: Gerrit =?ISO-8859-1?Q?K=FChn?= To: Marek Salwerowicz Subject: Re: NFS over LAGG / lacp poor performance Message-Id: <20140425113711.e7c7d1c2.gerrit.kuehn@aei.mpg.de> In-Reply-To: <535A1354.2040309@wp.pl> References: <535A1354.2040309@wp.pl> Organization: Max Planck Gesellschaft X-Mailer: Sylpheed 3.1.3 (GTK+ 2.24.19; amd64-portbld-freebsd8.2) Mime-Version: 1.0 X-MIMETrack: Itemize by SMTP Server on intranet/aei-hannover(Release 8.5.3FP6|November 21, 2013) at 04/25/2014 11:37:11, Serialize by Router on intranet/aei-hannover(Release 8.5.3FP6|November 21, 2013) at 04/25/2014 11:37:11, Serialize complete at 04/25/2014 11:37:11 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII X-PMX-Version: 6.0.2.2308539, Antispam-Engine: 2.7.2.2107409, Antispam-Data: 2014.4.25.92722 X-PerlMx-Spam: Gauge=IIIIIIII, Probability=8%, Report=' HTML_00_01 0.05, HTML_00_10 0.05, MIME_LOWER_CASE 0.05, BODYTEXTP_SIZE_3000_LESS 0, BODY_SIZE_2000_2999 0, BODY_SIZE_5000_LESS 0, BODY_SIZE_7000_LESS 0, __ANY_URI 0, __BOUNCE_CHALLENGE_SUBJ 0, __BOUNCE_NDR_SUBJ_EXEMPT 0, __C230066_P5 0, __CT 0, __CTE 0, __CT_TEXT_PLAIN 0, __HAS_FROM 0, __HAS_MSGID 0, __HAS_X_MAILER 0, __IN_REP_TO 0, __MIME_TEXT_ONLY 0, __MIME_VERSION 0, __SANE_MSGID 0, __SUBJ_ALPHA_NEGATE 0, __TO_MALFORMED_2 0, __URI_NO_PATH 0, __URI_NO_WWW 0, __URI_NS ' Cc: freebsd-net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Apr 2014 09:46:53 -0000 On Fri, 25 Apr 2014 09:48:36 +0200 Marek Salwerowicz wrote about NFS over LAGG / lacp poor performance: Hello Marek, MS> FreeBSD storage1 9.1-RELEASE-p10 FreeBSD 9.1-RELEASE-p10 #0: Sun Jan 12 MS> 20:11:23 UTC 2014 MS> root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64 Sorry for hijacking this (as I can probably not help you with your issue at hand), but I have a roughly similar setup here (9.1-REL nfs server on X9 mainboard with igb interfaces) and was wondering lately about some performance issues I have on a dedicated nfs link. While looking around on the system, I noticed that interrupts on the igb queues spread a bit strangely: --- root@storage:/root # vmstat -i interrupt total rate irq1: atkbd0 2743 0 irq18: ehci0 uhci5 4445137 2 irq21: uhci1 29 0 cpu0:timer 352597482 228 irq256: igb0:que 0 99396134 64 irq257: igb0:que 1 61496018 39 irq258: igb0:que 2 101687742 66 irq259: igb0:que 3 100824264 65 irq260: igb0:link 2 0 irq261: igb1:que 0 1666960 1 irq262: igb1:que 1 2325576555 1510 irq263: igb1:que 2 1563283 1 irq264: igb1:que 3 1897428 1 irq265: igb1:link 2 0 irq266: mps0 327734440 212 irq267: mps1 193113287 125 irq268: mps2 174367181 113 irq269: ahci0 59169140 38 cpu1:timer 416297615 270 cpu3:timer 327005767 212 cpu2:timer 325504623 211 Total 4874345832 3165 --- igb0 is only running standard/admin stuff, igb1 is running the mentioned dedicated nfs link. As you can see, igb1 gets interrupts on one queue only. I wonder if this is what it should be and if if has any impact on performace. I found several threads on the mailing lists about igb interfaces, queues and interrupts, but so far I could not really make out if this might be an issue for me. Could you have a look on your system and let me know how your interrupts are spread? cu Gerrit