From owner-freebsd-net@FreeBSD.ORG Fri Apr 25 07:48:54 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 5D97F82D for ; Fri, 25 Apr 2014 07:48:54 +0000 (UTC) Received: from mx3.wp.pl (mx3.wp.pl [212.77.101.10]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "smtp.wp.pl", Issuer "Thawte SSL CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D90531A4F for ; Fri, 25 Apr 2014 07:48:53 +0000 (UTC) Received: (wp-smtpd smtp.wp.pl 32662 invoked from network); 25 Apr 2014 09:48:37 +0200 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=wp.pl; s=1024a; t=1398412118; bh=zncxVZFf/DZogM7UoZbFmdKSAry9SjOFSJF6Hvexmak=; h=From:To:Subject; b=d2/+DFoanaPJiPz50FuV0ovPlqJf1dEgl0f0gEiH2A3PobbraiHQZ6CiolenMAXq5 bxqzASCVyr7+j1gR2FacTXtPZS3U+11qa8z53JBTu7AVriHX82XifCCeLGLNbPrUnH Tvt9q6ARYkorPIwI+ugOtUqYezHjTX7CYZyRGlZo= Received: from pb-d-128-141-237-169.cern.ch (HELO [128.141.237.169]) (marek_sal@[128.141.237.169]) (envelope-sender ) by smtp.wp.pl (WP-SMTPD) with AES128-SHA encrypted SMTP for ; 25 Apr 2014 09:48:37 +0200 Message-ID: <535A1354.2040309@wp.pl> Date: Fri, 25 Apr 2014 09:48:36 +0200 From: Marek Salwerowicz User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20100101 Thunderbird/24.4.0 MIME-Version: 1.0 To: freebsd-net@freebsd.org Subject: NFS over LAGG / lacp poor performance Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-WP-AV: skaner antywirusowy poczty Wirtualnej Polski S. A. X-WP-SPAM: NO 0000000 [EfPk] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Apr 2014 07:48:54 -0000 Hi list, I have two FreeBSD boxes (both based on SuperMicro X9DRD-7LN4F-JBOD motherboard, with 32GB RAM, 1 CPU :Intel(R) Xeon(R) CPU E5-2640 v2) storage1% uname -a FreeBSD storage1 9.1-RELEASE-p10 FreeBSD 9.1-RELEASE-p10 #0: Sun Jan 12 20:11:23 UTC 2014 root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64 storage2% uname -a FreeBSD storage2 10.0-RELEASE-p1 FreeBSD 10.0-RELEASE-p1 #0: Tue Apr 8 06:45:06 UTC 2014 root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64 They work as NFS storages for VMware ESXi. On both there are installed 4 igb 1Gbit NICs, with LACP and 2 VLANs (vlan900 is for NFS, vlan14 is for management/general purpose): on storage1: igb0: port 0xd060-0xd07f mem 0xfbb60000-0xfbb7ffff,0xfbb8c000-0xfbb8ffff irq 42 at device 0.0 on pci5 igb1: port 0xd040-0xd05f mem 0xfbb40000-0xfbb5ffff,0xfbb88000-0xfbb8bfff irq 45 at device 0.1 on pci5 igb2: port 0xd020-0xd03f mem 0xfbb20000-0xfbb3ffff,0xfbb84000-0xfbb87fff irq 44 at device 0.2 on pci5 igb3: port 0xd000-0xd01f mem 0xfbb00000-0xfbb1ffff,0xfbb80000-0xfbb83fff irq 46 at device 0.3 on pci5 on storage2: igb0: port 0xd060-0xd07f mem 0xfbb60000-0xfbb7ffff,0xfbb8c000-0xfbb8ffff irq 42 at device 0.0 on pci5 igb1: port 0xd040-0xd05f mem 0xfbb40000-0xfbb5ffff,0xfbb88000-0xfbb8bfff irq 45 at device 0.1 on pci5 igb2: port 0xd020-0xd03f mem 0xfbb20000-0xfbb3ffff,0xfbb84000-0xfbb87fff irq 44 at device 0.2 on pci5 igb3: port 0xd000-0xd01f mem 0xfbb00000-0xfbb1ffff,0xfbb80000-0xfbb83fff irq 46 at device 0.3 on pci5 storage1% ifconfig -a igb0: flags=8843 metric 0 mtu 9000 options=401bb ether 00:25:90:c1:1d:18 inet6 fe80::225:90ff:fec1:1d18%igb0 prefixlen 64 scopeid 0x1 nd6 options=29 media: Ethernet autoselect (1000baseT ) status: active igb1: flags=8843 metric 0 mtu 9000 options=401bb ether 00:25:90:c1:1d:18 inet6 fe80::225:90ff:fec1:1d19%igb1 prefixlen 64 scopeid 0x2 nd6 options=29 media: Ethernet autoselect (1000baseT ) status: active igb2: flags=8843 metric 0 mtu 9000 options=401bb ether 00:25:90:c1:1d:18 inet6 fe80::225:90ff:fec1:1d1a%igb2 prefixlen 64 scopeid 0x3 nd6 options=29 media: Ethernet autoselect (1000baseT ) status: active igb3: flags=8843 metric 0 mtu 9000 options=401bb ether 00:25:90:c1:1d:18 inet6 fe80::225:90ff:fec1:1d1b%igb3 prefixlen 64 scopeid 0x4 nd6 options=29 media: Ethernet autoselect (1000baseT ) status: active lo0: flags=8049 metric 0 mtu 16384 options=600003 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x7 inet 127.0.0.1 netmask 0xff000000 nd6 options=21 lagg0: flags=8843 metric 0 mtu 9000 options=401bb ether 00:25:90:c1:1d:18 inet6 fe80::225:90ff:fec1:1d18%lagg0 prefixlen 64 scopeid 0x8 nd6 options=21 media: Ethernet autoselect status: active laggproto lacp lagghash l2,l3,l4 laggport: igb3 flags=1c laggport: igb2 flags=1c laggport: igb1 flags=1c laggport: igb0 flags=1c vlan14: flags=8843 metric 0 mtu 9000 options=103 ether 00:25:90:c1:1d:18 inet 192.168.1.65 netmask 0xffffff00 broadcast 192.168.1.255 inet6 fe80::225:90ff:fec1:1d18%vlan14 prefixlen 64 scopeid 0x9 nd6 options=29 media: Ethernet autoselect status: active vlan: 14 parent interface: lagg0 vlan900: flags=8843 metric 0 mtu 9000 options=103 ether 00:25:90:c1:1d:18 inet 172.25.25.65 netmask 0xffffff00 broadcast 172.25.25.255 inet6 fe80::225:90ff:fec1:1d18%vlan900 prefixlen 64 scopeid 0xa nd6 options=29 media: Ethernet autoselect status: active vlan: 900 parent interface: lagg0 storage2% ifconfig -a igb0: flags=8843 metric 0 mtu 9000 options=403bb ether 00:25:90:ca:3b:e0 inet6 fe80::225:90ff:feca:3be0%igb0 prefixlen 64 scopeid 0x1 nd6 options=29 media: Ethernet autoselect (1000baseT ) status: active igb1: flags=8843 metric 0 mtu 9000 options=403bb ether 00:25:90:ca:3b:e0 inet6 fe80::225:90ff:feca:3be1%igb1 prefixlen 64 scopeid 0x2 nd6 options=29 media: Ethernet autoselect (1000baseT ) status: active igb2: flags=8843 metric 0 mtu 9000 options=403bb ether 00:25:90:ca:3b:e0 inet6 fe80::225:90ff:feca:3be2%igb2 prefixlen 64 scopeid 0x3 nd6 options=29 media: Ethernet autoselect (1000baseT ) status: active igb3: flags=8843 metric 0 mtu 9000 options=403bb ether 00:25:90:ca:3b:e0 inet6 fe80::225:90ff:feca:3be3%igb3 prefixlen 64 scopeid 0x4 nd6 options=29 media: Ethernet autoselect (1000baseT ) status: active lo0: flags=8049 metric 0 mtu 16384 options=600003 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5 inet 127.0.0.1 netmask 0xff000000 nd6 options=21 lagg0: flags=8843 metric 0 mtu 9000 options=403bb ether 00:25:90:ca:3b:e0 inet6 fe80::225:90ff:feca:3be0%lagg0 prefixlen 64 scopeid 0x6 nd6 options=21 media: Ethernet autoselect status: active laggproto lacp lagghash l2,l3,l4 laggport: igb3 flags=1c laggport: igb2 flags=1c laggport: igb1 flags=1c laggport: igb0 flags=1c vlan14: flags=8843 metric 0 mtu 9000 options=303 ether 00:25:90:ca:3b:e0 inet 192.168.1.72 netmask 0xffffff00 broadcast 192.168.1.255 inet6 fe80::225:90ff:feca:3be0%vlan14 prefixlen 64 scopeid 0x7 nd6 options=29 media: Ethernet autoselect status: active vlan: 14 parent interface: lagg0 vlan900: flags=8843 metric 0 mtu 9000 options=303 ether 00:25:90:ca:3b:e0 inet 172.25.25.72 netmask 0xffffff00 broadcast 172.25.25.255 inet6 fe80::225:90ff:feca:3be0%vlan900 prefixlen 64 scopeid 0x8 nd6 options=29 media: Ethernet autoselect status: active vlan: 900 parent interface: lagg0 Both boxes are connected to the same switch (HP 1910-48G) I need to transfer around 10 TB of data from storage1 to storage2 I obeserve that during copying, only one NIC (instead of all 4) is used. Boxes are not stressed during copying What's more, apart from having 1 NIC saturated (transfer around 120 MB/s), I observe transfer rate on level of 70-80 MB/s I haven't changed any kernel configuration, nor sysctl's - am I missing sth ? Regards, Marek -- Marek Salwerowicz