Date: Fri, 25 Apr 2014 09:48:36 +0200 From: Marek Salwerowicz <marek_sal@wp.pl> To: freebsd-net@freebsd.org Subject: NFS over LAGG / lacp poor performance Message-ID: <535A1354.2040309@wp.pl>
next in thread | raw e-mail | index | archive | help
Hi list,
I have two FreeBSD boxes (both based on SuperMicro X9DRD-7LN4F-JBOD
motherboard, with 32GB RAM, 1 CPU :Intel(R) Xeon(R) CPU E5-2640 v2)
storage1% uname -a
FreeBSD storage1 9.1-RELEASE-p10 FreeBSD 9.1-RELEASE-p10 #0: Sun Jan 12
20:11:23 UTC 2014
root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64
storage2% uname
-a
FreeBSD storage2 10.0-RELEASE-p1 FreeBSD 10.0-RELEASE-p1 #0: Tue Apr 8
06:45:06 UTC 2014
root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64
They work as NFS storages for VMware ESXi.
On both there are installed 4 igb 1Gbit NICs, with LACP and 2 VLANs
(vlan900 is for NFS, vlan14 is for management/general purpose):
on storage1:
igb0: <Intel(R) PRO/1000 Network Connection version - 2.3.4> port
0xd060-0xd07f mem 0xfbb60000-0xfbb7ffff,0xfbb8c000-0xfbb8ffff irq 42 at
device 0.0 on pci5
igb1: <Intel(R) PRO/1000 Network Connection version - 2.3.4> port
0xd040-0xd05f mem 0xfbb40000-0xfbb5ffff,0xfbb88000-0xfbb8bfff irq 45 at
device 0.1 on pci5
igb2: <Intel(R) PRO/1000 Network Connection version - 2.3.4> port
0xd020-0xd03f mem 0xfbb20000-0xfbb3ffff,0xfbb84000-0xfbb87fff irq 44 at
device 0.2 on pci5
igb3: <Intel(R) PRO/1000 Network Connection version - 2.3.4> port
0xd000-0xd01f mem 0xfbb00000-0xfbb1ffff,0xfbb80000-0xfbb83fff irq 46 at
device 0.3 on pci5
on storage2:
igb0: <Intel(R) PRO/1000 Network Connection version - 2.4.0> port
0xd060-0xd07f mem 0xfbb60000-0xfbb7ffff,0xfbb8c000-0xfbb8ffff irq 42 at
device 0.0 on pci5
igb1: <Intel(R) PRO/1000 Network Connection version - 2.4.0> port
0xd040-0xd05f mem 0xfbb40000-0xfbb5ffff,0xfbb88000-0xfbb8bfff irq 45 at
device 0.1 on pci5
igb2: <Intel(R) PRO/1000 Network Connection version - 2.4.0> port
0xd020-0xd03f mem 0xfbb20000-0xfbb3ffff,0xfbb84000-0xfbb87fff irq 44 at
device 0.2 on pci5
igb3: <Intel(R) PRO/1000 Network Connection version - 2.4.0> port
0xd000-0xd01f mem 0xfbb00000-0xfbb1ffff,0xfbb80000-0xfbb83fff irq 46 at
device 0.3 on pci5
storage1% ifconfig -a
igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
options=401bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO>
ether 00:25:90:c1:1d:18
inet6 fe80::225:90ff:fec1:1d18%igb0 prefixlen 64 scopeid 0x1
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
igb1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
options=401bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO>
ether 00:25:90:c1:1d:18
inet6 fe80::225:90ff:fec1:1d19%igb1 prefixlen 64 scopeid 0x2
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
igb2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
options=401bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO>
ether 00:25:90:c1:1d:18
inet6 fe80::225:90ff:fec1:1d1a%igb2 prefixlen 64 scopeid 0x3
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
igb3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
options=401bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO>
ether 00:25:90:c1:1d:18
inet6 fe80::225:90ff:fec1:1d1b%igb3 prefixlen 64 scopeid 0x4
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x7
inet 127.0.0.1 netmask 0xff000000
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
options=401bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO>
ether 00:25:90:c1:1d:18
inet6 fe80::225:90ff:fec1:1d18%lagg0 prefixlen 64 scopeid 0x8
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
media: Ethernet autoselect
status: active
laggproto lacp lagghash l2,l3,l4
laggport: igb3 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
laggport: igb2 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
laggport: igb1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
laggport: igb0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
vlan14: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
options=103<RXCSUM,TXCSUM,TSO4>
ether 00:25:90:c1:1d:18
inet 192.168.1.65 netmask 0xffffff00 broadcast 192.168.1.255
inet6 fe80::225:90ff:fec1:1d18%vlan14 prefixlen 64 scopeid 0x9
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect
status: active
vlan: 14 parent interface: lagg0
vlan900: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu
9000
options=103<RXCSUM,TXCSUM,TSO4>
ether 00:25:90:c1:1d:18
inet 172.25.25.65 netmask 0xffffff00 broadcast 172.25.25.255
inet6 fe80::225:90ff:fec1:1d18%vlan900 prefixlen 64 scopeid 0xa
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect
status: active
vlan: 900 parent interface: lagg0
storage2% ifconfig
-a
igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
options=403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO>
ether 00:25:90:ca:3b:e0
inet6 fe80::225:90ff:feca:3be0%igb0 prefixlen 64 scopeid 0x1
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
igb1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
options=403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO>
ether 00:25:90:ca:3b:e0
inet6 fe80::225:90ff:feca:3be1%igb1 prefixlen 64 scopeid 0x2
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
igb2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
options=403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO>
ether 00:25:90:ca:3b:e0
inet6 fe80::225:90ff:feca:3be2%igb2 prefixlen 64 scopeid 0x3
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
igb3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
options=403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO>
ether 00:25:90:ca:3b:e0
inet6 fe80::225:90ff:feca:3be3%igb3 prefixlen 64 scopeid 0x4
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
inet 127.0.0.1 netmask 0xff000000
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
options=403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO>
ether 00:25:90:ca:3b:e0
inet6 fe80::225:90ff:feca:3be0%lagg0 prefixlen 64 scopeid 0x6
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
media: Ethernet autoselect
status: active
laggproto lacp lagghash l2,l3,l4
laggport: igb3 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
laggport: igb2 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
laggport: igb1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
laggport: igb0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
vlan14: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
options=303<RXCSUM,TXCSUM,TSO4,TSO6>
ether 00:25:90:ca:3b:e0
inet 192.168.1.72 netmask 0xffffff00 broadcast 192.168.1.255
inet6 fe80::225:90ff:feca:3be0%vlan14 prefixlen 64 scopeid 0x7
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect
status: active
vlan: 14 parent interface: lagg0
vlan900: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu
9000
options=303<RXCSUM,TXCSUM,TSO4,TSO6>
ether 00:25:90:ca:3b:e0
inet 172.25.25.72 netmask 0xffffff00 broadcast 172.25.25.255
inet6 fe80::225:90ff:feca:3be0%vlan900 prefixlen 64 scopeid 0x8
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect
status: active
vlan: 900 parent interface: lagg0
Both boxes are connected to the same switch (HP 1910-48G)
I need to transfer around 10 TB of data from storage1 to storage2
I obeserve that during copying, only one NIC (instead of all 4) is used.
Boxes are not stressed during copying
What's more, apart from having 1 NIC saturated (transfer around 120
MB/s), I observe transfer rate on level of 70-80 MB/s
I haven't changed any kernel configuration, nor sysctl's - am I missing
sth ?
Regards,
Marek
--
Marek Salwerowicz
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?535A1354.2040309>