Date: Wed, 04 Sep 2013 11:26:46 -0400 From: Mike Tancsa <mike@sentex.net> To: Rick Macklem <rmacklem@uoguelph.ca> Cc: FreeBSD Net <net@freebsd.org>, David Wolfskill <david@catwhisker.org> Subject: TSO help or hindrance ? (was Re: TSO and FreeBSD vs Linux) Message-ID: <52275136.1010105@sentex.net> In-Reply-To: <979862494.17918795.1378299005617.JavaMail.root@uoguelph.ca> References: <979862494.17918795.1378299005617.JavaMail.root@uoguelph.ca>
next in thread | previous in thread | raw e-mail | index | archive | help
This is a multi-part message in MIME format. --------------090102030905000709060202 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit On 9/4/2013 8:50 AM, Rick Macklem wrote: > David Wolfskill wrote: >> >> >> I noticed that when I tried to write files to NFS, I could write >> small >> files OK, but larger ones seemed to ... hang. >> * "ifconfig -v em0" showed flags TSO4 & VLAN_HWTSO turned on. >> * "sysctl net.inet.tcp.tso" showed "1" -- enabled. >> >> As soon as I issued "sudo net.inet.tcp.tso=0" ... the copy worked >> without >> a hitch or a whine. And I was able to copy all 117709618 bytes, not >> just >> 2097152 (2^21). >> >> Is the above expected? It came rather as a surprise to me. >> > Not surprising to me, I'm afraid. When there are serious NFS problems > like this, it is often caused by a network fabric issue and broken > TSO is at the top of the list w.r.t. cause. I was just experimenting a bit with iSCSI via FreeNAS and was a little disappointed at the speeds I was getting. So, I tried disabling tso on both boxes and it did seem to speed things up a bit. Data and testing methods attached in a txt file. I did 3 cases. Just boot up FreeNAS and the initiator without tweaks. That had the worst performance. disable tso on the nic as well as via sysctl on both boxes. That had the best performance. re-enable tso on both boxes. That had better performance than the first case, but still not as good as totally disabling it. I am guessing something is not quite being re-enabled properly ? But its different than the other two cases ?!? tgt is FreeNAS-9.1.1-RELEASE-x64 (a752d35) and initiator is r254328 9.2 AMD64 The FreeNAS box has 16G of RAM, so the file is being served out of cache as gstat shows no activity when sending out the file ---Mike -- ------------------- Mike Tancsa, tel +1 519 651 3400 Sentex Communications, mike@sentex.net Providing Internet services since 1994 www.sentex.net Cambridge, Ontario Canada http://www.tancsa.com/ --------------090102030905000709060202 Content-Type: text/plain; charset=windows-1252; name="stat.txt" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="stat.txt" 3 data files. notso = tso disabled on tgt and initiator tso-boot = initiator is rebooted and tests are run tso = tso disabled and then re-enabled. For whatever reason its not as bad as post boot, but still worse than no tso 0{mdttestbox}# ministat notso tso-boot x notso + tso-boot +----------------------------------------------------------------------------------------------------------------------+ | + | | +++ x | |+ ++++ xxxx x x x x| | |AM| |___M__A_____| | +----------------------------------------------------------------------------------------------------------------------+ N Min Max Median Avg Stddev x 9 69086843 71749085 69475320 69810666 887628.9 + 9 55846191 56473897 56357956 56302165 208843.3 Difference at 95.0% confidence -1.35085e+07 +/- 644386 -19.3502% +/- 0.923048% (Student's t, pooled s = 644787) 0{mdttestbox}# # ministat tso notso x tso + notso +----------------------------------------------------------------------------------------------------------------------+ | x + | |x xxxxx x + x + + + + + + +| | |____MA___||___________________M___________A_________________________________| | +----------------------------------------------------------------------------------------------------------------------+ N Min Max Median Avg Stddev x 9 68632067 69126677 68779201 68808473 142050.01 + 9 69086843 71749085 69475320 69810666 887628.9 Difference at 95.0% confidence 1.00219e+06 +/- 635239 1.4565% +/- 0.923199% (Student's t, pooled s = 635635) 0{mdttestbox}# cat tso 68779201 68734143 68915705 68752827 68782212 69126677 68828520 68724901 68632067 0{mdttestbox}# cat notso 71749085 69256608 69097532 69086843 70587459 69179672 69475320 69754511 70108963 0{mdttestbox}# cat tso-boot 55846191 56314173 56284204 56095729 56466769 56446535 56357956 56434027 56473897 0{mdttestbox}# 0{mdttestbox}# cat t.sh #!/bin/sh command="dd if=/mnt/test of=/dev/null bs=4096k" for i in `jot 9 1`;do mount /dev/da0a /mnt eval $command 2>&1 | grep bytes | awk -F"[\( ]" '{print $8}' umount /mnt done 0{mdttestbox}# initiator is r254328 9.2 AMD64 The two boxes are linked via private gigE switch em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000 options=4209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWTSO> ether 00:1e:67:49:31:b8 inet 10.254.1.2 netmask 0xffffff00 broadcast 10.254.1.255 inet6 fe80::21e:67ff:fe49:31b8%em1 prefixlen 64 scopeid 0x3 nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL> media: Ethernet autoselect (1000baseT <full-duplex>) status: active ifconfig igb0 igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000 options=400bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO> ether 00:15:17:fe:a5:bc inet 10.254.1.1 netmask 0xffffff00 broadcast 10.254.1.255 inet6 fe80::215:17ff:fefe:a5bc%igb0 prefixlen 64 scopeid 0x1 nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> media: Ethernet autoselect (1000baseT <full-duplex>) status: active Initiator just does iscontrol -c /etc/iscsi.conf -n freebsd freebsd { authmethod = none initiatorname = mdttestbox.sentex.ca TargetName = iqn.2011-03.org.example.istgt:freebsd TargetAddress = 10.254.1.1:3260,2 } ifconfig igb0 -tso ifconfig em0 -tso and sysctl -w net.inet.tcp.tso=0 --------------090102030905000709060202--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?52275136.1010105>