Date: Sat, 23 Nov 2019 09:51:15 +0100 From: Vincenzo Maffione <vmaffione@freebsd.org> To: BulkMailForRudy <crapsh@monkeybrains.net> Cc: "freebsd-net@freebsd.org" <freebsd-net@freebsd.org> Subject: Re: ix0 and ix1 ifconfig options different on Supermicro board Message-ID: <CA%2B_eA9iV=MZSrC%2BQHWnM6kQcTj35wpkeaUfC5ON39eUj4ZevSQ@mail.gmail.com> In-Reply-To: <c0cd7a45-d986-2f40-db65-610b543ffebe@monkeybrains.net> References: <e4f76015-a661-30aa-a9ef-166f683a6082@monkeybrains.net> <c0cd7a45-d986-2f40-db65-610b543ffebe@monkeybrains.net>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi, TSO/LRO (for IPv4 and/or IPv6) will increase TCP bulk throughput on machine X for those TCP connection where X is one of the two endpoints, that is TCP connections that are local to X. That's why you are seing iperf achieving higher throughput with TSO/LRO enabled. TSO means that your local TCP stack will pass down large (e.g. 32K) packets to the NIC driver, and the NIC will take care of segmentation. This is beneficial for two reasons: (1) the segmentation work is done in hardware rather than in the CPU, and this is typically faster (and also, you save the CPU time for other stuff); (2) the per-packet cost of protocol processing (TCP, IP, Ethernet) is amortized over a large amount of bytes, which means that your total per-byte CPU time will be way lower. Most of the gain actually comes from (2). LRO is similar, but in the receive direction. However, if your device is a router it means that it forwards packets. Therefore the local TCP stack is not involved, so TSO simply does not apply (at least in FreeBSD). I think LRO applies, but there is a latency hit, as suggested by the wiki page you pointed. So no, enabling TSO/LRO will not increase the forwarding rate, but possibly increase latency. You should keep it disabled. Cheers, Vincenzo Il giorno ven 22 nov 2019 alle ore 22:47 BulkMailForRudy < crapsh@monkeybrains.net> ha scritto: > > I just did another test to a machine with a Chelsio card. > > Server D (cxl3) -> Server A = 3.5Gbps > > Turning on flags lro tso4 tso6 vlanhwtso , yields > > Server D (cxl3) -> Server A = 9.1 Gbps > > Oddly, this was an ipv4 iperf, but tso6 seems to help. > > I had settings turned off per > https://wiki.freebsd.org/10gFreeBSD/Router#Disabling_LRO_and_TSO > > Servers A,B, and C are all running services. Server D is acting as a > router. Are the LRO and TSO only for TCP to the box, or will it > increase speeds for forwarding if I enable it? > > > Thanks, > > Rudy > > > On 11/22/19 1:30 PM, BulkMailForRudy wrote: > > > > I have nearly identical setups, but ix0 and ix1 are getting different > > options at boot. This seems to be the only difference I see between > > machines and I am trying to answer the question, Why can Server A > > iperf close to line rate while the other servers can not? > > > > The Test: iperf -P 3 -c REMOTE_ADDR > > > > Server A (ix1) -> Server C (ix0) = 9.4Gbps > > Server B (ix0)-> Server C (ix0) = 5.6Gbps > > Server C (ix0)-> A (ix1) or B (ix0) = 5.0Gbps > > > > > > The motherboards are identical between A,B and C and the configs very > > similar. The only difference is that Server A is plugged into ix1 > > while Server B and C are using ix0. > > > > > > I am not modifying the flags at boot (eg ifconfig -tso), yet ix0 lacks > > TXCSUM,TSO4,TSO6,LRO,WOL. > > > > ix0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric > > 0 mtu 1500 > > > options=a538b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6> > > > > > ether *ac:1f:6b:6a:14:6*4 > > media: Ethernet autoselect (10Gbase-T <full-duplex>) > > ix1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 > > > options=e53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6> > > > > > ether *ac:1f:6b:6a:14:6*5 > > media: Ethernet autoselect (10Gbase-T <full-duplex>) > > > > I did try adding some flags to ix0 and -- not sure if this was the > > reason -- the box started acting oddly and I ended up rebooting it. > > > > > > My hunch has is that there is somethign with the TSO4. > > > > > > Rudy > > > > _______________________________________________ > > freebsd-net@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-net > > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > > > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CA%2B_eA9iV=MZSrC%2BQHWnM6kQcTj35wpkeaUfC5ON39eUj4ZevSQ>