Date: Thu, 4 Feb 2016 13:35:02 +0000 From: Hongjiang Zhang <honzhan@microsoft.com> To: "Meyer, Wolfgang" <wolfgang.meyer@hob.de>, "'freebsd-net@FreeBSD.org'" <freebsd-net@FreeBSD.org> Cc: "'freebsd-performance@FreeBSD.org'" <freebsd-performance@FreeBSD.org> Subject: RE: ixgbe: Network performance tuning (#TCP connections) Message-ID: <af25d963881e4080ab5a946c05aee077@SG2PR3002MB0106.064d.mgd.msft.net> In-Reply-To: <EC88118611AE564AB0B10C6A4569004D0137D57AEB@HOBEX11.hob.de> References: <EC88118611AE564AB0B10C6A4569004D0137D57AEB@HOBEX11.hob.de>
next in thread | previous in thread | raw e-mail | index | archive | help
Did you enable LRO on FreeBSD side (check 'ifconfig' output)? Linux default= enables GRO (see the output of 'ethtool -k eth0'). Thanks Hongjiang Zhang -----Original Message----- From: owner-freebsd-net@freebsd.org [mailto:owner-freebsd-net@freebsd.org] = On Behalf Of Meyer, Wolfgang Sent: Wednesday, February 3, 2016 9:37 PM To: 'freebsd-net@FreeBSD.org' <freebsd-net@FreeBSD.org> Cc: 'freebsd-performance@FreeBSD.org' <freebsd-performance@FreeBSD.org> Subject: ixgbe: Network performance tuning (#TCP connections) Hello, we are evaluating network performance on a DELL-Server (PowerEdge R930 with= 4 Sockets, hw.model: Intel(R) Xeon(R) CPU E7-8891 v3 @ 2.80GHz) with 10 Gb= E-Cards. We use programs that on server side accepts connections on a IP-ad= dress+port from the client side and after establishing the connection data = is sent in turns between server and client in a predefined pattern (server = side sends more data than client side) with sleeps in between the send phas= es. The test set-up is chosen in such way that every client process initiat= es 500 connections handled in threads and on the server side each process r= epresenting an IP/Port pair also handles 500 connections in threads. The number of connections is then increased and the overall network througp= ut is observed using nload. On FreeBSD (on server side) roughly at 50,000 c= onnections errors begin to occur and the overall throughput won't increase = further with more connections. With Linux on the server side it is possible= to establish more than 120,000 connections and at 50,000 connections the o= verall throughput ist double that of FreeBSD with the same sending pattern.= Furthermore system load on FreeBSD is much higher with 50 % system usage o= n each core and 80 % interrupt usage on the 8 cores handling the interrupt = queues for the NIC. In comparison Linux has <10 % system usage, <10 % user = usage and about 15 % interrupt usage on the 16 cores handling the network i= nterrupts for 50,000 connections. Varying the numbers for the NIC interrupt queues won't change the performan= ce (rather worsens the situation). Disabling Hyperthreading (utilising 40 c= ores) degrades the performance. Increasing MAXCPU to utilise all 80 cores w= on't improve compared to 64 cores, atkbd and uart had to be disabled to avo= id kernel panics with increased MAXCPU (thanks to Andre Oppermann for inves= tigating this). Initiallly the tests were made on 10.2 Release, later I swi= tched to 10 Stable (later with ixgbe driver version 3.1.0) but that didn't = change the numbers. Some sysctl configurables were modified along the network performance guide= lines found on the net (e.g. https://na01.safelinks.protection.outlook.com/= ?url=3Dhttps%3a%2f%2fcalomel.org%2ffreebsd_network_tuning.html%2c&data=3D01= %7c01%7chonzhan%40064d.mgd.microsoft.com%7cf827a05328ca4ca9781608d32c9f5b12= %7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=3DxsMoC%2b1ZcnoHBnPqhLUMDIr8V= LBcLejnrXgkRyDWzYc%3d https://na01.safelinks.protection.outlook.com/?url=3D= https%3a%2f%2fwww.freebsd.org%2fdoc%2fhandbook%2fconfigtuning-kernel-limits= .html%2c&data=3D01%7c01%7chonzhan%40064d.mgd.microsoft.com%7cf827a05328ca4c= a9781608d32c9f5b12%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=3DXNqvrYfTN= zfe2btrip%2f5FoX3iTTpTSbNrDjbhtVBevo%3d https://na01.safelinks.protection.o= utlook.com/?url=3Dhttps%3a%2f%2fpleiades.ucsc.edu%2fhyades%2fFreeBSD_Networ= k_Tuning&data=3D01%7c01%7chonzhan%40064d.mgd.microsoft.com%7cf827a05328ca4c= a9781608d32c9f5b12%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=3D%2bQ66X%2= frnqNakX%2fSGcK08QTTrsDjUUWBHOXu6%2fOBIBNQ%3d) but most of them didn't have= any measuarable impact. Final sysctl.conf and loader.conf settings see bel= ow. Actually the only tunables that provided any improvement were identifie= d to be hw.ix.txd, and hw.ix.rxd that were reduced (!) to the minimum value= of 64 and hw.ix.tx_process_limit and hw.ix.rx_process_limit that were set = to -1. Any ideas what tunables might be changed to get a higher number of TCP conn= ections (it's not a question of the overall throughput as changing the send= ing pattern allows me to fully utilise the 10Gb bandwidth)? How can I deter= mine where the kernel is spending its time that causes the high CPU load? A= ny pointers are highly appreciated, I can't believe that there is such a bl= atant difference in network performance compared to Linux. Regards, Wolfgang <loader.conf>: cc_htcp_load=3D"YES" hw.ix.txd=3D"64" hw.ix.rxd=3D"64" hw.ix.tx_process_limit=3D"-1" hw.ix.rx_process_limit=3D"-1" hw.ix.num_queues=3D"8" #hw.ix.enable_aim=3D"0" #hw.ix.max_interrupt_rate=3D"31250" #net.isr.maxthreads=3D"16" <sysctl.conf>: kern.ipc.soacceptqueue=3D1024 kern.ipc.maxsockbuf=3D16777216 net.inet.tcp.sendbuf_max=3D16777216 net.inet.tcp.recvbuf_max=3D16777216 net.inet.tcp.tso=3D0 net.inet.tcp.mssdflt=3D1460 net.inet.tcp.minmss=3D1300 net.inet.tcp.nolocaltimewait=3D1 net.inet.tcp.syncache.rexmtlimit=3D0 #net.inet.tcp.syncookies=3D0 net.inet.tcp.drop_synfin=3D1 net.inet.tcp.fast_finwait2_recycle=3D1 net.inet.tcp.icmp_may_rst=3D0 net.inet.tcp.msl=3D5000 net.inet.tcp.path_mtu_discovery=3D0 net.inet.tcp.blackhole=3D1 net.inet.udp.blackhole=3D1 net.inet.tcp.cc.algorithm=3Dhtcp net.inet.tcp.cc.htcp.adaptive_backoff=3D1 net.inet.tcp.cc.htcp.rtt_scaling=3D1 net.inet.ip.forwarding=3D1 net.inet.ip.fastforwarding=3D1 net.inet.ip.rtexpire=3D1 net.inet.ip.rtminexpire=3D1 ________________________________ Follow HOB: - HOB: https://na01.safelinks.protection.outlook.com/?url=3Dhttp%3a%2f%2fww= w.hob.de%2fredirect%2fhob.html&data=3D01%7c01%7chonzhan%40064d.mgd.microsof= t.com%7cf827a05328ca4ca9781608d32c9f5b12%7c72f988bf86f141af91ab2d7cd011db47= %7c1&sdata=3D%2fenM%2fs72BvfP9H6CnrFeXqhrZoetovqoIMB%2bk0RFfQM%3d - Xing: https://na01.safelinks.protection.outlook.com/?url=3Dhttp%3a%2f%2fw= ww.hob.de%2fredirect%2fxing.html&data=3D01%7c01%7chonzhan%40064d.mgd.micros= oft.com%7cf827a05328ca4ca9781608d32c9f5b12%7c72f988bf86f141af91ab2d7cd011db= 47%7c1&sdata=3DSi0aQmWV%2ba%2fWx5%2f6tZtOWQDj5cmq9t57DhA2h0qsa7Y%3d - LinkedIn: https://na01.safelinks.protection.outlook.com/?url=3Dhttp%3a%2f= %2fwww.hob.de%2fredirect%2flinkedin.html&data=3D01%7c01%7chonzhan%40064d.mg= d.microsoft.com%7cf827a05328ca4ca9781608d32c9f5b12%7c72f988bf86f141af91ab2d= 7cd011db47%7c1&sdata=3DBc0iX1u2twaazr%2fzi2wuTRqTZOk2rwtu61lNqJrui14%3d - HOBLink Mobile: https://na01.safelinks.protection.outlook.com/?url=3Dhttp= %3a%2f%2fwww.hob.de%2fredirect%2fhoblinkmobile.html&data=3D01%7c01%7chonzha= n%40064d.mgd.microsoft.com%7cf827a05328ca4ca9781608d32c9f5b12%7c72f988bf86f= 141af91ab2d7cd011db47%7c1&sdata=3DI1ViplWsQ2HMWM%2fLnhWRKseziJBoNlyVBj2wiWF= x1wM%3d - Facebook: https://na01.safelinks.protection.outlook.com/?url=3Dhttp%3a%2f= %2fwww.hob.de%2fredirect%2ffacebook.html&data=3D01%7c01%7chonzhan%40064d.mg= d.microsoft.com%7cf827a05328ca4ca9781608d32c9f5b12%7c72f988bf86f141af91ab2d= 7cd011db47%7c1&sdata=3D1WliqTPX5YN3AdvK6DzAB7yDkQmjyC3jh%2f47PZ2uU7Y%3d - Twitter: https://na01.safelinks.protection.outlook.com/?url=3Dhttp%3a%2f%= 2fwww.hob.de%2fredirect%2ftwitter.html&data=3D01%7c01%7chonzhan%40064d.mgd.= microsoft.com%7cf827a05328ca4ca9781608d32c9f5b12%7c72f988bf86f141af91ab2d7c= d011db47%7c1&sdata=3DgaLO9OTaBis4F3IGoFY55nwMIOGPaZ0ri%2fK7N7nx7kI%3d - YouTube: https://na01.safelinks.protection.outlook.com/?url=3Dhttp%3a%2f%= 2fwww.hob.de%2fredirect%2fyoutube.html&data=3D01%7c01%7chonzhan%40064d.mgd.= microsoft.com%7cf827a05328ca4ca9781608d32c9f5b12%7c72f988bf86f141af91ab2d7c= d011db47%7c1&sdata=3DT%2fhNogZgLHoTCOj%2fKjsSJhPDlBqyxCAD8tJj5fueiqw%3d - E-Mail: https://na01.safelinks.protection.outlook.com/?url=3Dhttp%3a%2f%2= fwww.hob.de%2fredirect%2fmail.html&data=3D01%7c01%7chonzhan%40064d.mgd.micr= osoft.com%7cf827a05328ca4ca9781608d32c9f5b12%7c72f988bf86f141af91ab2d7cd011= db47%7c1&sdata=3DQnWAw2culSVFforLRWYgXfyHaj4PyB4Hn1rj4jVQvjU%3d HOB GmbH & Co. KG Schwadermuehlstr. 3 D-90556 Cadolzburg Geschaeftsfuehrung: Klaus Brandstaetter, Zoran Adamovic AG Fuerth, HRA 5180 Steuer-Nr. 218/163/00107 USt-ID-Nr. DE 132747002 Komplementaerin HOB electronic Beteiligungs GmbH AG Fuerth, HRB 3416 ______= _________________________________________ freebsd-net@freebsd.org mailing list https://na01.safelinks.protection.outlook.com/?url=3Dhttps%3a%2f%2flists.fr= eebsd.org%2fmailman%2flistinfo%2ffreebsd-net&data=3D01%7c01%7chonzhan%40064= d.mgd.microsoft.com%7cf827a05328ca4ca9781608d32c9f5b12%7c72f988bf86f141af91= ab2d7cd011db47%7c1&sdata=3DGU28hzKU5SSyA3hbP1PNtUNh7G5ut%2fMD6mdRmuJNkEs%3d To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?af25d963881e4080ab5a946c05aee077>