Date: Wed, 05 Nov 2014 17:12:30 -0800 From: "George Neville-Neil" <gnn@neville-neil.com> To: "Andrey V. Elsukov" <bu7cher@yandex.ru> Cc: Evandro Nunes <evandronunes12@gmail.com>, "Alexander V. Chernikov" <melifaro@freebsd.org>, "freebsd-net@freebsd.org" <net@freebsd.org> Subject: Re: netmap in GENERIC, by default, on HEAD Message-ID: <7382F19D-A4E9-4831-8261-952629231B1A@neville-neil.com> In-Reply-To: <545A57A8.2010904@yandex.ru> References: <92D22BEA-DDE5-4C6E-855C-B8CACB0319AC@neville-neil.com> <545A47A5.4010601@yandex.ru> <CAG4HiT46ZT1dDTt7Asv8gagPKrwH%2BTS-uUsvKDhNKiXMDM68nA@mail.gmail.com> <545A57A8.2010904@yandex.ru>
next in thread | previous in thread | raw e-mail | index | archive | help
This is an OpenPGP/MIME signed message (RFC 3156 and 4880).
--=_MailMate_270F14BB-B8AD-4903-9533-4BF594370719_=
Content-Type: text/plain
On 5 Nov 2014, at 9:00, Andrey V. Elsukov wrote:
> On 05.11.2014 19:18, Evandro Nunes wrote:
>> On Wed, Nov 5, 2014 at 1:52 PM, Andrey V. Elsukov <bu7cher@yandex.ru> wrote:
>>
>>> On 05.11.2014 18:39, George Neville-Neil wrote:
>>>> Howdy,
>>>>
>>>> Last night (Pacific Time) I committed a change so that GENERIC, on HEAD
>>>> has the netmap
>>>> device enabled. This is to increase the breadth of our testing of that
>>>> feature prior
>>>> to the release of FreeBSD 11.
>>>>
>>>> In two weeks I will enable IPSec by default, again in preparation for 11.
>>>
>>> Hi,
>>>
>>> recently we did some IP forwarding tests and the GENERIC kernel is
>>> several times faster than GENERIC+IPSEC. Even when IPSEC has no SA.
>>>
>>> I didn't do test on vanilla kernel, but our kernel is able forward
>>> IPv4/IPv6 on rate close to 8.6 Mpps. The same kernel compiled with IPSEC
>>> can forward only 180 kpps. I think this problem should be solved before
>>> enabling it in GENERIC.
>>>
>>
>> this forward rate you mention is related to netmap? or usual
>> forwarding/fastforwarding? this is a huge number, do you mind sharing your
>> dmesg output and top -PSH output so I can check for interrupt CPU usage and
>> other relevant stuff?
>
> This is patched kernel without netmap and fastforwarding. We removed all
> lock contention on the forwarding path to be sure that it doesn't affect
> IPSEC.
>
> Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz (2200.05-MHz K8-class CPU)
> FreeBSD/SMP: Multiprocessor System Detected: 32 CPUs
> FreeBSD/SMP: 2 package(s) x 8 core(s) x 2 SMT threads
> real memory = 68736253952 (65552 MB)
> avail memory = 66370662400 (63295 MB)
> ix0: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 2.5.15>
> port 0x7020-0x703f mem 0xde680000-0xde6fffff,0xde704000-0xde707fff irq
> 32 at device 0.0 on pci4
> ix0: Using MSIX interrupts with 16 vectors
> ix0: Ethernet address: 90:e2:ba:0d:73:54
> ix0: PCI Express Bus: Speed 5.0GT/s Width x8
>
> This is IPv6 forwarding test - 6 /64 prefixes each has 200 random
> addresses. They are routed between 6 vlans.
>
> # netstat -I ix0 -w 1
> input (ix0) output
> packets errs idrops bytes packets errs bytes colls
> 8917043 0 0 571436864 8149880 0 522587200 0
> 8943391 0 0 571598336 8179318 0 525085504 0
> 8928155 0 0 571262144 8168254 0 522712192 0
> 8921342 0 937 571693504 8128132 0 521997184 0
> 8924322 0 0 571170048 8211500 0 520264320 0
> 8934564 0 0 571483584 8180040 0 524475264 0
> 8937039 0 0 571384640 8234779 0 525686080 0
> 8926528 0 0 571481728 8160380 0 524265920 0
> 8923160 0 0 571397248 8229839 0 522569408 0
> 8930070 0 1705 571594944 8216092 0 528481152 0
> 8916249 0 0 571294784 8184286 0 524399360 0
> 8937301 0 0 571391040 8221895 0 526383744 0
> 8927967 0 0 571613312 8164779 0 524997760 0
> 8936306 0 0 571251712 8167960 0 519575744 0
> 8922983 0 306 571430528 8216466 0 525893056 0
> 8916209 0 0 571434240 8202692 0 526046336 0
> 8945608 0 0 571426624 8265756 0 524815552 0
> 8925548 0 1045 571444480 8229681 0 530935232 0
> 8932145 0 0 571747200 8149710 0 523409536 0
> 8929339 0 0 571683200 8186790 0 520719040 0
> 8917697 0 0 571585152 8212635 0 525775680 0
>
> # top -PSH
> last pid: 2788; load averages: 12.01, 4.76, 1.92
>
> up 0+00:04:38 20:58:48
> 471 processes: 45 running, 344 sleeping, 82 waiting
> CPU 0: 0.0% user, 0.0% nice, 21.6% system, 68.2% interrupt, 10.2% idle
> CPU 1: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
> CPU 2: 0.0% user, 0.0% nice, 2.7% system, 84.3% interrupt, 12.9% idle
> CPU 3: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
> CPU 4: 0.0% user, 0.0% nice, 3.9% system, 86.7% interrupt, 9.4% idle
> CPU 5: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
> CPU 6: 0.0% user, 0.0% nice, 5.5% system, 88.6% interrupt, 5.9% idle
> CPU 7: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
> CPU 8: 0.0% user, 0.0% nice, 3.5% system, 90.2% interrupt, 6.3% idle
> CPU 9: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
> CPU 10: 0.0% user, 0.0% nice, 3.1% system, 87.1% interrupt, 9.8% idle
> CPU 11: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
> CPU 12: 0.0% user, 0.0% nice, 27.5% system, 62.0% interrupt, 10.6% idle
> CPU 13: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
> CPU 14: 0.0% user, 0.0% nice, 6.3% system, 85.9% interrupt, 7.8% idle
> CPU 15: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
> CPU 16: 0.0% user, 0.0% nice, 17.6% system, 79.6% interrupt, 2.7% idle
> CPU 17: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
> CPU 18: 0.0% user, 0.0% nice, 2.4% system, 92.2% interrupt, 5.5% idle
> CPU 19: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
> CPU 20: 0.0% user, 0.0% nice, 7.8% system, 86.7% interrupt, 5.5% idle
> CPU 21: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
> CPU 22: 0.0% user, 0.0% nice, 6.3% system, 87.5% interrupt, 6.3% idle
> CPU 23: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
> CPU 24: 0.0% user, 0.0% nice, 1.6% system, 89.4% interrupt, 9.0% idle
> CPU 25: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
> CPU 26: 0.0% user, 0.0% nice, 2.0% system, 91.8% interrupt, 6.3% idle
> CPU 27: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
> CPU 28: 0.0% user, 0.0% nice, 2.7% system, 87.8% interrupt, 9.4% idle
> CPU 29: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
> CPU 30: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
> CPU 31: 0.0% user, 0.0% nice, 0.4% system, 0.0% interrupt, 99.6% idle
> Mem: 17M Active, 13M Inact, 716M Wired, 61G Free
> ARC: 28M Total, 6102K MFU, 20M MRU, 16K Anon, 373K Header, 1492K Other
> Swap: 16G Total, 16G Free
>
>
> # top -HSIzts1
> last pid: 2805; load averages: 13.88, 7.92, 3.56
>
> up 0+00:06:40 21:00:50
> 469 processes: 45 running, 342 sleeping, 82 waiting
> CPU: 0.0% user, 0.0% nice, 3.5% system, 39.3% interrupt, 57.1% idle
> Mem: 17M Active, 13M Inact, 716M Wired, 61G Free
> ARC: 28M Total, 6105K MFU, 20M MRU, 16K Anon, 373K Header, 1494K Other
> Swap: 16G Total, 16G Free
>
> PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND
> 12 root -92 - 0K 1504K CPU22 22 3:38 87.35%
> intr{irq293: ix0:que }
> 12 root -92 - 0K 1504K WAIT 26 3:41 86.57%
> intr{irq295: ix0:que }
> 12 root -92 - 0K 1504K CPU18 18 3:41 86.47%
> intr{irq291: ix0:que }
> 12 root -92 - 0K 1504K CPU28 28 3:41 86.47%
> intr{irq296: ix0:que }
> 12 root -92 - 0K 1504K CPU24 24 3:39 86.18%
> intr{irq294: ix0:que }
> 12 root -92 - 0K 1504K WAIT 4 3:35 86.18%
> intr{irq284: ix0:que }
> 12 root -92 - 0K 1504K CPU8 8 3:35 86.18%
> intr{irq286: ix0:que }
> 12 root -92 - 0K 1504K CPU6 6 3:33 85.60%
> intr{irq285: ix0:que }
> 12 root -92 - 0K 1504K CPU20 20 3:32 85.50%
> intr{irq292: ix0:que }
> 12 root -92 - 0K 1504K CPU10 10 3:32 84.86%
> intr{irq287: ix0:que }
> 12 root -92 - 0K 1504K CPU2 2 3:32 84.67%
> intr{irq283: ix0:que }
> 12 root -92 - 0K 1504K WAIT 14 3:31 84.38%
> intr{irq289: ix0:que }
> 12 root -92 - 0K 1504K CPU16 16 3:11 77.59%
> intr{irq290: ix0:que }
> 12 root -92 - 0K 1504K CPU0 0 2:42 64.70%
> intr{irq282: ix0:que }
> 12 root -92 - 0K 1504K CPU12 12 2:44 63.67%
> intr{irq288: ix0:que }
> 0 root -92 0 0K 4672K - 12 1:01 26.66%
> kernel{ix0 que}
> 0 root -92 0 0K 4672K - 0 0:58 23.00%
> kernel{ix0 que}
> 0 root -92 0 0K 4672K - 16 0:48 18.16%
> kernel{ix0 que}
> 0 root -92 0 0K 4672K - 20 0:20 7.86%
> kernel{ix0 que}
> 0 root -92 0 0K 4672K - 14 0:15 5.57%
> kernel{ix0 que}
> 0 root -92 0 0K 4672K CPU6 6 0:14 5.57%
> kernel{ix0 que}
> 0 root -92 0 0K 4672K - 22 0:12 4.59%
> kernel{ix0 que}
> 0 root -92 0 0K 4672K - 18 0:06 4.59%
> kernel{ix0 que}
> 0 root -92 0 0K 4672K - 28 0:06 4.49%
> kernel{ix0 que}
> 0 root -92 0 0K 4672K - 26 0:06 4.39%
> kernel{ix0 que}
> 0 root -92 0 0K 4672K - 24 0:05 4.05%
> kernel{ix0 que}
> 0 root -92 0 0K 4672K - 8 0:09 3.47%
> kernel{ix0 que}
> 0 root -92 0 0K 4672K - 4 0:08 3.17%
> kernel{ix0 que}
> 0 root -92 0 0K 4672K - 10 0:07 2.98%
> kernel{ix0 que}
> 0 root -92 0 0K 4672K - 2 0:05 2.39%
> kernel{ix0 que}
>
> # pmcstat -TS llc-misses -w10
> PMC: [llc-misses] Samples: 3949 (100.0%) , 0 unresolved
>
> %SAMP IMAGE FUNCTION CALLERS
> 28.6 kernel ixgbe_rxeof ixgbe_msix_que:25.9
> ixgbe_handle_que:2.8
> 14.7 kernel bcmp netisr_dispatch_src:13.5
> rtalloc_fib_nolock:1.0
> 5.9 kernel mb_ctor_mbuf uma_zalloc_arg
> 5.8 kernel ixgbe_mq_start vlan_transmit
> 4.6 kernel _mtx_trylock ixgbe_mq_start
> 3.8 kernel ether_nh_input netisr_dispatch_src
> 2.6 kernel ether_input ixgbe_msix_que
> 2.4 kernel m_tag_delete_chain m_tag_copy_chain:1.2 uma_zfree_arg:0.9
> 2.4 kernel netisr_dispatch_src ixgbe_rxeof:1.6 ixgbe_msix_que:0.6
> 2.2 kernel cpu_search_highest cpu_search_highest
> 2.2 kernel ixgbe_txeof ixgbe_msix_que
> 2.1 kernel uma_zfree_arg m_freem
> 2.0 kernel netisr_dispatch ixgbe_msix_que:1.1 ixgbe_rxeof:0.7
> 2.0 kernel critical_exit uma_zfree_arg
> 1.9 kernel _thread_lock_flags ithread_loop:1.2
> intr_event_schedule_thread:0.6
> 1.9 kernel _bus_dmamap_sync ixgbe_rxeof
> 1.3 kernel ixgbe_xmit ixgbe_mq_start_locked
> 0.9 kernel ixgbe_msix_que intr_event_execute_handlers
>
Thanks! This looks like it might be related to issues in the ixgbe driver specifically.
I will test against this and other drivers in the lab.
Also, did you cpuset your ixgbe interrupts to package 0? (cpu 0..7) and turn off
SMT?
Best,
George
--=_MailMate_270F14BB-B8AD-4903-9533-4BF594370719_=
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename=signature.asc
Content-Type: application/pgp-signature; name=signature.asc
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org
iEYEARECAAYFAlRayv4ACgkQYdh2wUQKM9IepQCdHcbJSzrSZn58IVXhIR+YmFZD
4V0AoOBuWbqXuwz6UDUjgnm47ZHiB6Wj
=IxlB
-----END PGP SIGNATURE-----
--=_MailMate_270F14BB-B8AD-4903-9533-4BF594370719_=--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?7382F19D-A4E9-4831-8261-952629231B1A>