Date: Wed, 05 Nov 2014 20:00:24 +0300 From: "Andrey V. Elsukov" <bu7cher@yandex.ru> To: Evandro Nunes <evandronunes12@gmail.com> Cc: "Alexander V. Chernikov" <melifaro@freebsd.org>, "freebsd-net@freebsd.org" <net@freebsd.org> Subject: Re: netmap in GENERIC, by default, on HEAD Message-ID: <545A57A8.2010904@yandex.ru> In-Reply-To: <CAG4HiT46ZT1dDTt7Asv8gagPKrwH%2BTS-uUsvKDhNKiXMDM68nA@mail.gmail.com> References: <92D22BEA-DDE5-4C6E-855C-B8CACB0319AC@neville-neil.com> <545A47A5.4010601@yandex.ru> <CAG4HiT46ZT1dDTt7Asv8gagPKrwH%2BTS-uUsvKDhNKiXMDM68nA@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
This is an OpenPGP/MIME signed message (RFC 4880 and 3156)
--w6HrlTCB1wwmluQU9FwTkbOBVFpAuU9P2
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: quoted-printable
On 05.11.2014 19:18, Evandro Nunes wrote:
> On Wed, Nov 5, 2014 at 1:52 PM, Andrey V. Elsukov <bu7cher@yandex.ru> w=
rote:
>=20
>> On 05.11.2014 18:39, George Neville-Neil wrote:
>>> Howdy,
>>>
>>> Last night (Pacific Time) I committed a change so that GENERIC, on HE=
AD
>>> has the netmap
>>> device enabled. This is to increase the breadth of our testing of th=
at
>>> feature prior
>>> to the release of FreeBSD 11.
>>>
>>> In two weeks I will enable IPSec by default, again in preparation for=
11.
>>
>> Hi,
>>
>> recently we did some IP forwarding tests and the GENERIC kernel is
>> several times faster than GENERIC+IPSEC. Even when IPSEC has no SA.
>>
>> I didn't do test on vanilla kernel, but our kernel is able forward
>> IPv4/IPv6 on rate close to 8.6 Mpps. The same kernel compiled with IPS=
EC
>> can forward only 180 kpps. I think this problem should be solved befor=
e
>> enabling it in GENERIC.
>>
>=20
> this forward rate you mention is related to netmap? or usual
> forwarding/fastforwarding? this is a huge number, do you mind sharing y=
our
> dmesg output and top -PSH output so I can check for interrupt CPU usage=
and
> other relevant stuff?
This is patched kernel without netmap and fastforwarding. We removed all
lock contention on the forwarding path to be sure that it doesn't affect
IPSEC.
Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz (2200.05-MHz K8-class CPU)
FreeBSD/SMP: Multiprocessor System Detected: 32 CPUs
FreeBSD/SMP: 2 package(s) x 8 core(s) x 2 SMT threads
real memory =3D 68736253952 (65552 MB)
avail memory =3D 66370662400 (63295 MB)
ix0: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 2.5.15>
port 0x7020-0x703f mem 0xde680000-0xde6fffff,0xde704000-0xde707fff irq
32 at device 0.0 on pci4
ix0: Using MSIX interrupts with 16 vectors
ix0: Ethernet address: 90:e2:ba:0d:73:54
ix0: PCI Express Bus: Speed 5.0GT/s Width x8
This is IPv6 forwarding test - 6 /64 prefixes each has 200 random
addresses. They are routed between 6 vlans.
# netstat -I ix0 -w 1
input (ix0) output
packets errs idrops bytes packets errs bytes colls
8917043 0 0 571436864 8149880 0 522587200 0
8943391 0 0 571598336 8179318 0 525085504 0
8928155 0 0 571262144 8168254 0 522712192 0
8921342 0 937 571693504 8128132 0 521997184 0
8924322 0 0 571170048 8211500 0 520264320 0
8934564 0 0 571483584 8180040 0 524475264 0
8937039 0 0 571384640 8234779 0 525686080 0
8926528 0 0 571481728 8160380 0 524265920 0
8923160 0 0 571397248 8229839 0 522569408 0
8930070 0 1705 571594944 8216092 0 528481152 0
8916249 0 0 571294784 8184286 0 524399360 0
8937301 0 0 571391040 8221895 0 526383744 0
8927967 0 0 571613312 8164779 0 524997760 0
8936306 0 0 571251712 8167960 0 519575744 0
8922983 0 306 571430528 8216466 0 525893056 0
8916209 0 0 571434240 8202692 0 526046336 0
8945608 0 0 571426624 8265756 0 524815552 0
8925548 0 1045 571444480 8229681 0 530935232 0
8932145 0 0 571747200 8149710 0 523409536 0
8929339 0 0 571683200 8186790 0 520719040 0
8917697 0 0 571585152 8212635 0 525775680 0
# top -PSH
last pid: 2788; load averages: 12.01, 4.76, 1.92
up 0+00:04:38 20:58:48
471 processes: 45 running, 344 sleeping, 82 waiting
CPU 0: 0.0% user, 0.0% nice, 21.6% system, 68.2% interrupt, 10.2% idle=
CPU 1: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle=
CPU 2: 0.0% user, 0.0% nice, 2.7% system, 84.3% interrupt, 12.9% idle=
CPU 3: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle=
CPU 4: 0.0% user, 0.0% nice, 3.9% system, 86.7% interrupt, 9.4% idle=
CPU 5: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle=
CPU 6: 0.0% user, 0.0% nice, 5.5% system, 88.6% interrupt, 5.9% idle=
CPU 7: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle=
CPU 8: 0.0% user, 0.0% nice, 3.5% system, 90.2% interrupt, 6.3% idle=
CPU 9: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle=
CPU 10: 0.0% user, 0.0% nice, 3.1% system, 87.1% interrupt, 9.8% idle=
CPU 11: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle=
CPU 12: 0.0% user, 0.0% nice, 27.5% system, 62.0% interrupt, 10.6% idle=
CPU 13: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle=
CPU 14: 0.0% user, 0.0% nice, 6.3% system, 85.9% interrupt, 7.8% idle=
CPU 15: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle=
CPU 16: 0.0% user, 0.0% nice, 17.6% system, 79.6% interrupt, 2.7% idle=
CPU 17: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle=
CPU 18: 0.0% user, 0.0% nice, 2.4% system, 92.2% interrupt, 5.5% idle=
CPU 19: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle=
CPU 20: 0.0% user, 0.0% nice, 7.8% system, 86.7% interrupt, 5.5% idle=
CPU 21: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle=
CPU 22: 0.0% user, 0.0% nice, 6.3% system, 87.5% interrupt, 6.3% idle=
CPU 23: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle=
CPU 24: 0.0% user, 0.0% nice, 1.6% system, 89.4% interrupt, 9.0% idle=
CPU 25: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle=
CPU 26: 0.0% user, 0.0% nice, 2.0% system, 91.8% interrupt, 6.3% idle=
CPU 27: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle=
CPU 28: 0.0% user, 0.0% nice, 2.7% system, 87.8% interrupt, 9.4% idle=
CPU 29: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle=
CPU 30: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle=
CPU 31: 0.0% user, 0.0% nice, 0.4% system, 0.0% interrupt, 99.6% idle=
Mem: 17M Active, 13M Inact, 716M Wired, 61G Free
ARC: 28M Total, 6102K MFU, 20M MRU, 16K Anon, 373K Header, 1492K Other
Swap: 16G Total, 16G Free
# top -HSIzts1
last pid: 2805; load averages: 13.88, 7.92, 3.56
up 0+00:06:40 21:00:50
469 processes: 45 running, 342 sleeping, 82 waiting
CPU: 0.0% user, 0.0% nice, 3.5% system, 39.3% interrupt, 57.1% idle
Mem: 17M Active, 13M Inact, 716M Wired, 61G Free
ARC: 28M Total, 6105K MFU, 20M MRU, 16K Anon, 373K Header, 1494K Other
Swap: 16G Total, 16G Free
PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND
12 root -92 - 0K 1504K CPU22 22 3:38 87.35%
intr{irq293: ix0:que }
12 root -92 - 0K 1504K WAIT 26 3:41 86.57%
intr{irq295: ix0:que }
12 root -92 - 0K 1504K CPU18 18 3:41 86.47%
intr{irq291: ix0:que }
12 root -92 - 0K 1504K CPU28 28 3:41 86.47%
intr{irq296: ix0:que }
12 root -92 - 0K 1504K CPU24 24 3:39 86.18%
intr{irq294: ix0:que }
12 root -92 - 0K 1504K WAIT 4 3:35 86.18%
intr{irq284: ix0:que }
12 root -92 - 0K 1504K CPU8 8 3:35 86.18%
intr{irq286: ix0:que }
12 root -92 - 0K 1504K CPU6 6 3:33 85.60%
intr{irq285: ix0:que }
12 root -92 - 0K 1504K CPU20 20 3:32 85.50%
intr{irq292: ix0:que }
12 root -92 - 0K 1504K CPU10 10 3:32 84.86%
intr{irq287: ix0:que }
12 root -92 - 0K 1504K CPU2 2 3:32 84.67%
intr{irq283: ix0:que }
12 root -92 - 0K 1504K WAIT 14 3:31 84.38%
intr{irq289: ix0:que }
12 root -92 - 0K 1504K CPU16 16 3:11 77.59%
intr{irq290: ix0:que }
12 root -92 - 0K 1504K CPU0 0 2:42 64.70%
intr{irq282: ix0:que }
12 root -92 - 0K 1504K CPU12 12 2:44 63.67%
intr{irq288: ix0:que }
0 root -92 0 0K 4672K - 12 1:01 26.66%
kernel{ix0 que}
0 root -92 0 0K 4672K - 0 0:58 23.00%
kernel{ix0 que}
0 root -92 0 0K 4672K - 16 0:48 18.16%
kernel{ix0 que}
0 root -92 0 0K 4672K - 20 0:20 7.86%
kernel{ix0 que}
0 root -92 0 0K 4672K - 14 0:15 5.57%
kernel{ix0 que}
0 root -92 0 0K 4672K CPU6 6 0:14 5.57%
kernel{ix0 que}
0 root -92 0 0K 4672K - 22 0:12 4.59%
kernel{ix0 que}
0 root -92 0 0K 4672K - 18 0:06 4.59%
kernel{ix0 que}
0 root -92 0 0K 4672K - 28 0:06 4.49%
kernel{ix0 que}
0 root -92 0 0K 4672K - 26 0:06 4.39%
kernel{ix0 que}
0 root -92 0 0K 4672K - 24 0:05 4.05%
kernel{ix0 que}
0 root -92 0 0K 4672K - 8 0:09 3.47%
kernel{ix0 que}
0 root -92 0 0K 4672K - 4 0:08 3.17%
kernel{ix0 que}
0 root -92 0 0K 4672K - 10 0:07 2.98%
kernel{ix0 que}
0 root -92 0 0K 4672K - 2 0:05 2.39%
kernel{ix0 que}
# pmcstat -TS llc-misses -w10
PMC: [llc-misses] Samples: 3949 (100.0%) , 0 unresolved
%SAMP IMAGE FUNCTION CALLERS
28.6 kernel ixgbe_rxeof ixgbe_msix_que:25.9
ixgbe_handle_que:2.8
14.7 kernel bcmp netisr_dispatch_src:13.5
rtalloc_fib_nolock:1.0
5.9 kernel mb_ctor_mbuf uma_zalloc_arg
5.8 kernel ixgbe_mq_start vlan_transmit
4.6 kernel _mtx_trylock ixgbe_mq_start
3.8 kernel ether_nh_input netisr_dispatch_src
2.6 kernel ether_input ixgbe_msix_que
2.4 kernel m_tag_delete_chain m_tag_copy_chain:1.2 uma_zfree_arg:=
0.9
2.4 kernel netisr_dispatch_src ixgbe_rxeof:1.6 ixgbe_msix_que:0.6
2.2 kernel cpu_search_highest cpu_search_highest
2.2 kernel ixgbe_txeof ixgbe_msix_que
2.1 kernel uma_zfree_arg m_freem
2.0 kernel netisr_dispatch ixgbe_msix_que:1.1 ixgbe_rxeof:0.7
2.0 kernel critical_exit uma_zfree_arg
1.9 kernel _thread_lock_flags ithread_loop:1.2
intr_event_schedule_thread:0.6
1.9 kernel _bus_dmamap_sync ixgbe_rxeof
1.3 kernel ixgbe_xmit ixgbe_mq_start_locked
0.9 kernel ixgbe_msix_que intr_event_execute_handlers
--=20
WBR, Andrey V. Elsukov
--w6HrlTCB1wwmluQU9FwTkbOBVFpAuU9P2
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQEcBAEBAgAGBQJUWlesAAoJEAHF6gQQyKF6aJsH/2bQCyYPCS6dllqsj/o81eMX
814K6t14Q85xvfDuTlfjxqjvTBjxPWrbZK2SBQ1qWB0WZvBQeOnFWn4Whf0QQ46f
WRm2ft2YE7tTyBtdY5B9EltErH9MnbXqHGhGMQhrhTOlgmVo6dVVcGi2aXLmiUhm
L62dKMpifedM9Ecfm9aZpXZhn0pdVyqriJ+QTND2xeynqC24Lu871CbiHr5UCE8j
8KpKCnS7c3uLcNH4j/pCvMmDcNc/igi9tyVC2wjx4rWm/sZlK+dG74p2UfjZGKow
W+f8ITbpnE8OSIQ6OphlJFTdv4Hakc4AtCHrXLceqWixU4Q5i2+ntm/YhNjnJLs=
=Xifd
-----END PGP SIGNATURE-----
--w6HrlTCB1wwmluQU9FwTkbOBVFpAuU9P2--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?545A57A8.2010904>