Date: Fri, 25 Jul 2014 12:20:38 -0400 From: John Jasen <jjasen@gmail.com> To: FreeBSD Net <freebsd-net@freebsd.org> Subject: Re: fastforward/routing: a 3 million packet-per-second system? Message-ID: <CAACLuR1r0axCYWXeLDSa-m07eAVgTMBVW5sNbt%2By_Lt2ss1r7Q@mail.gmail.com> In-Reply-To: <53CE80DD.9090109@gmail.com> References: <53CE80DD.9090109@gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Based on advice I received, I've collected lock profile debugging output,
and pmcannotate'd data from the system while it was processing about 3
million packets/second.
Combined, the files are about 325k in size, so I'll submit highlights here.
I can provide the raw files to interested parties privately.
pmcannotate summary output:
grep ^Profile pmcannotate.20140725
Profile trace for function: __rw_rlock() [17.04%]
Profile trace for function: __mtx_unlock_flags() [9.10%]
Profile trace for function: _rw_runlock_cookie() [7.67%]
Profile trace for function: sched_idletd() [5.73%]
Profile trace for function: memcpy() [5.64%]
Profile trace for function: bcopy() [5.04%]
Profile trace for function: bcmp() [5.01%]
Profile trace for function: __mtx_lock_flags() [3.66%]
Profile trace for function: t4_eth_tx() [3.25%]
Profile trace for function: lock_profile_release_lock() [2.73%]
Profile trace for function: ip_fastforward() [2.68%]
Profile trace for function: ether_output() [2.50%]
Profile trace for function: get_scatter_segment() [1.75%]
Profile trace for function: rn_match() [1.74%]
Profile trace for function: _mtx_lock_spin_cookie() [1.53%]
Profile trace for function: lock_profile_obtain_lock_success() [1.49%]
Profile trace for function: cxgbe_transmit() [1.37%]
Profile trace for function: uma_zalloc_arg() [1.31%]
Profile trace for function: bzero() [1.30%]
Profile trace for function: service_iq() [1.26%]
Profile trace for function: ether_nh_input() [1.23%]
Profile trace for function: __mtx_lock_sleep() [1.19%]
Profile trace for function: arpresolve() [1.07%]
Profile trace for function: uma_zfree_arg() [0.95%]
Profile trace for function: reclaim_tx_descs() [0.87%]
Profile trace for function: _mtx_trylock_flags_() [0.80%]
Profile trace for function: bounce_bus_dmamap_load_buffer() [0.72%]
Profile trace for function: ether_demux() [0.64%]
Profile trace for function: mb_ctor_mbuf() [0.63%]
Profile trace for function: rtalloc1_fib() [0.54%]
sysctl debug.lock.prof.stats summary: (some of the highest hit counts,
especially in cnt_hold:
7 17 3271389 2258698 63952832 0 0 0
6761237 /usr/src/sys/netinet/in_rmx.c:114 (sleep mutex:rtentry)
7 13 11543138 470545 63952832 0 0 0
815777 /usr/src/sys/net/route.c:439 (sleep mutex:rtentry)
6 15 3943582 1545195 63952779 0 0 0
3439254 /usr/src/sys/netinet/ip_fastfwd.c:593 (sleep mutex:rtentry
On Tue, Jul 22, 2014 at 11:18 AM, John Jasen <jjasen@gmail.com> wrote:
> Feedback and/or tips and tricks more than welcome.
>
<snip>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAACLuR1r0axCYWXeLDSa-m07eAVgTMBVW5sNbt%2By_Lt2ss1r7Q>