Date: Mon, 28 Jul 2014 10:30:48 -0700 From: Adrian Chadd <adrian@freebsd.org> To: John Jasen <jjasen@gmail.com> Cc: FreeBSD Net <freebsd-net@freebsd.org> Subject: Re: fastforward/routing: a 3 million packet-per-second system? Message-ID: <CAJ-Vmo=Ma2_D%2BAya6jCmrSJC9u5QruVuzd9E2c4qvWVnuV-wkA@mail.gmail.com> In-Reply-To: <CAACLuR09etiyujR7-qOGSUQpQFH1o6S38_96_uSd9Cy57OnYYQ@mail.gmail.com> References: <53CE80DD.9090109@gmail.com> <CAACLuR1r0axCYWXeLDSa-m07eAVgTMBVW5sNbt%2By_Lt2ss1r7Q@mail.gmail.com> <CAJ-Vmonsc79ULDOT9trtOotq7mRh1XJkhL2JfDNxXP16OFWMFg@mail.gmail.com> <CAJ-Vmok6vyrD-2%2BswCBPP_VeMqH6t7CXuVBPUaDfwyRrb6aiTg@mail.gmail.com> <53D4600A.1010505@gmail.com> <CAJ-Vmonf_krdxrkrKbjkC-a=4x0M7y9mXeTGMvs1qOSGNbqdGA@mail.gmail.com> <53D4F77B.9020009@gmail.com> <CAJ-VmonMYi-7ZCw-PqiuOp5euAdxbyiHXW-ZW96MWQ8%2B-RV8MA@mail.gmail.com> <CAACLuR09etiyujR7-qOGSUQpQFH1o6S38_96_uSd9Cy57OnYYQ@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 28 July 2014 07:51, John Jasen <jjasen@gmail.com> wrote: > in_input crept up into the top 5, versus fastforward. > > > Would PMC counters help? Not at the moment. This is a lock contention thing, not a pmc thing. I bet if you ran pmc the mutex/rwlock things would be up high. :) > > cat debug.lock.pref.stats.out-20140728-1 | sort -nk 4 | tail -5 > 5 4 413 115 160 2 0 0 > 63 /usr/src/sys/kern/kern_condvar.c:145 (sleep mutex:Giant) > 1 1 148858 4095 650072 0 0 0 > 11184 /usr/src/sys/kern/subr_turnstile.c:552 (spin mutex:turnstile chain) > 8 14 13747639 561636 72520256 0 0 0 > 689603 /usr/src/sys/net/route.c:439 (sleep mutex:rtentry) > 3 20 3907071 2322975 72520256 0 0 0 > 2529589 /usr/src/sys/netinet/ip_input.c:1315 (sleep mutex:rtentry) > 3 17 3665247 3715117 72520256 0 0 0 > 8425384 /usr/src/sys/netinet/in_rmx.c:114 (sleep mutex:rtentry) Try disabling net.inet.ip.redirect (sysctl net.inet.ip.redirect=0). That'll eliminate that in_rmx.c check. Oh look! The ip_output() path doesn't know about flowtable either. I'm kind of surprised that the 2-tuple flowtable (ie, only ipv4 and only ipv6 addresses, not TCP/UDP ports) isn't used in the ip forwarding case. All ip_rtaddr() is doing is doing a route lookup and taking a reference to the ifa. It's using that for things like network/netmask on that interface. Anyway - yeah, it looks like you've hit lock contention on that particular setup. You'll likely get a little more throughout out by disabling redirects for now. The real solution is to make the whole rtentry locking less stupid and bottleneck-y as well as extending the flowtable support to include the ip_forward() path. -a
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAJ-Vmo=Ma2_D%2BAya6jCmrSJC9u5QruVuzd9E2c4qvWVnuV-wkA>