Date: Mon, 3 Feb 2020 14:39:03 -0800 From: Navdeep Parhar <np@FreeBSD.org> To: Slawa Olhovchenkov <slw@zxy.spb.ru> Cc: freebsd-net@freebsd.org Subject: Re: Chelsio NETMAP performance Message-ID: <6868f207-d054-3d45-b60d-eaf7115760c1@FreeBSD.org> In-Reply-To: <20200203222321.GB8012@zxy.spb.ru> References: <20200203201728.GC8028@zxy.spb.ru> <863de9e1-42cc-6f3a-5c1f-1bf737714c9f@FreeBSD.org> <20200203222321.GB8012@zxy.spb.ru>
next in thread | previous in thread | raw e-mail | index | archive | help
On 2/3/20 2:23 PM, Slawa Olhovchenkov wrote: > On Mon, Feb 03, 2020 at 01:39:52PM -0800, Navdeep Parhar wrote: >=20 >> On 2/3/20 12:17 PM, Slawa Olhovchenkov wrote: >>> I am try to use Chelsio T540-CR in netmap mode and see poor (compared= >>> to Intel 82599ES) performance. >> >> What approximate FreeBSD version is this? >=20 > 12.1-STABLE >=20 >>> >>> Same application ac receive only about 8.9Mpss, compared to 12.5Mpps >>> at Intel. >>> >>> pmc profile show mostly time spend in: >>> >>> 49.76% [17802] service_nm_rxq @ /boot/kernel/if_cxgbe.ko >>> 100.0% [17802] t4_vi_intr >>> 100.0% [17802] ithread_loop @ /boot/kernel/kernel >>> 100.0% [17802] fork_exit >>> >>> >>> to be exact at line >>> >>> while ((d->rsp.u.type_gen & F_RSPD_GEN) =3D=3D nm_rxq->iq_gen= ) { >>> >>> Is this maximum limit for this vendor? >> >> No, a T540 should be able to sink full 10Gbps (14.88Mpps) on a single = rx >> queue. Try adding this to your loader.conf: >> >> hw.cxgbe.toecaps_allowed=3D"0" >> >> Then try simple netmap "pkt-gen -f rx" instead of any custom app and s= ee >> how many pps it's able to sink. >=20 > Thanks! `hw.cxgbe.toecaps_allowed=3D"0"` allow recive 14Mpps for may > application too! >=20 > Now I am got only 10% less performance compared to Intel, as I see by > higher Chelsio interrupt cpu time (top show about 30% for every > interrupt handler). Is this normal? Is this posible to optimize? Try changing the interrupt holdoff timer for the netmap rx queues. This shows the list of timers available (in microseconds): # sysctl dev.t5nex.0.holdoff_timers nm_holdoff_tmr_idx is a 0-based index into the list above. So if the tmr idx is 0 you are using the 0th (first) value from the list of timers. Try increasing nm_holdoff_tmr_idx and see if that brings down the interrupt rate under control. # sysctl hw.cxgbe.nm_holdoff_tmr_idx=3D3/4/5 Regards, Navdeep
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?6868f207-d054-3d45-b60d-eaf7115760c1>