Date: Tue, 9 Mar 2010 15:00:39 -0800 From: "David Christensen" <davidch@broadcom.com> To: "Ryan Stone" <rysto32@gmail.com> Cc: "pyunyh@gmail.com" <pyunyh@gmail.com>, Ian FREISLICH <ianf@clue.co.za>, "current@freebsd.org" <current@freebsd.org> Subject: RE: dev.bce.X.com_no_buffers increasing and packet loss Message-ID: <5D267A3F22FD854F8F48B3D2B52381933AF90EEE0E@IRVEXCHCCR01.corp.ad.broadcom.com> In-Reply-To: <bc2d971003091430g236806edy4cf3bb873665fc5@mail.gmail.com> References: <20100305210435.GF14818@michelle.cdnetworks.com> <20100305175639.GB14818@michelle.cdnetworks.com> <E1NnVaT-0003Ft-3p@clue.co.za> <E1Nnc4d-0003mB-6e@clue.co.za> <E1Nne0Q-0003uZ-OR@clue.co.za> <E1Noulp-0007Rc-Ro@clue.co.za> <20100309212139.GO1311@michelle.cdnetworks.com> <5D267A3F22FD854F8F48B3D2B52381933AF90EED69@IRVEXCHCCR01.corp.ad.broadcom.com> <20100309214012.GQ1311@michelle.cdnetworks.com> <5D267A3F22FD854F8F48B3D2B52381933AF90EEDA7@IRVEXCHCCR01.corp.ad.broadcom.com> <bc2d971003091430g236806edy4cf3bb873665fc5@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
=20 > -----Original Message----- > From: Ryan Stone [mailto:rysto32@gmail.com]=20 > Sent: Tuesday, March 09, 2010 2:31 PM > To: David Christensen > Cc: pyunyh@gmail.com; Ian FREISLICH; current@freebsd.org > Subject: Re: dev.bce.X.com_no_buffers increasing and packet loss >=20 > > What's the traffic look like? =A0Jumbo, standard, short frames? =A0Any= =20 > > good ideas on profiling the code? =A0I haven't figured out how to use=20 > > the CPU TSC but there is a free running timer on the device=20 > that might=20 > > be usable to calculate where the driver's time is spent. > > > > Dave >=20 > In my experience hwpmc is the best and easiest way to profile=20 > anything on FreeBSD. Here's something I sent to a different=20 > thread a couple of months ago explaining how to use it: >=20 > 1) If device hwpmc is not compiled into your kernel, kldload=20 > hwpmc(you will need the HWPMC_HOOKS option in either case) > 2) Run pmcstat to begin taking samples(make sure that=20 > whatever you are profiling is busy doing work first!): >=20 > pmcstat -S unhalted-cycles -O /tmp/samples.out >=20 > The -S option specifies what event you want to use to trigger=20 > sampling. The unhalted-cycles is the best event to use if=20 > 3) After you've run pmcstat for "long enough"(a proper=20 > definition of long enough requires a statistician, which I=20 > most certainly am not, but I find that for a busy system 10=20 > seconds is enough), Control-C it to stop it*. You can use=20 > pmcstat to post-process the samples into human-readable text: >=20 > pmcstat -R /tmp/samples.out -G /tmp/graph.txt >=20 > The graph.txt file will show leaf functions on the left and=20 > their callers beneath them, indented to reflect the=20 > callchain. It's not too easy to describe and I don't have=20 > sample output available right now. Below is a quick sample I obtained running netperf. We're=20 interested in the bce(4) driver so I assume I'm interested in the time spent in bce and the functions it calls. Looks to me like memory allocation/freeing is a major source of CPU cycles in this test. Am I reading this right? @ CPU_CLK_UNHALTED_CORE [1091924 samples] 49.25% [537739] sched_idletd @ /boot/kernel/kernel 100.0% [537739] fork_exit 20.89% [228070] trash_dtor @ /boot/kernel/kernel 85.45% [194883] mb_dtor_clust 100.0% [194883] uma_zfree_arg 100.0% [194883] mb_free_ext 14.55% [33186] mb_dtor_mbuf 100.0% [33186] uma_zfree_arg 84.27% [27966] mb_free_ext 15.73% [5220] m_freem 00.00% [1] mb_dtor_pack 100.0% [1] uma_zfree_arg 100.0% [1] mb_free_ext 02.34% [25542] bce_intr @ /boot/kernel/if_bce.ko 100.0% [25542] intr_event_execute_handlers @ /boot/kernel/kernel 100.0% [25542] ithread_loop 100.0% [25542] fork_exit 02.20% [24055] trash_ctor @ /boot/kernel/kernel 96.41% [23192] mb_ctor_clust 100.0% [23192] uma_zalloc_arg 100.0% [23192] bce_fill_rx_chain @ /boot/kernel/if_bce.ko 03.39% [815] mb_ctor_mbuf @ /boot/kernel/kernel 100.0% [815] uma_zalloc_arg 99.39% [810] bce_fill_rx_chain @ /boot/kernel/if_bce.ko 00.49% [4] m_copym @ /boot/kernel/kernel 00.12% [1] tcp_output 00.20% [48] uma_zalloc_arg 100.0% [48] bce_fill_rx_chain @ /boot/kernel/if_bce.ko 100.0% [48] bce_intr Dave=
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?5D267A3F22FD854F8F48B3D2B52381933AF90EEE0E>