Date: Sun, 14 Dec 2003 03:10:42 -0800 From: Luigi Rizzo <rizzo@icir.org> To: Eugene Grosbein <eugen@www.svzserv.kemerovo.su> Cc: net@freebsd.org Subject: Re: how to saturate 100Mbit Message-ID: <20031214031042.A99005@xorpc.icir.org> In-Reply-To: <20031214042812.GC2249@svzserv.kemerovo.su>; from eugen@www.svzserv.kemerovo.su on Sun, Dec 14, 2003 at 11:29:07AM %2B0700 References: <FE045D4D9F7AED4CBFF1B3B813C85337035E418B@mail.sandvine.com> <20031213101706.A79791@xorpc.icir.org> <20031214042812.GC2249@svzserv.kemerovo.su>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sun, Dec 14, 2003 at 11:29:07AM +0700, Eugene Grosbein wrote: > On Sat, Dec 13, 2003 at 10:17:06AM -0800, Luigi Rizzo wrote: > > > the fxp has a problem which does not allow it to go above 103/110/120kpps > > depending on which descriptor model you use, no matter how fast > > the CPU is. > > Can you explain the problem, please? Sorry if i sound a bit vague but did these tests a couple of years ago, and i do not have access to the fxp specs or contacts with people who designed the chip so i cannot say what is exactly the problem. The problem is in the hardware, not in the driver. Apparently the chip wastes a lot of time (couple of microseconds if i remember well) between reading the descriptor and then transmitting the data. This extra delay is in parallel with some other chip activity, so for larger frames (say 256 bytes) it is completely masked and packets are transmitted at the nominal rate, but smaller packet are sent as if the interframe gap were larger than the standard. At the time i did a lot of experiments to make sure that the NIC was not stalled by the cpu not supplying frames in time, etc., and all tests confirmed the above diagnosis. BTW it is not something that has to do with flow control frames, because enabling/disabling them does not change the throughput (and you can see the frames when they are not disabled). Changing the way descriptor are used (there are two different formats, one has something like the data right after the descriptor) helped increase the performance to 120kpps, whereas the standard freebsd driver is stuck at something between 103 and 110kpps. Not that it matters terribly, just useful to know when you have to push small frames at line rate and you find you can't with those cards. There are others (e.g. intel/dec 2114x, 3com's xl, possibly others, plus probably all of the gbit units when used at 100Mbit) which work fine at line rate with 64-byte frames. > > Even not using any special kernel modules, a simple loop over > > a sendto() on a udp socket can achieve around 500kpps on a 2.4GHz > > box (em or bge). With some tricks and a sufficiently fast PCI > > bus you can reach some 750kpps but then it really depends > > on how fast is your PCI bus. A > > 100*1024*1024/8/1500=8738.1(3) > > It seems one does not need hundred of thousand pps to achive 100Mbps. certainly not, but because most of the time the overhead is mostly per packet, if you want to run performance tests you care about packets, not data rate. cheers luigi
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20031214031042.A99005>