Date: Fri, 18 Oct 2002 12:17:33 -0700 (PDT) From: Kelly Yancey <kbyanc@posi.net> To: Luigi Rizzo <rizzo@icir.org> Cc: Petri Helenius <pete@he.iki.fi>, Lars Eggert <larse@ISI.EDU>, <freebsd-net@FreeBSD.ORG> Subject: Re: ENOBUFS Message-ID: <20021018114829.Y1611-100000@gateway.posi.net> In-Reply-To: <20021018112652.A83405@carp.icir.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Fri, 18 Oct 2002, Luigi Rizzo wrote:
> Oh, I *thought* the numbers you reported were pps but now i see that
> nowhere you mentioned that.
>
Sorry. I just checked with our tester. Those are the total number of
packets sent during the test. Each test lasted 10 seconds, so divde by 10 to
get pps.
> But if things are as you say, i am seriously puzzled on what you
> are trying to measure -- the output interface (fxp) is a 100Mbit/s
> card which cannot possibly support the load you are trying to offer
> to saturate the input link.
>
We don't want to saturate the input link, only saturate the outbound link
(100Mps). Oddly enough, the em card cannot do this with any packets less than
333 bytes and drops ~50% of the packets. But clearly this isn't a bottlenext
issue because the drop-off isn't smooth. 332 byte backs cause ~50% packet
loss; 333 byte packets cause 0% packet loss.
> You should definitely clarify how fast the smartbits unit is pushing
> out traffic, and whether its speed depends on the measured RTT.
>
It doesn't sound like the box is that smart. As it was explained to me, the
test setup includes a desired 'load' to put on the wire: it is measured as a
percentage of the wire speed. Since our SmartBit unit only supports
100base-T and doesn't understand vlans, we have to use 7 separate outbound
ports, each configured for 14.25% load. To the GigE interface, this should
appear as 99.75 megabits of data (including all headers/framing).
> It might well be that what you are
> seeing is saturation of ipintrq, which happens because of some
> strange timing issue -- nothing to do with the board.
>
I don't understand why it would only happen with the em card and not with
the bge under the exact same traffic (or even more demanding traffic, i.e.
64byte frames). Also, wouldn't packet gradually subside as we approached the
333 byte magic limit rather than the sudden drop-off we are seeing?
> In any case, at least in my experience, a 1GHz box with two em
> cards can easily forward between 350 and 400kpps (64-byte frames) with a
> 4.6-ish kernel, and a 2.4GHz box goes above 650kpps.
>
We expect our kernel to be slower than that (we typically see ~120kpps for
64-byte frames using the bge driver and a 5701-based card) because we are
using an fxp card for outbound traffic and have added additional code to the
ip_input() processing. The point isn't absolute numbers, though, but trying
to figure out why when using the em driver (and only with the em driver!) we
see ~50% packet loss with packets smaller than 333 bytes (no matter what size,
just that it is smaller). That is, 64 byte frames: ~50% packet loss; 332 byte
frames: ~50% packet loss; 333 byte frames: 0% packet loss. That sort of
sudden drop doesn't look like a bottleneck to me.
We've mostly written the em driver off because of this. The bge driver
works just fine performance wise; it was the sporadic watchdog timeouts
that led us to investigate the Intel cards to begin with. I only mentioned it
on-list because earlier Jim McGrath alluded to similar performance issues with
the Intel GigE cards and small frames.
Actually, at this point, I'm hoping that your polling patches for the em
driver workaround whatever problem is causing the packet loss and am eagerly
awaiting them to be committed. :)
Thanks,
Kelly
>
> On Fri, Oct 18, 2002 at 11:13:54AM -0700, Kelly Yancey wrote:
> > On Fri, 18 Oct 2002, Luigi Rizzo wrote:
> >
> > > How is the measurement done, does the box under test act as a router
> > > with the smartbit pushing traffic in and expecting it back ?
> > >
> > The box has 2 interfaces, a fxp and a em (or bge). The GigE interface is
> > configured with 7 VLANs. THe SmartBit produces X byte UDP datagrams that go
> > through a Foundry ServerIron switch for VLAN tagging and then to the GigE
> > interface (where they are untagged). The box is acting as a router and all
> > traffic is directed out the fxp interface where it returns to the SmartBit.
> >
> > > The numbers are strange, anyways.
> > >
> > > A frame of N bytes takes (N*8+160) nanoseconds on the wire, which
> > > for 330-byte frames should amount to 1000000/(330*8+160) ~= 357kpps,
> > > not the 249 or so you are seeing. Looks as if the times were 40% off.
> > >
> >
> > Yeah, I've never made to much sense of the actual numbers myself. Our
> > resident SmartBit expert runs the tests and provides me with the results. I
> > use them more for getting an idea of the relative performance of one
> > configuration over another and not as absolute numbers themselves. I'll check
> > with our resident expert and see if he can explain how it calculates those
> > numbers. The point being, though, that there is an undeniable drop-off with
> > 332 byte or smaller packets. We have never seen any such drop-off using the
> > bge driver.
> >
> > Thanks,
> >
> > Kelly
> >
> > > cheers
> > > luigi
> > >
> > > On Fri, Oct 18, 2002 at 10:45:13AM -0700, Kelly Yancey wrote:
> > > ...
> > > > > can push out over 400kpps (64byte frames) on a 2.4GHz box.
> > > > >
> > > > > luigi
> > > > >
> > > >
> > > > Using a SmartBit to push traffic across a 1.8Ghz P4; 82543 chipset card
> > > > plugged into PCI-X bus:
> > > >
> > > > FrameSize TxFrames RxFrames LostFrames Lost (%)
> > > > 330 249984 129518 120466 48.19
> > > > 331 249144 127726 121418 48.73
> > > > 332 248472 140817 107655 43.33
> > > > 333 247800 247800 0 0
> > > >
> > > > It has no trouble handling frames 333 bytes or larger. But for any frame
> > > > 332 bytes or smaller we consistently see ~50% packet loss. This same machine
> > > > easily pushes ~100Mps with the very same frame sizes using a bge card rather
> > > > than em.
> > > >
> > > > I've gotten the same results with both em driver version 1.3.14 and 1.3.15
> > > > on both FreeBSD 4.5 and 4.7 (all 4 combinations, that is).
> > > >
> > > > Kelly
> > > >
> > > > --
> > > > Kelly Yancey -- kbyanc@{posi.net,FreeBSD.org}
> > > > FreeBSD, The Power To Serve: http://www.freebsd.org/
> > > >
> > > >
> > > > To Unsubscribe: send mail to majordomo@FreeBSD.org
> > > > with "unsubscribe freebsd-net" in the body of the message
> > >
> > > To Unsubscribe: send mail to majordomo@FreeBSD.org
> > > with "unsubscribe freebsd-net" in the body of the message
> > >
> >
> > --
> > Kelly Yancey -- kbyanc@{posi.net,FreeBSD.org}
> > Join distributed.net Team FreeBSD: http://www.posi.net/freebsd/Team-FreeBSD/
> >
> >
> > To Unsubscribe: send mail to majordomo@FreeBSD.org
> > with "unsubscribe freebsd-net" in the body of the message
>
> To Unsubscribe: send mail to majordomo@FreeBSD.org
> with "unsubscribe freebsd-net" in the body of the message
>
--
Kelly Yancey -- kbyanc@{posi.net,FreeBSD.org}
Join distributed.net Team FreeBSD: http://www.posi.net/freebsd/Team-FreeBSD/
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-net" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20021018114829.Y1611-100000>