Date: Sun, 11 Nov 2001 23:45:55 -0800 From: Luigi Rizzo <rizzo@aciri.org> To: Martin Karsten <Martin.Karsten@KOM.tu-darmstadt.de> Cc: net@FreeBSD.ORG Subject: Re: NEW CODE: polling support for device drivers. Message-ID: <20011111234555.B38855@iguana.aciri.org> In-Reply-To: <200111091005.fA9A5Vo09707@KOM.tu-darmstadt.de> References: <20011027005905.A72758@iguana.aciri.org> <200111091005.fA9A5Vo09707@KOM.tu-darmstadt.de>
next in thread | previous in thread | raw e-mail | index | archive | help
On Fri, Nov 09, 2001 at 11:05:31AM +0100, Martin Karsten wrote:
> Thanks, this is fantastic! On all FreeBSD 4.x versions, the performance of
> end systems receiving large amounts of small packets used to be much worse
> than on FreeBSD 3.4. I'm not a driver expert, but as you described, the
> vanilla systems seem to spend too much time in the interrupt context, such
> that both the IP interrupt and the UDP socket queues are not served fast
> enough. Enabling your polling code reduces this kind of loss quite
> impressively and by tuning the net.xorp.poll_burst_max parameter, it seems
> possible to further avoid end system losses. This is quite important to me
> for certain experiments.
>
> Would you mind explaining the other sysctl xorp parameters?
most of them are for statistical purposes, and furthermore I am
revising some of them to make them simpler to use and (to some
degree) autoconfiguring.
For the time being, take the code I posted just as a proof of
concept, a cleaned up version should come out soon.
In that version you will mostly have knobs to enable/disable polling,
and set the max number of packets you will grab from the interface
at each clock tick (this is currently poll_burst_max) and whenever
a process issues a system call. As i complete some other part of
the work, you might have an additional parameter which tells you
what is the maximum fraction of the CPU reserved to process
"interrupts" (or the equivalent in terms of polling).
cheers
luigi
> Thanks again,
> Martin
>
> > Maybe this can be of some interest to some of you using BSD boxes
> > as routers, and who are concerned with performance and robustness
> > to attacks. I would be grateful if some of you could have a look
> > at this and possibly provide feedback.
> >
> > The patches attached here (for 4.4, i386 non-SMP) implement
> > polling-mode operation for (a couple at the moment) network device
> > drivers.
> >
> > This mode of operation gives huge performance and stability benefits
> > under heavy load, at the price of a very modest (in many cases not
> > noticeable) increase in latency.
> >
> > On our test box (Pentium 750 MHz, 21143 with "dc" driver), a stock
> > FreeBSD 4.4 was barely able to forward a single stream of 130kpps
> > through a 100Mbit interfac (well ok, that is not too bad!), but
> > the system was very unresponsive, and livelock as soon as you start
> > a second stream on a second interface.
> >
> > With polling enabled, the same box could forward 180Kpps and be
> > completely responsive even when bombed with 4 full speed 100Mbit
> > stream (about 590Kpps). Your mileage might vary, depending on the
> > number of PCI buses, card type, etc.
> >
> > Note, this is really a minimal set of diffs, providing the basic
> > mechanism for polling in the kernel, and very basic patches for a
> > couple of device driver ("dc" and "fxp"). Next in the works is a
> > system that limits to a programmable value the fraction of CPU time
> > spent in a driver context -- right now the tuning is semi-manual.
> >
> > Instruction follow.
> >
> > cheers
> > luigi
> >
> > ----------------------------------+-----------------------------------------
> > Luigi RIZZO, luigi@iet.unipi.it . ACIRI/ICSI (on leave from Univ. di Pisa)
> > http://www.iet.unipi.it/~luigi/ . 1947 Center St, Berkeley CA 94704
> > Phone: (510) 666 2927
> > ----------------------------------+-----------------------------------------
> >
> > INSTALLATION AND USE
> >
> > + make sure you have a device supported by the "dc" or "fxp" driver
> > (the latter is not tested 100%, but should work)
> >
> > + (cd /sys ; patch < polling.patches )
> > + add the following two lines to the kernel config:
> > options HZ=1000
> > options XORP_ETHER_POLLING
> > + build and install the kernel
> > + at runtime use the following sysctl variables to control polling:
> >
> > sysctl net.xorp.polling=1 # 1 to enable polling, 0 to disable
> > sysctl net.xorp.poll_burst_min=10 # controls cpu allocation
> >
> > CPU permitting, the system will forward a minimum of (
> > poll_burst_min*HZ ) packets per second per interface, and more
> > if there are any CPU cycles available.
> >
> > This parameter is only important if you expect to have CPU
> > intensive tasks running on the router box -- otherwise, you can
> > leave it set to a low value (5...20), and the system will use
> > all the spare CPU cycles for forwarding.
> >
> > If you expect the system to be loaded, check the max forwarding
> > speed setting the parameter to a low value when the system is
> > unloaded, and then set the parameter to a suitable value to
> > "reserve" the desired amount of CPU to forwarding.
> >
> > The additional latency introduced by polling can be as large as
> > 1/HZ seconds, which is why i suggest using HZ=1000 (corresponding
> > to 1ms additional latency in the worst case).
> >
> > Just for reference, the files touched by this diff are:
> >
> > sys/conf/options.i386
> > option definition
> >
> > sys/i386/include/asnames.h
> > sys/net/if.h
> > sys/net/netisr.h
> > sys/sys/systm.h
> > constants, variable and prototypes (mostly one-line changes)
> >
> > sys/i386/i386/swtch.s
> > sys/i386/i386/trap.c
> > calls to ether_poll from the idle loop and traps
> >
> > sys/kern/kern_clock.c
> > main polling routines
> >
> > sys/dev/fxp/if_fxp.c
> > sys/pci/if_dc.c
> > sys/pci/if_dcreg.h
> > device driver changes
> >
> > Supporting polling for other devices should be reasonably simple,
> > following the example of the other two drivers.
> > ---------------------------------------------------------------------
>
> [Attachment, skipping...]
>
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-net" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20011111234555.B38855>