Date: Fri, 9 Nov 2001 11:05:31 +0100 (CET) From: Martin Karsten <Martin.Karsten@KOM.tu-darmstadt.de> To: Luigi Rizzo <rizzo@aciri.org> Cc: net@FreeBSD.ORG Subject: Re: NEW CODE: polling support for device drivers. Message-ID: <200111091005.fA9A5Vo09707@KOM.tu-darmstadt.de> In-Reply-To: <20011027005905.A72758@iguana.aciri.org> from Luigi Rizzo at "Oct 27, 2001 00:59:05 am"
next in thread | previous in thread | raw e-mail | index | archive | help
Thanks, this is fantastic! On all FreeBSD 4.x versions, the performance of
end systems receiving large amounts of small packets used to be much worse
than on FreeBSD 3.4. I'm not a driver expert, but as you described, the
vanilla systems seem to spend too much time in the interrupt context, such
that both the IP interrupt and the UDP socket queues are not served fast
enough. Enabling your polling code reduces this kind of loss quite
impressively and by tuning the net.xorp.poll_burst_max parameter, it seems
possible to further avoid end system losses. This is quite important to me
for certain experiments.
Would you mind explaining the other sysctl xorp parameters?
Thanks again,
Martin
> Maybe this can be of some interest to some of you using BSD boxes
> as routers, and who are concerned with performance and robustness
> to attacks. I would be grateful if some of you could have a look
> at this and possibly provide feedback.
>
> The patches attached here (for 4.4, i386 non-SMP) implement
> polling-mode operation for (a couple at the moment) network device
> drivers.
>
> This mode of operation gives huge performance and stability benefits
> under heavy load, at the price of a very modest (in many cases not
> noticeable) increase in latency.
>
> On our test box (Pentium 750 MHz, 21143 with "dc" driver), a stock
> FreeBSD 4.4 was barely able to forward a single stream of 130kpps
> through a 100Mbit interfac (well ok, that is not too bad!), but
> the system was very unresponsive, and livelock as soon as you start
> a second stream on a second interface.
>
> With polling enabled, the same box could forward 180Kpps and be
> completely responsive even when bombed with 4 full speed 100Mbit
> stream (about 590Kpps). Your mileage might vary, depending on the
> number of PCI buses, card type, etc.
>
> Note, this is really a minimal set of diffs, providing the basic
> mechanism for polling in the kernel, and very basic patches for a
> couple of device driver ("dc" and "fxp"). Next in the works is a
> system that limits to a programmable value the fraction of CPU time
> spent in a driver context -- right now the tuning is semi-manual.
>
> Instruction follow.
>
> cheers
> luigi
>
> ----------------------------------+-----------------------------------------
> Luigi RIZZO, luigi@iet.unipi.it . ACIRI/ICSI (on leave from Univ. di Pisa)
> http://www.iet.unipi.it/~luigi/ . 1947 Center St, Berkeley CA 94704
> Phone: (510) 666 2927
> ----------------------------------+-----------------------------------------
>
> INSTALLATION AND USE
>
> + make sure you have a device supported by the "dc" or "fxp" driver
> (the latter is not tested 100%, but should work)
>
> + (cd /sys ; patch < polling.patches )
> + add the following two lines to the kernel config:
> options HZ=1000
> options XORP_ETHER_POLLING
> + build and install the kernel
> + at runtime use the following sysctl variables to control polling:
>
> sysctl net.xorp.polling=1 # 1 to enable polling, 0 to disable
> sysctl net.xorp.poll_burst_min=10 # controls cpu allocation
>
> CPU permitting, the system will forward a minimum of (
> poll_burst_min*HZ ) packets per second per interface, and more
> if there are any CPU cycles available.
>
> This parameter is only important if you expect to have CPU
> intensive tasks running on the router box -- otherwise, you can
> leave it set to a low value (5...20), and the system will use
> all the spare CPU cycles for forwarding.
>
> If you expect the system to be loaded, check the max forwarding
> speed setting the parameter to a low value when the system is
> unloaded, and then set the parameter to a suitable value to
> "reserve" the desired amount of CPU to forwarding.
>
> The additional latency introduced by polling can be as large as
> 1/HZ seconds, which is why i suggest using HZ=1000 (corresponding
> to 1ms additional latency in the worst case).
>
> Just for reference, the files touched by this diff are:
>
> sys/conf/options.i386
> option definition
>
> sys/i386/include/asnames.h
> sys/net/if.h
> sys/net/netisr.h
> sys/sys/systm.h
> constants, variable and prototypes (mostly one-line changes)
>
> sys/i386/i386/swtch.s
> sys/i386/i386/trap.c
> calls to ether_poll from the idle loop and traps
>
> sys/kern/kern_clock.c
> main polling routines
>
> sys/dev/fxp/if_fxp.c
> sys/pci/if_dc.c
> sys/pci/if_dcreg.h
> device driver changes
>
> Supporting polling for other devices should be reasonably simple,
> following the example of the other two drivers.
> ---------------------------------------------------------------------
[Attachment, skipping...]
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-net" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200111091005.fA9A5Vo09707>