Date: Tue, 8 Oct 2002 22:46:49 -0600 From: "Kenneth D. Merry" <ken@kdm.org> To: Christopher Smith <csmith@its.uq.edu.au> Cc: hardware@FreeBSD.ORG, net@FreeBSD.ORG Subject: Re: High interrupt load on firewalls Message-ID: <20021008224649.A39689@panzer.kdm.org> In-Reply-To: <20021008224313.A39509@panzer.kdm.org>; from ken@kdm.org on Tue, Oct 08, 2002 at 10:43:13PM -0600 References: <B9C9E292.30E56%csmith@its.uq.edu.au> <20021008224313.A39509@panzer.kdm.org>
next in thread | previous in thread | raw e-mail | index | archive | help
[ taking -questions out of the CC list, please don't send things to more
than 2 lists, the mail servers don't usually allow it in any case. ]
On Wed, Oct 09, 2002 at 13:41:38 +1000, Christopher Smith wrote:
> We have two firewalls sitting on gigabit links. Each has 2 Netgear GA620
> (ti driver) fibre cards with about 7 vlans spread across them. Both these
> machines run at *very* high interrupt loads (95 - 100% during business hours
> (mostly 100%), 80 - 90 % during off hours). They are 1GHz P3 machines (Dell
> 1550s) with 256MB of RAM. They're actually dual machines, but enabling the
> second CPU doesn't help in terms of load, it just halves the numbers top
> reports.
>
> Obviously, these machines process a lot of traffic. However, the interrupt
> load seems to me to be very, very high and the main reason we are seeing
> such high rates of packet loss (up to 10%, constantly) through these
> machines - is there any way it can be lessened, either with a better driver,
> different network cards, or some other way ? We are currently testing with
> a dual 2.4GHz P4 (Dell 2650) using the same network cards, and are peaking
> at around 40% (really 80%). However, that doesn't seem to leave much room
> to grow, and it's a very expensive way to ease the load.
The Tigon II boards have a number of parameters you can tweak to change the
intererupt coalescing parameters.
It may be that you can tweak the parameters and decrease your load
somewhat, but it will require some experimentation.
In -stable, you'll have to recompile your kernel with the new values. In
-current, there's an ioctl interface, and I can give you a program (I think
I still have it) that lets you tweak the parameters on the fly.
The parameters you want to tweak are in ti_attach() (src/sys/pci/if_ti.c):
/* Set default tuneable values. */
sc->ti_stat_ticks = 2 * TI_TICKS_PER_SEC;
sc->ti_rx_coal_ticks = TI_TICKS_PER_SEC / 5000;
sc->ti_tx_coal_ticks = TI_TICKS_PER_SEC / 500;
sc->ti_rx_max_coal_bds = 64;
sc->ti_tx_max_coal_bds = 128;
sc->ti_tx_buf_ratio = 21;
ti_stat_ticks is the card statistics update interval. I wouldn't recommend
bothering with it.
ti_{rx,tx}_coal_ticks is the number of clock ticks (on the card) that have
to go by before you get interrupted for a send or receive.
ti_{rx,tx}_coal_bds is the number of buffers that have to accumulate before
you get interrupted for a send or receive.
The time and number of buffer limits are a logical or operation. i.e. when
the timeout or the buffer threshold is reached, an interrupt is generated.
The ti_tx_buf_ratio variable controls the ratio of space allocated to send
buffers on the card versus receive buffers. It is in 1/64th increments.
So with the default setting of 21, 21/64 of the available buffer space,
or 32.8%, is allocated to transmit buffers. The remaining space is
allocated to receive buffers.
My suggestion is to increase the number of ticks, and the number of buffers
coalesced some, and see if you can decrease your interrupt load.
I assume you're using 1500 byte packets. If so, beware that the Tigon II
boards aren't as efficient with 1500 byte packets as some other boards.
They're great with jumbo frames, but with 1500 byte packets you'll probably
be pretty hard pressed to get really high throughput.
> Will FreeBSD 5.0 be able to spread the interrupts across both CPUs ? Is
> this high interrupt load a problem with the driver, the hardware, FreeBSD
> itself, or is it something that is normal ?
In theory it will, assuming the locks are pushed down on the network stack.
At the moment I think you'll find the performance will likely be worse than
-stable.
> What hardware are other people using to firewall high-volume gigabit links ?
Most of my work has been with jumbo frames, and I have a couple of GA620T
boards. This isn't firewall work though, but rather geared towards maximum
bandwidth.
You might want to try out some of the Intel gigabit boards. At least we've
got an engineer from Intel who maintains the driver.
I haven't tried them out, though, so I can't comment on the boards or the
driver. Other folks seem to have good things to say about them, though.
Ken
--
Kenneth Merry
ken@kdm.org
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-net" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20021008224649.A39689>