Date: Fri, 3 Mar 2006 16:26:16 +0200 From: Artis Caune <Artis.Caune@latnet.lv> To: freebsd-net@freebsd.org Subject: dequeueing outgoing packets on gigabit speed (if_em) Message-ID: <84FA2AE9-087B-4CCF-A884-D74232BC0936@latnet.lv>
next in thread | raw e-mail | index | archive | help
Hello freebsd-net!
We are trying to build freebsd traffic rate limiter, which will
work in bridge mode. It must forward frames as fast as possible.
When packet is received and it is not over rate limit, it is
forwarded to another interface.
The easiest way is to replace interface if_input function pointer
with our function, which check packet for errors (like ether_input)
and if rate for given IP is not over, forward it to another
interface:
IFQ_ENQUEUE(&ifp->if_snd, m, err);
...
if ((ifp->if_drv_flags & IFF_DRV_OACTIVE) == 0)
(*ifp->if_start)(ifp);
...
kernel module source:
http://freebsd.latnet.lv/bsdcar/
Traffic is going from cisco monitoring port to em0
currently it's:
30 second output rate 627852000 bits/sec, 169622 packets/sec
and not changing much. It's customers traffic, no generated with
some packet generator.
It's okay with incomming traffic, but there are lot of drops with
outgoing traffic. em0 can receive 800Mbits/s; 400Kpps and more.
about the box:
FreeBSD 7.0-CURRENT #0: Fri Feb 24 12:57:50 EET 2006
device = '82546EB Dual Port Gigabit Ethernet Controller' (pcix)
VM_KMEM_SIZE_MAX=536870912 (512M)
kern.ipc.nmbclusters=100000
dual 2.8GHz xeon
2G RAM
changes in if_em.h:
#define EM_DEFAULT_TXD EM_MAX_TXD //(4096)
#define EM_DEFAULT_RXD EM_MAX_RXD //(4096)
#define EM_TIDV 0
#define EM_TADV 0
#define EM_RADV 0
I made 3 test.
In every test I did:
# kldunload bsdcar; kldunload if_em
# kldload if_em; kldload bsdcar; bsdcar -W em1 -H em0
# sleep 300; kldunload bsdcar;
# netstat -nibd
# sysctl dev.em.1.debug_info=1
In second test I changed if_snd qlen from 4095->65536,
in third: from 4095->131072
In third test there was no dropped output packets, but all clusters were
exhausted so input packets got dropped.
If I set clusters very high (200000), I get "kmem_map too small" panic.
Is it bad to set ifq_maxlen very high? (delay?)
How can we speed up dequeueing of outgoing packets?
--- test #1 ---
# netstat -w1 -bdhI em1
input (em1) output
packets errs bytes packets errs bytes colls drops
0 0 0 134K 0 61M 0 26K
0 0 0 134K 0 61M 0 25K
0 0 0 138K 0 61M 0 21K
0 0 0 136K 0 60M 0 28K
1 0 419 140K 0 61M 0 20K
0 0 0 135K 0 61M 0 25K
0 0 0 135K 0 61M 0 24K
0 0 0 134K 0 60M 0 30K
0 0 0 137K 0 60M 0 29K
0 0 0 136K 0 60M 0 30K
# netstat -nibd
Name Mtu Ipkts Ierrs Ibytes Opkts Oerrs Obytes Coll
Drop
em0 1500 49301683 0 974718376 8 0 3352
0 0
em1 1500 8 0 3352 41258403 0 1622430098 0
8038451
# sysctl dev.em.1.debug_info=1
em1: Adapter hardware address = 0xc84e9128
em1: CTRL = 0x8300249 RCTL = 0x8002
em1: Packet buffer = Tx=16k Rx=48k
em1: Flow control watermarks high = 47104 low = 45604
em1: tx_int_delay = 0, tx_abs_int_delay = 0
em1: rx_int_delay = 0, rx_abs_int_delay = 0
em1: fifo workaround = 0, fifo_reset_count = 0
em1: hw tdh = 107, hw tdt = 107
em1: Num Tx descriptors avail = 4096
em1: Tx Descriptors not avail1 = 2285698
em1: Tx Descriptors not avail2 = 0
em1: Std mbuf failed = 0
em1: Std mbuf cluster failed = 0
em1: Driver dropped packets = 0
--- test #2 --- (18% packet loss)
BSD CAR: changing em1 ifq_maxlen from 4095 to 65536
# netstat -w1 -bdhI em1
input (em1) output
packets errs bytes packets errs bytes colls drops
0 0 0 136K 0 60M 0 26K
0 0 0 134K 0 61M 0 27K
0 0 0 134K 0 61M 0 25K
0 0 0 133K 0 61M 0 28K
0 0 0 132K 0 60M 0 28K
0 0 0 134K 0 61M 0 26K
0 0 0 135K 0 60M 0 28K
0 0 0 134K 0 61M 0 29K
0 0 0 136K 0 60M 0 27K
0 0 0 137K 0 62M 0 23K
# netstat -nibd
Name Mtu Ipkts Ierrs Ibytes Opkts Oerrs Obytes Coll
Drop
em0 1500 48879911 0 1158747782 8 0 3352
0 0
em1 1500 8 0 3352 40342501 0 1543561097 0
8471870
# sysctl dev.em.1.debug_info=1
em1: Adapter hardware address = 0xc8505928
em1: CTRL = 0x8300249 RCTL = 0x8002
em1: Packet buffer = Tx=16k Rx=48k
em1: Flow control watermarks high = 47104 low = 45604
em1: tx_int_delay = 0, tx_abs_int_delay = 0
em1: rx_int_delay = 0, rx_abs_int_delay = 0
em1: fifo workaround = 0, fifo_reset_count = 0
em1: hw tdh = 3695, hw tdt = 3695
em1: Num Tx descriptors avail = 4096
em1: Tx Descriptors not avail1 = 285906
em1: Tx Descriptors not avail2 = 0
em1: Std mbuf failed = 0
em1: Std mbuf cluster failed = 0
em1: Driver dropped packets = 0
--- test #3 ---
BSD CAR: changing em1 ifq_maxlen from 4095 to 131072
# netstat -w1 -bdhI em1
input (em1) output
packets errs bytes packets errs bytes colls drops
0 0 0 143K 0 61M 0 0
0 0 0 141K 0 62M 0 0
0 0 0 142K 0 62M 0 0
0 0 0 140K 0 61M 0 0
0 0 0 142K 0 62M 0 0
0 0 0 142K 0 62M 0 0
0 0 0 143K 0 62M 0 0
0 0 0 142K 0 62M 0 0
0 0 0 141K 0 62M 0 0
0 0 0 142K 0 61M 0 0
# netstat -nibd
Name Mtu Ipkts Ierrs Ibytes Opkts Oerrs Obytes
Coll Drop
em0 1500 43284041 17188917 1870968566 8 0
3352 0 0
em1 1500 8 0 3352 43197781 0
1869131369 0 0
# sysctl dev.em.1.debug_info=1
em1: Adapter hardware address = 0xc64c0128
em1: CTRL = 0x8300249 RCTL = 0x8002
em1: Packet buffer = Tx=16k Rx=48k
em1: Flow control watermarks high = 47104 low = 45604
em1: tx_int_delay = 0, tx_abs_int_delay = 0
em1: rx_int_delay = 0, rx_abs_int_delay = 0
em1: fifo workaround = 0, fifo_reset_count = 0
em1: hw tdh = 1497, hw tdt = 1497
em1: Num Tx descriptors avail = 4096
em1: Tx Descriptors not avail1 = 2310650
em1: Tx Descriptors not avail2 = 0
em1: Std mbuf failed = 0
em1: Std mbuf cluster failed = 0
em1: Driver dropped packets = 0
# netstat -m
12290/88225/100515 mbufs in use (current/cache/total)
12288/87712/100000/100000 mbuf clusters in use (current/cache/total/max)
12288/87707 mbuf+clusters out of packet secondary zone in use
(current/cache)
0/0/0/0 4k (page size) jumbo clusters in use (current/cache/total/max)
0/0/0/0 9k jumbo clusters in use (current/cache/total/max)
0/0/0/0 16k jumbo clusters in use (current/cache/total/max)
27648K/197480K/225128K bytes allocated to network (current/cache/total)
0/21520839/10760419 requests for mbufs denied (mbufs/clusters/mbuf
+clusters)
0/0/0 requests for jumbo clusters denied (4k/9k/16k)
0/5/6656 sfbufs in use (current/peak/max)
0 requests for sfbufs denied
0 requests for sfbufs delayed
0 requests for I/O initiated by sendfile
0 calls to protocol drain routines
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?84FA2AE9-087B-4CCF-A884-D74232BC0936>