Date: Wed, 12 Mar 2008 10:33:04 -0700 From: Wade Klaver <wadeklaver@itiva.com> To: freebsd-hackers@freebsd.org Cc: freebsd-ipfw@freebsd.org Subject: On the trail of a dummynet/bridge/ipfw bug. Message-ID: <1205343184.4032.44.camel@wade-linux.itiva.com>
next in thread | raw e-mail | index | archive | help
--=-L/S4LNxbzLpvKH3MrrIG
Content-Type: text/plain
Content-Transfer-Encoding: quoted-printable
PROBLEM DESCRIPTION
I have a bridge set up on a 7.0 box and am attempting to use it to limit
HTTP connections outgoing from a box behind it to 192Kbit/s for testing.
During this testing I ran into some problems. At first, I found that
the number of simultaneous pipes was limited to 1024, allowing only 1024
192Kbit/s clients. Additional clients were simply blocked. I am using
a very simple firewall config:
ipfw pipe 1 config bw 192Kbits/s mask all
ipfw add 00051 skipto 99 ip from 192.168.0.0/16 to 192.168.0.0/16
ipfw add 00052 skipto 1000 ip from any to any
ipfw add 00100 pipe 1 ip from 192.168.10.88 80 to any via bridge0
ipfw add 00200 pipe 1 ip from any 25111 to any via bridge
Regardless of how many clients I threw at the box, I had the limit:
[root@ibm3550b ~]# ipfw pipe show | wc -l
1028
We managed to track this down to a problem in the ipfw2 userland app.
The following patch to /usr/src/sbin/ipfw/ipfw2.c allowed this limit to
be surpassed. It would appear that ipfw does not dynamically resize the
pipe array beyond the initial 1024 elements allocated.
# diff ipfw2.c ipfw2.c.orig=20
2507c2507
< int nalloc =3D 8192; /* start somewhere... */
---
> int nalloc =3D 1024; /* start somewhere... */
However, this just revealed a bigger problem, potentially do to the
above patch, potentially due to something worse. Now the bridge will
allow more connections, up to around 2300 where the bridge just dies.
and no more traffic passes. It is worth noting that I can still connect
to the bridge itself if it has an IP assigned to it, but traffic through
the bridge ceases. It is also remedied by a /etc/rc.d/netif restart.
ADDITIONAL NOTES
Please let me know if there is any additional information I can provide.
In the kernel options below, HZ=3D2000 was just something I was trying.
The problem exhibits itself with HZ=3D1000 as well.
I posted this to -hackers and to -ipfw. Please direct me and future corresp=
ondence=20
on this issue to the most appropriate list. I just felt it was not
solid enough to go to -bugs yet.
SYSTEM INFORMATION
IBM 3550
XEON 5345
4GB Memory
[root@ibm3550b /usr/src/sys]# uname -a
FreeBSD ibm3550b.itivalabs.net 7.0-STABLE FreeBSD 7.0-STABLE #13: Wed
Mar 12 03:26:08 PDT 2008
root@ibm3550b.itivalabs.net:/usr/obj/usr/src/sys/WADE amd64
Bridge members:
bce0: <Broadcom NetXtreme II BCM5708 1000Base-T (B2)> mem
0xc8000000-0xc9ffffff irq 18 at device 0.0 on pci4
bce1: <Broadcom NetXtreme II BCM5708 1000Base-T (B2)> mem
0xce000000-0xcfffffff irq 16 at device 0.0 on pci6
Kernel options:
# Make an SMP-capable kernel by default
options SMP # Symmetric MultiProcessor Kernel
options LIBALIAS
options IPFIREWALL #firewall
options IPFIREWALL_VERBOSE #enable logging to syslogd(8)
options IPFIREWALL_VERBOSE_LIMIT=3D100 #limit verbosity
options IPFIREWALL_DEFAULT_TO_ACCEPT #allow everything by
default
options IPFIREWALL_FORWARD #packet destination changes
options IPFIREWALL_NAT #ipfw kernel nat support
options IPDIVERT #divert sockets
#options IPFILTER #ipfilter support
#options IPFILTER_LOG #ipfilter logging
#options IPFILTER_LOOKUP #ipfilter pools
#options IPFILTER_DEFAULT_BLOCK #block all packets by default
options IPSTEALTH #support for stealth forwarding
options MBUF_STRESS_TEST
options DUMMYNET
options HZ=3D2000
options EXT2FS
/etc/sysctl.conf
net.inet.ip.fw.dyn_max=3D8192
net.inet.ip.dummynet.hash_size=3D4096
net.inet.ip.fw.dyn_buckets=3D1024
net.inet.ip.dummynet.max_chain_len=3D64
kern.ipc.nmbclusters=3D32768
--=-L/S4LNxbzLpvKH3MrrIG
Content-Type: application/pgp-signature; name=signature.asc
Content-Description: This is a digitally signed message part
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
iD8DBQBH2BPKne3UhGESRwURAkMVAKCVjfXweg6+gXn1c9kAM2v07o4+JgCfXZga
Ebsj07bJ4wcROQhzdqVcBOQ=
=B200
-----END PGP SIGNATURE-----
--=-L/S4LNxbzLpvKH3MrrIG--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1205343184.4032.44.camel>