Date: Fri, 28 Jan 2005 18:18:19 +0100 From: Thomas Vogt <freebsdlists@bsdunix.ch> To: freebsd-performance@freebsd.org Cc: freebsd-net@freebsd.org Subject: freebsd router project. Problems with polling? Message-ID: <1106932700.48903.58.camel@bert.mlan.solnet.ch>
next in thread | raw e-mail | index | archive | help
Hello Overview: ATM, I've a 4.10 router (xeon 2.4ghz UP, dual on board em gigE interfaces) in a productive enviroment. The routing software is quagga. It's doing quite well. It has an average of 200k pakets per seconds with the full routing table. The System has polling enabled and interrupt is almost idle and cpu too. New one: Now I'm building a FreeBSD 5.3 or -Stable based router system on CF cards too. Intel whitepaper shows a max pps for the em cards at 1.4mpps. I doubt that I can reach more than ~600k pps with the fully routing table loaded! But this would be ok great. My system: Xeon 2.4Ghz UP, 2x onboard em GigE interface, 5.3-RELEASE-p4 with if_em.c,v 1.44.2.4. my configs: sysctl.conf: vm.swap_enabled=0 kern.ipc.somaxconn=1024 kern.polling.enable=1 kern.random.sys.harvest.ethernet=0 kern.random.sys.harvest.interrupt=0 net.inet.udp.recvspace=65536 net.inet.tcp.sendspace=65536 net.inet.tcp.delayed_ack=0 net.inet.tcp.msl=5000 net.inet.ip.maxfragpackets=40 net.inet.ip.maxfragsperpacket=4 net.inet.ip.fw.one_pass=0 net.inet.ip.fw.dyn_max=8192 net.inet.ip.fw.dyn_udp_lifetime=15 net.inet.ip.fastforwarding=1 loader.conf: kern.ipc.nmbclusters=32768 Kernel config: http://www.bsdunix.ch/public/ROUTER5 Setup: I've 3 similar machines. I use my own udpsend.c and udprec.c to send packages and to count the pakages. It can send up to 300-330k pps (udp size 64 bytes) ATM, the full routing table is not loaded. It's a very basic setup. The goal is to find the maximum pps throughput for the router with small pakets. But atm I've problems with device polling. Graphic: ------------------ |10.0.1.2 udp send | ------------------ | | -------em0------ |freebsd router | -------em1------ | | ----------------------- | 192.168.1.2 udp recv | ------------------------ netstat -w 1 (polling disabled) input (Total) output packets errs bytes packets errs bytes colls 300531 0 23441438 300585 0 23444792 0 300939 0 25872898 300895 0 25870184 0 300738 0 23457584 300768 0 23460626 0 300304 10 25826466 300304 0 25826876 0 ... Interrupt load is about 70% with net.inet.ip.fastforwarding enabled. If I disable this the system becomes unusable. The system hasn't reach the limit yet. But the interrupt is much to high. It's not worthy to add a second udp sender machine, at the moment. netstat -w 1 (polling enabled) input (Total) output packets errs bytes packets errs bytes colls 150151 47647 12910330 150150 0 12911806 0 150151 0 11711798 150152 0 11711876 0 150151 47665 12910986 150151 0 12911810 0 150151 0 11711798 150151 0 11711798 0 ... Interrupt load is about 10%. CPU is about 60% and with kern.polling.idle_poll enabled it goes to 100% (as expected). As you see the speed is droping down to 50% with polling enable and and I got a lot of errors. kern.polling.lost_polls: 188748 and kern.polling.suspect: 186919 are also very high. I don't know why polling is so bad on this machine. All SMP option are disabled in the kernel and bios. I tried to do as much as in http://lists.freebsd.org/pipermail/freebsd-questions/2004-November/064427.html described. I will prepare others tests with -STABLE and -CURRENT in the next few days. At the mean time, are they some other magic things config option I can try? Perhaps increase the HZ to 2000 in the kernel or remove polling and try smp machine? I doubt that I can run the machine without polling. If you see 70% interrupt load with 300k pps without polling. regards Thomas Vogt
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1106932700.48903.58.camel>