Date: Fri, 23 Jun 95 18:03:26 MDT From: terry@cs.weber.edu (Terry Lambert) To: hsu@cs.hut.fi (Heikki Suonsivu) Cc: freebsd-hackers@freefall.cdrom.com Subject: Re: FreeBSD as a router Message-ID: <9506240003.AA07109@cs.weber.edu> In-Reply-To: <199506232158.AAA00252@shadows.cs.hut.fi> from "Heikki Suonsivu" at Jun 24, 95 00:58:03 am
next in thread | previous in thread | raw e-mail | index | archive | help
> I suspect there are bottlenecks everywhere. > > You can probably start with linear traversal of the routing table > and non-seperate reader/writer locks coupled with a lack of kernel > preemption. > > Its not CPU or bus performance problem, if there is no performance > difference between a 386-40 and a 486-66. And both route at 400kB/s, at > least with SMC 8013s onboard. Straight machine-machine throughput is > 800-900kB/s. I understood that SMC's can't have more than one outgoing > packet at a time which would be a good excuse, but how about better boards > on PCI bus which can have multiple packets going both ways without CPU > intervention? If you are hitting the theoretical max, you can't go faster. Otherwise, factors I'd consider important for routing would be: o Referencing the routing table, especially it if's large o Stack latency o Hardware latency. Rod's note about interrupting before the packet is fully revieved is well taken. o Concurrency. If you wait on a spin loop rather than a timer and supporting kernel preemption, you will lose on concurrency. o Bus transfer contention for multiple cards. o Buffer allocation and deallocation. Better boards, to make a real difference above two boards, which should be as concurrent (CSMA/CD means no bidirection use of the same wire; FDDI is another issue), would need to perhaps have the router (and therefore ISO loayers up through 4) on the board itself. In which case, your PC is a glorified power supply, which I don't see how that could save you more than $40, and assumes you have a dead and otherwise useless PC (ie: it's a way of getting good cash from bad on your PC "investment"). The latency looms exceedingly big as a factor in determining throughput since you can't really process a packet unless you are done with the previous one (unless you support kernel level premption and input ring buffering). Even this is less than useful without transmitter interrupt driven output (and again, kernel preemption). At which point it's *all* latency, card prebuffering, and pool rentention time issues. Primary hardware latency issues in this case are bus transfer and header preinterrupt to start interpreting packet contents as quickly as possible. PCI has a burst rate of 4 bytes * 33 MHz for the 32 bit version (the 64 bit version is not standardized yet; there is no such thing as a 64 bit PCI card), yielding burst rates of 132MB/s. Assuming a processing engine capable of handling this and minimal bus interaction overhead other than data transfers, and we are talking ~60MB/s or 4-6 100Mb/s cards for a box. That's just considering all other problems as "solved" and just considering PCI on it's own merit. The sustained rate goes down the more memory you have in the machine, of course, since bus-on time must be interleaved with DRAM refresh (one of the reasons that dedicated routers seldom use DRAM instead of SRAM for time critical buffering issues). The person to talk this over with is Garrett, probably, since he is well aware of the issues involved and is in fact the person who has made most of the tradeoff decisions, where applicable. He may not be interested in discussing this at all; the scope is rather large and has little immediate and tangible benefit. You also can't discount the inadequacies of the inherited code base to the task of simply packet-pumping. A lot of the 4.4 code simply was not designed with that environment in mind; also, as I have posted in the past, there are apparently some significant issues that have been introduced in the 4.4 code that weren't there in the 4.3 code (dig the University of Arizona paper reference out of the list archives for more info on this). All in all, Rod's evaluation is correct: BSD on a PC is not really up to the task of replacing heavily loaded router hardware; in point of fact, much of the existing router hardware sold by the companies in the business isn't either. Most of the US backbone is routed by dedicated RS/6000 boxes, which are one of the few pieces of hardware fast enough to break a T3 up into T1's (or put T1's back together). Some, but not all, of these issues can be overcome in software, potentially at a loss of usefulness as a general purpose OS. Terry Lambert terry@cs.weber.edu --- Any opinions in this posting are my own and not those of my present or previous employers.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?9506240003.AA07109>