Date: Thu, 31 Jul 1997 19:54:22 -0400 (EDT) From: Andrew Gallatin <gallatin@CS.Duke.EDU> To: Chris Csanady <ccsanady@friley01.res.iastate.edu> Cc: freebsd-current@FreeBSD.ORG Subject: Re: Myrinet, etc.. (Re: code talks: announcing EIDE bus master patches) Message-ID: <199707312354.TAA29288@hurricane.cs.duke.edu> In-Reply-To: <199707311845.NAA14531@friley01.res.iastate.edu> References: <199707310317.MAA25355@genesis.atrad.adelaide.edu.au> <199707311845.NAA14531@friley01.res.iastate.edu>
next in thread | previous in thread | raw e-mail | index | archive | help
Chris Csanady writes: > > >Stefan Esser stands accused of saying: > >> > (*) PCI device == a Myrinet M2F-PCI32 card. This is a programmable > >> > gigabit networking card. It has a 256k bank of SRAM on the card, and > >> > is very good for doing things like measuring PCI b/w. The tests were > >> > done from user space operating on mmap'ed device memory & a kernel > >> > allocated chunk of RAM to do DMA xfers to/from. It also runs IP > >> > traffic at better than 300Mb/sec. > >> > > I'd also like to point out that the 300Mb's is acheived using an IP > stack layered on top of some active messaging protocol. (which is > implemented on the io processor on the nic.) > > As far as the TCP/IP stack under FreeBSD, all you can push through it > is about 150Mb/s. This is somewhat unfortunate, although surprisingly > linux doesn't seem to manage much better. I think it would be quite > nice if we could correctly implement a zero copy architecture.. > > Chris Csanady Actually, that's not true. I'm using an in-kernel IP driver layered on top of Myricom's MyriApi general-purpose messaging software (which is kernel resident). It hooks into the network stack the way any ethernet driver would (and gets hit with a copy for each xmit from or receive into user space like anything else does). There is software that's downloaded onto the interface card, but it has no protocol specific knowledge. When its sitting under the IP driver, it is used the same way the firwmware on any ethernet card would be used -- the driver tells it a MAC address that it wants something sent to, and it sends it, etc. The fact that we're running over Myrinet & not ethernet is transparent to any application. In this configuration, with the hardware I currently have available (memory b/w challenged Pentium Pros) I can receive UDP traffic at about 356 Mb/sec and send UDP traffic at a rate of a about 280Mb/sec. TCP streams are around 275Mb/sec. I think my transmission speed is limited both by the memory b/w limitations of Pentium Pro's (from the uiomove copy to/from user space) & the fact that the myri MTU is more than twice the 4k hw pagesize of the Intel platform, so I'm incurring a lot of dma setup overhead on transmits. (I manage my own set of MTU-sized receive buffers, so there's only 1 dma required for each receive). I've just noticed that -current appears to support having MCLBYTES > page size; this should improve transmit speed some. Please don't get me wrong: I agree that it really would be nice to have a zero-copy IP stack. Drew PS: For more performance numbers check out the netperf pages (http://www.cup.hp.com/netperf/numbers/NetperfBrowse.html)..I'm happy to report that FreeBSD w/Myrinet is #2 (behind a pair of 10 & 16 processor SGI Power Challenges connected via HIPPI), and/or our group's pages (http://www.cs.duke.edu/ari/manic/ip). ------------------------------------------------------------------------------ Andrew Gallatin http://www.cs.duke.edu/~gallatin Duke University Email: gallatin@cs.duke.edu Department of Computer Science Phone: (919) 660-6590
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?199707312354.TAA29288>