From owner-freebsd-atm Sun Jan 3 14:50:50 1999 Return-Path: Received: (from majordom@localhost) by hub.freebsd.org (8.8.8/8.8.8) id OAA17677 for freebsd-atm-outgoing; Sun, 3 Jan 1999 14:50:50 -0800 (PST) (envelope-from owner-freebsd-atm@FreeBSD.ORG) Received: from marcos.networkcs.com (marcos.networkcs.com [137.66.16.1]) by hub.freebsd.org (8.8.8/8.8.8) with ESMTP id OAA17672 for ; Sun, 3 Jan 1999 14:50:47 -0800 (PST) (envelope-from salo@us.networkcs.com) Received: from us.networkcs.com (us.networkcs.com [137.66.11.15]) by marcos.networkcs.com (8.9.0.Beta5/8.9.0.Beta5) with ESMTP id QAA28751 for ; Sun, 3 Jan 1999 16:50:19 -0600 (CST) Received: (from salo@localhost) by us.networkcs.com (8.8.7/8.8.7) id QAA29347; Sun, 3 Jan 1999 16:50:17 -0600 (CST) Date: Sun, 3 Jan 1999 16:50:17 -0600 (CST) From: Tim Salo Message-Id: <199901032250.QAA29347@us.networkcs.com> To: atm@FreeBSD.ORG Subject: Re: ATM in FreeBSD, status requested! Cc: johnc@networkcs.com, jpt@networkcs.com, mks@networkcs.com Sender: owner-freebsd-atm@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org I don't know what the future role of Chuck Cranor's ATM software ought to be in FreeBSD, but we would like to continue to make our HARP software (http://www.msci.magic.net/harp) more useful to researchers, (to the extent that funding is available, of course). Thank you everyone for your thoughts on features that might be added to HARP. I have a few questions about possible QoS enhancements for HARP. o Would support for ALTQ in HARP meet all of the identified needs for QoS support over ATM? If not, what else should we consider? o Is there need for RSVP over ATM support, as described in RFCs 2379, 2380, 2381 and 2382? o Is there a need for a user-level, native ATM API that allows QoS parameters to be set? (HARP 3, the version in FreeBSD 3.0, supports the XNS Issue 5 ATM Transport Protocol Information for Sockets interface, but the QoS calls don't do anything. Note that this API is a non-IP API; that is, it allows applications to communicate with HARP directly, rather than through the operating system's IP stack.) - Is there need for a user-level, native ATM API that can set rate shaping parameters for data transmitted out through an ATM interface? - Is there need for a user-level, native ATM API that can set QoS parameters for outgoing SVCs? o Is there need to be able to set administratively (i.e., hand configure) rate shaping parameters for data transmitted through the operating system's IP stack out through an ATM interface? o Is there need to be able to set administratively (i.e., hand configure) QoS parameters for outgoing SVCs that are used by the operating system's IP stack? o Is there need to be able to set dynamically (e.g., via RSVP or some similar mechanism) rate shaping parameters for data transmitted through the operating system's IP stack out through an ATM interface? o Is there need to be able to set dynamically (e.g., via RSVP or some similar mechanism) QoS parameters for outgoing SVCs that will carry IP traffic that is transmitted through the operating system's IP stack out through an ATM interface? o Is anything [known at this time to be] required for IPv6 QoS support beyond ALTQ? o What other QoS facilities should we consider for HARP? Thanks in advance, -tjs To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-atm" in the body of the message From owner-freebsd-atm Fri Jan 8 12:56:30 1999 Return-Path: Received: (from majordom@localhost) by hub.freebsd.org (8.8.8/8.8.8) id MAA23972 for freebsd-atm-outgoing; Fri, 8 Jan 1999 12:56:30 -0800 (PST) (envelope-from owner-freebsd-atm@FreeBSD.ORG) Received: from mail-blue.research.att.com (mail-blue.research.att.com [135.207.30.102]) by hub.freebsd.org (8.8.8/8.8.8) with ESMTP id MAA23949; Fri, 8 Jan 1999 12:56:09 -0800 (PST) (envelope-from chuck@research.att.com) Received: from corona.research.att.com (corona.research.att.com [135.207.26.21]) by mail-blue.research.att.com (Postfix) with ESMTP id 6466C4CE09; Fri, 8 Jan 1999 15:55:38 -0500 (EST) Received: (from chuck@localhost) by corona.research.att.com (980427.SGI.8.8.8/8.8.7) id PAA81839; Fri, 8 Jan 1999 15:55:37 -0500 (EST) Message-ID: <19990108155537.A1179348@research.att.com> Date: Fri, 8 Jan 1999 15:55:37 -0500 From: Chuck Cranor To: freebsd-atm@FreeBSD.ORG Cc: Poul-Henning Kamp , Tim Salo , "John D. DeHart" , Zubin Dittia , guru@arl.wustl.edu Subject: Re: ATM in FreeBSD, status requested! Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Mailer: Mutt 0.93.2i Organization: AT&T Labs-Research Sender: owner-freebsd-atm@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org greetings- i've received several copies of messages from this list regarding the status of my BSDATM code WRT FreeBSD over the holidays. i have not had time to reply until now due to the holidays and the context switch overhead of starting a new job (I'm now at AT&T Labs-Research and am still trying to get my world back in order). from reading the archive of this list it seems pretty clear to me that there are some misconceptions about the goals of the BSDATM project that has lead to some unfair and inaccurate criticism of it. this is partly my fault for not providing better documentation for what i was trying to do with BSDATM. over the past couple of days i've spent some time looking at the HARP code in -current and re-reading the HARP web page to get a better handle on the goals of HARP. i'd like to now clarify the goals of the BSDATM project and how it fits in with respect to HARP. Goals of the BSDATM project: at the time i started BSDATM there were no freely available ATM device drivers for BSD, the only thing out there was Werner Almesberger's stuff for Linux. this was a problem for us (wustl.edu) because we were using BSD as a research platform, but we were unable to do anything with ATM under BSD. i decided to address this problem by creating BSDATM. my goals in creating BSDATM were: [1] to write a device driver for the Efficient Networks midway-based ATM cards that: - is robust and not likely to crash your system. - is portable across BSD operating systems and ENI card types (e.g. both SBus and PCI). - is well commented and easy to debug (even via e-mail). - provides reasonable performance with minimal data copying. - supports the midway's bandwidth settings (QoS) [2] to write an upper-level ATM networking framework that has as small a footprint on the BSD kernel and operating system as possible while still meeting our research needs. our needs include: - being able to transmit and receive plain AAL0 cells and AAL5 frames between our host systems and our ATM-based multimedia devices. for example, we have a relatively simple video source/sink box called an "MMX" that communicates with our BSD-based video server using AAL0 netnatm sockets. the AAL0 connection used for transmitting data from the video server to the MMX requires the use of the ENI's bandwidth control because the MMX box has limited buffering and no way to throttle the sender. the AAL0 connection used for recording real-time video data from an MMX puts a heavy interrupt load on the receiving system (interrupt per cell) and is a good stress test for the ATM device driver. another example is Jon Turner's next generation of ATM switch which is being distributed as part of the "Gigabit Network Technology Distribution" sponsored by the NSF and wustl. the new switch's control processor will be a PC running BSD. the PC communicates with the switch by sending and receiving control data over a netnatm socket. - being able to transmit and receive IP packets over PVCs without clashing with VCs currently in use by netnatm sockets (and vice-versa). - being able to take advantage of the hardware demux feature of the ATM host interface (netnatm does this). again, note that the goals of BSDATM are to produce a robust lower-level device layer capable of supporting all of midway's features combined with a lean (minimal footprint) upper-level ATM layer. thus, supporting SVCs, complex ATM signaling protocols, etc... was never a goal of BSDATM and is not included in BSDATM's higher-level ATM networking framework. i chose to do it this way because i was working on BSDATM alone in my spare time (when not working on my doctoral research) and i did not have the time to do the standards research necessary to adequately support those upper-layer features in BSDATM. instead, i focused my efforts on the lower levels of ATM support with the idea that future work in the upper layers could build on my lower-level work and either replace or build on my lean upper-layer ATM framework. so despite the assertions made on this list that BSDATM is a poorly written and poorly supported piece of work i believe that BSDATM has been quite successful in reaching the goals set for it. i have had great success in the area of getting other people to use and/or build on it (both in the research area and in the ISP business). for example, in addition to the ATM projects at wustl.edu, Kenjiro Cho has done the ALTQ work and helped maintain the FreeBSD version of the driver. I worked with Anne Hutton and the folks at ISI/CARIN vie e-mail to add support for the Adaptec version of the midway card to the driver and get the bandwidth limiting stuff going for them. [The Adaptec version of the card has a better DMA interface than the older ENI version.] The driver has been ported to all the free BSD systems and also to other operating systems such as Mach. The number of pending bug reports relating to the handling the midway device is zero --- both Kenjiro and I have worked hard to eliminate the final kinks from the driver and we have not had any problems with it for many months. In addition to my midway work, I have been working with some folks who want to add support for ENI's Lanai-based 25Mbps ATM card to FreeBSD, NetBSD, and OpenBSD. for that project it appears that we will need to eventually get PPP over ATM working. Goals of HARP: from looking at the HARP web page (http://www.msci.magic.net/harp/) it should be pretty clear to everyone that the HARP folks' major focus is on supporting the standard protocols at the upper layers of the ATM framework and the bulk of their work lies in that area rather than in lower-level driver support (where BSDATM's focus lies). i believe that in some sense the HARP work and the BSDATM work complement each other rather than conflict. looking at the code size footprint of the the projects confirms this: HARP: total lines of code = ~80000 (see http://www.msci.magic.net/harp/usenix/ppframe.htm, the "lessons learned I" slide) the HARP midway driver is only ~6% of those ~80000 lines of code BSDATM: total lines of code = ~5700 the midway driver is about 70% of BSDATM I spent some time looking at the HARP version of the midway driver and comparing it to my driver. some points of interest include: - the HARP driver uses structures to access midway device registers, the BSDATM midway driver uses the more portable bus_space macros with offsets to access the card. i believe FreeBSD is moving in the bus_space/bus_dma direction (according to conversations i've had with justin gibbs). - the HARP driver does not support Adaptec versions of the card. - the HARP driver does not support sbus versions of the card and may not handle byte ordering conflicts properly. - the HARP driver does not test the DMA engine at boot time to verify proper operation. - the HARP driver does not dynamically determine what DMA burst sizes are valid on a specific system and does not handle the strict DMA alignment required on some systems (e.g. the "alburst" restriction common on sparcs). Instead the HARP driver hardwares the max DMA burst to 8 words and doesn't make use of 16 word DMA bursts. - the HARP driver doesn't support setting bandwidth limits on VCs and thus only uses the channel 0 transmit buffer (BSDATM supports multiple transmission channels). - the HARP driver doesn't appear to check for bogus AAL5 pdu length values in the AAL5 frame. - the HARP driver requires the allocation of an mbuf even if the received frame is in error (e.g. CRC error). the "receiver to lock" if the mbuf allocation fails. - the HARP driver clears the "VCI_IN_SERVICE" bit without advancing its servread pointer in error conditions. strictly speaking, i don't believe this should be allowed. - the HARP driver may not handle DMA descriptor wraparound properly (Kenjiro detected that the empty and full conditions for the DMA descriptors are the same and thus you've got to be careful.) The transmit buffer in HARP is handled properly. - the HARP driver doesn't have a VC drain state to drain off DMA and I/O operations when a VC is being closed and thus may prematurely free resources (often happens with our multimedia ATM devices). - the HARP driver handles unaligned data by doing an in-place bcopy (KM_COPY)... it appears to be re-writing a cluster mbuf's data area (M_EXT). i believe that is illegal (especially when the data area is referenced by more than one mbuf header). - BSDATM driver doesn't have code to read the seeprom on the ENI card (the upper layer ATM code doesn't require it) - HARP dynamically allocates receive buffers on the card, but the allocations are of a fixed size. BSDATM allocates fixed sized buffers on the card at boot time (doesn't have a memory allocator). - HARP has code to talk to the SUNI chip (PHY chip) to read the counters and clear SUNI interrupts, BSDATM doesn't. - both BSDATM and HARP use a header in front of the data to pass data around (in BSDATM it is the atm_pseudohdr structure, in HARP it is un-named). - BSDATM caches allocated mbufs if an "out of DMA descriptor" error condition occurs so that they don't have to be reallocated later. - BSDATM has a DDB-based debugging interface that is useful for debugging other people's problems via e-mail. that's a partial list of differences i came up with after spending a day or two looking at code. my impression is that there is a significant amount of duplicate work to be done at the driver level to get HARP's midway support up to speed with BSDATM. likewise, there is a significant amount of duplicate work to be done to get BSDATM's upper layer ATM support caught up with HARP. based on that, i believe that it may be worthwhile to look at the driver/upper-layer interface differences between the two systems and seeing if the two interfaces can be merged so that both HARP and BSDATM can share lower level drivers. if that can be done, then it may be possible to get "the best of both worlds"? it isn't fully clear to me how ALTQ fits into this picture. chuck -- Chuck Cranor Senior Technical Staff Member, AT&T Labs-Research Room B180, 180 Park Ave, Florham Park NJ, 07932 To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-atm" in the body of the message From owner-freebsd-atm Fri Jan 8 13:13:01 1999 Return-Path: Received: (from majordom@localhost) by hub.freebsd.org (8.8.8/8.8.8) id NAA26325 for freebsd-atm-outgoing; Fri, 8 Jan 1999 13:13:01 -0800 (PST) (envelope-from owner-freebsd-atm@FreeBSD.ORG) Received: from critter.freebsd.dk (critter.freebsd.dk [212.242.40.131]) by hub.freebsd.org (8.8.8/8.8.8) with ESMTP id NAA26320 for ; Fri, 8 Jan 1999 13:12:57 -0800 (PST) (envelope-from phk@critter.freebsd.dk) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.9.1/8.8.5) with ESMTP id WAA11630; Fri, 8 Jan 1999 22:12:13 +0100 (CET) To: Chuck Cranor cc: freebsd-atm@FreeBSD.ORG, Tim Salo , "John D. DeHart" , Zubin Dittia , guru@arl.wustl.edu Subject: Re: ATM in FreeBSD, status requested! In-reply-to: Your message of "Fri, 08 Jan 1999 15:55:37 EST." <19990108155537.A1179348@research.att.com> Date: Fri, 08 Jan 1999 22:12:12 +0100 Message-ID: <11628.915829932@critter.freebsd.dk> From: Poul-Henning Kamp Sender: owner-freebsd-atm@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org In message <19990108155537.A1179348@research.att.com>, Chuck Cranor writes: > i've received several copies of messages from this list regarding >the status of my BSDATM code WRT FreeBSD over the holidays. Hi Chuck, Thanks for a very nice analyzis. I can personally add one item to your list: If the adaptec card is in a dockingstation, it will not probe correctly unless a powercycle is performed. I've given up on figuring out why (heisenbug, whenever I tried to reproduce it worked fine, whenever I was in a hurry it failed, sometimes also after several powercycles), and plugged that particular machine on a 100mbit ethernet instead. >based on that, i believe that it may be worthwhile to look at the >driver/upper-layer interface differences between the two systems and >seeing if the two interfaces can be merged so that both HARP and >BSDATM can share lower level drivers. This is basically the point I'm most interested in because of the influx of 25.6Mbit ATM based ADSL stuff we are starting to hear about. I would hate for us to have two drivers for every single atm device out there... Wrt, ALTQ I'm informed that no matter what IPV6 FreeBSD choose, ALTQ will be part of the packet at this time, so I think that will be sorted out at that time. -- Poul-Henning Kamp FreeBSD coreteam member phk@FreeBSD.ORG "Real hackers run -current on their laptop." FreeBSD -- It will take a long time before progress goes too far! To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-atm" in the body of the message