From owner-freebsd-scsi@FreeBSD.ORG Mon Sep 24 11:08:37 2007 Return-Path: Delivered-To: freebsd-scsi@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B4DB616A4C4 for ; Mon, 24 Sep 2007 11:08:37 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 8F73813C481 for ; Mon, 24 Sep 2007 11:08:37 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) by freefall.freebsd.org (8.14.1/8.14.1) with ESMTP id l8OB8b2S064308 for ; Mon, 24 Sep 2007 11:08:37 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.1/8.14.1/Submit) id l8OB8aP5064304 for freebsd-scsi@FreeBSD.org; Mon, 24 Sep 2007 11:08:36 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 24 Sep 2007 11:08:36 GMT Message-Id: <200709241108.l8OB8aP5064304@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-scsi@FreeBSD.org Cc: Subject: Current problem reports assigned to you X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Sep 2007 11:08:37 -0000 Current FreeBSD problem reports Critical problems Serious problems S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/39388 scsi ncr/sym drivers fail with 53c810 and more than 256MB m o kern/40895 scsi wierd kernel / device driver bug o kern/52638 scsi [panic] SCSI U320 on SMP server won't run faster than s kern/57398 scsi [mly] Current fails to install on mly(4) based RAID di o kern/60598 scsi wire down of scsi devices conflicts with config o kern/60641 scsi [sym] Sporadic SCSI bus resets with 53C810 under load s kern/61165 scsi [panic] kernel page fault after calling cam_send_ccb o kern/74627 scsi [ahc] [hang] Adaptec 2940U2W Can't boot 5.3 o kern/81887 scsi [aac] Adaptec SCSI 2130S aac0: GetDeviceProbeInfo comm o kern/90282 scsi [sym] SCSI bus resets cause loss of ch device o kern/92798 scsi [ahc] SCSI problem with timeouts o kern/93128 scsi [sym] FreeBSD 6.1 BETA 1 has problems with Symbios/LSI o kern/94838 scsi Kernel panic while mounting SD card with lock switch o o kern/99954 scsi [ahc] reading from DVD failes on 6.x (regression) o kern/110847 scsi [ahd] Tyan U320 onboard problem with more than 3 disks 15 problems total. Non-critical problems S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/23314 scsi [aic] aic driver fails to detect Adaptec 1520B unless o kern/35234 scsi World access to /dev/pass? (for scanner) requires acce o kern/38828 scsi [feature request] DPT PM2012B/90 doesn't work o kern/44587 scsi dev/dpt/dpt.h is missing defines required for DPT_HAND o kern/76178 scsi [ahd] Problem with ahd and large SCSI Raid system o kern/114597 scsi [sym] System hangs at SCSI bus reset with dual HBAs 6 problems total. From owner-freebsd-scsi@FreeBSD.ORG Wed Sep 26 00:31:39 2007 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2A33116A421; Wed, 26 Sep 2007 00:31:39 +0000 (UTC) (envelope-from hselasky@c2i.net) Received: from swip.net (mailfe08.swip.net [212.247.154.225]) by mx1.freebsd.org (Postfix) with ESMTP id 5450113C4A8; Wed, 26 Sep 2007 00:31:38 +0000 (UTC) (envelope-from hselasky@c2i.net) X-Cloudmark-Score: 0.000000 [] Received: from [85.19.218.45] (account mc467741@c2i.net [85.19.218.45] verified) by mailfe08.swip.net (CommuniGate Pro SMTP 5.1.10) with ESMTPA id 624282714; Wed, 26 Sep 2007 01:31:35 +0200 From: Hans Petter Selasky To: freebsd-scsi@freebsd.org, freebsd-usb@freebsd.org, freebsd-net@freebsd.org Date: Wed, 26 Sep 2007 01:31:47 +0200 User-Agent: KMail/1.9.7 MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200709260131.49156.hselasky@c2i.net> Cc: Subject: Request for feedback on common data backstore in the kernel X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Sep 2007 00:31:39 -0000 Hi, Please keep me CC'ed, hence I'm not on all these lists. In the kernel we currently have two different data backstores: struct mbuf and struct buf These two backstores serve two different device types. "mbufs" are for network devices and "buf" is for disk devices. Problem: The current backstores are loaded into DMA by using the BUS-DMA framework. This appears not to be too fast according to Kip Macy. See: http://perforce.freebsd.org/chv.cgi?CH=126455 Some ideas I have: When a buffer is out out of range for a hardware device and a data-copy is needed I want to simply copy that data in smaller parts to/from a pre-allocated bounce buffer. I want to avoid allocating this buffer when "bus_dmamap_load()" is called. For pre-allocated USB DMA memory I currently have: struct usbd_page struct usbd_page { void *buffer; // virtual address bus_size_t physaddr; // as seen by one of my devices bus_dma_tag_t tag; bus_dmamap_t map; uint32_t length; }; Mostly only "length == PAGE_SIZE" is allowed. When USB allocates DMA memory it allocates the same size all the way and that is PAGE_SIZE bytes. If two different PCI controllers want to communicate directly passing DMA buffers, technically one would need to translate the physical address for device 1 to the physical address as seen by device 2. If this translation table is sorted, the search will be rather quick. Another approach is to limit the number of translations: #define N_MAX_PCI_TRANSLATE 4 struct usbd_page { void *buffer; // virtual address bus_size_t physaddr[N_MAX_PCI_TRANSLATE]; bus_dma_tag_t tag; bus_dmamap_t map; uint32_t length; }; Then PCI device 1 on bus X can use physaddr[0] and PCI device 2 on bus Y can use physaddr[1]. If the physaddr[] is equal to some magic then the DMA buffer is not reachable and must be bounced. Then when two PCI devices talk together all they need to pass is a structure like this: struct usbd_page_cache { struct usbd_page *page_start; uint32_t page_offset_buf; uint32_t page_offset_end; }; And the required DMA address is looked up in some nanos. Has someone been thinking about this topic before ? --HPS From owner-freebsd-scsi@FreeBSD.ORG Wed Sep 26 05:09:24 2007 Return-Path: Delivered-To: freebsd-scsi@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0E17016A468; Wed, 26 Sep 2007 05:09:24 +0000 (UTC) (envelope-from jmg@hydrogen.funkthat.com) Received: from hydrogen.funkthat.com (gate.funkthat.com [69.17.45.168]) by mx1.freebsd.org (Postfix) with ESMTP id A9FC813C48A; Wed, 26 Sep 2007 05:09:23 +0000 (UTC) (envelope-from jmg@hydrogen.funkthat.com) Received: from hydrogen.funkthat.com (gkvdzsioz46jvzap@localhost.funkthat.com [127.0.0.1]) by hydrogen.funkthat.com (8.13.6/8.13.3) with ESMTP id l8Q4s261083635; Tue, 25 Sep 2007 21:54:02 -0700 (PDT) (envelope-from jmg@hydrogen.funkthat.com) Received: (from jmg@localhost) by hydrogen.funkthat.com (8.13.6/8.13.3/Submit) id l8Q4s2hA083634; Tue, 25 Sep 2007 21:54:02 -0700 (PDT) (envelope-from jmg) Date: Tue, 25 Sep 2007 21:54:01 -0700 From: John-Mark Gurney To: Hans Petter Selasky Message-ID: <20070926045401.GB47467@funkthat.com> Mail-Followup-To: Hans Petter Selasky , freebsd-arch@FreeBSD.org References: <200709260131.49156.hselasky@c2i.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <200709260131.49156.hselasky@c2i.net> User-Agent: Mutt/1.4.2.1i X-Operating-System: FreeBSD 5.4-RELEASE-p6 i386 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-3.0 (hydrogen.funkthat.com [127.0.0.1]); Tue, 25 Sep 2007 21:54:03 -0700 (PDT) X-Mailman-Approved-At: Wed, 26 Sep 2007 13:16:14 +0000 Cc: freebsd-arch@FreeBSD.org Subject: Re: Request for feedback on common data backstore in the kernel X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: John-Mark Gurney List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Sep 2007 05:09:24 -0000 Hans Petter Selasky wrote this message on Wed, Sep 26, 2007 at 01:31 +0200: > Please keep me CC'ed, hence I'm not on all these lists. > > In the kernel we currently have two different data backstores: > > struct mbuf > > and > > struct buf > > These two backstores serve two different device types. "mbufs" are for network > devices and "buf" is for disk devices. I don't see how this relates to the rest of your email, but even though they are used similarly, their normal size is quite different... mbufs normally contain 64-256 byte packets, w/ large file transfers attaching a 2k cluster (which comes from a different pool than the core mbuf) to the mbuf... buf is usually something like 16k-64k... > Problem: > > The current backstores are loaded into DMA by using the BUS-DMA framework. > This appears not to be too fast according to Kip Macy. See: > > http://perforce.freebsd.org/chv.cgi?CH=126455 This only works on x86/amd64 because of the direct mapped memory that they support.. This would complete break arches like sparc64 that require an iommu to translate the addresses... and also doesn't address keeping the buffers in sync on arches like arm... sparc64 may have many gigs of memory, but only a 2GB window for mapping main memory... It sounds like the x86/amd64 bus_dma implementation needs to be improved to run more quickly... As w/ all things, you can hardcode stuff, but then you loose portability... > Some ideas I have: > > When a buffer is out out of range for a hardware device and a data-copy is > needed I want to simply copy that data in smaller parts to/from a > pre-allocated bounce buffer. I want to avoid allocating this buffer > when "bus_dmamap_load()" is called. > > For pre-allocated USB DMA memory I currently have: > > struct usbd_page > > struct usbd_page { > void *buffer; // virtual address > bus_size_t physaddr; // as seen by one of my devices > bus_dma_tag_t tag; > bus_dmamap_t map; > uint32_t length; > }; > > Mostly only "length == PAGE_SIZE" is allowed. When USB allocates DMA memory it > allocates the same size all the way and that is PAGE_SIZE bytes. I could see attaching preallocated memory to a tag, and having maps that attempt to use this memory, but that's something else... > If two different PCI controllers want to communicate directly passing DMA > buffers, technically one would need to translate the physical address for > device 1 to the physical address as seen by device 2. If this translation > table is sorted, the search will be rather quick. Another approach is to > limit the number of translations: > > #define N_MAX_PCI_TRANSLATE 4 > > struct usbd_page { > void *buffer; // virtual address > bus_size_t physaddr[N_MAX_PCI_TRANSLATE]; > bus_dma_tag_t tag; > bus_dmamap_t map; > uint32_t length; > }; > > Then PCI device 1 on bus X can use physaddr[0] and PCI device 2 on bus Y can > use physaddr[1]. If the physaddr[] is equal to some magic then the DMA buffer > is not reachable and must be bounced. > > Then when two PCI devices talk together all they need to pass is a structure > like this: > > struct usbd_page_cache { > struct usbd_page *page_start; > uint32_t page_offset_buf; > uint32_t page_offset_end; > }; > > And the required DMA address is looked up in some nanos. > > Has someone been thinking about this topic before ? There is no infastructure to support passing dma address between hardware devices, and is complete unrelated to the issues raised above... This requires the ability to pass in a map to a tag and create a new map... It is possible, as on the sun4v where you have two iommu's.. You'd have to program on iommu to point to the other one, to support that... But it is rare to see devices to dma directly to each other... You usually end up dma'ing to main memory, and then having the other device dma it out of memory.. The only time you need to dma between devices is if one has local memory, and the other device is able to sanely populate it... This is very rare... Also, the PCI bus length can get quite long.. With PCIe, each device is now it's own PCI bus, so you're starting to see PCI bus counts in the 10's and 20's, if not higher.. having an area of all of those, and calculating them and filling them out sounds like a huge expense... I'm a bit puzzeled as to what you wanted to solve, as the problem you stated doesn't relate to the solutions you were thinking about... Maybe I'm missing something? Can you give me an example of where cxgb is writing to the memory on another pci bus, and not main memory? P.S. I redirected to -arch as this seems more related than the other lists... -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." From owner-freebsd-scsi@FreeBSD.ORG Wed Sep 26 17:51:26 2007 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3360D16A418; Wed, 26 Sep 2007 17:51:26 +0000 (UTC) (envelope-from scottl@samsco.org) Received: from pooker.samsco.org (pooker.samsco.org [168.103.85.57]) by mx1.freebsd.org (Postfix) with ESMTP id 9FD0813C458; Wed, 26 Sep 2007 17:51:25 +0000 (UTC) (envelope-from scottl@samsco.org) Received: from scoo-longs-computer.local (wireless.prattprop.com [209.97.224.24] (may be forged)) (authenticated bits=0) by pooker.samsco.org (8.13.8/8.13.8) with ESMTP id l8QHp52N063752; Wed, 26 Sep 2007 11:51:12 -0600 (MDT) (envelope-from scottl@samsco.org) Message-ID: <46FA9C04.9060603@samsco.org> Date: Wed, 26 Sep 2007 11:51:00 -0600 From: Scott Long User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US; rv:1.8.1.6) Gecko/20070802 SeaMonkey/1.1.4 MIME-Version: 1.0 To: Hans Petter Selasky References: <200709260131.49156.hselasky@c2i.net> In-Reply-To: <200709260131.49156.hselasky@c2i.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Greylist: Sender succeeded SMTP AUTH authentication, not delayed by milter-greylist-2.0.2 (pooker.samsco.org [168.103.85.57]); Wed, 26 Sep 2007 11:51:12 -0600 (MDT) X-Spam-Status: No, score=0.0 required=5.5 tests=none autolearn=failed version=3.1.8 X-Spam-Checker-Version: SpamAssassin 3.1.8 (2007-02-13) on pooker.samsco.org Cc: freebsd-scsi@freebsd.org, freebsd-usb@freebsd.org, freebsd-net@freebsd.org Subject: Re: Request for feedback on common data backstore in the kernel X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Sep 2007 17:51:26 -0000 Hans Petter Selasky wrote: > Hi, > > Please keep me CC'ed, hence I'm not on all these lists. > > In the kernel we currently have two different data backstores: > > struct mbuf > > and > > struct buf > > These two backstores serve two different device types. "mbufs" are for network > devices and "buf" is for disk devices. > > Problem: > > The current backstores are loaded into DMA by using the BUS-DMA framework. > This appears not to be too fast according to Kip Macy. See: > > http://perforce.freebsd.org/chv.cgi?CH=126455 > Busdma isn't fast enough for 1Gb and 10Gb network drivers that are trying to max out their packet rates. It's still fine for storage drivers and other slow/medium speed device drivers, like USB and Firewire. I am working on techniques to make the API easier to use and the implementation fast enough for the aforementioned devices. > Some ideas I have: > > When a buffer is out out of range for a hardware device and a data-copy is > needed I want to simply copy that data in smaller parts to/from a > pre-allocated bounce buffer. I want to avoid allocating this buffer > when "bus_dmamap_load()" is called. I think you've missed the point of having architecture portable drivers. John-Mark describes this further. It also makes little sense to push the responsibility for handling bounce buffers out of a central subsystem and back into every driver. Scott From owner-freebsd-scsi@FreeBSD.ORG Wed Sep 26 19:56:57 2007 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9A11716A419; Wed, 26 Sep 2007 19:56:57 +0000 (UTC) (envelope-from hselasky@c2i.net) Received: from swip.net (mailfe04.swip.net [212.247.154.97]) by mx1.freebsd.org (Postfix) with ESMTP id 7ADDF13C4A6; Wed, 26 Sep 2007 19:56:56 +0000 (UTC) (envelope-from hselasky@c2i.net) X-Cloudmark-Score: 0.000000 [] Received: from [85.19.218.45] (account mc467741@c2i.net [85.19.218.45] verified) by mailfe04.swip.net (CommuniGate Pro SMTP 5.1.10) with ESMTPA id 626638699; Wed, 26 Sep 2007 21:56:43 +0200 From: Hans Petter Selasky To: Scott Long Date: Wed, 26 Sep 2007 21:57:00 +0200 User-Agent: KMail/1.9.7 References: <200709260131.49156.hselasky@c2i.net> <46FA9C04.9060603@samsco.org> In-Reply-To: <46FA9C04.9060603@samsco.org> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200709262157.02712.hselasky@c2i.net> Cc: freebsd-scsi@freebsd.org, freebsd-arch@freebsd.org, freebsd-usb@freebsd.org, freebsd-net@freebsd.org Subject: Re: Request for feedback on common data backstore in the kernel X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Sep 2007 19:56:57 -0000 Hi Scott, The discussion has been moved to "freebsd-arch@freebsd.org". Please only reply there next time. On Wednesday 26 September 2007, Scott Long wrote: > Hans Petter Selasky wrote: > > Hi, > > > > Please keep me CC'ed, hence I'm not on all these lists. > > > > In the kernel we currently have two different data backstores: > > > > struct mbuf > > > > and > > > > struct buf > > > > These two backstores serve two different device types. "mbufs" are for > > network devices and "buf" is for disk devices. > > > > Problem: > > > > The current backstores are loaded into DMA by using the BUS-DMA > > framework. This appears not to be too fast according to Kip Macy. See: > > > > http://perforce.freebsd.org/chv.cgi?CH=126455 > > Busdma isn't fast enough for 1Gb and 10Gb network drivers that are > trying to max out their packet rates. It's still fine for storage > drivers and other slow/medium speed device drivers, like USB and > Firewire. I am working on techniques to make the API easier to use > and the implementation fast enough for the aforementioned devices. That's great! > > > Some ideas I have: > > > > When a buffer is out out of range for a hardware device and a data-copy > > is needed I want to simply copy that data in smaller parts to/from a > > pre-allocated bounce buffer. I want to avoid allocating this buffer when > > "bus_dmamap_load()" is called. > > I think you've missed the point of having architecture portable drivers. > John-Mark describes this further. I use the bus_dma framework to allocate and sync all DMA memory, and I assume that is correct. > It also makes little sense to push > the responsibility for handling bounce buffers out of a central > subsystem and back into every driver. I'm thinking about putting some wrappers around the "bus_dmamap_load()" function like: void usbd_rx_buf_load(struct usbd_xfer *xfer, void *buf, uint32_t framebuffer_offset, uint32_t framebuffer_index, uint32_t len); void usbd_tx_buf_load(struct usbd_xfer *xfer, void *buf, uint32_t framebuffer_offset, uint32_t framebuffer_index, uint32_t len); But I haven't figured out all the details yet. The "usbd_xxx_load()" functions should automagically figure out what is best to do and it won't be more than a few lines of code. --HPS From owner-freebsd-scsi@FreeBSD.ORG Thu Sep 27 01:13:52 2007 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EE3ED16A418; Thu, 27 Sep 2007 01:13:52 +0000 (UTC) (envelope-from scottl@samsco.org) Received: from pooker.samsco.org (pooker.samsco.org [168.103.85.57]) by mx1.freebsd.org (Postfix) with ESMTP id 7BF9B13C48A; Thu, 27 Sep 2007 01:13:52 +0000 (UTC) (envelope-from scottl@samsco.org) Received: from phobos.samsco.home (phobos.samsco.home [192.168.254.11]) (authenticated bits=0) by pooker.samsco.org (8.13.8/8.13.8) with ESMTP id l8R1DPix065986; Wed, 26 Sep 2007 19:13:26 -0600 (MDT) (envelope-from scottl@samsco.org) Message-ID: <46FB03B1.6020100@samsco.org> Date: Wed, 26 Sep 2007 19:13:21 -0600 From: Scott Long User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US; rv:1.8.1.6) Gecko/20070802 SeaMonkey/1.1.4 MIME-Version: 1.0 To: Hans Petter Selasky References: <200709260131.49156.hselasky@c2i.net> <46FA9C04.9060603@samsco.org> <200709262157.02712.hselasky@c2i.net> In-Reply-To: <200709262157.02712.hselasky@c2i.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Greylist: Sender succeeded SMTP AUTH authentication, not delayed by milter-greylist-2.0.2 (pooker.samsco.org [168.103.85.57]); Wed, 26 Sep 2007 19:13:26 -0600 (MDT) X-Spam-Status: No, score=-1.4 required=5.5 tests=ALL_TRUSTED autolearn=failed version=3.1.8 X-Spam-Checker-Version: SpamAssassin 3.1.8 (2007-02-13) on pooker.samsco.org Cc: freebsd-scsi@freebsd.org, freebsd-arch@freebsd.org, freebsd-usb@freebsd.org, freebsd-net@freebsd.org Subject: Re: Request for feedback on common data backstore in the kernel X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Sep 2007 01:13:53 -0000 Hans Petter Selasky wrote: > Hi Scott, > > The discussion has been moved to "freebsd-arch@freebsd.org". Please only reply > there next time. > > On Wednesday 26 September 2007, Scott Long wrote: >> Hans Petter Selasky wrote: >>> Hi, >>> >>> Please keep me CC'ed, hence I'm not on all these lists. >>> >>> In the kernel we currently have two different data backstores: >>> >>> struct mbuf >>> >>> and >>> >>> struct buf >>> >>> These two backstores serve two different device types. "mbufs" are for >>> network devices and "buf" is for disk devices. >>> >>> Problem: >>> >>> The current backstores are loaded into DMA by using the BUS-DMA >>> framework. This appears not to be too fast according to Kip Macy. See: >>> >>> http://perforce.freebsd.org/chv.cgi?CH=126455 >> Busdma isn't fast enough for 1Gb and 10Gb network drivers that are >> trying to max out their packet rates. It's still fine for storage >> drivers and other slow/medium speed device drivers, like USB and >> Firewire. I am working on techniques to make the API easier to use >> and the implementation fast enough for the aforementioned devices. > > That's great! > Well, the point is that I'm not sure why you're so worried about performance issues with USB and busdma. Do you have any test data that shows that it's a problem? >>> Some ideas I have: >>> >>> When a buffer is out out of range for a hardware device and a data-copy >>> is needed I want to simply copy that data in smaller parts to/from a >>> pre-allocated bounce buffer. I want to avoid allocating this buffer when >>> "bus_dmamap_load()" is called. >> I think you've missed the point of having architecture portable drivers. >> John-Mark describes this further. > > I use the bus_dma framework to allocate and sync all DMA memory, and I assume > that is correct. > That is one of the uses of busdma, yes. The other is to handle buffers that have been allocated elsewhere in the system. >> It also makes little sense to push >> the responsibility for handling bounce buffers out of a central >> subsystem and back into every driver. > > I'm thinking about putting some wrappers around the "bus_dmamap_load()" > function like: > > void usbd_rx_buf_load(struct usbd_xfer *xfer, void *buf, > uint32_t framebuffer_offset, uint32_t framebuffer_index, uint32_t len); > > void usbd_tx_buf_load(struct usbd_xfer *xfer, void *buf, > uint32_t framebuffer_offset, uint32_t framebuffer_index, uint32_t len); > > But I haven't figured out all the details yet. The "usbd_xxx_load()" functions > should automagically figure out what is best to do and it won't be more than > a few lines of code. Can you describe what these two wrappers are supposed to do? Are you expecting bus_dmamap_load() to operate synchronously? Scott