Date: Mon, 22 Jan 2001 12:10:33 -0800 From: Alfred Perlstein <bright@wintelcom.net> To: Robert Lipe <robertlipe@usa.net> Cc: freebsd-hackers@FreeBSD.ORG Subject: Re: contigmalloc, M_WAITOK, & leaks. Message-ID: <20010122121033.A26076@fw.wintelcom.net> In-Reply-To: <20010122132647.I10504@rjlhome.sco.com>; from robertlipe@usa.net on Mon, Jan 22, 2001 at 01:26:47PM -0600 References: <20010122110642.B10504@rjlhome.sco.com> <20010122100524.D7240@fw.wintelcom.net> <20010122124539.F10504@rjlhome.sco.com> <20010122105227.E7240@fw.wintelcom.net> <20010122132647.I10504@rjlhome.sco.com>
next in thread | previous in thread | raw e-mail | index | archive | help
* Robert Lipe <robertlipe@usa.net> [010122 11:21] wrote: > > > In this case, the resource failure isn't temporary. Once it starts > > > failing, it fails until the system is rebooted. Since this is on a > > > dedicated thread, I could handle it if it really was temporary. Of > > > course, putting me to sleep so that someone else (or even another of my > > > own threads) could run around and free the resources would be ideal. As > > > it stands, once I get the failure notice, I'm hosed. > > > > Yup, it's sort of the fault of the memory allocator, afaik except > > for allocations that are > (PAGE_SIZE/2), you can't free back to > > the kernel map, meaning that if you allocate several thousand < > > (PAGE_SIZE/2) chunks you've pretty much broken contigmalloc. > > Aaaah. There's a hint. Yes, by the time we get into trouble, I've > allocated (and freed) several thousand chunks that are < PAGE_SIZE/2. > This is happening in periods of a few dozen. (I don't know the number, > but I think it's 48 or 64.) The test program allocates a few dozen tx > buffers, fills them up, then the tx completion handler releases them. > There are never very many of these outstanding at any time. All the > allocs (in this specific case) are always the same size, and there are > no intervening calls to contigmalloc. This should stack the deck pretty > well for fragmentation. > > So if I sandbagged the allocs to be *larger* the KMA would behave more > consistently with what I'd expect becuase it would then reclaim? I > didn't see that one coming. :-) > > Would rounding them up to PAGE_SIZE/2 suffice, or did you really mean "> > PAGE_SIZE/2"? On i386, PAGE_SIZE is 4096, so the allocations need to be 2048+1 :). so it's > PAGE_SIZE/2. but... this is a terrible workaround, I'm not sure it would work and you shouldn't do it. :) > > Not hinting much, the only reason I can guess that you need contigmalloc > > is because you're allocating > PAGE_SIZE chunks and you don't want to > > tell your hardware to do scatter/gather IO and would rather point it > > at some physically contig memory and be done with it. > > Is there an s/g memory interface in FreeBSD? This was my first choice, > but since I couldn't find a set of functions to build a list of buffers > that satisfied a set of constraints, I fell back to contigmalloc to get > things off the ground. I'd be delighted to use an interface to get me > an s/g list with a given set of constraints that pointed to a list of > buffers with a given set of constraints. See the busdma stuff, there's a problem because there's no busdma for mbufs (well actually there is but it fails on really small unaligned blocks which basically breaks it for mbufs). NetBSD has some stuff for setting up s/g on mbufs (more busdma) but it looks pretty inefficient and I havne't had a chance to look at emulating or providing a different interface for it. > > This is possible (using contigmalloc) if you cache the contigmalloc'd > > chunks, but not if you keep discarding them and allow the kernel memory > > map to become more fragmented. > > Implementing a cache for these is a few lines of code and will improve > the robustness of what I have. I'm clearly going to have to do this. There's no way around it if you want to use contigmalloc. However if you were to use busdma you could do it with normal malloc'd blocks. > > My question is, why exactly do you need contigmalloc, is this some > > device you're trying to write a driver for? Can you explain it some? > > That would help me suggest a workaround. > > Sure. I'm trying to write a driver for arbitrary devices. :-) This is > kinky, I know. > > UDI (www.projectudi.org) allows a driver writer to specify a very rich > set of constraints on DMA objects. I'm implementing the piece of the > UDI environment that provides this to the drivers. So while I know > that the test program and driver I'm using right now don't *really* > need contigmalloc, we'll eventually need the ability to ensure that DMA > buffers for 32-bit PCI cards are in the bottom 32 bits, (or perhaps > even not the bottom if you have intervening bus bridges) that stupid > hardware providing 24 bits of addressing but bolted to a PCI controller > gets allocated in the bottom 16Mb, etc. > > If I was trying to support a very specific piece of hardware, I could > divine workarounds, but it's something I'm probably going to have to > determine programmatically. ack, I'd review patches if you wanted to write/port busdma for mbufs.. see: http://cvsweb.netbsd.org/bsdweb.cgi/syssrc/sys/arch/i386/i386/bus_machdep.c for a starting point. -- -Alfred Perlstein - [bright@wintelcom.net|alfred@freebsd.org] "I have the heart of a child; I keep it in a jar on my desk." To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-hackers" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20010122121033.A26076>