Date: Sun, 1 Feb 2004 00:01:07 -0500 (EST) From: Robert Watson <rwatson@FreeBSD.org> To: Dag-Erling =?iso-8859-1?q?Sm=F8rgrav?= <des@des.no> Cc: current@FreeBSD.org Subject: Re: Coalescing pipe allocation Message-ID: <Pine.NEB.3.96L.1040131234955.17012E-100000@fledge.watson.org> In-Reply-To: <xzp1xpfiezn.fsf@dwp.des.no>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sun, 1 Feb 2004, Dag-Erling Sm=F8rgrav wrote: > Robert Watson <rwatson@FreeBSD.org> writes: > > I have not yet macro-benchmarked this change. It will either have a > > slight performance benefit if half-closed pipes aren't used extensively= or > > it doesn't make a difference, be a wash, or if half-closed pipes turn u= p a > > lot, potentially slightly negatively impact performance (seems unlikely= ). >=20 > [you may want to check the -current archives for silby@'s excellent > message about pipe memory usage last Tuesday] Yeah, I just chatted with Silby and he sent me a copy of the message. His observations all look pretty on-target to me, and suggested strategies look good. > I think our biggest problem with pipes is not half-open pipes but rather > their absence; most of the code that uses pipes only uses one direction > and does not bother to shut the other down. This is certainly the case > for sh(1) and make(1), which I suspect are the heaviest pipe consumers > in a typical system (especially in one running our favorite benchmark > tool, 'make buildworld'.) The result is that we use almost twice as > much memory for pipes as we need to. >=20 > * solution: allocate pipe buffers on first-use rather than at pipe > creation time. Yeah, I was interested by this observation -- many applications don't even think pipes are bi-directional, meaning that doing work allocating pipe map space for both directions basically wastes half the work. This change should be pretty easy, given that we already have code in to do a resize on write. It's probably as simply dropping the call to pipe_create() in my version of the code, and perhaps making sure that pipespace() checks that case. FYI, the vast majority of the cost of pipe creation appears to be in setting up those mappings. I ran some mutex profiling and the amount of work getting done is pretty high -- we should be able to reap a lot of benefit by doing even a little less work there. Doing half the work would do wonders. Caching sets of mappings for reuse would also help a lot, I suspect, and we may be able to use UMA to help manage that (not sure yet how well that will work).=20 > Another problem with pipes is that the default pipe buffer size is too > large.=20 >=20 > * solution: reduce the initial allocation to a single page, and grow > the buffer later if necessary (up to a limit og four or eight > pages) >=20 > One problem we can't solve easily is the fact that pipe buffers are > allocated directly from the pipe vm map, which is a very inefficient > operation in FreeBSD. I have a tentative design in my notebook for a > more efficient data structure for vm maps, allowing vm_map_find() to run > in logarithmic rather than linear time, but this hasn't been high on my > priority list as it's fairly hard to get right.=20 >=20 > BTW, as I am currently working on some of these issues, I'd be happy if > you could commit your patches fairly quickly so I don't have to start > all over again once they hit the tree.=20 Ok, sounds good. I got Juli to do a technical review, and have been running a variety of tests. I should probably just commit it so it's there and then we can refine. I would imagine we can get another 30% of the cost out of pipe creation (at least) through a bit of refinement of just the sort we're discussing.=20 Look for a commit sometime in the next hour or two -- I've got a couple of other people running big builds with it, etc, to make sure nothing shakes loose.=20 Robert N M Watson FreeBSD Core Team, TrustedBSD Projects robert@fledge.watson.org Senior Research Scientist, McAfee Research
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?Pine.NEB.3.96L.1040131234955.17012E-100000>