Date: Fri, 12 Jul 2002 14:58:35 -0700 (PDT) From: John Polstra <jdp@polstra.com> To: net@freebsd.org Cc: bmilekic@unixdaemons.com Subject: Re: mbuf external buffer reference counters Message-ID: <200207122158.g6CLwZax009408@vashon.polstra.com> In-Reply-To: <20020712142021.A44645@unixdaemons.com> References: <20020711162026.A18717@unixdaemons.com> <200207121803.g6CI3je9008944@vashon.polstra.com> <20020712142021.A44645@unixdaemons.com>
next in thread | previous in thread | raw e-mail | index | archive | help
In article <20020712142021.A44645@unixdaemons.com>, Bosko Milekic <bmilekic@unixdaemons.com> wrote: > > I've thought about the cache issue with regards to the ref. counts > before, actually, and initially, I also thought the exact same thing > as you bring up here. However, there are a few things you need to > remember: > > 1) SMP; counters are typically referenced by several different threads > which may be running on different CPUs at any given point in time, and > this means that we'll probably end up having corresponding cache lines > invalidated back and forth anyway; Agreed. The PII and newer CPUs do have some short cuts built in that mitigate this somewhat by doing direct cache-to-cache updates in the SMP case. But quantitatively I don't know how much that helps. > 2) Using more cache lines may not be better overall, we may be doing > write-backs of other data already there; in any case, we would really > have to measure this; The research that led to the slab allocator demonstrated pretty conclusively that, at least in general, it's better to spread out the usage across all cache lines rather than compete for just a few. Measurements trump research, though, as long as the measurements reflect real-world usage patterns. If you decide to pack the refcounts into the clusters themselves, it might be better to put the recount at the front of each cluster, and offset the packet data by 16 bytes to make room for it. That way, the reference count would be in the same cache line as the first part of the packet header -- a cache line which is almost certain to be accessed (though probably not dirtied) anyway. > 3) By far the most important: all modifications to the ref. count are > atomic, bus-locked, ops. I spoke to Peter a little about this and > although I'm not 100% sure, we think that bus-locked > fetch-inc/dec-stores need the bus anyway. If that's the case, > then we really don't care about whether or not they get cached, right? I'm afraid I don't know the answer to that. The majority of systems will be uniprocessor for a good long time, and I would hate to see their performance sacrificed needlessly. John -- John Polstra John D. Polstra & Co., Inc. Seattle, Washington USA "Disappointment is a good sign of basic intelligence." -- Chögyam Trungpa To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-net" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200207122158.g6CLwZax009408>