Date: Fri, 1 Dec 2006 01:39:24 +0000 (GMT) From: Robert Watson <rwatson@FreeBSD.org> To: Ivan Voras <ivoras@fer.hr> Cc: freebsd-arch@freebsd.org Subject: Re: a proposed callout API Message-ID: <20061201012221.J79653@fledge.watson.org> In-Reply-To: <ekmr76$23g$1@sea.gmane.org> References: <200611292147.kATLll4m048223@apollo.backplane.com> <11606.1164837711@critter.freebsd.dk> <ekmr76$23g$1@sea.gmane.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Thu, 30 Nov 2006, Ivan Voras wrote:
> No trying to take sides here, but for us willing to learn here, what exactly
> are the problems in Matt Dillon's suggestions? From a novice's POV, having
> per-cpu queues looks (emphasis: looks) very scalable and performant.
The implications of adopting the model Matt proposes are quite far-reaching:
callouts don't exist in isolation, but occur in the context of data structures
and work occuring in many threads. If callouts are pinned to a particular
CPU, and can only be scheduled, rescheduled, and cancelled from that CPU, that
implies either that all work associated with that callout is also pinned to
the CPU, or that migration or message-passing be involved if the requirement
comes up in a thread on another CPU.
Consider the case of TCP timers: a number of TCP timers get regularly
rescheduled (delack, retransmit, etc). If they can only be manipulated from
cpu0 (i.e., protected by a synchronization primitive that can't be acquired
from another CPU -- i.e., critical sections instead of mutexes), how do you
handle the case where the a TCP packet for that connection is processed on
cpu1 and needs to change the scheduling of the timer? In a strict work/data
structure pinning model, you would pin the TCP connection to cpu0, and only
process any data leading to timer changes on that CPU. Alternatively, you
might pass a message from cpu1 to cpu0 to change the scheduling.
The idea of processing timers in multiple threads and pinning them to multiple
CPUs clearly isn't a bad idea: we could likely benefit from parallelism (and
generally, concurrency) in timer processing. One of the things we discussed
at the recent developer summit was subsystem callout threads (introducing the
opportunity for parallism without committing to a particular CPU scheduling
model), as well as per-CPU callout threads but protected using mutexes so that
reschedule/cancel/etc can be performed form other CPUs still. Changing the
API so that scheduling/rescheduling/etc activities themselves must occur on a
particular CPU has serious implications and commits us to an architectural
approach for which there is little concensus. If the goal is simply
parallelism, it's possible to accomplish that without embedding assumptions
about the synchronization model at this point. Take a look at the USENIX
paper by Paul Willmann (et al) at Rice for some rather interesting
experimentation, measurement, and discussion precisely along these lines:
http://www.ece.rice.edu/~willmann/pubs/paranet_tr06-872.pdf
Robert N M Watson
Computer Laboratory
University of Cambridge
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20061201012221.J79653>