Skip site navigation (1)Skip section navigation (2)
Date:      Thu, 15 Aug 2013 18:12:05 -0700
From:      Adrian Chadd <>
To:        Alexander Motin <>
Cc:        Scott Long <>, Jeff Roberson <>, ken <>, "" <>, FreeBSD SCSI <>, Steven Hartland <>, "Justin T. Gibbs" <>
Subject:   Re: New CAM locking preview
Message-ID:  <>
In-Reply-To: <>
References:  <>

Next in thread | Previous in thread | Raw E-Mail | Index | Archive | Help

I assume you've run this with full witness debugging enabled, to catch lock
ordering issues?

This is great. I look forward to per-CPU, pinned, completion threads that I
can do interesting things with (like schedule aio-sendfile completions..)


On 15 August 2013 14:40, Alexander Motin <> wrote:

> Hi.
> Last weeks I've made substantial progress on my CAM locking work. In fact,
> at this moment I think I've tied all loose ends good enough to consider the
> new design viable and implementation worth further testing and bug fixing.
> So I would like to ask for review of my work from everybody who interested
> in CAM internals.
> In short, my idea was to split single per-SIM lock, that creates huge
> congestion under high IOPS, into several smaller ones. So design I've
> finally chosen includes such locks:
>  1) New per-device (per-LUN) locks to protect state of the devices and
> respective periphs. In most cases peripheral drivers just use that lock
> instead of SIM lock used before, so code modification is minimal and
> straightforward.
>  2) New per-target lock to protect list of LUNs fetched from the device.
>  3) Old single per-SIM lock to protect SIM driver internals, but only
> that. No parts of CAM itself use that lock. Keeping it for SIMs allows to
> keep API and hopefully ABI compatibility. Reducing its scope allows to
> reduce congestion.
>  4) New per-SIM lock to protect SIM and device command queues. That allows
> execute queued commands from any context unrelated to other locks. Also
> this lock serializes accesses to sim_action() method for the most of
> commands, this allows to mostly avoid busy spilling on SIM lock collision.
>  5) New per-bus locks to protect target, device and periphs reference
> counters. It allows to create and destroy paths unrelated to other locks in
> any possible context.
> Numbers above also define supposed lock ordering: while holding per-device
> lock 1) is allowed to request SIM lock 3), but not backward. Cases where
> opposite is required (command completions and async events) are handled via
> queuing events via several completion threads. The rest of locks are
> self-contained and does not really suppose cascading.
> All these changes combined with GEOM direct dispatch (it will be next
> separate project) allow to double system performance in disk I/O
> microbenchmarks, comparing to present head, same as it was announced on
> 2013-05 DevSummit:**mav/camlock.pdf<>. Tests without GEOM changes also show performance improvement, but limited
> by heavy bottleneck at the GEOM g_up/g_down threads at the level of 5-20%.
> Project sources could be found at SVN projects/camlock branch:
>**base/projects/camlock/<>. Many early changes from that branch are already integrated to head, so to
> simplify review the rest patches for changes before r254059 were manually
> remade and could be found here:**
> mav/camlock_patches/ <>; .
> These changes do not require controller driver modifications, keeping KPIs
> and hopefully KBIs intact, but create base for later work to use multiqueue
> capabilities of new controllers.
> This work is sponsored by iXsystems, Inc.
> --
> Alexander Motin
> ______________________________**_________________
> mailing list
> To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@**
> <>"

Want to link to this message? Use this URL: <>