Skip site navigation (1)Skip section navigation (2)
Date:      Fri, 16 Aug 2013 09:51:20 +0300
From:      Alexander Motin <mav@FreeBSD.org>
To:        Adrian Chadd <adrian@freebsd.org>
Cc:        Scott Long <scottl@freebsd.org>, Jeff Roberson <jeff@freebsd.org>, ken <ken@freebsd.org>, "freebsd-hackers@freebsd.org" <freebsd-hackers@freebsd.org>, FreeBSD SCSI <freebsd-scsi@freebsd.org>, Steven Hartland <smh@freebsd.org>, "Justin T. Gibbs" <gibbs@freebsd.org>
Subject:   Re: New CAM locking preview
Message-ID:  <520DCBE8.9050703@FreeBSD.org>
In-Reply-To: <CAJ-VmonD_MJSdc6M_H14k-nWotg1cN1GG1=U7B2OtOnhGDn02g@mail.gmail.com>
References:  <520D4ADB.50209@FreeBSD.org> <CAJ-VmonD_MJSdc6M_H14k-nWotg1cN1GG1=U7B2OtOnhGDn02g@mail.gmail.com>

Next in thread | Previous in thread | Raw E-Mail | Index | Archive | Help
On 16.08.2013 04:12, Adrian Chadd wrote:
> Cool!
>
> I assume you've run this with full witness debugging enabled, to catch
> lock ordering issues?

Of course! I've endless times switched between debug and normal builds 
to test both correctness and performance after each change. But more 
external tests are welcome.

> This is great. I look forward to per-CPU, pinned, completion threads
> that I can do interesting things with (like schedule aio-sendfile
> completions..)
>
> On 15 August 2013 14:40, Alexander Motin <mav@freebsd.org
> <mailto:mav@freebsd.org>> wrote:
>
>     Hi.
>
>     Last weeks I've made substantial progress on my CAM locking work. In
>     fact, at this moment I think I've tied all loose ends good enough to
>     consider the new design viable and implementation worth further
>     testing and bug fixing. So I would like to ask for review of my work
>     from everybody who interested in CAM internals.
>
>     In short, my idea was to split single per-SIM lock, that creates
>     huge congestion under high IOPS, into several smaller ones. So
>     design I've finally chosen includes such locks:
>       1) New per-device (per-LUN) locks to protect state of the devices
>     and respective periphs. In most cases peripheral drivers just use
>     that lock instead of SIM lock used before, so code modification is
>     minimal and straightforward.
>       2) New per-target lock to protect list of LUNs fetched from the
>     device.
>       3) Old single per-SIM lock to protect SIM driver internals, but
>     only that. No parts of CAM itself use that lock. Keeping it for SIMs
>     allows to keep API and hopefully ABI compatibility. Reducing its
>     scope allows to reduce congestion.
>       4) New per-SIM lock to protect SIM and device command queues. That
>     allows execute queued commands from any context unrelated to other
>     locks. Also this lock serializes accesses to sim_action() method for
>     the most of commands, this allows to mostly avoid busy spilling on
>     SIM lock collision.
>       5) New per-bus locks to protect target, device and periphs
>     reference counters. It allows to create and destroy paths unrelated
>     to other locks in any possible context.
>
>     Numbers above also define supposed lock ordering: while holding
>     per-device lock 1) is allowed to request SIM lock 3), but not
>     backward. Cases where opposite is required (command completions and
>     async events) are handled via queuing events via several completion
>     threads. The rest of locks are self-contained and does not really
>     suppose cascading.
>
>     All these changes combined with GEOM direct dispatch (it will be
>     next separate project) allow to double system performance in disk
>     I/O microbenchmarks, comparing to present head, same as it was
>     announced on 2013-05 DevSummit:
>     http://people.freebsd.org/~__mav/camlock.pdf
>     <http://people.freebsd.org/~mav/camlock.pdf>; . Tests without GEOM
>     changes also show performance improvement, but limited by heavy
>     bottleneck at the GEOM g_up/g_down threads at the level of 5-20%.
>
>     Project sources could be found at SVN projects/camlock branch:
>     http://svnweb.freebsd.org/__base/projects/camlock/
>     <http://svnweb.freebsd.org/base/projects/camlock/>; . Many early
>     changes from that branch are already integrated to head, so to
>     simplify review the rest patches for changes before r254059 were
>     manually remade and could be found here:
>     http://people.freebsd.org/~__mav/camlock_patches/
>     <http://people.freebsd.org/~mav/camlock_patches/>; .
>
>     These changes do not require controller driver modifications,
>     keeping KPIs and hopefully KBIs intact, but create base for later
>     work to use multiqueue capabilities of new controllers.
>
>     This work is sponsored by iXsystems, Inc.


-- 
Alexander Motin



Want to link to this message? Use this URL: <http://docs.FreeBSD.org/cgi/mid.cgi?520DCBE8.9050703>