From owner-freebsd-scsi@FreeBSD.ORG Fri Aug 16 01:12:09 2013 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 39F69F86; Fri, 16 Aug 2013 01:12:09 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-wg0-x22e.google.com (mail-wg0-x22e.google.com [IPv6:2a00:1450:400c:c00::22e]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id AF1E02472; Fri, 16 Aug 2013 01:12:07 +0000 (UTC) Received: by mail-wg0-f46.google.com with SMTP id k13so1073334wgh.13 for ; Thu, 15 Aug 2013 18:12:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=PHq+4h2tXFQIradobpqSO0JKQm6cWJqoq9eumez+nrI=; b=Pt/Gkz6gbsbEVj37cdNsoGvKIs60uRHqhlS4IciYXcT8OavkzuWUs2yHGjz5xvOXut ZmIp13BkFSx29IVlKd/PqW92zmiuQm2gc+V+LnfNvTYBuYOzsj1Tkbadb5171WYlx4IX kO5Y7cMFuLTp9rpfsjZcABwngj+qJ2kOdMEIGjT+bKgW+qH1k0AOKM2adLua+p7x8bTP IGZV8WNfca+rYwG7b2SqC4zWTgi4GokBQhvusgiyQWhT+G5Li/4To7ezDgyIfqDXGTGN gAp8GwNcBebY4VUxHY6ai7TlaGrLNSog1g2NqgsPYfaKtY937mauZjsyW0ZXDf7QRoiM D/UA== MIME-Version: 1.0 X-Received: by 10.180.37.16 with SMTP id u16mr226772wij.46.1376615525882; Thu, 15 Aug 2013 18:12:05 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.217.116.136 with HTTP; Thu, 15 Aug 2013 18:12:05 -0700 (PDT) In-Reply-To: <520D4ADB.50209@FreeBSD.org> References: <520D4ADB.50209@FreeBSD.org> Date: Thu, 15 Aug 2013 18:12:05 -0700 X-Google-Sender-Auth: D6pqo5evTjWdHMuJBBeiW4ha7JA Message-ID: Subject: Re: New CAM locking preview From: Adrian Chadd To: Alexander Motin Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: Scott Long , Jeff Roberson , ken , "freebsd-hackers@freebsd.org" , FreeBSD SCSI , Steven Hartland , "Justin T. Gibbs" X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Aug 2013 01:12:09 -0000 Cool! I assume you've run this with full witness debugging enabled, to catch lock ordering issues? This is great. I look forward to per-CPU, pinned, completion threads that I can do interesting things with (like schedule aio-sendfile completions..) -adrian On 15 August 2013 14:40, Alexander Motin wrote: > Hi. > > Last weeks I've made substantial progress on my CAM locking work. In fact, > at this moment I think I've tied all loose ends good enough to consider the > new design viable and implementation worth further testing and bug fixing. > So I would like to ask for review of my work from everybody who interested > in CAM internals. > > In short, my idea was to split single per-SIM lock, that creates huge > congestion under high IOPS, into several smaller ones. So design I've > finally chosen includes such locks: > 1) New per-device (per-LUN) locks to protect state of the devices and > respective periphs. In most cases peripheral drivers just use that lock > instead of SIM lock used before, so code modification is minimal and > straightforward. > 2) New per-target lock to protect list of LUNs fetched from the device. > 3) Old single per-SIM lock to protect SIM driver internals, but only > that. No parts of CAM itself use that lock. Keeping it for SIMs allows to > keep API and hopefully ABI compatibility. Reducing its scope allows to > reduce congestion. > 4) New per-SIM lock to protect SIM and device command queues. That allows > execute queued commands from any context unrelated to other locks. Also > this lock serializes accesses to sim_action() method for the most of > commands, this allows to mostly avoid busy spilling on SIM lock collision. > 5) New per-bus locks to protect target, device and periphs reference > counters. It allows to create and destroy paths unrelated to other locks in > any possible context. > > Numbers above also define supposed lock ordering: while holding per-device > lock 1) is allowed to request SIM lock 3), but not backward. Cases where > opposite is required (command completions and async events) are handled via > queuing events via several completion threads. The rest of locks are > self-contained and does not really suppose cascading. > > All these changes combined with GEOM direct dispatch (it will be next > separate project) allow to double system performance in disk I/O > microbenchmarks, comparing to present head, same as it was announced on > 2013-05 DevSummit: http://people.freebsd.org/~**mav/camlock.pdf. Tests without GEOM changes also show performance improvement, but limited > by heavy bottleneck at the GEOM g_up/g_down threads at the level of 5-20%. > > Project sources could be found at SVN projects/camlock branch: > http://svnweb.freebsd.org/**base/projects/camlock/. Many early changes from that branch are already integrated to head, so to > simplify review the rest patches for changes before r254059 were manually > remade and could be found here: http://people.freebsd.org/~** > mav/camlock_patches/ . > > These changes do not require controller driver modifications, keeping KPIs > and hopefully KBIs intact, but create base for later work to use multiqueue > capabilities of new controllers. > > This work is sponsored by iXsystems, Inc. > > -- > Alexander Motin > ______________________________**_________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/**mailman/listinfo/freebsd-**hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@** > freebsd.org " >