From owner-freebsd-net@FreeBSD.ORG Sat Aug 24 14:16:34 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 7A8C197D; Sat, 24 Aug 2013 14:16:34 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [198.74.231.69]) by mx1.freebsd.org (Postfix) with ESMTP id 557DD205E; Sat, 24 Aug 2013 14:16:34 +0000 (UTC) Received: from fledge.watson.org (fledge.watson.org [198.74.231.63]) by cyrus.watson.org (Postfix) with ESMTPS id 040C446B49; Sat, 24 Aug 2013 10:16:34 -0400 (EDT) Date: Sat, 24 Aug 2013 15:16:33 +0100 (BST) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: "Alexander V. Chernikov" Subject: Re: [rfc] migrate lagg to an rmlock In-Reply-To: <5218AA36.1080807@ipfw.ru> Message-ID: References: <5218AA36.1080807@ipfw.ru> User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: FreeBSD Net , Adrian Chadd , freebsd-current X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 24 Aug 2013 14:16:34 -0000 On Sat, 24 Aug 2013, Alexander V. Chernikov wrote: > On 24.08.2013 00:54, Adrian Chadd wrote: >> >> I'd like to commit this to -10. It migrates the if_lagg locking >> from a rw lock to a rm lock. We see a bit of contention between the >> transmit and > > We're running lagg with rmlock on several hundred heavily loaded machines, > it really works better. However, there should not be any contention between > receive and transmit side since there is actually no _real_ need to lock RX > (and even use lagg receive code at all): > > http://lists.freebsd.org/pipermail/svn-src-all/2013-April/067570.html We should distinguish "lock contention" from "line contention". When acquiring a rwlock on multiple CPUs concurrently, the cache lines used to implement the lock are contended, as they must bounce between caches via the cache coherence protocol, also referred to as "contention". In the if_lagg code, I assume that the read-only acquire of the rwlock (and perhaps now rmlock) is for data stability rather than mutual exclusion -- e.g., to allow processing to completion against a stable version of the lagg configuration. As such, indeed, there should be no lock contention unless a configuration update takes place, and any line contention is a property of the locking primitive rather than data model. There are a number of other places in the kernel where migration to an rmlock makes sense -- however, some care must be taken for four reasons: (1) while read locks don't experience line contention, write locking becomes observably e.g., rmlocks might not be suitable for tcbinfo; (2) rmlocks, unlike rwlocks, more expensive so is not suitable for all rwlock line contention spots -- implement reader priority propagation, so you must reason about; and (3) historically, rmlocks have not fully implemented WITNESS so you may get less good debugging output. if_lagg is a nice place to use rmlocks, as reconfigurations are very rare, and it's really all about long-term data stability. Robert