From owner-freebsd-current@FreeBSD.ORG Mon Aug 26 18:45:30 2013 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 2B15D7D4; Mon, 26 Aug 2013 18:45:30 +0000 (UTC) (envelope-from davide.italiano@gmail.com) Received: from mail-vc0-x229.google.com (mail-vc0-x229.google.com [IPv6:2607:f8b0:400c:c03::229]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id A6C80273A; Mon, 26 Aug 2013 18:45:29 +0000 (UTC) Received: by mail-vc0-f169.google.com with SMTP id ib11so2362043vcb.14 for ; Mon, 26 Aug 2013 11:45:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=1eN6suHFYVPG/ZPNbyyUSstgvMT2vDNQnDD+m7G3FIU=; b=PS0HPAq5tNsxGTR6LSYTpce17cMfwJSWvYLZcpilsAmDAywC/1X4ZBUbsA1Z4xr3pc FHfFJ8w+sOwdFxh/L8K/QzYJFIUCr1/3itLxNVMBW5C6I0wsJuUVy887DTrGOfHIfqk/ ilqtOskFYN0z2uzxuCZCIttpRsU/UWp81/p0DdUSPzU0FIPy1fsshTUBCiyeAyLz7CqS l+jI/3wMGG8kdORenv35b84pvs+DyEiK9UWsG7OnnDGw3PNqIEVENYcxCKgUJAPziXGi Ov5L+MNijeuIx7EwOlzrU26BLjrzDti/ndSbLVv9CAIEk5M02L2YDCT1f6o5DemBTSPp tS1A== MIME-Version: 1.0 X-Received: by 10.220.174.200 with SMTP id u8mr16166622vcz.6.1377542728764; Mon, 26 Aug 2013 11:45:28 -0700 (PDT) Sender: davide.italiano@gmail.com Received: by 10.220.65.132 with HTTP; Mon, 26 Aug 2013 11:45:28 -0700 (PDT) In-Reply-To: References: <5218AA36.1080807@ipfw.ru> Date: Mon, 26 Aug 2013 11:45:28 -0700 X-Google-Sender-Auth: uAH71tZSdvuPxJdT5_x8nHM2C0g Message-ID: Subject: Re: [rfc] migrate lagg to an rmlock From: Davide Italiano To: Robert Watson Content-Type: text/plain; charset=ISO-8859-1 Cc: FreeBSD Net , Adrian Chadd , freebsd-current , "Alexander V. Chernikov" X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Aug 2013 18:45:30 -0000 On Sat, Aug 24, 2013 at 7:16 AM, Robert Watson wrote: > On Sat, 24 Aug 2013, Alexander V. Chernikov wrote: > >> On 24.08.2013 00:54, Adrian Chadd wrote: >>> >>> >>> I'd like to commit this to -10. It migrates the if_lagg locking >>> from a rw lock to a rm lock. We see a bit of contention between the >>> transmit and >> >> >> We're running lagg with rmlock on several hundred heavily loaded machines, >> it really works better. However, there should not be any contention between >> receive and transmit side since there is actually no _real_ need to lock RX >> (and even use lagg receive code at all): >> >> http://lists.freebsd.org/pipermail/svn-src-all/2013-April/067570.html > > > We should distinguish "lock contention" from "line contention". When > acquiring a rwlock on multiple CPUs concurrently, the cache lines used to > implement the lock are contended, as they must bounce between caches via the > cache coherence protocol, also referred to as "contention". In the if_lagg > code, I assume that the read-only acquire of the rwlock (and perhaps now > rmlock) is for data stability rather than mutual exclusion -- e.g., to allow > processing to completion against a stable version of the lagg configuration. > As such, indeed, there should be no lock contention unless a configuration > update takes place, and any line contention is a property of the locking > primitive rather than data model. > > There are a number of other places in the kernel where migration to an > rmlock makes sense -- however, some care must be taken for four reasons: (1) > while read locks don't experience line contention, write locking becomes > observably e.g., rmlocks might not be suitable for tcbinfo; (2) rmlocks, > unlike rwlocks, more expensive so is not suitable for all rwlock line > contention spots -- implement reader priority propagation, so you must > reason about; and (3) historically, rmlocks have not fully implemented > WITNESS so you may get less good debugging output. if_lagg is a nice place I'm not sure what you mean here with (3), because from my understanding of the code WITNESS is implemented both in the sleepable and non-sleepable case, but there could be something I'm missing. Something I think we lack in rmlock code is fully supported LOCK_PROFILING as we have in all the other primitives, but again, if I'm wrong feel free to correct me. > to use rmlocks, as reconfigurations are very rare, and it's really all about > long-term data stability. > > Robert > > _______________________________________________ > freebsd-current@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" Thanks, -- Davide "There are no solved problems; there are only problems that are more or less solved" -- Henri Poincare