Date: Sat, 24 Aug 2013 12:21:40 -0600 From: Scott Long <scottl@netflix.com> To: Alfred Perlstein <bright@mu.org> Cc: FreeBSD Net <freebsd-net@freebsd.org>, Adrian Chadd <adrian@freebsd.org>, freebsd-current <freebsd-current@freebsd.org>, "Robert N. M. Watson" <rwatson@FreeBSD.org>, "Alexander V. Chernikov" <melifaro@ipfw.ru> Subject: Re: [rfc] migrate lagg to an rmlock Message-ID: <E263D234-751F-4948-83CE-48F883F85A59@yahoo.com> In-Reply-To: <5218F803.7000405@mu.org> References: <CAJ-Vmo=VKVDEmmPrTbob6Ft%2B7FWypodNoL36Og=7p_CXBSfktg@mail.gmail.com> <5218AA36.1080807@ipfw.ru> <alpine.BSF.2.00.1308241511400.92711@fledge.watson.org> <5218E108.6090901@mu.org> <A4BA90F3-AB8A-41A6-B931-20AF81F903E0@FreeBSD.org> <5218F803.7000405@mu.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Aug 24, 2013, at 12:14 PM, Alfred Perlstein <bright@mu.org> wrote: > On 8/24/13 10:47 AM, Robert N. M. Watson wrote: >> On 24 Aug 2013, at 17:36, Alfred Perlstein wrote: >>=20 >>>> We should distinguish "lock contention" from "line contention". = When acquiring a rwlock on multiple CPUs concurrently, the cache lines = used to implement the lock are contended, as they must bounce between = caches via the cache coherence protocol, also referred to as = "contention". In the if_lagg code, I assume that the read-only acquire = of the rwlock (and perhaps now rmlock) is for data stability rather than = mutual exclusion -- e.g., to allow processing to completion against a = stable version of the lagg configuration. As such, indeed, there should = be no lock contention unless a configuration update takes place, and any = line contention is a property of the locking primitive rather than data = model. >>>>=20 >>>> There are a number of other places in the kernel where migration to = an rmlock makes sense -- however, some care must be taken for four = reasons: (1) while read locks don't experience line contention, write = locking becomes observably e.g., rmlocks might not be suitable for = tcbinfo; (2) rmlocks, unlike rwlocks, more expensive so is not suitable = for all rwlock line contention spots -- implement reader priority = propagation, so you must reason about; and (3) historically, rmlocks = have not fully implemented WITNESS so you may get less good debugging = output. if_lagg is a nice place to use rmlocks, as reconfigurations are = very rare, and it's really all about long-term data stability. >>> Robert, what do you think about a quick swap of the ifnet structures = to counter before 10.x? >> Could you be more specific about the proposal you're making? >>=20 >> Robert >=20 > The lagg patch referred to in the thread seems to indicate that zero = locking is needed if we just switched to counter(9), that makes me = wonder if we could do better with locking in other places if we switched = to counter(9) while we have the chance. >=20 > This is the thread: >=20 > http://lists.freebsd.org/pipermail/svn-src-all/2013-April/067570.html >=20 >> / =20 > />/ Perfect solution would be to convert ifnet(9) to counters(9), = but this > />/ requires much more work, and unfortunately ABI change, so = temporarily > />/ patch lagg(4) manually. > />/ />/ We store counters in the softc, and once per second push = their values > />/ to legacy ifnet counters./ >=20 Some sort of gatekeeper semantic is needed to ensure that configuration = changes to the lagg state don't cause incorrect behavior to the data = path. It's not about protecting the integrity of counters. This can be = done several ways, but right now it's via a read/write semantic. Scott
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?E263D234-751F-4948-83CE-48F883F85A59>