From owner-freebsd-atm@FreeBSD.ORG Sun May 28 23:04:49 2006 Return-Path: X-Original-To: freebsd-atm@FreeBSD.org Delivered-To: freebsd-atm@FreeBSD.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id AFC3C16AF5B; Sun, 28 May 2006 23:01:01 +0000 (UTC) (envelope-from skip.ford@verizon.net) Received: from vms046pub.verizon.net (vms046pub.verizon.net [206.46.252.46]) by mx1.FreeBSD.org (Postfix) with ESMTP id 6A5FC43D46; Sun, 28 May 2006 23:01:01 +0000 (GMT) (envelope-from skip.ford@verizon.net) Received: from pool-70-17-33-65.pskn.east.verizon.net ([70.17.33.65]) by vms046.mailsrvcs.net (Sun Java System Messaging Server 6.2-4.02 (built Sep 9 2005)) with ESMTPA id <0IZZ00MIDZXOYM86@vms046.mailsrvcs.net>; Sun, 28 May 2006 18:01:01 -0500 (CDT) Date: Sun, 28 May 2006 19:00:59 -0400 From: Skip Ford To: Robert Watson Mail-followup-to: Robert Watson , freebsd-atm@FreeBSD.org, freebsd-arch@FreeBSD.org Message-id: <20060528230058.GA836@lucy.pool-70-17-33-65.pskn.east.verizon.net> MIME-version: 1.0 Content-type: text/plain; charset=us-ascii Content-disposition: inline User-Agent: Mutt/1.4.2.1i Cc: freebsd-atm@FreeBSD.org, freebsd-arch@FreeBSD.org Subject: Locking netatm X-BeenThere: freebsd-atm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: ATM for FreeBSD! List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 28 May 2006 23:04:52 -0000 In my attempt to make netatm MPSAFE, I've run into just a couple situations I'm not sure how to handle. I've more-or-less replaced the splnet() calls with a single subsystem lock. Nearly all of them were protecting structures within the netatm subsystem. But I'm not sure how to handle the existing splimp() calls. There seem to be a few different reasons why splimp() calls were made. First, for timing. Three netatm functions use splimp() to protect access to the list of atm_timeq structures (struct atm_time *). Here is the usage from one of those three functions. I've removed much of the code to simplify what it does: s = splimp(); FOREACH(atm_timeq); ... ... possibly modify structures within atm_timeq ... ... modify the struct atm_time * passed to this function ... (void) splx(s); So my question is, were network interrupts disabled when mucking with the atm_timeq list because a generated interrupt can modify structures within the list? This use is probably very netatm-specific. I'm still studying the timeout code to understand what it's doing. A second situation where network interrupts were disabled was for netatm memory allocation for devices: in atm_dev_alloc() s = splimp(); FOREACH(atm_mem_head) ... malloc (...) (void) splx(s); and in atm_dev_free() s = splimp(); FOREACH(atm_mem_head) ... free (...); (void) splx(s); I'm not sure how these should be protected. Presumably, we don't want to receive interrupts until the netatm memory for the device is allocated. Would a global subsystem lock protect these calls? I can protect atm_mem_head, so maybe that'd be enough? Another use is to protect calls to other subsystems. For example: within atm_nif_attach(struct atm_nif *nip) ifp = nip->nif_ifp; s = splimp(); if_attach(ifp); bpfattach(ifp, DLT_ATM_CLIP, T_ATM_LLC_MAX_LEN); (void) splx(s); } and within atm_nif_detach(struct atm_nif *nip) ifp = nip->nif_ifp; s = splimp(); bpfdetach(ifp); if_detach(ifp); if_free(ifp); (void) splx(s); Holding a new netatm subsystem lock won't protect those calls so I'm not sure how to handle those. Other non-netatm code in the tree seems to not do any locking at all around those calls. These are really the only uses I've yet to convert so if someone can provide some pointers, I'd appreciate it. I'm pretty new to FreeBSD locking, either the old way or the new way. I'm still studying the code, including other network stacks and the netatm stack itself, but a pointer or two would be appreciated. I feel like it's mostly converted, though I've done no testing at all yet. Once I finish removing splimp(), I can test with the single subsystem lock, then move on to finer-grained locking where necessary. -- Skip From owner-freebsd-atm@FreeBSD.ORG Mon May 29 04:54:31 2006 Return-Path: X-Original-To: freebsd-atm@FreeBSD.org Delivered-To: freebsd-atm@FreeBSD.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9D5DE16A508; Mon, 29 May 2006 04:54:31 +0000 (UTC) (envelope-from gnn@neville-neil.com) Received: from mrout2.yahoo.com (mrout2.yahoo.com [216.145.54.172]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5851043D4C; Mon, 29 May 2006 04:54:31 +0000 (GMT) (envelope-from gnn@neville-neil.com) Received: from traveling-laptop-140.corp.yahoo.com.neville-neil.com (proxy7.corp.yahoo.com [216.145.48.98]) by mrout2.yahoo.com (8.13.6/8.13.4/y.out) with ESMTP id k4T4quVX018117; Sun, 28 May 2006 21:52:57 -0700 (PDT) Date: Mon, 29 May 2006 13:52:48 +0900 Message-ID: From: gnn@FreeBSD.org To: Robert Watson , freebsd-atm@FreeBSD.org, freebsd-arch@FreeBSD.org In-Reply-To: <20060528230058.GA836@lucy.pool-70-17-33-65.pskn.east.verizon.net> References: <20060528230058.GA836@lucy.pool-70-17-33-65.pskn.east.verizon.net> User-Agent: Wanderlust/2.14.0 (Africa) SEMI/1.14.6 (Maruoka) FLIM/1.14.8 (=?ISO-8859-4?Q?Shij=F2?=) APEL/10.6 Emacs/22.0.50 (i386-apple-darwin8.5.1) MULE/5.0 (SAKAKI) MIME-Version: 1.0 (generated by SEMI 1.14.6 - "Maruoka") Content-Type: text/plain; charset=US-ASCII Cc: Subject: Re: Locking netatm X-BeenThere: freebsd-atm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: ATM for FreeBSD! List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 May 2006 04:54:31 -0000 At Sun, 28 May 2006 19:00:59 -0400, Skip Ford wrote: > > In my attempt to make netatm MPSAFE, I've run into just a couple > situations I'm not sure how to handle. I've more-or-less replaced > the splnet() calls with a single subsystem lock. Nearly all of them > were protecting structures within the netatm subsystem. But I'm not > sure how to handle the existing splimp() calls. > Hi Skip, I'm not familiar with netatm but I am somewhat familiar with our locking code, and I hope that others will also join in. > There seem to be a few different reasons why splimp() calls were > made. First, for timing. Three netatm functions use splimp() to > protect access to the list of atm_timeq structures (struct atm_time > *). Here is the usage from one of those three functions. I've > removed much of the code to simplify what it does: > > s = splimp(); > > FOREACH(atm_timeq); > ... > > ... possibly modify structures within atm_timeq ... > ... modify the struct atm_time * passed to this function ... > > (void) splx(s); > > So my question is, were network interrupts disabled when mucking > with the atm_timeq list because a generated interrupt can modify > structures within the list? This use is probably very > netatm-specific. I'm still studying the timeout code to understand > what it's doing. > The spl() calls haven't disabled real interrupts, as far as I know, for quite a while. They acted as general code locks to prevent simultaneous access to data structures while an update was in progress. In terms of the timeq, the locks were acting as a mutex now would to protect the list during an update. > A second situation where network interrupts were disabled was for > netatm memory allocation for devices: > > in atm_dev_alloc() > > s = splimp(); > > FOREACH(atm_mem_head) > ... > malloc (...) > > (void) splx(s); > > and in atm_dev_free() > > s = splimp(); > > FOREACH(atm_mem_head) > ... > free (...); > > (void) splx(s); > > I'm not sure how these should be protected. Presumably, we don't > want to receive interrupts until the netatm memory for the device is > allocated. Would a global subsystem lock protect these calls? I > can protect atm_mem_head, so maybe that'd be enough? I would assume, again, that these are prtections of the list, and not the memory allocation routines. A mutex protecting the list, like the one that protects, say, the UDP protocol list is probably what you want. > Another use is to protect calls to other subsystems. For > example: > > within atm_nif_attach(struct atm_nif *nip) > > ifp = nip->nif_ifp; > > s = splimp(); > > if_attach(ifp); > bpfattach(ifp, DLT_ATM_CLIP, T_ATM_LLC_MAX_LEN); > > (void) splx(s); > } > > and within atm_nif_detach(struct atm_nif *nip) > > ifp = nip->nif_ifp; > > s = splimp(); > > bpfdetach(ifp); > if_detach(ifp); > if_free(ifp); > > (void) splx(s); > > Holding a new netatm subsystem lock won't protect those calls so I'm > not sure how to handle those. Other non-netatm code in the tree > seems to not do any locking at all around those calls. I believe these are now unnecessary since the ifnet lists now have their own locks. > These are really the only uses I've yet to convert so if someone can > provide some pointers, I'd appreciate it. I'm pretty new to FreeBSD > locking, either the old way or the new way. I'm still studying the > code, including other network stacks and the netatm stack itself, > but a pointer or two would be appreciated. I feel like it's mostly > converted, though I've done no testing at all yet. Once I finish > removing splimp(), I can test with the single subsystem lock, then > move on to finer-grained locking where necessary. I would look at the UDP and TCP protocol list locking as being somewhat similar to what you have here. UDP is the easiest to understand as it's mostly in one file, netinet/udp_usrreq.c. Later, George From owner-freebsd-atm@FreeBSD.ORG Mon May 29 05:18:13 2006 Return-Path: X-Original-To: freebsd-atm@FreeBSD.org Delivered-To: freebsd-atm@FreeBSD.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2614716A50F; Mon, 29 May 2006 05:18:13 +0000 (UTC) (envelope-from kris@obsecurity.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.FreeBSD.org (Postfix) with ESMTP id D961F43D46; Mon, 29 May 2006 05:18:12 +0000 (GMT) (envelope-from kris@obsecurity.org) Received: from obsecurity.dyndns.org (elvis.mu.org [192.203.228.196]) by elvis.mu.org (Postfix) with ESMTP id A99D61A3C1F; Sun, 28 May 2006 22:18:12 -0700 (PDT) Received: by obsecurity.dyndns.org (Postfix, from userid 1000) id 11B4D5153E; Mon, 29 May 2006 01:18:12 -0400 (EDT) Date: Mon, 29 May 2006 01:18:11 -0400 From: Kris Kennaway To: gnn@FreeBSD.org Message-ID: <20060529051811.GA60877@xor.obsecurity.org> References: <20060528230058.GA836@lucy.pool-70-17-33-65.pskn.east.verizon.net> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="Kj7319i9nmIyA2yE" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.1i Cc: freebsd-atm@FreeBSD.org, Robert Watson , freebsd-arch@FreeBSD.org Subject: Re: Locking netatm X-BeenThere: freebsd-atm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: ATM for FreeBSD! List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 May 2006 05:18:13 -0000 --Kj7319i9nmIyA2yE Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, May 29, 2006 at 01:52:48PM +0900, gnn@FreeBSD.org wrote: > > So my question is, were network interrupts disabled when mucking > > with the atm_timeq list because a generated interrupt can modify > > structures within the list? This use is probably very > > netatm-specific. I'm still studying the timeout code to understand > > what it's doing. > >=20 >=20 > The spl() calls haven't disabled real interrupts, as far as I know, > for quite a while. They acted as general code locks to prevent > simultaneous access to data structures while an update was in > progress. In terms of the timeq, the locks were acting as a mutex now > would to protect the list during an update. Actually the spl calls have all been NOPs since the early 5.0 days, and they were just left in an mnemonic placeholders showing which code/data sections need to be protected. Kris --Kj7319i9nmIyA2yE Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.3 (FreeBSD) iD8DBQFEeoQTWry0BWjoQKURAk94AJ41qSIQle1p3QJpxfY+eZ6nrN21jgCeLlvP auY7U0HK7miLNQHDLpfOgMc= =WY96 -----END PGP SIGNATURE----- --Kj7319i9nmIyA2yE-- From owner-freebsd-atm@FreeBSD.ORG Mon May 29 12:02:57 2006 Return-Path: X-Original-To: freebsd-atm@freebsd.org Delivered-To: freebsd-atm@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4046F16A93E; Mon, 29 May 2006 12:02:57 +0000 (UTC) (envelope-from bde@zeta.org.au) Received: from mailout1.pacific.net.au (mailout1.pacific.net.au [61.8.0.84]) by mx1.FreeBSD.org (Postfix) with ESMTP id A388143D7B; Mon, 29 May 2006 12:02:42 +0000 (GMT) (envelope-from bde@zeta.org.au) Received: from mailproxy2.pacific.net.au (mailproxy2.pacific.net.au [61.8.0.87]) by mailout1.pacific.net.au (Postfix) with ESMTP id A60353283C3; Mon, 29 May 2006 22:02:41 +1000 (EST) Received: from katana.zip.com.au (katana.zip.com.au [61.8.7.246]) by mailproxy2.pacific.net.au (8.13.4/8.13.4/Debian-3sarge1) with ESMTP id k4TC2c3T028279; Mon, 29 May 2006 22:02:39 +1000 Date: Mon, 29 May 2006 22:02:38 +1000 (EST) From: Bruce Evans X-X-Sender: bde@delplex.bde.org To: Kris Kennaway In-Reply-To: <20060529051811.GA60877@xor.obsecurity.org> Message-ID: <20060529213903.G24267@delplex.bde.org> References: <20060528230058.GA836@lucy.pool-70-17-33-65.pskn.east.verizon.net> <20060529051811.GA60877@xor.obsecurity.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: Robert Watson , gnn@freebsd.org, freebsd-atm@freebsd.org, freebsd-arch@freebsd.org Subject: Re: Locking netatm X-BeenThere: freebsd-atm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: ATM for FreeBSD! List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 May 2006 12:03:16 -0000 On Mon, 29 May 2006, Kris Kennaway wrote: > On Mon, May 29, 2006 at 01:52:48PM +0900, gnn@FreeBSD.org wrote: >>> So my question is, were network interrupts disabled when mucking >>> with the atm_timeq list because a generated interrupt can modify >>> structures within the list? This use is probably very >>> netatm-specific. I'm still studying the timeout code to understand >>> what it's doing. >> >> The spl() calls haven't disabled real interrupts, as far as I know, >> for quite a while. They acted as general code locks to prevent >> simultaneous access to data structures while an update was in >> progress. In terms of the timeq, the locks were acting as a mutex now >> would to protect the list during an update. > > Actually the spl calls have all been NOPs since the early 5.0 days, > and they were just left in an mnemonic placeholders showing which > code/data sections need to be protected. Actually the spl calls were sort of replaced by Giant locking. The Giant locking is just implicit and magic where the non-null spl "locking" was explicit and less magic. The effect was initially similar to replacing all spl calls by a non-null splhigh() (giving very coarse-grained locking to interrupt handlers) and also adding splhigh() to all kernel entry points (giving even coarser-grained locking). Now many things are not under Giant, the Giant-locked things are not locked out so much by other things, but they still lock out each other and still depend very much on the Giant locking having much the same effect as splhigh(). To the extent that subsystems called by netatm are MPSAFE, you can just remove the spl "locking". It hasn't done anything for a long time, and the Giant locking is also null for other MPSAFE subsystems. However, the other subsystems might not be fully MPSAFE yet. Some have to be run in !MPSAFE mode because their callers are not MPSAFE, and this might hide bugs. Locking for timing probably isn't very important. Giant and other locking can give very large latency in hard-to-predict ways, but some things have been running for a long time without this causing many obvious problems other than reduced efficiency. Bruce From owner-freebsd-atm@FreeBSD.ORG Mon May 29 13:26:03 2006 Return-Path: X-Original-To: freebsd-atm@freebsd.org Delivered-To: freebsd-atm@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 06FDA16A4F1 for ; Mon, 29 May 2006 13:26:03 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: from wx-out-0102.google.com (wx-out-0102.google.com [66.249.82.200]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5A23943D4C for ; Mon, 29 May 2006 13:26:01 +0000 (GMT) (envelope-from asmrookie@gmail.com) Received: by wx-out-0102.google.com with SMTP id i31so133705wxd for ; Mon, 29 May 2006 06:26:00 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:message-id:date:from:reply-to:to:subject:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=rlb9n0e5e1uTN9lig42k7Cmb6Qii8Kxu/wZEZZcbhSCVLlfWOV6VJLI+xHpsxXaxrZSKplnzhWCrS2Q4SDGI7vAGw5AQ0JF2QnZwbNuAnJuHuhH3eXp5kPqr9i9GSe4d3w8jk2DbFw2mdranAvWjvkDbpefWPFdieKd28i83pXM= Received: by 10.70.54.11 with SMTP id c11mr2322899wxa; Mon, 29 May 2006 06:25:59 -0700 (PDT) Received: by 10.70.11.2 with HTTP; Mon, 29 May 2006 06:25:59 -0700 (PDT) Message-ID: <3bbf2fe10605290625p63221f95i6e17466ab5db6812@mail.gmail.com> Date: Mon, 29 May 2006 15:25:59 +0200 From: "Attilio Rao" To: "Robert Watson" , freebsd-atm@freebsd.org, freebsd-arch@freebsd.org In-Reply-To: <20060528230058.GA836@lucy.pool-70-17-33-65.pskn.east.verizon.net> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: base64 Content-Disposition: inline References: <20060528230058.GA836@lucy.pool-70-17-33-65.pskn.east.verizon.net> Cc: Subject: Re: Locking netatm X-BeenThere: freebsd-atm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: rookie@gufi.org List-Id: ATM for FreeBSD! List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 May 2006 13:26:03 -0000 MjAwNi81LzI5LCBTa2lwIEZvcmQgPHNraXAuZm9yZEB2ZXJpem9uLm5ldD46Cgpbc25pcF0KCj4g U28gbXkgcXVlc3Rpb24gaXMsIHdlcmUgbmV0d29yayBpbnRlcnJ1cHRzIGRpc2FibGVkIHdoZW4g bXVja2luZwo+IHdpdGggdGhlIGF0bV90aW1lcSBsaXN0IGJlY2F1c2UgYSBnZW5lcmF0ZWQgaW50 ZXJydXB0IGNhbiBtb2RpZnkKPiBzdHJ1Y3R1cmVzIHdpdGhpbiB0aGUgbGlzdD8gIFRoaXMgdXNl IGlzIHByb2JhYmx5IHZlcnkKPiBuZXRhdG0tc3BlY2lmaWMuICBJJ20gc3RpbGwgc3R1ZHlpbmcg dGhlIHRpbWVvdXQgY29kZSB0bwo+IHVuZGVyc3RhbmQgd2hhdCBpdCdzIGRvaW5nLgoKVGhpcyBp cyB0aGUgd2hvbGUgcG9pbnQgb2YgbG9ja2luZyBwcmltaXRpdmVzOiBhdG9taWMgb3BlcmF0aW9u cy4KSW50ZXJydXB0aW5nIHRoZSBjb2RlIHdoaWxlIGdldHRpbmcgYSBsb2NrIGNvdWxkIHJlc3Vs dCBpbiBhIHJhY2UgY29uZGl0aW9uLgoKSWYgeW91IGhhdmUgdG8gcHJvdGVjdCBkYXRhcyB0aHJv dWdoIGEgVFNMIChUZXN0IGFuZCBTZXQgTG9jaykKb3BlcmF0aW9uLCB3aGF0IGhhcHBlbiBpcyBz b21ldGhpbmcgbGlrZSAoaHlwb3RldGljYWxseSk6CgppZiAobG9jayA9PSAwKQogICAgbG9jayA9 IDE7CmVsc2UKICAgdGhyZWFkX3NsZWVwKCk7CgpUaGFuIHlvdSBpbW1hZ2luZSB0aHJlYWRBIHZl cmlmeSBjb21wYXJpc29ucyBhcyB0cnVlLCBnZXQgaW50ZXJydXB0ZWQKYnkgaW50ZXJydXB0IHRp bWVyIGFuZCB0aHJlYWRCIGlzIHNjaGVkdWxlZCB0byBydW4uIHRocmVhZEIgZXhlY3V0ZXMKdGhl IHdob2xlIHByb2NlZHVyZSAoc2V0dGluZyBsb2NrID0gMSkgdGhhbiB0aHJlYWRBIGlzIHNjaGVk dWxlZCBhZ2Fpbgp0byBydW4gYW5kIHRoaW5rcyB0aGUgY3JpdGljYWwgc2VjdGlvbiBpcyBmcmVl IChzaW5jZSBpdHMgY29tcGFyaXNvbgp3YXMgdHJ1ZSkgc28gc2V0cyBhZ2FpbiBsb2NrPTEgYW5k IGdvIG9uLiBBdCB0aGlzIHBvaW50IHlvdSBoYXZlIGFuCmluY29ycmVjdCBiZWhhdmlvdXIgZm9y IHlvdXIgT1MgKGEgcmFjZSBjb25kaXRpb24pLiBUaGlzIGlzIHdoeQppbnRlcnJ1cHRzIGFyZSBw YWluZnVsIGZvciBhdG9taWMgb3BlcmF0aW9ucyBhbmQgd2h5IHRoZXkncmUgbmV2ZXIKYWxsb3dl ZC4KCj4gQSBzZWNvbmQgc2l0dWF0aW9uIHdoZXJlIG5ldHdvcmsgaW50ZXJydXB0cyB3ZXJlIGRp c2FibGVkIHdhcyBmb3IKPiBuZXRhdG0gbWVtb3J5IGFsbG9jYXRpb24gZm9yIGRldmljZXM6Cj4K PiBpbiBhdG1fZGV2X2FsbG9jKCkKPgo+ICAgICAgICBzID0gc3BsaW1wKCk7Cj4KPiAgICAgICAg Rk9SRUFDSChhdG1fbWVtX2hlYWQpCj4gICAgICAgICAgICAgICAgLi4uCj4gICAgICAgIG1hbGxv YyAoLi4uKQo+Cj4gICAgICAgICh2b2lkKSBzcGx4KHMpOwo+Cj4gYW5kIGluIGF0bV9kZXZfZnJl ZSgpCj4KPiAgICAgICAgcyA9IHNwbGltcCgpOwo+Cj4gICAgICAgIEZPUkVBQ0goYXRtX21lbV9o ZWFkKQo+ICAgICAgICAgICAgICAgIC4uLgo+ICAgICAgICBmcmVlICguLi4pOwo+Cj4gICAgICAg ICh2b2lkKSBzcGx4KHMpOwo+Cj4gSSdtIG5vdCBzdXJlIGhvdyB0aGVzZSBzaG91bGQgYmUgcHJv dGVjdGVkLiAgUHJlc3VtYWJseSwgd2UgZG9uJ3QKPiB3YW50IHRvIHJlY2VpdmUgaW50ZXJydXB0 cyB1bnRpbCB0aGUgbmV0YXRtIG1lbW9yeSBmb3IgdGhlCj4gZGV2aWNlIGlzIGFsbG9jYXRlZC4g IFdvdWxkIGEgZ2xvYmFsIHN1YnN5c3RlbSBsb2NrIHByb3RlY3QgdGhlc2UKPiBjYWxscz8gIEkg Y2FuIHByb3RlY3QgYXRtX21lbV9oZWFkLCBzbyBtYXliZSB0aGF0J2QgYmUgZW5vdWdoPwoKVGhl IGJldHRlciB3YXkgdG8gcHJvdGVjdCBhIGxpc3QgaXMgYSByd2xvY2suCkV2ZW4gaWYgdGhleSdy ZSBzdGlsbCBpbmNvbXBsZXRlLCBhY3R1YWxseSwgdGhleSdyZSB0aGUgb25seSBwcmltaXRpdmUK d2hpY2ggY2FuIGFzc3VyZSBhIHR3by1sZXZlbHMgcHJvdGVjdGlvbiBhbmQgYSBzb3J0IG9mIHBy aW9yaXR5CnByb3BhZ2F0aW9uLCBzbyBpdCB3b3VsZCBiZSBwcmVmZXJhYmxlIHVzaW5nIGl0ICho b2xkaW5nIGFjY29yZGluZyB0bwp5b3VyIG9wZXJhdGlvbnMpLgoKSSB0aGluayB0aGF0IGEgbG90 IG9mIHF1ZXVlcyB0aGF0IHdlIGhhdmUgcHJvdGVjdGVkIGJ5IGEgc3ggbG9jayBtaWdodApzd2l0 Y2ggdG8gcndsb2Nrcy4uLgoKPiBBbm90aGVyIHVzZSBpcyB0byBwcm90ZWN0IGNhbGxzIHRvIG90 aGVyIHN1YnN5c3RlbXMuICBGb3IKPiBleGFtcGxlOgo+Cj4gd2l0aGluIGF0bV9uaWZfYXR0YWNo KHN0cnVjdCBhdG1fbmlmICpuaXApCj4KPiAgICAgICAgaWZwID0gbmlwLT5uaWZfaWZwOwo+Cj4g ICAgICAgIHMgPSBzcGxpbXAoKTsKPgo+ICAgICAgICBpZl9hdHRhY2goaWZwKTsKPiAgICAgICAg YnBmYXR0YWNoKGlmcCwgRExUX0FUTV9DTElQLCBUX0FUTV9MTENfTUFYX0xFTik7Cj4KPiAgICAg ICAgKHZvaWQpIHNwbHgocyk7Cj4gfQo+Cj4gYW5kIHdpdGhpbiBhdG1fbmlmX2RldGFjaChzdHJ1 Y3QgYXRtX25pZiAqbmlwKQo+Cj4gICAgICAgIGlmcCA9IG5pcC0+bmlmX2lmcDsKPgo+ICAgICAg ICBzID0gc3BsaW1wKCk7Cj4KPiAgICAgICAgYnBmZGV0YWNoKGlmcCk7Cj4gICAgICAgIGlmX2Rl dGFjaChpZnApOwo+ICAgICAgICBpZl9mcmVlKGlmcCk7Cj4KPiAgICAgICAgKHZvaWQpIHNwbHgo cyk7Cj4KPiBIb2xkaW5nIGEgbmV3IG5ldGF0bSBzdWJzeXN0ZW0gbG9jayB3b24ndCBwcm90ZWN0 IHRob3NlIGNhbGxzIHNvCj4gSSdtIG5vdCBzdXJlIGhvdyB0byBoYW5kbGUgdGhvc2UuICBPdGhl ciBub24tbmV0YXRtIGNvZGUgaW4gdGhlCj4gdHJlZSBzZWVtcyB0byBub3QgZG8gYW55IGxvY2tp bmcgYXQgYWxsIGFyb3VuZCB0aG9zZSBjYWxscy4KCkhlcmUgYSBtdXRleCg5KSBpcyBlbm91Z2gu Cgo+IFRoZXNlIGFyZSByZWFsbHkgdGhlIG9ubHkgdXNlcyBJJ3ZlIHlldCB0byBjb252ZXJ0IHNv IGlmIHNvbWVvbmUKPiBjYW4gcHJvdmlkZSBzb21lIHBvaW50ZXJzLCBJJ2QgYXBwcmVjaWF0ZSBp dC4gIEknbSBwcmV0dHkgbmV3IHRvCj4gRnJlZUJTRCBsb2NraW5nLCBlaXRoZXIgdGhlIG9sZCB3 YXkgb3IgdGhlIG5ldyB3YXkuICBJJ20gc3RpbGwKPiBzdHVkeWluZyB0aGUgY29kZSwgaW5jbHVk aW5nIG90aGVyIG5ldHdvcmsgc3RhY2tzIGFuZCB0aGUgbmV0YXRtCj4gc3RhY2sgaXRzZWxmLCBi dXQgYSBwb2ludGVyIG9yIHR3byB3b3VsZCBiZSBhcHByZWNpYXRlZC4gIEkgZmVlbAo+IGxpa2Ug aXQncyBtb3N0bHkgY29udmVydGVkLCB0aG91Z2ggSSd2ZSBkb25lIG5vIHRlc3RpbmcgYXQgYWxs Cj4geWV0LiAgT25jZSBJIGZpbmlzaCByZW1vdmluZyBzcGxpbXAoKSwgSSBjYW4gdGVzdCB3aXRo IHRoZSBzaW5nbGUKPiBzdWJzeXN0ZW0gbG9jaywgdGhlbiBtb3ZlIG9uIHRvIGZpbmVyLWdyYWlu ZWQgbG9ja2luZyB3aGVyZQo+IG5lY2Vzc2FyeS4KClRoZXJlIGlzIG5vdCBhIGxvdCBvZiB1cGRh dGUgZG9jdW1lbnRhdGlvbiBhYm91dCBpdCwgSSBjYW4gc2F5IHlvdSBpbgpwYXJ0aWN1bGFyOgpo dHRwOi8vd3d3LmZyZWVic2Qub3JnL2RvYy9lbi9ib29rcy9hcmNoLWhhbmRib29rL2xvY2tpbmcu aHRtbApodHRwOi8vd3d3LmxlbWlzLmNvbS9ncm9nL1NNUG5nL1NpbmdhcG9yZS9wYXBlci5wZGYK aHR0cDovL3Blb3BsZS5mcmVlYnNkLm9yZy9+ZnNtcC9TTVAvU01QLmh0bWwKCkJUVywgc29tZSBv ZiB0aGlzIGlzIG5vdCBjdXJyZW50bHkgYXBwbGllZCBpbnRvIHRoZSBrZXJuZWwgc28gbG9vayBh dAp0aGUgY29kZSBhcyB1c3VhbCA6UAoKQXR0aWxpbwoKLS0gClBlYWNlIGNhbiBvbmx5IGJlIGFj aGlldmVkIGJ5IHVuZGVyc3RhbmRpbmcgLSBBLiBFaW5zdGVpbgo=