From owner-freebsd-arch@FreeBSD.ORG Sun Mar 23 01:16:34 2008 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B7A2E106564A; Sun, 23 Mar 2008 01:16:34 +0000 (UTC) (envelope-from dillon@apollo.backplane.com) Received: from apollo.backplane.com (apollo.backplane.com [216.240.41.2]) by mx1.freebsd.org (Postfix) with ESMTP id 99E3A8FC12; Sun, 23 Mar 2008 01:16:34 +0000 (UTC) (envelope-from dillon@apollo.backplane.com) Received: from apollo.backplane.com (localhost [127.0.0.1]) by apollo.backplane.com (8.14.1/8.13.7) with ESMTP id m2N14xWK026185; Sat, 22 Mar 2008 18:04:59 -0700 (PDT) Received: (from dillon@localhost) by apollo.backplane.com (8.14.1/8.13.4/Submit) id m2N14xlf026184; Sat, 22 Mar 2008 18:04:59 -0700 (PDT) Date: Sat, 22 Mar 2008 18:04:59 -0700 (PDT) From: Matthew Dillon Message-Id: <200803230104.m2N14xlf026184@apollo.backplane.com> To: Robert Watson References: <20080322105145.GA41672@team.vega.ru> <20080322135637.Y6961@fledge.watson.org> Cc: arch@freebsd.org Subject: Re: Disposal of a misleading M_TRYWAIT X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Mar 2008 01:16:34 -0000 :> Since the advent of MBUMA in FreeBSD (whatever), M_TRYWAIT has meant :> M_WAITOK. (The reason for M_TRYWAIT itself was that an original mbuf's :> M_WAIT could return NULL.) :> :.. : :This seems reasonable to me for exactly the reasons you stte. We might :simultaneously want to complete the M_DONTWAIT -> M_NOWAIT conversion. And :you can then remove the XXX comment in mbuf.h about phasing out M_TRYWAIT and :M_DONTWAIT. :-) : :Robert N M Watson The real issue is the fact that both the kernel malloc and the mbuf allocation APIs are using the same M_ prefix for their flags. We converted our mbuf allocator flags (aka M_DONTWAIT, M_TRYWAIT, M_WAIT) from M_ to MB_ and the code became a whole lot easier to read. I would not recommend converting the mbuf allocator to actually *USE* kernel malloc flags. The problem there is that you then have no clear delineation between M_ flags supported by malloc and M_ flags supported by the mbuf allocator. -Matt From owner-freebsd-arch@FreeBSD.ORG Sun Mar 23 01:36:17 2008 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2D3911065673; Sun, 23 Mar 2008 01:36:17 +0000 (UTC) (envelope-from jroberson@chesapeake.net) Received: from webaccess-cl.virtdom.com (webaccess-cl.virtdom.com [216.240.101.25]) by mx1.freebsd.org (Postfix) with ESMTP id C01808FC1A; Sun, 23 Mar 2008 01:36:16 +0000 (UTC) (envelope-from jroberson@chesapeake.net) Received: from [10.0.1.200] (cpe-24-94-72-120.hawaii.res.rr.com [24.94.72.120]) (authenticated bits=0) by webaccess-cl.virtdom.com (8.13.6/8.13.6) with ESMTP id m2N1aAYq026125; Sat, 22 Mar 2008 21:36:11 -0400 (EDT) (envelope-from jroberson@chesapeake.net) Date: Sat, 22 Mar 2008 15:37:23 -1000 (HST) From: Jeff Roberson X-X-Sender: jroberson@desktop To: Matthew Dillon In-Reply-To: <200803230104.m2N14xlf026184@apollo.backplane.com> Message-ID: <20080322153658.K910@desktop> References: <20080322105145.GA41672@team.vega.ru> <20080322135637.Y6961@fledge.watson.org> <200803230104.m2N14xlf026184@apollo.backplane.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: arch@freebsd.org, Robert Watson Subject: Re: Disposal of a misleading M_TRYWAIT X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Mar 2008 01:36:17 -0000 On Sat, 22 Mar 2008, Matthew Dillon wrote: > > :> Since the advent of MBUMA in FreeBSD (whatever), M_TRYWAIT has meant > :> M_WAITOK. (The reason for M_TRYWAIT itself was that an original mbuf's > :> M_WAIT could return NULL.) > :> > :.. > : > :This seems reasonable to me for exactly the reasons you stte. We might > :simultaneously want to complete the M_DONTWAIT -> M_NOWAIT conversion. And > :you can then remove the XXX comment in mbuf.h about phasing out M_TRYWAIT and > :M_DONTWAIT. :-) > : > :Robert N M Watson > > The real issue is the fact that both the kernel malloc and the mbuf > allocation APIs are using the same M_ prefix for their flags. > > We converted our mbuf allocator flags (aka M_DONTWAIT, M_TRYWAIT, M_WAIT) > from M_ to MB_ and the code became a whole lot easier to read. > > I would not recommend converting the mbuf allocator to actually *USE* > kernel malloc flags. The problem there is that you then have no clear > delineation between M_ flags supported by malloc and M_ flags supported > by the mbuf allocator. They have been the same allocator for some time now. It makes more sense for them to use the same flags. Jeff > > -Matt > > _______________________________________________ > freebsd-arch@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-arch > To unsubscribe, send any mail to "freebsd-arch-unsubscribe@freebsd.org" > From owner-freebsd-arch@FreeBSD.ORG Sun Mar 23 04:37:13 2008 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8C54F106566B for ; Sun, 23 Mar 2008 04:37:13 +0000 (UTC) (envelope-from sam@freebsd.org) Received: from ebb.errno.com (ebb.errno.com [69.12.149.25]) by mx1.freebsd.org (Postfix) with ESMTP id 5FC978FC21 for ; Sun, 23 Mar 2008 04:37:13 +0000 (UTC) (envelope-from sam@freebsd.org) Received: from Macintosh-2.local ([10.0.0.196]) (authenticated bits=0) by ebb.errno.com (8.13.6/8.12.6) with ESMTP id m2N41NnF028309 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sat, 22 Mar 2008 21:01:23 -0700 (PDT) (envelope-from sam@freebsd.org) Message-ID: <47E5D613.4060005@freebsd.org> Date: Sat, 22 Mar 2008 21:01:23 -0700 From: Sam Leffler Organization: FreeBSD Project User-Agent: Thunderbird 2.0.0.12 (Macintosh/20080213) MIME-Version: 1.0 To: Jeff Roberson References: <20080322105145.GA41672@team.vega.ru> <20080322135637.Y6961@fledge.watson.org> <200803230104.m2N14xlf026184@apollo.backplane.com> <20080322153658.K910@desktop> In-Reply-To: <20080322153658.K910@desktop> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-DCC-sonic.net-Metrics: ebb.errno.com; whitelist Cc: arch@freebsd.org Subject: Re: Disposal of a misleading M_TRYWAIT X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Mar 2008 04:37:13 -0000 Jeff Roberson wrote: > On Sat, 22 Mar 2008, Matthew Dillon wrote: > >> >> :> Since the advent of MBUMA in FreeBSD (whatever), M_TRYWAIT has meant >> :> M_WAITOK. (The reason for M_TRYWAIT itself was that an original >> mbuf's >> :> M_WAIT could return NULL.) >> :> >> :.. >> : >> :This seems reasonable to me for exactly the reasons you stte. We might >> :simultaneously want to complete the M_DONTWAIT -> M_NOWAIT >> conversion. And >> :you can then remove the XXX comment in mbuf.h about phasing out >> M_TRYWAIT and >> :M_DONTWAIT. :-) >> : >> :Robert N M Watson >> >> The real issue is the fact that both the kernel malloc and the mbuf >> allocation APIs are using the same M_ prefix for their flags. >> >> We converted our mbuf allocator flags (aka M_DONTWAIT, M_TRYWAIT, >> M_WAIT) >> from M_ to MB_ and the code became a whole lot easier to read. >> >> I would not recommend converting the mbuf allocator to actually *USE* >> kernel malloc flags. The problem there is that you then have no clear >> delineation between M_ flags supported by malloc and M_ flags >> supported >> by the mbuf allocator. > > They have been the same allocator for some time now. It makes more > sense for them to use the same flags. We made a decision a while back to not use malloc flags for mbuf routine arguments. There are even assertions to verify it. Changing the flag names however would be a painful change for little gain. Sam From owner-freebsd-arch@FreeBSD.ORG Sun Mar 23 09:08:33 2008 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DF0F1106566C for ; Sun, 23 Mar 2008 09:08:33 +0000 (UTC) (envelope-from rermilov@team.vega.ru) Received: from mail.vega.ru (infra.dev.vega.ru [90.156.167.14]) by mx1.freebsd.org (Postfix) with ESMTP id 8A6F78FC16 for ; Sun, 23 Mar 2008 09:08:33 +0000 (UTC) (envelope-from rermilov@team.vega.ru) Received: from [87.242.97.68] (port=64187 helo=edoofus.dev.vega.ru) by mail.vega.ru with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.68 (FreeBSD)) (envelope-from ) id 1JdMBq-0003nk-1M; Sun, 23 Mar 2008 09:08:30 +0000 Received: from edoofus.dev.vega.ru (localhost [127.0.0.1]) by edoofus.dev.vega.ru (8.14.2/8.14.2) with ESMTP id m2N98OZd037359; Sun, 23 Mar 2008 12:08:24 +0300 (MSK) (envelope-from rermilov@team.vega.ru) Received: (from ru@localhost) by edoofus.dev.vega.ru (8.14.2/8.14.2/Submit) id m2N98Nuq037358; Sun, 23 Mar 2008 12:08:23 +0300 (MSK) (envelope-from rermilov@team.vega.ru) X-Authentication-Warning: edoofus.dev.vega.ru: ru set sender to rermilov@team.vega.ru using -f Date: Sun, 23 Mar 2008 12:08:23 +0300 From: Ruslan Ermilov To: Sam Leffler Message-ID: <20080323090823.GA37326@team.vega.ru> References: <20080322105145.GA41672@team.vega.ru> <20080322135637.Y6961@fledge.watson.org> <200803230104.m2N14xlf026184@apollo.backplane.com> <20080322153658.K910@desktop> <47E5D613.4060005@freebsd.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <47E5D613.4060005@freebsd.org> User-Agent: Mutt/1.5.17 (2007-11-01) Cc: arch@freebsd.org Subject: Re: Disposal of a misleading M_TRYWAIT X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Mar 2008 09:08:34 -0000 On Sat, Mar 22, 2008 at 09:01:23PM -0700, Sam Leffler wrote: > We made a decision a while back to not use malloc flags for mbuf routine > arguments. While I can see the point in this (if we ever change the mbuf allocator from UMA to something else), like I said in my email, a lot of code already uses malloc flags for this. > There are even assertions to verify it. I think this was OBE, now mbuf.h says: : /* : * Flags specifying how an allocation should be made. : * : * The flag to use is as follows: : * - M_DONTWAIT or M_NOWAIT from an interrupt handler to not block allocation. : * - M_WAIT or M_WAITOK from wherever it is safe to block. : * : * M_DONTWAIT/M_NOWAIT means that we will not block the thread explicitly and : * if we cannot allocate immediately we may return NULL, whereas : * M_WAIT/M_WAITOK means that if we cannot allocate resources we : * will block until they are available, and thus never return NULL. : * : * XXX Eventually just phase this out to use M_WAITOK/M_NOWAIT. : */ : #define MBTOM(how) (how) : #define M_DONTWAIT M_NOWAIT : #define M_TRYWAIT M_WAITOK : #define M_WAIT M_WAITOK Cheers, -- Ruslan Ermilov ru@FreeBSD.org FreeBSD committer From owner-freebsd-arch@FreeBSD.ORG Sun Mar 23 09:15:38 2008 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3EB7A106566B; Sun, 23 Mar 2008 09:15:38 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.freebsd.org (Postfix) with ESMTP id 0E3E08FC19; Sun, 23 Mar 2008 09:15:38 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id 41C9446B6C; Sun, 23 Mar 2008 05:15:37 -0400 (EDT) Date: Sun, 23 Mar 2008 09:15:37 +0000 (GMT) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: Sam Leffler In-Reply-To: <47E5D613.4060005@freebsd.org> Message-ID: <20080323091107.Y79412@fledge.watson.org> References: <20080322105145.GA41672@team.vega.ru> <20080322135637.Y6961@fledge.watson.org> <200803230104.m2N14xlf026184@apollo.backplane.com> <20080322153658.K910@desktop> <47E5D613.4060005@freebsd.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: arch@freebsd.org Subject: Re: Disposal of a misleading M_TRYWAIT X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Mar 2008 09:15:38 -0000 On Sat, 22 Mar 2008, Sam Leffler wrote: >> They have been the same allocator for some time now. It makes more sense >> for them to use the same flags. > > We made a decision a while back to not use malloc flags for mbuf routine > arguments. There are even assertions to verify it. Changing the flag names > however would be a painful change for little gain. However, this isn't just a search and replace patch on flag names. M_TRYWAIT in fact does a Yoda suggests -- it doesn't "try", it just "does", and as a result none of the intermittently applied NULL-checking for M_TRYWAIT return paths can ever be executed. Hence the are never tested, and new code is turning up that makes the assumption that these paths will never see failure. A significant part of Ruslan's patch is improving consistency in error handling by making all code consider the wait case unable to fail, and there's nothing quite like removing unused exceptional case handling to make network code easier to read. I would also be fine with a proposal that replaces the existing M_DONTWAIT / M_TRYWAIT with mbuf allocator specific flags that read FOO_WAIT and FOO_NOWAIT (be it the same as in Dragonfly, MB_WAIT and MB_NOWAIT or otherwise). It's certainl true that in other cases, we've tried hard to have allocator wrappers with enhanced functionality use their own flag namespace -- SF_WAIT, etc, but since the mbuf allocator really is UMA these days, I am also alright with using the UMA/malloc flags. Robert N M Watson Computer Laboratory University of Cambridge From owner-freebsd-arch@FreeBSD.ORG Sun Mar 23 14:40:22 2008 Return-Path: Delivered-To: freebsd-arch@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C851F1065670 for ; Sun, 23 Mar 2008 14:40:22 +0000 (UTC) (envelope-from fjoe@samodelkin.net) Received: from atlas57.myplace.ru (atlas57.myplace.ru [80.66.68.57]) by mx1.freebsd.org (Postfix) with ESMTP id C8D4A8FC20 for ; Sun, 23 Mar 2008 14:40:21 +0000 (UTC) (envelope-from fjoe@samodelkin.net) Received: (qmail 89289 invoked from network); 23 Mar 2008 19:55:16 +0600 Received: from gw.nsib.ru (HELO husky.fjoe.local) (217.117.80.2) by atlas57.myplace.ru with (DHE-RSA-AES256-SHA encrypted) SMTP; 23 Mar 2008 19:55:16 +0600 Message-ID: <47E66591.2060804@samodelkin.net> Date: Sun, 23 Mar 2008 20:13:37 +0600 From: Max Khon User-Agent: Thunderbird 2.0.0.6 (X11/20071028) MIME-Version: 1.0 To: freebsd-arch@FreeBSD.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Subject: make -k does not exit with appropriate error code if errors are encountered X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Mar 2008 14:40:22 -0000 Hi! BSD make does not exit with appropriate error code (2) if it is run with "-k" and errors are encountered. Please review this patch that fixes this bug: http://people.freebsd.org/~fjoe/make.diff Sample BSDmakefile is: --- cut here --- all: a b a: exit 1 b: touch ${.TARGET} --- cut here --- I'd like to commit the to HEAD soon. /fjoe From owner-freebsd-arch@FreeBSD.ORG Sun Mar 23 15:17:17 2008 Return-Path: Delivered-To: freebsd-arch@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5294C1065676 for ; Sun, 23 Mar 2008 15:17:17 +0000 (UTC) (envelope-from ed@hoeg.nl) Received: from palm.hoeg.nl (mx0.hoeg.nl [IPv6:2001:610:652::211]) by mx1.freebsd.org (Postfix) with ESMTP id C675D8FC5C for ; Sun, 23 Mar 2008 15:17:15 +0000 (UTC) (envelope-from ed@hoeg.nl) Received: by palm.hoeg.nl (Postfix, from userid 1000) id D957A1CC50; Sun, 23 Mar 2008 16:17:14 +0100 (CET) Date: Sun, 23 Mar 2008 16:17:14 +0100 From: Ed Schouten To: Max Khon Message-ID: <20080323151714.GO51074@hoeg.nl> References: <47E66591.2060804@samodelkin.net> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="nWEzmRaGLXxZdI3i" Content-Disposition: inline In-Reply-To: <47E66591.2060804@samodelkin.net> User-Agent: Mutt/1.5.17 (2007-11-01) Cc: freebsd-arch@FreeBSD.org Subject: Re: make -k does not exit with appropriate error code if errors are encountered X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Mar 2008 15:17:17 -0000 --nWEzmRaGLXxZdI3i Content-Type: multipart/mixed; boundary="ssZxAlvqSOvXAj81" Content-Disposition: inline --ssZxAlvqSOvXAj81 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable * Max Khon wrote: > BSD make does not exit with appropriate error code (2) if it is run with= =20 > "-k" and errors are encountered. > > Please review this patch that fixes this bug: > > http://people.freebsd.org/~fjoe/make.diff > > Sample BSDmakefile is: > > --- cut here --- > all: a b > > a: > exit 1 > > b: > touch ${.TARGET} > --- cut here --- > > I'd like to commit the to HEAD soon. > > /fjoe Could you take at the attached patches as well? One patch adds support for the -p flag, which is described in POSIX. The other patch adds a small optimization, by ignoring execution when the command is a comment. Thanks! --=20 Ed Schouten WWW: http://g-rave.nl/ --ssZxAlvqSOvXAj81 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="freebsd-make-graph-posix.diff" Content-Transfer-Encoding: quoted-printable --- src/usr.bin/make/main.c Sun Jul 9 14:13:59 2006 +++ src/usr.bin/make/main.c Sun Jul 9 14:16:59 2006 @@ -127,6 +127,7 @@ Boolean jobsRunning; /* TRUE if the jobs might be running */ Boolean keepgoing; /* -k flag */ Boolean noExecute; /* -n flag */ +Boolean printGraphOnly; /* -p flag */ Boolean queryFlag; /* -q flag */ Boolean touchFlag; /* -t flag */ Boolean usePipes; /* !-P flag */ @@ -335,7 +336,7 @@ rearg: optind =3D 1; /* since we're called more than once */ optreset =3D 1; -#define OPTFLAGS "ABC:D:E:I:PSV:Xd:ef:ij:km:nqrstvx:" +#define OPTFLAGS "ABC:D:E:I:PSV:Xd:ef:ij:km:npqrstvx:" for (;;) { if ((optind < argc) && strcmp(argv[optind], "--") =3D=3D 0) { found_dd =3D TRUE; @@ -477,6 +478,10 @@ noExecute =3D TRUE; MFLAGS_append("-n", NULL); break; + case 'p': + printGraphOnly =3D TRUE; + debug |=3D DEBUG_GRAPH1; + break; case 'q': queryFlag =3D TRUE; /* Kind of nonsensical, wot? */ @@ -685,6 +690,7 @@ beSilent =3D FALSE; /* Print commands as executed */ ignoreErrors =3D FALSE; /* Pay attention to non-zero returns */ noExecute =3D FALSE; /* Execute all commands */ + printGraphOnly =3D FALSE; /* Do not print processed output */ keepgoing =3D FALSE; /* Stop on error */ allPrecious =3D FALSE; /* Remove targets when interrupted */ queryFlag =3D FALSE; /* This is not just a check-run */ @@ -1026,7 +1032,7 @@ Targ_PrintGraph(1); =20 /* print the values of any variables requested by the user */ - if (Lst_IsEmpty(&variables)) { + if (Lst_IsEmpty(&variables) && !printGraphOnly) { /* * Since the user has not requested that any variables * be printed, we can build targets. @@ -1069,7 +1075,7 @@ } Lst_Destroy(&targs, NOFREE); =20 - } else { + } else if (!printGraphOnly) { Var_Print(&variables, expandVars); } =20 --- src/usr.bin/make/make.1 Sun Jul 9 14:14:02 2006 +++ src/usr.bin/make/make.1 Sun Jul 9 14:22:53 2006 @@ -40,7 +40,7 @@ .Nd maintain program dependencies .Sh SYNOPSIS .Nm -.Op Fl ABPSXeiknqrstv +.Op Fl ABPSXeiknpqrstv .Op Fl C Ar directory .Op Fl D Ar variable .Op Fl d Ar flags @@ -242,6 +242,15 @@ This option has no effect unless .Fl j is used too. +.It Fl p +Only print the input graph, do not executing any commands. The output is t= he +same as +.Fl d Ar g1 . +When combined with +.Fl f Pa /dev/null , +only the builtin rules of +.Nm +are displayed. .It Fl q Do not execute any commands, but exit 0 if the specified targets are up-to-date and 1, otherwise. --ssZxAlvqSOvXAj81 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="freebsd-make-comment.diff" Content-Transfer-Encoding: quoted-printable Index: job.c =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D RCS file: /datadump/cvs/freebsd/src/usr.bin/make/job.c,v retrieving revision 1.126 diff -u -r1.126 job.c --- job.c 8 Mar 2007 09:16:10 -0000 1.126 +++ job.c 10 Jun 2007 11:39:45 -0000 @@ -2817,6 +2817,24 @@ } =20 /** + * shellskip + * + * Results: + * Returns a non-zero value if execution of the command wouldn't + * have an effect at all, which allows us to skip this line. This + * is useful for lines starting with '#'. + */ +static int +shellskip(char *cmd) +{ + if (commandShell->skipfirst !=3D NULL && + strchr(commandShell->skipfirst, cmd[0]) !=3D NULL) + return (1); + + return (0); +} + +/** * shellneed * * Results: @@ -2958,6 +2976,13 @@ return (0); } =20 + /* + * Don't execute the command if it's just a comment. + */ + if (shellskip(cmd)) { + return (0); + } + ps.in =3D STDIN_FILENO; ps.out =3D STDOUT_FILENO; ps.err =3D STDERR_FILENO; Index: shell.c =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D RCS file: /datadump/cvs/freebsd/src/usr.bin/make/shell.c,v retrieving revision 1.1 diff -u -r1.1 shell.c --- shell.c 24 May 2005 15:30:03 -0000 1.1 +++ shell.c 10 Jun 2007 11:51:31 -0000 @@ -82,7 +82,7 @@ "name=3Dcsh path=3D'" PATH_DEFSHELLDIR "/csh' " "quiet=3D'unset verbose' echo=3D'set verbose' filter=3D'unset verbose' " "hasErrCtl=3DN check=3D'echo \"%s\"\n' ignore=3D'csh -c \"%s || exit 0\"'= " - "echoFlag=3Dv errFlag=3De " + "echoFlag=3Dv errFlag=3De skipfirst=3D# " "meta=3D'" CSH_META "' builtins=3D'" CSH_BUILTINS "'", =20 /* @@ -92,7 +92,7 @@ "name=3Dsh path=3D'" PATH_DEFSHELLDIR "/sh' " "quiet=3D'set -' echo=3D'set -v' filter=3D'set -' " "hasErrCtl=3DY check=3D'set -e' ignore=3D'set +e' " - "echoFlag=3Dv errFlag=3De " + "echoFlag=3Dv errFlag=3De skipfirst=3D# " "meta=3D'" SH_META "' builtins=3D'" SH_BUILTINS "'", =20 /* @@ -103,7 +103,7 @@ "name=3Dksh path=3D'" PATH_DEFSHELLDIR "/ksh' " "quiet=3D'set -' echo=3D'set -v' filter=3D'set -' " "hasErrCtl=3DY check=3D'set -e' ignore=3D'set +e' " - "echoFlag=3Dv errFlag=3De " + "echoFlag=3Dv errFlag=3De skipfirst=3D# " "meta=3D'" SH_META "' builtins=3D'" SH_BUILTINS "' unsetenv=3DT", =20 NULL @@ -150,6 +150,7 @@ free(sh->echo); free(sh->exit); ArgArray_Done(&sh->builtins); + free(sh->skipfirst); free(sh->meta); free(sh); } @@ -174,7 +175,8 @@ fprintf(stderr, " builtins=3D%d\n", sh->builtins.argc - 1); for (i =3D 1; i < sh->builtins.argc; i++) fprintf(stderr, " '%s'", sh->builtins.argv[i]); - fprintf(stderr, "\n meta=3D'%s'\n", sh->meta); + fprintf(stderr, "\n skipfirst=3D'%s'\n", sh->skipfirst); + fprintf(stderr, " meta=3D'%s'\n", sh->meta); fprintf(stderr, " unsetenv=3D%d\n", sh->unsetenv); } =20 @@ -260,6 +262,10 @@ qsort(sh->builtins.argv + 1, sh->builtins.argc - 1, sizeof(char *), sort_builtins); *fullSpec =3D TRUE; + } else if (strcmp(keyw, "skipfirst") =3D=3D 0) { + free(sh->skipfirst); + sh->skipfirst =3D estrdup(eq); + *fullSpec =3D TRUE; } else if (strcmp(keyw, "meta") =3D=3D 0) { free(sh->meta); sh->meta =3D estrdup(eq); @@ -379,6 +385,8 @@ * execute the command directly. If this list is empty * it is assumed, that the command must always be * handed over to the shell. + * skipfirst Skip execution of the command if the first + * character has a certain value. * meta The shell meta characters. If this is not specified * or empty, commands are alway passed to the shell. * Otherwise they are not passed when they contain Index: shell.h =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D RCS file: /datadump/cvs/freebsd/src/usr.bin/make/shell.h,v retrieving revision 1.1 diff -u -r1.1 shell.h --- shell.h 24 May 2005 15:30:03 -0000 1.1 +++ shell.h 10 Jun 2007 11:39:45 -0000 @@ -96,6 +96,7 @@ char *exit; /* command line flag: exit on error */ =20 ArgArray builtins; /* ordered list of shell builtins */ + char *skipfirst; /* characters that determine if we can skip the line */ char *meta; /* shell meta characters */ =20 Boolean unsetenv; /* unsetenv("ENV") before exec */ --ssZxAlvqSOvXAj81-- --nWEzmRaGLXxZdI3i Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (FreeBSD) iEYEARECAAYFAkfmdHoACgkQ52SDGA2eCwVwaACfc+REXUn8H298o0gQoqKVnm9h 0dUAnjVM7fww74UaBZvKV1UIOQwcH8im =drjK -----END PGP SIGNATURE----- --nWEzmRaGLXxZdI3i-- From owner-freebsd-arch@FreeBSD.ORG Sun Mar 23 16:47:38 2008 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7D70C106566C for ; Sun, 23 Mar 2008 16:47:38 +0000 (UTC) (envelope-from sam@freebsd.org) Received: from ebb.errno.com (ebb.errno.com [69.12.149.25]) by mx1.freebsd.org (Postfix) with ESMTP id 45D8D8FC12 for ; Sun, 23 Mar 2008 16:47:38 +0000 (UTC) (envelope-from sam@freebsd.org) Received: from trouble.errno.com (trouble.errno.com [10.0.0.248]) (authenticated bits=0) by ebb.errno.com (8.13.6/8.12.6) with ESMTP id m2NGlTgF031945 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sun, 23 Mar 2008 09:47:29 -0700 (PDT) (envelope-from sam@freebsd.org) Message-ID: <47E689A1.1050203@freebsd.org> Date: Sun, 23 Mar 2008 09:47:29 -0700 From: Sam Leffler Organization: FreeBSD Project User-Agent: Thunderbird 2.0.0.9 (X11/20071125) MIME-Version: 1.0 To: Ruslan Ermilov References: <20080322105145.GA41672@team.vega.ru> <20080322135637.Y6961@fledge.watson.org> <200803230104.m2N14xlf026184@apollo.backplane.com> <20080322153658.K910@desktop> <47E5D613.4060005@freebsd.org> <20080323090823.GA37326@team.vega.ru> In-Reply-To: <20080323090823.GA37326@team.vega.ru> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-DCC-Rhyolite-Metrics: ebb.errno.com; whitelist Cc: arch@freebsd.org Subject: Re: Disposal of a misleading M_TRYWAIT X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Mar 2008 16:47:38 -0000 Ruslan Ermilov wrote: > On Sat, Mar 22, 2008 at 09:01:23PM -0700, Sam Leffler wrote: > >> We made a decision a while back to not use malloc flags for mbuf routine >> arguments. >> > > While I can see the point in this (if we ever change the > mbuf allocator from UMA to something else), like I said > in my email, a lot of code already uses malloc flags for > this. > > >> There are even assertions to verify it. >> > > I think this was OBE, now mbuf.h says: > > : /* > : * Flags specifying how an allocation should be made. > : * > : * The flag to use is as follows: > : * - M_DONTWAIT or M_NOWAIT from an interrupt handler to not block allocation. > : * - M_WAIT or M_WAITOK from wherever it is safe to block. > : * > : * M_DONTWAIT/M_NOWAIT means that we will not block the thread explicitly and > : * if we cannot allocate immediately we may return NULL, whereas > : * M_WAIT/M_WAITOK means that if we cannot allocate resources we > : * will block until they are available, and thus never return NULL. > : * > : * XXX Eventually just phase this out to use M_WAITOK/M_NOWAIT. > : */ > : #define MBTOM(how) (how) > : #define M_DONTWAIT M_NOWAIT > : #define M_TRYWAIT M_WAITOK > : #define M_WAIT M_WAITOK > > Yes, looks like someone broke it. Hence the confusion/pollution reappeared. The original suggestion was to purge M_TRYWAIT which was fine with me. If someone wants to sweep the tree and rename everything I'm not going to argue but it makes diffs against old code noisier and (as I said) for little gain. Sam From owner-freebsd-arch@FreeBSD.ORG Mon Mar 24 05:05:23 2008 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9E9F1106566C for ; Mon, 24 Mar 2008 05:05:23 +0000 (UTC) (envelope-from jroberson@chesapeake.net) Received: from webaccess-cl.virtdom.com (webaccess-cl.virtdom.com [216.240.101.25]) by mx1.freebsd.org (Postfix) with ESMTP id 6E7798FC26 for ; Mon, 24 Mar 2008 05:05:23 +0000 (UTC) (envelope-from jroberson@chesapeake.net) Received: from [10.0.1.200] (cpe-24-94-72-120.hawaii.res.rr.com [24.94.72.120]) (authenticated bits=0) by webaccess-cl.virtdom.com (8.13.6/8.13.6) with ESMTP id m2O55Kgs022202 for ; Mon, 24 Mar 2008 01:05:21 -0400 (EDT) (envelope-from jroberson@chesapeake.net) Date: Sun, 23 Mar 2008 19:06:40 -1000 (HST) From: Jeff Roberson X-X-Sender: jroberson@desktop To: arch@freebsd.org Message-ID: <20080323185402.C910@desktop> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: Subject: buffer queue lock contention X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Mar 2008 05:05:23 -0000 http://people.freebsd.org/~jeff/bqlock.diff This patch makes a seperate lock for each queue of the buffer cache. It is intended to reduce contention on the single bqlock. In most cases this is quite trivial however getnewbuffer() warrants some discussion. In getnewbuffer() we walk several queues in order looking for a buffer to recycle or use directly. With a single lock you're guaranteed that as you're looking at one queue a new buffer won't come available on an earlier queue. You're also guaranteed that you can sleep atomically while waiting for new buffers. In my patch the queues are independently synchronized and so these guarantees are no longer present. A thread could scan each list sucessively and not find any suitable buffers and then go to sleep even after a new buffer had been posted to one of the other queues. I don't believe this is a significant problem because the very next brelse is likely to wakeup the getnewbuf sleeper. With a buffer cache under suitable pressure the wakeups are likely to happen quite frequently. The other part of this diff is to avoid acquiring the global 'needs buffer' and 'running buf request' locks when the variables they protect are not set. This would make it possible for one thread to go to sleep simultaneously with another thread releasing a buffer that would wake it up and this wakeup would be missed. However, as in the bqlock case, another brelse is not far behind to unstick this process. Presently we call bufcountwakeup() and bufspacewakeup() on almost every call to brelse() and thus acquire these global locks several times. When we've actually done io we acquire a third global lock with runningbufwakeup(). All of these global locks effectively add latency and cost to each io. Thanks, Jeff From owner-freebsd-arch@FreeBSD.ORG Mon Mar 24 11:07:01 2008 Return-Path: Delivered-To: freebsd-arch@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C7F8510656DE for ; Mon, 24 Mar 2008 11:07:01 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id AC42E8FC30 for ; Mon, 24 Mar 2008 11:07:01 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.2/8.14.2) with ESMTP id m2OB712f087725 for ; Mon, 24 Mar 2008 11:07:01 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.2/8.14.1/Submit) id m2OB70GY087721 for freebsd-arch@FreeBSD.org; Mon, 24 Mar 2008 11:07:00 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 24 Mar 2008 11:07:00 GMT Message-Id: <200803241107.m2OB70GY087721@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-arch@FreeBSD.org Cc: Subject: Current problem reports assigned to freebsd-arch@FreeBSD.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Mar 2008 11:07:01 -0000 Current FreeBSD problem reports Critical problems Serious problems Non-critical problems S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/120749 arch [request] Suggest upping the default kern.ps_arg_cache 1 problem total. From owner-freebsd-arch@FreeBSD.ORG Mon Mar 24 14:06:24 2008 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D06EC106566B; Mon, 24 Mar 2008 14:06:24 +0000 (UTC) (envelope-from csjp@sub.vaned.net) Received: from sub.vaned.net (sub.vaned.net [205.200.235.40]) by mx1.freebsd.org (Postfix) with ESMTP id 9B9138FC14; Mon, 24 Mar 2008 14:06:24 +0000 (UTC) (envelope-from csjp@sub.vaned.net) Received: by sub.vaned.net (Postfix, from userid 1001) id E6F6B2E1; Mon, 24 Mar 2008 09:06:23 -0500 (CDT) Date: Mon, 24 Mar 2008 09:06:23 -0500 From: "Christian S.J. Peron" To: Robert Watson Message-ID: <20080324140623.GA14941@sub.vaned.net> References: <20080317133029.GA19369@sub.vaned.net> <20080317134335.A3253@fledge.watson.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20080317134335.A3253@fledge.watson.org> User-Agent: Mutt/1.4.2.3i Cc: arch@freebsd.org, freebsd-current@freebsd.org, "Christian S.J. Peron" , net@freebsd.org Subject: Re: HEADS UP: zerocopy bpf commits impending X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Mar 2008 14:06:25 -0000 I just want everyone to know that I have completed the zerocopy bpf commit. Please be on the "lookout" for any strange bpf related issues. For people that want to test the new zerocopy bpf implementation, a patch can be found here: http://people.freebsd.org/~csjp/pcap.1206364304.diff Any comments, patches etc can be sent to Robert Watson (rwatson@) or myself. Thanks! From owner-freebsd-arch@FreeBSD.ORG Mon Mar 24 14:15:52 2008 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D4E881065670; Mon, 24 Mar 2008 14:15:52 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.freebsd.org (Postfix) with ESMTP id 7753A8FC13; Mon, 24 Mar 2008 14:15:52 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id 4E55746C0D; Mon, 24 Mar 2008 10:15:51 -0400 (EDT) Date: Mon, 24 Mar 2008 14:15:51 +0000 (GMT) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: "Christian S.J. Peron" In-Reply-To: <20080324140623.GA14941@sub.vaned.net> Message-ID: <20080324141334.T7797@fledge.watson.org> References: <20080317133029.GA19369@sub.vaned.net> <20080317134335.A3253@fledge.watson.org> <20080324140623.GA14941@sub.vaned.net> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: arch@freebsd.org, freebsd-current@freebsd.org, net@freebsd.org Subject: Re: HEADS UP: zerocopy bpf commits impending X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Mar 2008 14:15:53 -0000 On Mon, 24 Mar 2008, Christian S.J. Peron wrote: > I just want everyone to know that I have completed the zerocopy bpf commit. > Please be on the "lookout" for any strange bpf related issues. > > For people that want to test the new zerocopy bpf implementation, a patch > can be found here: > > http://people.freebsd.org/~csjp/pcap.1206364304.diff > > Any comments, patches etc can be sent to Robert Watson (rwatson@) or myself. FYI, right now there is a known issue in which only one of the two BPF buffers can be owned by user processes at a time. As a result, when acking one buffer, it's almost always the case that userspace will enter select() even though another buffer is essentially ready, leading to a system call being generated for each buffer even though that's undesirable. I'm working on some changing allowing both buffers to be owned by userspace at a time, but it will be a couple of weeks before that enters CVS. I believe that the current libpcap patches should keep working with that fine, although of course, we'll see. :-) The bpf.4 documentation is very careful to warn that applications should not assume that there are any invariants about the number of buffers assigned to userspace at a time. Robert N M Watson Computer Laboratory University of Cambridge From owner-freebsd-arch@FreeBSD.ORG Mon Mar 24 14:53:37 2008 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A7066106566B; Mon, 24 Mar 2008 14:53:37 +0000 (UTC) (envelope-from petri@helenius.fi) Received: from silver.he.iki.fi (helenius.fi [83.150.107.219]) by mx1.freebsd.org (Postfix) with ESMTP id 5C5158FC18; Mon, 24 Mar 2008 14:53:37 +0000 (UTC) (envelope-from petri@helenius.fi) Received: from localhost (localhost [127.0.0.1]) by silver.he.iki.fi (Postfix) with ESMTP id 69359DF49; Mon, 24 Mar 2008 16:31:00 +0200 (EET) Received: from silver.he.iki.fi ([127.0.0.1]) by localhost (silver.he.iki.fi [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ymvqJQKR1Yfl; Mon, 24 Mar 2008 16:30:53 +0200 (EET) Received: from [83.150.107.194] (d194.helenius.fi [83.150.107.194]) by silver.he.iki.fi (Postfix) with ESMTP; Mon, 24 Mar 2008 16:30:53 +0200 (EET) Message-ID: <47E7BB1C.4020703@helenius.fi> Date: Mon, 24 Mar 2008 16:30:52 +0200 From: Petri Helenius User-Agent: Thunderbird 2.0.0.12 (Windows/20080213) MIME-Version: 1.0 To: Robert Watson References: <20080317133029.GA19369@sub.vaned.net> <20080317134335.A3253@fledge.watson.org> <20080324140623.GA14941@sub.vaned.net> <20080324141334.T7797@fledge.watson.org> In-Reply-To: <20080324141334.T7797@fledge.watson.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: arch@freebsd.org, freebsd-current@freebsd.org, net@freebsd.org Subject: Re: HEADS UP: zerocopy bpf commits impending X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Mar 2008 14:53:38 -0000 Pardon the basic question, but is the current patchset "zero copy" or "one copy"? The paper I saw a link to described a mechanism to eliminate one of the two copies the traditional bpf approach makes but I haven't taken a look into the actual code. Pete From owner-freebsd-arch@FreeBSD.ORG Mon Mar 24 15:26:37 2008 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 17BE2106566B; Mon, 24 Mar 2008 15:26:37 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.freebsd.org (Postfix) with ESMTP id B22198FC21; Mon, 24 Mar 2008 15:26:36 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id 9388746B0C; Mon, 24 Mar 2008 11:26:35 -0400 (EDT) Date: Mon, 24 Mar 2008 15:26:35 +0000 (GMT) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: Petri Helenius In-Reply-To: <47E7BB1C.4020703@helenius.fi> Message-ID: <20080324151814.Y12107@fledge.watson.org> References: <20080317133029.GA19369@sub.vaned.net> <20080317134335.A3253@fledge.watson.org> <20080324140623.GA14941@sub.vaned.net> <20080324141334.T7797@fledge.watson.org> <47E7BB1C.4020703@helenius.fi> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: arch@freebsd.org, freebsd-current@freebsd.org, net@freebsd.org Subject: Re: HEADS UP: zerocopy bpf commits impending X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Mar 2008 15:26:37 -0000 On Mon, 24 Mar 2008, Petri Helenius wrote: > Pardon the basic question, but is the current patchset "zero copy" or "one > copy"? The paper I saw a link to described a mechanism to eliminate one of > the two copies the traditional bpf approach makes but I haven't taken a look > into the actual code. The short answer is "one-copy". This eliminates the copy between the kernel and user space, but not the possibility of in-kernel copying. In practice, that in-kernel copying is frequently desirable as: (1) It allows packing of headers into a buffer when a small snaplen is used, which greatly reduces memory overhead when capturing, for example, just TCP headers and not payloads. (2) It allows us to more easily maintain independence between separate BPF sessions, and in particular, to avoid leaking memory between kernel, userspace, and different BPF consumers. If doing full capture of all packet data to userspace, the approach we took would improve performance, but would still involve one full copy of packet data in kernel. Further work would be required to eliminate that copy. Robert N M Watson Computer Laboratory University of Cambridge From owner-freebsd-arch@FreeBSD.ORG Mon Mar 24 20:10:43 2008 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1E0581065676 for ; Mon, 24 Mar 2008 20:10:43 +0000 (UTC) (envelope-from jb@what-creek.com) Received: from what-creek.com (what-creek.com [66.111.37.70]) by mx1.freebsd.org (Postfix) with ESMTP id E9DA38FC14 for ; Mon, 24 Mar 2008 20:10:42 +0000 (UTC) (envelope-from jb@what-creek.com) Received: by what-creek.com (Postfix, from userid 102) id 864F573303; Mon, 24 Mar 2008 19:55:23 +0000 (GMT) Date: Mon, 24 Mar 2008 19:55:23 +0000 From: John Birrell To: arch@freebsd.org Message-ID: <20080324195523.GA97312@what-creek.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.4.2.3i Cc: Subject: Patch for CDDL file repo copy X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Mar 2008 20:10:43 -0000 In line with the core@ decision to have separate trees for CDDL, GPLv2 and GPLv3 files, here is a patch to deal with the existing CDDL files in current: This patch assumes the following directory trees are repo copied: src/contrib/opensolaris => src/cddl/contrib/opensolaris src/compat/opensolaris => src/cddl/compat/opensolaris src/sys/contrib/opensolaris => src/sys/cddl/contrib/opensolaris src/sys/compat/opensolaris => src/sys/cddl/compat/opensolaris Note that the compat/opensolaris directories contain some files that actually have a BSD license. These files have no use outside their current location. They have a BSD license simply because they were created from scratch by the author. It is possible in future that these BSD licensed files may required additional code which comes with a CDDL. In that event the CDDL will be added and the file will remain in it's current location. If this move is to happen at all, I'd like it to occur now because it affects the addition of DTrace to current. The next step after this is to add the extra compat files that DTrace needs and move some of the things that are currently built into the ZFS kernel module to an 'opensolaris' kernel module so that they can be shared with DTrace. -- John Birrell From owner-freebsd-arch@FreeBSD.ORG Thu Mar 27 07:34:33 2008 Return-Path: Delivered-To: arch@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 502BD1065672 for ; Thu, 27 Mar 2008 07:34:33 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from harmony.bsdimp.com (bsdimp.com [199.45.160.85]) by mx1.freebsd.org (Postfix) with ESMTP id F1D8B8FC19 for ; Thu, 27 Mar 2008 07:34:32 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from localhost (localhost [127.0.0.1]) by harmony.bsdimp.com (8.14.2/8.14.1) with ESMTP id m2R7VoqX034325 for ; Thu, 27 Mar 2008 01:31:50 -0600 (MDT) (envelope-from imp@bsdimp.com) Date: Thu, 27 Mar 2008 01:32:29 -0600 (MDT) Message-Id: <20080327.013229.1649766744.imp@bsdimp.com> To: arch@FreeBSD.ORG From: "M. Warner Losh" X-Mailer: Mew version 5.2 on Emacs 21.3 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit Cc: Subject: AsiaBSDCon DEVSUMMIT patch X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Mar 2008 07:34:33 -0000 Greetings, We've been talking about the situation with suspend/resume in the tree. Here's a quick hack to allow one to suspend/resume an individual device. This may or may not work too well, but it is offered up for testing and criticism. http://people.freebsd.org/~imp/devctl.diff devctl -s ath 0 suspend ath0 devctl -r ath 0 resume ath0 Warner From owner-freebsd-arch@FreeBSD.ORG Thu Mar 27 09:04:15 2008 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 71FBC1065670 for ; Thu, 27 Mar 2008 09:04:15 +0000 (UTC) (envelope-from jroberson@chesapeake.net) Received: from webaccess-cl.virtdom.com (webaccess-cl.virtdom.com [216.240.101.25]) by mx1.freebsd.org (Postfix) with ESMTP id 38C088FC2C for ; Thu, 27 Mar 2008 09:04:15 +0000 (UTC) (envelope-from jroberson@chesapeake.net) Received: from [10.0.1.200] (cpe-24-94-72-120.hawaii.res.rr.com [24.94.72.120]) (authenticated bits=0) by webaccess-cl.virtdom.com (8.13.6/8.13.6) with ESMTP id m2R943J4089927; Thu, 27 Mar 2008 05:04:04 -0400 (EDT) (envelope-from jroberson@chesapeake.net) Date: Wed, 26 Mar 2008 23:04:43 -1000 (HST) From: Jeff Roberson X-X-Sender: jroberson@desktop To: "M. Warner Losh" In-Reply-To: <20080327.013229.1649766744.imp@bsdimp.com> Message-ID: <20080326230322.H72156@desktop> References: <20080327.013229.1649766744.imp@bsdimp.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: arch@freebsd.org Subject: Re: AsiaBSDCon DEVSUMMIT patch X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Mar 2008 09:04:15 -0000 On Thu, 27 Mar 2008, M. Warner Losh wrote: > Greetings, > > We've been talking about the situation with suspend/resume in the > tree. Here's a quick hack to allow one to suspend/resume an > individual device. This may or may not work too well, but it is > offered up for testing and criticism. > > http://people.freebsd.org/~imp/devctl.diff > > devctl -s ath 0 suspend ath0 > devctl -r ath 0 resume ath0 Hey Warner, This is a great idea. Would it be possible to provide a little more background about what the expected failure/success modes are? If we had some easy to follow steps we could ask for testers on current@ and create a wiki with a list of known working/broken hardware. That'd be a great step towards having widespread suspend/resume support. Thanks, Jeff > > Warner > > > _______________________________________________ > freebsd-arch@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-arch > To unsubscribe, send any mail to "freebsd-arch-unsubscribe@freebsd.org" > From owner-freebsd-arch@FreeBSD.ORG Thu Mar 27 20:00:57 2008 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EEAF81065672 for ; Thu, 27 Mar 2008 20:00:57 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from speedfactory.net (mail.speedfactory.net [66.23.216.219]) by mx1.freebsd.org (Postfix) with ESMTP id 8EC0A8FC22 for ; Thu, 27 Mar 2008 20:00:57 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from server.baldwin.cx (unverified [66.23.211.162]) by speedfactory.net (SurgeMail 3.8s) with ESMTP id 236963333-1834499 for multiple; Thu, 27 Mar 2008 15:59:33 -0400 Received: from localhost.corp.yahoo.com (john@localhost [127.0.0.1]) (authenticated bits=0) by server.baldwin.cx (8.14.2/8.14.2) with ESMTP id m2RK0aAb005153; Thu, 27 Mar 2008 16:00:39 -0400 (EDT) (envelope-from jhb@freebsd.org) From: John Baldwin To: freebsd-arch@freebsd.org Date: Thu, 27 Mar 2008 11:05:18 -0400 User-Agent: KMail/1.9.7 References: <20080327.013229.1649766744.imp@bsdimp.com> In-Reply-To: <20080327.013229.1649766744.imp@bsdimp.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200803271105.18401.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH authentication, not delayed by milter-greylist-2.0.2 (server.baldwin.cx [127.0.0.1]); Thu, 27 Mar 2008 16:00:40 -0400 (EDT) X-Virus-Scanned: ClamAV 0.91.2/6430/Thu Mar 27 13:39:43 2008 on server.baldwin.cx X-Virus-Status: Clean X-Spam-Status: No, score=-4.1 required=4.2 tests=ALL_TRUSTED,AWL,BAYES_00, DATE_IN_PAST_03_06 autolearn=ham version=3.1.3 X-Spam-Checker-Version: SpamAssassin 3.1.3 (2006-06-01) on server.baldwin.cx Cc: Subject: Re: AsiaBSDCon DEVSUMMIT patch X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Mar 2008 20:00:58 -0000 On Thursday 27 March 2008 03:32:29 am M. Warner Losh wrote: > Greetings, > > We've been talking about the situation with suspend/resume in the > tree. Here's a quick hack to allow one to suspend/resume an > individual device. This may or may not work too well, but it is > offered up for testing and criticism. > > http://people.freebsd.org/~imp/devctl.diff > > devctl -s ath 0 suspend ath0 > devctl -r ath 0 resume ath0 Unfortunately, what you really need is to power down the device to D3 for suspend and then bring it up. Otherwise you might not lose enough state to notice that resume isn't restoring all of it. bge(4) doesn't survive resume on my laptop I think because brgphy doesn't re-patch the firmware on resume, and you'd need a full power down to run into that sort of thing. What I would actually prefer would be this: devctl ath0 power off (maps to D3 on PCI/ACPI) devctl ath0 power D1 (PCI/ACPI-specific) devctl ath0 power on (maps to D0 on PCI/APCI) You'd probably need a 'int BUS_SET_POWERSTATE(device_t parent, device_t child, const char *state)' and implement it for ACPI and PCI. You would then have the ioctl handler do: /* copy in state string */ /* lookup device */ error = BUS_SET_POWERSTATE(device_get_parent(device), device, state); I would also make devctl take a plain device name and figure out the devclass/unit from that. Either that or pass the device name as a string to the kernel and have it do a lookup (for a userland ioctl I don't think an O(n) walk over the device list is all that evil). If you want to do named commands (like 'power') rather than getopt args for everything you can use a linker set to build a table of commands (I've done this for RAID management utils at work) that let you do something like: struct devctl_power_request { const char device[MAXDEVNAME]; const char state[32]; } #define DEVIOC_POWER _IOW('d', 1, struct devctl_power_request) /* av[0] will be 'power' */ static void power_command(int ac, char **av) { struct devctl_power_request req; if (ac != 3) errx(1, "Usage: devctl power "); strlcpy(req.device, av[1], sizeof(req.device)); strlcpy(req.state, av[2], sizeof(req.state)); if (ioctl(fd, DEVIOC_POWER, &req) < 0) err(1, "Set power state failed"); } DEVCTL_COMMAND(power); (Using a linker set makes it easier to add new commands later and have them all be self-contained.) -- John Baldwin From owner-freebsd-arch@FreeBSD.ORG Thu Mar 27 20:23:35 2008 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1B7FC1065673; Thu, 27 Mar 2008 20:23:35 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from harmony.bsdimp.com (bsdimp.com [199.45.160.85]) by mx1.freebsd.org (Postfix) with ESMTP id C12DA8FC2B; Thu, 27 Mar 2008 20:23:34 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from localhost (localhost [127.0.0.1]) by harmony.bsdimp.com (8.14.2/8.14.1) with ESMTP id m2RKKBRd050067; Thu, 27 Mar 2008 14:20:12 -0600 (MDT) (envelope-from imp@bsdimp.com) Date: Thu, 27 Mar 2008 14:20:52 -0600 (MDT) Message-Id: <20080327.142052.-1337017421.imp@bsdimp.com> To: jhb@freebsd.org From: "M. Warner Losh" In-Reply-To: <200803271105.18401.jhb@freebsd.org> References: <20080327.013229.1649766744.imp@bsdimp.com> <200803271105.18401.jhb@freebsd.org> X-Mailer: Mew version 5.2 on Emacs 21.3 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit Cc: freebsd-arch@freebsd.org Subject: Re: AsiaBSDCon DEVSUMMIT patch X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Mar 2008 20:23:35 -0000 In message: <200803271105.18401.jhb@freebsd.org> John Baldwin writes: : On Thursday 27 March 2008 03:32:29 am M. Warner Losh wrote: : > Greetings, : > : > We've been talking about the situation with suspend/resume in the : > tree. Here's a quick hack to allow one to suspend/resume an : > individual device. This may or may not work too well, but it is : > offered up for testing and criticism. : > : > http://people.freebsd.org/~imp/devctl.diff : > : > devctl -s ath 0 suspend ath0 : > devctl -r ath 0 resume ath0 Wow, you have a lot of comments for a simple test program :-) : Unfortunately, what you really need is to power down the device to D3 for : suspend and then bring it up. Otherwise you might not lose enough state to : notice that resume isn't restoring all of it. bge(4) doesn't survive resume : on my laptop I think because brgphy doesn't re-patch the firmware on resume, : and you'd need a full power down to run into that sort of thing. True. I was going to implement this next as a bus method to have the bus to the right thing. : What I would actually prefer would be this: : : devctl ath0 power off (maps to D3 on PCI/ACPI) : devctl ath0 power D1 (PCI/ACPI-specific) : devctl ath0 power on (maps to D0 on PCI/APCI) I'm not sure I like this at all. This is about completely suspending a device, or completely resuming the device for testing purposes. Randomly putting the device into D1 state is a bad idea. The device driver itself should do that level of detail. : You'd probably need a 'int BUS_SET_POWERSTATE(device_t parent, device_t child, : const char *state)' and implement it for ACPI and PCI. You would then have : the ioctl handler do: : : /* copy in state string */ : /* lookup device */ : error = BUS_SET_POWERSTATE(device_get_parent(device), device, state); : I would also make devctl take a plain device name and figure out the : devclass/unit from that. Either that or pass the device name as a string to : the kernel and have it do a lookup (for a userland ioctl I don't think an : O(n) walk over the device list is all that evil). I'll leave that for someone else to implement :-). : If you want to do named commands (like 'power') rather than getopt args for : everything you can use a linker set to build a table of commands (I've done : this for RAID management utils at work) that let you do something like: : : struct devctl_power_request { : const char device[MAXDEVNAME]; : const char state[32]; : } : : #define DEVIOC_POWER _IOW('d', 1, struct devctl_power_request) : : /* av[0] will be 'power' */ : static void : power_command(int ac, char **av) : { : struct devctl_power_request req; : : if (ac != 3) : errx(1, "Usage: devctl power "); : strlcpy(req.device, av[1], sizeof(req.device)); : strlcpy(req.state, av[2], sizeof(req.state)); : if (ioctl(fd, DEVIOC_POWER, &req) < 0) : err(1, "Set power state failed"); : } : DEVCTL_COMMAND(power); : : (Using a linker set makes it easier to add new commands later and have them : all be self-contained.) Wow! that's a lot more complicated than I had in mind :-) Warner From owner-freebsd-arch@FreeBSD.ORG Thu Mar 27 21:37:34 2008 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 72858106566B for ; Thu, 27 Mar 2008 21:37:34 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from speedfactory.net (mail.speedfactory.net [66.23.216.219]) by mx1.freebsd.org (Postfix) with ESMTP id 107478FC22 for ; Thu, 27 Mar 2008 21:37:33 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from server.baldwin.cx (unverified [66.23.211.162]) by speedfactory.net (SurgeMail 3.8s) with ESMTP id 236972932-1834499 for multiple; Thu, 27 Mar 2008 17:36:04 -0400 Received: from localhost.corp.yahoo.com (john@localhost [127.0.0.1]) (authenticated bits=0) by server.baldwin.cx (8.14.2/8.14.2) with ESMTP id m2RLbGiN005837; Thu, 27 Mar 2008 17:37:23 -0400 (EDT) (envelope-from jhb@freebsd.org) From: John Baldwin To: "M. Warner Losh" Date: Thu, 27 Mar 2008 17:27:51 -0400 User-Agent: KMail/1.9.7 References: <20080327.013229.1649766744.imp@bsdimp.com> <200803271105.18401.jhb@freebsd.org> <20080327.142052.-1337017421.imp@bsdimp.com> In-Reply-To: <20080327.142052.-1337017421.imp@bsdimp.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200803271727.51811.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH authentication, not delayed by milter-greylist-2.0.2 (server.baldwin.cx [127.0.0.1]); Thu, 27 Mar 2008 17:37:24 -0400 (EDT) X-Virus-Scanned: ClamAV 0.91.2/6431/Thu Mar 27 15:33:57 2008 on server.baldwin.cx X-Virus-Status: Clean X-Spam-Status: No, score=-4.4 required=4.2 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.1.3 X-Spam-Checker-Version: SpamAssassin 3.1.3 (2006-06-01) on server.baldwin.cx Cc: freebsd-arch@freebsd.org Subject: Re: AsiaBSDCon DEVSUMMIT patch X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Mar 2008 21:37:34 -0000 On Thursday 27 March 2008 04:20:52 pm M. Warner Losh wrote: > In message: <200803271105.18401.jhb@freebsd.org> > John Baldwin writes: > : On Thursday 27 March 2008 03:32:29 am M. Warner Losh wrote: > : > Greetings, > : > > : > We've been talking about the situation with suspend/resume in the > : > tree. Here's a quick hack to allow one to suspend/resume an > : > individual device. This may or may not work too well, but it is > : > offered up for testing and criticism. > : > > : > http://people.freebsd.org/~imp/devctl.diff > : > > : > devctl -s ath 0 suspend ath0 > : > devctl -r ath 0 resume ath0 > > Wow, you have a lot of comments for a simple test program :-) > > : Unfortunately, what you really need is to power down the device to D3 for > : suspend and then bring it up. Otherwise you might not lose enough state to > : notice that resume isn't restoring all of it. bge(4) doesn't survive resume > : on my laptop I think because brgphy doesn't re-patch the firmware on resume, > : and you'd need a full power down to run into that sort of thing. > > True. I was going to implement this next as a bus method to have the > bus to the right thing. > > : What I would actually prefer would be this: > : > : devctl ath0 power off (maps to D3 on PCI/ACPI) > : devctl ath0 power D1 (PCI/ACPI-specific) > : devctl ath0 power on (maps to D0 on PCI/APCI) > > I'm not sure I like this at all. This is about completely suspending > a device, or completely resuming the device for testing purposes. > Randomly putting the device into D1 state is a bad idea. The device > driver itself should do that level of detail. It is useful for doing some of what phk suggested earlier though. For example, putting devices in D1 and seeing if you really get an interrupt on link state events or USB device insertion, etc. That is, for debugging some of the power management stuff. I imagine that on/off will be used the vast majority of the time, but I can see D[12] being useful for debugging. I wouldn't mind being able to manually turn devices off when I'm on an airplane to conserve battery until such time as we get smarter drivers that automatically manage the power. For example, I'd like to turn off my sound card sometimes, or power down bge0 when iwi0 associates (since I only ever use 1 of them). > : If you want to do named commands (like 'power') rather than getopt args for > : everything you can use a linker set to build a table of commands (I've done > : this for RAID management utils at work) that let you do something like: > : > : struct devctl_power_request { > : const char device[MAXDEVNAME]; > : const char state[32]; > : } > : > : #define DEVIOC_POWER _IOW('d', 1, struct devctl_power_request) > : > : /* av[0] will be 'power' */ > : static void > : power_command(int ac, char **av) > : { > : struct devctl_power_request req; > : > : if (ac != 3) > : errx(1, "Usage: devctl power "); > : strlcpy(req.device, av[1], sizeof(req.device)); > : strlcpy(req.state, av[2], sizeof(req.state)); > : if (ioctl(fd, DEVIOC_POWER, &req) < 0) > : err(1, "Set power state failed"); > : } > : DEVCTL_COMMAND(power); > : > : (Using a linker set makes it easier to add new commands later and have them > : all be self-contained.) > > Wow! that's a lot more complicated than I had in mind :-) But is more extensible so you can have 'devctl eject foo0' (think ACPI _EJx methods) or other commands that are a bit more user friendly in syntax than a plethora of getopt options. It's also not that hard esp. since I've basically sent you the implementation via private e-mail. :) -- John Baldwin From owner-freebsd-arch@FreeBSD.ORG Thu Mar 27 23:08:11 2008 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B5CBA106567A; Thu, 27 Mar 2008 23:08:11 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from harmony.bsdimp.com (bsdimp.com [199.45.160.85]) by mx1.freebsd.org (Postfix) with ESMTP id 6231F8FC21; Thu, 27 Mar 2008 23:08:11 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from localhost (localhost [127.0.0.1]) by harmony.bsdimp.com (8.14.2/8.14.1) with ESMTP id m2RN5oZU052054; Thu, 27 Mar 2008 17:05:51 -0600 (MDT) (envelope-from imp@bsdimp.com) Date: Thu, 27 Mar 2008 17:06:30 -0600 (MDT) Message-Id: <20080327.170630.-365729353.imp@bsdimp.com> To: jhb@freebsd.org From: "M. Warner Losh" In-Reply-To: <200803271727.51811.jhb@freebsd.org> References: <200803271105.18401.jhb@freebsd.org> <20080327.142052.-1337017421.imp@bsdimp.com> <200803271727.51811.jhb@freebsd.org> X-Mailer: Mew version 5.2 on Emacs 21.3 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit Cc: freebsd-arch@freebsd.org Subject: Re: AsiaBSDCon DEVSUMMIT patch X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Mar 2008 23:08:11 -0000 In message: <200803271727.51811.jhb@freebsd.org> John Baldwin writes: : On Thursday 27 March 2008 04:20:52 pm M. Warner Losh wrote: : > In message: <200803271105.18401.jhb@freebsd.org> : > John Baldwin writes: : > : On Thursday 27 March 2008 03:32:29 am M. Warner Losh wrote: : > : > Greetings, : > : > : > : > We've been talking about the situation with suspend/resume in the : > : > tree. Here's a quick hack to allow one to suspend/resume an : > : > individual device. This may or may not work too well, but it is : > : > offered up for testing and criticism. : > : > : > : > http://people.freebsd.org/~imp/devctl.diff : > : > : > : > devctl -s ath 0 suspend ath0 : > : > devctl -r ath 0 resume ath0 : > : > Wow, you have a lot of comments for a simple test program :-) : > : > : Unfortunately, what you really need is to power down the device to D3 for : > : suspend and then bring it up. Otherwise you might not lose enough state : to : > : notice that resume isn't restoring all of it. bge(4) doesn't survive : resume : > : on my laptop I think because brgphy doesn't re-patch the firmware on : resume, : > : and you'd need a full power down to run into that sort of thing. : > : > True. I was going to implement this next as a bus method to have the : > bus to the right thing. : > : > : What I would actually prefer would be this: : > : : > : devctl ath0 power off (maps to D3 on PCI/ACPI) : > : devctl ath0 power D1 (PCI/ACPI-specific) : > : devctl ath0 power on (maps to D0 on PCI/APCI) : > : > I'm not sure I like this at all. This is about completely suspending : > a device, or completely resuming the device for testing purposes. : > Randomly putting the device into D1 state is a bad idea. The device : > driver itself should do that level of detail. : : It is useful for doing some of what phk suggested earlier though. For : example, putting devices in D1 and seeing if you really get an interrupt on : link state events or USB device insertion, etc. That is, for debugging some : of the power management stuff. I imagine that on/off will be used the vast : majority of the time, but I can see D[12] being useful for debugging. I : wouldn't mind being able to manually turn devices off when I'm on an airplane : to conserve battery until such time as we get smarter drivers that : automatically manage the power. For example, I'd like to turn off my sound : card sometimes, or power down bge0 when iwi0 associates (since I only ever : use 1 of them). The problem is that turning off the device w/o the cooperation of the device is a non-starter. You can already unload the driver from the system and have the device be powered down, for example, by setting hw.pci.do_power_nodriver to 1. In the short term, you'll be much happier with this approach, as it is more general and drivers cope with that already. The main reason I did suspend/resume is (a) to test out suspend/resume and (b) drivers are already supposed to handle it correctly. I'm very leery of controlling the power of a device behind its back, even for something as test program. The documentation for the chip will tell you what modes it gets interrupts in, and that's a lot more reliable than just trying it, which may or may not work for different revisions of the same chip... : > : If you want to do named commands (like 'power') rather than getopt args : for : > : everything you can use a linker set to build a table of commands (I've : done : > : this for RAID management utils at work) that let you do something like: : > : : > : struct devctl_power_request { : > : const char device[MAXDEVNAME]; : > : const char state[32]; : > : } : > : : > : #define DEVIOC_POWER _IOW('d', 1, struct devctl_power_request) : > : : > : /* av[0] will be 'power' */ : > : static void : > : power_command(int ac, char **av) : > : { : > : struct devctl_power_request req; : > : : > : if (ac != 3) : > : errx(1, "Usage: devctl power "); : > : strlcpy(req.device, av[1], sizeof(req.device)); : > : strlcpy(req.state, av[2], sizeof(req.state)); : > : if (ioctl(fd, DEVIOC_POWER, &req) < 0) : > : err(1, "Set power state failed"); : > : } : > : DEVCTL_COMMAND(power); : > : : > : (Using a linker set makes it easier to add new commands later and have : them : > : all be self-contained.) : > : > Wow! that's a lot more complicated than I had in mind :-) : : But is more extensible so you can have 'devctl eject foo0' (think ACPI _EJx : methods) or other commands that are a bit more user friendly in syntax than a : plethora of getopt options. It's also not that hard esp. since I've : basically sent you the implementation via private e-mail. :) I like the idea of having different commands.... Warner From owner-freebsd-arch@FreeBSD.ORG Fri Mar 28 08:22:45 2008 Return-Path: Delivered-To: freebsd-arch@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 782D21065670; Fri, 28 Mar 2008 08:22:45 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from harmony.bsdimp.com (bsdimp.com [199.45.160.85]) by mx1.freebsd.org (Postfix) with ESMTP id 34C858FC26; Fri, 28 Mar 2008 08:22:45 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from localhost (localhost [127.0.0.1]) by harmony.bsdimp.com (8.14.2/8.14.1) with ESMTP id m2S8MSlb071952; Fri, 28 Mar 2008 02:22:29 -0600 (MDT) (envelope-from imp@bsdimp.com) Date: Fri, 28 Mar 2008 02:23:08 -0600 (MDT) Message-Id: <20080328.022308.1645215000.imp@bsdimp.com> To: jhb@FreeBSD.org From: "M. Warner Losh" In-Reply-To: <200803271727.51811.jhb@freebsd.org> References: <200803271105.18401.jhb@freebsd.org> <20080327.142052.-1337017421.imp@bsdimp.com> <200803271727.51811.jhb@freebsd.org> X-Mailer: Mew version 5.2 on Emacs 21.3 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit Cc: freebsd-arch@FreeBSD.org Subject: Re: AsiaBSDCon DEVSUMMIT patch X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Mar 2008 08:22:45 -0000 OK. I've updated the kernel side of this patch. It still only supports suspend/resume, but tells the bus to do it. The PCI bus will now set the state properly. We can extend this to include detach and rescan (and somehow attach, but that's harder), but I haven't had time to do this yet. I drank way too much beer last night to do more (or I'd have merged in the suggested userland changes too). I'm still leery about doing things directly powering devices, but so long as it translates to suspend/resume, I'm happy. Power state management when a device isn't suspended also needs careful thought. This patch mimics the suspend and resume paths better, so we can do more testing on drivers. This has already found, or rediscovered, bugs in the usb code's integration into newbus. I've not yet fixed those, since they are hard (but also affect eject of USB Cardbus cards with devices attached). Warner From owner-freebsd-arch@FreeBSD.ORG Fri Mar 28 19:09:30 2008 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 95FF2106564A for ; Fri, 28 Mar 2008 19:09:30 +0000 (UTC) (envelope-from phk@critter.freebsd.dk) Received: from phk.freebsd.dk (phk.freebsd.dk [130.225.244.222]) by mx1.freebsd.org (Postfix) with ESMTP id 5B24C8FC1D for ; Fri, 28 Mar 2008 19:09:30 +0000 (UTC) (envelope-from phk@critter.freebsd.dk) Received: from critter.freebsd.dk (unknown [192.168.61.3]) by phk.freebsd.dk (Postfix) with ESMTP id D624B17105 for ; Fri, 28 Mar 2008 19:09:28 +0000 (UTC) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.14.2/8.14.2) with ESMTP id m2SJ9SfN006473 for ; Fri, 28 Mar 2008 19:09:28 GMT (envelope-from phk@critter.freebsd.dk) To: arch@freebsd.org From: Poul-Henning Kamp Date: Fri, 28 Mar 2008 19:09:28 +0000 Message-ID: <6472.1206731368@critter.freebsd.dk> Sender: phk@critter.freebsd.dk Cc: Subject: Flash disks and FFS layout heuristics X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Mar 2008 19:09:30 -0000 I've laid my hands on a "M-Tron MOBI3000 32GB" flash disk (2.5" format, it'll be in my laptop before long :-) Here is a naive benchmark sequence, comparing it to a WD Raptor () Flash Disk --------------------------------------------------------------- Empty fsck: 0.83 2.47 -66% restore -rf 839 1251 -33% loaded fsck: 10.34 78.81 -87% dump 0f /dev/null: 563.21 1094.91 -49% --------------------------------------------------------------- So far so good, it's clearly the seektime that dominates the flash-advantage. But this reproducible observation by fsck concerns me a bit: Flash: (205727 frags, 896270 blocks, 1.4% fragmentation) Disk: (197095 frags, 1193944 blocks, 1.1% fragmentation) I might indicate that the flash is fast enough to confuse some of FFS's layout heuristics. Any aspiring filesystems hackers should start to consider the implications for filesystemlayout, if there is in essence no seek-time penalty for reads and a fair seek pentalty for writes. Poul-Henning -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From owner-freebsd-arch@FreeBSD.ORG Sat Mar 29 06:52:10 2008 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D0DBD106566B for ; Sat, 29 Mar 2008 06:52:10 +0000 (UTC) (envelope-from rosemoore7@satellite-email.com) Received: from mout3.freenet.de (mout3.freenet.de [IPv6:2001:748:100:40::2:5]) by mx1.freebsd.org (Postfix) with ESMTP id E19818FC17 for ; Sat, 29 Mar 2008 06:52:09 +0000 (UTC) (envelope-from rosemoore7@satellite-email.com) Received: from [195.4.92.20] (helo=10.mx.freenet.de) by mout3.freenet.de with esmtpa (Exim 4.69) (envelope-from ) id 1JfUvB-0001zi-3H for freebsd-arch@freebsd.org; Sat, 29 Mar 2008 07:52:09 +0100 Received: from ip165-138-173-82.adsl2.versatel.nl ([82.173.138.165]:2502 helo=BV-55721B5E014) by 10.mx.freenet.de with esmtpa (ID rosemoore16@freenet.de) (port 587) (Exim 4.69 #12) id 1JfUvA-0008UP-4M for freebsd-arch@freebsd.org; Sat, 29 Mar 2008 07:52:09 +0100 From: "ROSE MOORE" To: freebsd-arch@freebsd.org Content-Type: multipart/mixed; charset="ISO-8859-1"; boundary="lnSXiGey81P9nGQ=_Kiw6O53aaHOiHeSwk0" MIME-Version: 1.0 Date: Sat, 29 Mar 2008 07:52:08 +0100 Message-ID: <31764057962571@mx.freenet.de> X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: PAYMENT FORM ATTACHED. X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: rosemoore7@satellite-email.com List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 29 Mar 2008 06:52:10 -0000 This is a multi-part message in MIME format --lnSXiGey81P9nGQ=_Kiw6O53aaHOiHeSwk0 Content-Type: text/plain ; charset="ISO-8859-1" Content-Transfer-Encoding: quoted-printable Verify your claim amount as you have won. Contact Mr. Moh Constance on= email: mrmohconstance@inbox.com and= call +31-619323044. Your's=20 Rose Moore. Cordinator. --lnSXiGey81P9nGQ=_Kiw6O53aaHOiHeSwk0 Content-Type: application/octet-stream; name="FORMULAR..pdf" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="FORMULAR..pdf" JVBERi0xLjMKJcfsj6IKNSAwIG9iago8PC9MZW5ndGggNiAwIFIvRmlsdGVyIC9GbGF0ZURlY29k ZT4+CnN0cmVhbQp4nNVd23Ikx3EN+W7IHzFvnpEDrbp3Fd+wC5KiuFwuAdAyg/aDgxYpOXapIBUO +zv8xc66ZHX1TJ7GYAGEY+nwqrtPnurKrKyszJpG9087NWmzU/n/+OC7dxe/vpl3P/z5Qk1RqTlq woLSPtid3t18Kl7++YeLny7iZPN/pZXx+Lt3uxd31Gja0S3uvr/Q5areOTellPxu1nHSQe/u3l18 u98dLg34J67OzBz/7e63Fx/fXXx18RO1bGuzjv43mNxy2PkU3RTmqpLWu+s/HcuanY7+PEGr4iwL FvMVy5SDs2yh7S6NtvDFBqMpSMW7/6zqZYFiY73744VxmvoRY5xC3OmZTJp81pYEfn/xfe7D5Ht3 yuF5HfK7NJHNpPFRaTU+dlLaKP/Uh2lSNugPqQFt9ZP1wHrzodngSFT5PEHznNRmnmhWqd3dfzyR x3xQdjFe9chUQ6vv0YEO33c6GqtKiICz8YMy0qMaiIPo9eGSVqSkTdj/10FPczQ0k35/uOTDgf8m HzqfiPXzIvCnw2WYFIXV/XfLxaGBPx+YtBz98cD3/LHf84feznDLLw+XftLWRLf//nDpJusVCQyH S0vy3X8WNbk7MH8Qfdub+nWdhvM82T4NP2Hl7f7fe0//5+BVcVbyuHnxOEIj8z7qrQ73/6eDD67x Bk9VUzLMs/0mmo/C/nJRPHbcHdrBPFziI8WgEcDGVHagTocTfYLKB6VfdVUtaE1GLp1Xk4u7S6Mm 5RaRo4VX27jzIYa66FJaoF1Zdfui+4CpPR/lQdqGHC5DnKn1tvaj/1rvKZVypbvfX8x+MjZ5Nvsv Dmaak07z/i8oT6LxN1rv/5KcNCoT4/6vyHGMTsbv/5qu+aSU2/9NHteoFQ3x3+ZDb100+7/L88oa 743e/32X/dcf82XtdExu/0u6l3KJbrA0+w80JCG5LPpz7oD3kdod+vLLQ1BxinHOw0Rt5c6TPZej D1CN3vcPfBA+5DEY6guThrg31XJhvSLnmWopSylVQowmT2ubz1Tq/1tm9nlCVq+EZh/L+UpIq6S3 myEJv0g4KjdWsCP1txu4X6Kr00IZgzoN0az2dEui3mhL4qEREdUeIQRaU3BBVOKy13OJy5qip9bu qeNycORjQxfKmldFLXvZ9cHsPyaP3+9e5X+/zE53l4/u+unuzYF89oYEy+kXB9euf0ZL+P71oRGm 3bKAtf7kEL8Lkzldno5dOZzjpeFeL91qpnppk9CODLOCjZ/v6YfxUXUJo5SRnHhTEda2jXt3wOCO nXhLovZ0S6J2ZUvioTl9bIv2iZs7ImA3t3Pb0DCUyuS+6NIbGsJ15f80fbGU5cVWYLw++MlHneL+ qh99QWnwTHlgzB7vpuCMbckpJUY2ezUtEbRG+JxzUurlSNad1EXHqulos2ouT/isWngO1QzZjWun F7XzOgbWIx9WhY1zdJHyWROpxh/V+OzQD1920eHiVb9404++aaab2zqUylR+lYf6H+s/Ke++8RXZ QLQe1HCbs+znMpCnHFSbZqBlxK+PjtbqDV5wm10j0JrOR1pS+RxtgyvaOu2fUVufJs/a3o3uUEfe 71+d+EjWkgI5+VH0dv+b7uqL078+NoxO72sE1xY2659vuvu8LLAR2NP9nIbh/5cDFUdezbbrMXR9 pRTQQ5fSJoevZ9RDOQ6h3w4Ou2hxtwwgjFuBclsfjuJWF3ix0IYZf9Oss7pDcww3v6fJKA0vJnNz fD6TOZpVs9mI9HeiwpKrX/UQ90rkDC2dRsMHmmY21Zu8e0bTuLQEwmF5WNa+Kyn6vxIdaxG9Fo42 AulypHoQWafLaordXCuF0hTvNaSbq4+ZZ1xyKfrRkDZDvhnCy033Mjlf2FpV6GjDH8FCe0bAtcbV XC8+4xprk87OVC3S+j4P00+eQPKq0l1s0fzLpdSdlW1q+ZI2el3V8WM2/V66JDXNeq1LzVsvnaeQ srtMUrVybG2VXPU/84zWni0t5c3aL499bt5/3e36ZnA+KcQN1u6HccBHw7fNQdJwfmLDk3n9Wq1m eBpiqmeODI8nwGkOKg+SCbYOEpcczzJIOi4lx5CXLyvRYubP+9CcuP+yEOWjkzJFzsO2dHeq6j4/ YzgwKe8/n6TcL4+Oqoex6ovXLprfCWtPvD9GJMu6PrGraqUpux71Y08N+fqJp94bnY2eW7xwzzgc wfBuz7e89ia//90h7ysZv1rtx+VKsr4Tx+5ajOTvs1zp5FzNq5/TP6n4MI+Zm++d/TyojDwjcdTB locmAk/n50gcySCT49V9CUF9vTkZ/Y3NE64yB3hcpBbDfnMU9Ay51ND865MKJS4G3VwTMHychibk rHWL1rpi+jpzn2WXlgYgtkxkKhGFEs+41qb90/59/H9H7Z73j2Adkx/7KbuplrxUPWJ/UevjIFyt Y01ccs4X+bfSpE32EfIwH0JOP8kvaAn2w7VPDmayVMLmmHfJhy+74HDxql+86Uff9KP/ze5bm7wl 7w5ep+zTuXYJKQzdWZqp3UkuUkgJkwlKUYy87heXPt71o6uOto5ZCh+vDsudPa07Sullc73baibx 9rv16e/DDb3MdcwiAZ2L/qFpC7HRBY43772292/eN6GNzfutZurmfZNAm/dbDdTN+yaBNu83FWna au3jKOSs94MWGC63wPCTBJP8ANMsPIZ3NGLUC3fviLEQHrHNZsqIsYSO6ehHQZOS2W7AjJ0gswVh xLYVadqmSHN3Xuh5Yqx/btmUKD3dlChd2ZR47K+KPLxK/rXlaHhT3h4ZunI6cvdJ+C4BRm6zgWqP LQnuaVRpbbQwHw3NpkTpyqZE6cqmxGm2peoUdameHz+l+qhn4Xq2lX/qNz1bz09CWRPyChDzUwbe 7D895MXF2/zLTn9Q7OpwqcmtbGpPUrkU8k5Ax28W/OOKW06Y1n6lKTW+J+uMoQ5g3oN5pB3wz+Ra hYmSwqPfqFna5Hqr/1BdtQxlC7NrvJhBVLhGyZjyipmflPIGbu4oS56hTE1ojEtPm2bbUOOY8cuu orQxs9qjoVxDT1QsjfCvDvPkgglpdXR5fFHfB9/7u+fDfjzkRF3eSF7K/laZujT8TCl2ZSnOXo1b 8LMxfQvA2tnSkNmyRxja1ufj638XQ36wbjVUggmlAZjn+2BhMtKUJaUpUQs5/XqqJ4+f6kFXfjZa mC1z/qkzrxHa2F18yvDQZkvKGzEq/H8/wN9+/6+hqTw90/7opAzW1SHnS3GmiE2dDnPKoUiRnkbZ /HhMQ++Wa6XxYgdR8poPvuwN3vClW5oLI+kLCvjld19KXIN3p7dz28+QReXvSRXuk/BdAqQKmw20 9XlDgntK1dd6Efd2WcS7kDsj6dgWatkvC+Hsd7NDVWLrRsV0LAFMt3mLKrF1CzMqAjLozVtUia1b 3C+h+46BM3P+0T7mv0PSXpX/zyK/uqBK2e/++yLuviDx3y5/l8SUdES5vdB+zk8AGHPaGIJuc708 IxKAiBRnSAIQkTS+E4Bu81+fQBKAbi+MglZl6NR6RkPrIYhIFuqEICI5qBOCiJRo2V10iiOpQekI uqWYnBf/3txIQhCRjIckABHJaYdIACKdgvGAhCAiUVWASACi7vnZAOsxJFhvNnFobh5J66kxkmKE pBgBySmHDIEgmhp5Z9rrEE/vxFA6gnKMKI/nzcKdEJRjhIckABEp5WHvyKp7ACoxwqM7ASjHiDzs MglAOUbMo/VWbtSgU48wJg+7qBOCcoyIyHoIIpJ3DpEAlKe7MoNOY2Bh6DTu2Xy0NLciAYhItjQn kgCUY4SHJADl6Q7vhCAiRXgnBFH3ArZegNabsfUARKSErQegHCOgTgjKMYImjUtOWj4BVPIIm5Eg 5hEiVPIIj0gAKnkEJAGoxQhAAlDJIyAJQCWPQNZDUMkjkPUQVPII1D0ElTwCkgCU8wiKU4RYgdQg d+qwqjY3S4FFhnIeUZuTSADKgWWGJACVPAKREJQDC7wTgnIeAa3HkGC9gK0HoJxHYOsBKAcWbD0A 3daCwuVHR6QYUaBTnUpBQYhwJwS1WgOQANRqDUACUKs1AAlArdYAJAC1WkO2HkOn1isFhXwnBLVa A5AA1GoNQAIQxwhZp4R0qoFAvBOCOEbIJABxjJBJAOIYIRsCQBwjZBKAOEaI1mMIxQhZJwBxjJBJ AOIYIZMAxDHCS7kRQ2A/wnkpY0EQxwiZBCCOETIJQBwjZBKAOEbIJABxjBCtxxDYj5DvhCCOETIJ QBwjZBKAOLO0OLM8hTiztDizPIU4s5RJAOLMUiYBiDNLmQQgzixlEoA4sxSthyDOLMU7IYgzS5kE IM4sZRKAOEZoadeNIRQjNNyzFCCOETIJQBwjZBKAOEbIJABxjJBJAOIYIVqPIRQjxDshiGOETAIQ xwiZBKDmETbCzJIgkFkSgjJLAWoeAUgAah4BSABqHgFIAGoeAUgAah4hW48hkFnKd0JQ8whAAlDz CEACEHtEgHkEQSBGUBaEYoQAsUfIJACxR8gkALFHyCQAsUfIJACxR4jWYwjECPlOCGKPkEkAYo+Q SQBij3BSRc0QihFOqnMRxB4hkwDEHiGTAMQeIZMAxB4hkwDEHiFajyEUI8Q7IYg9QiYBiD1CJgGI PcLAPIIgFCMMzCMEiD1CJgGIPUImAYg9QiYBiD1CJgGIPUK0HkMoRoh3QhB7hEwCEHuETAJQ8wgd h5TYpdEjBKh5BCFBIAGoeQQgAah5BCABqHkEIAGoeQQgAah5hGw9BDWPkO+EoOYRgASg5hGABKBW feaHPcQYIUKt+szPf4gxQoRa9QlIAGrVJyABqFWfgASgVn0CEoBa9SlbD0Gt+pTvhKBWfQISgFr1 CUgA4j1LeY8loT2WujEp7nwgiPcsZRKAeM9SJgGI9yzljRkA8Z6lTAIQ71mK1mNIsF7A1gMQ71nK JADxnqVMAlDzCLASJrQSlmGX1ycENY8AJAA1jwAkADWPAMsngJpHABKAmkfI1mMIeATQCUDNIwAJ QM0jAAlA7BFytpxQtlyHXcxhEcQeIZMAxB4hkwDEHiGn2ABij5BJAGKPEK3HEPhdA+gEIPYImQQg 9giZBCD2CLmiTqiirsMu1rkIYo+QSQBij5BJAGKPkMtwALFHyCQAsUeI1mMIxQhZJwCxR8gkALFH yCQAtVrDeLgfkZ++kjJLnxG0HyFArdYAJAC1WgOQANRqDUACUKs1AAlArdaQrccQ2I+Q74SgVmsA EoBarQFIAGoxgnJOtGoQBFYNQtCqIUAtRgASgFqMACQAtRghkxDUYgQgAajFCNl6DIFVA+gEoBYj AAlALUYAEoDYI+TnyxsEVg0jPvWNIPYImQQg9giZBCD2CPmhdACxR8gkALFHiNZjCKwaQCcAsUfI JACxR8gkAPGqIerEENizlO+EIF418N9ryKQIdUIQrxr47zVkksF3AhCvGvjvNUTr1aUB/72GfCeL vRxAvGrgv9fAHqED2KESId6zDGCHSoR4z1ImAYj3LGUSgHjPUiYBiPcsZRKAeM9StB6CeM9SvBOC eM9SJgGI9yxlEoB4h8riJ28tfPLW4idvTyHeoZJJAOIdKpkEIN6hEkkI4h0qmQQg3qESrccQeqpO 1glAvEMlkwDEO1QyCUDsEfIzHw1Ce5bikxgIYo+QSQBij5BJAGKPkB8UARB7hEwCEHuEaD2G0J6l rBOA2CNkEoDYI2QSgHg/Qn7mo0Foh0p8EgNBvB8hkwDE+xEyCUC8HyE/KAIg3o+QSQDi/QjRegyh HSpZJwDxfoRMAhDvR8gkAHGtIVfUCVXUtaAQ61wEca0hkwDEtYZMAhDXGnIZDiCuNWQSgLjWEK3H EKo+ZZ0AxLWGTAIQ1xoyCUBcayj4VJ0Aca2hYGYpQFxryCQAca0hkwDEtYZMAhDXGjIJQFxriNZD ENca4p0QxLWGTAIQ1xoyCUAtRqx+xl/HCAFqMWL14/o6RghQixGABKAWIwAJQC1GyCQEtRgBSABq MUK2HoJajAA6AajFCEACUIsRgAQgXjVkLwcQrxqi7yGIVw2ZBCBeNWQSgHjVkKcGgHjVkEkA4lVD tB6CeNWQdQIQrxoyCUC8asgkAHGMkCtqAHGMEOtcBHGMkEkA4hghkwDEMUIuwwHEMUImAYhjhGg9 BHGMkHUCEMcImQQgjhEyCUB5zc0vd/Oq1E8mpTSwIHZ78RMtrspQk6bMgXf5HQu6n75tp95pPs3C /fQPF7/b/figb2v79be1NSVsxlHUyy9/7C9aN/uPDsTs73T6qnRz6Bd10yo1drOcDt1c9frx3fQ2 fzph6Obwfr3Su7E71LuQ3Ni7crr0bt3ZJ+hdWFvwxeFS22LH/Om6z8s7tgK/d6++lau+PMsGn8pn 765Wb50rGo0qvKPEKNpBo3q6aLRW8NEaGZU/obBWKqtS3g32SXmHsLUptteEdS0G/dChVk8sNX45 LdttZSiym/FptFs57YY6MuPj7RZ0fs3cqTPktyvbOqv4RctLj8cu5h7PcdXjfLr0eK3A43s86/xB YzyzVt2h3sW08sNyOvRu1dkn6J3Nnytd2fPNsfuVt9TdHLTPx9VFl9fa9bfV9XfovRLfkje0+tmB j14y5/PDPMfTgYtpjNwm5beBLqYpp4tpivDTmSbvgKpV5N5frgdu7M47yjBWvaunS+/WnX1877w6 6t23+6/Zmm8OQa8tuercu/w+qHENrKe9c0ddf3xfgznp64MDU3eVjw482wd0iAHdD1+zx36e38gb grfzSOmvaLw58eAhAA+vfJxOQuHKjNmqYW3VsLLq2siPtqpz5VMmKLDUlMjY0hlaj9XOmLBKl4x1 A+aHdIkiTv4uF+chcz99206tUz1dssPpg5VKp9mSLR/LIIfp4T1/M4+CzXr1Hrv17sKp6IZe1tOl l+tOP7qXPuQPnK16+br0si/Wn4zR7tKQ95p6cFWd0s5lPS+XBtrtckieypJrzscrGVPeF63zlTFa 3y6Hb5apIUTtQTB/CrhefLGEcmkp2HXBQU3hRaesQtfaLPf9Rmquz/HfNJ1cWKg3y7VPlwkt67w0 uSj1On/lW+nk9/+8CC69Hi6utM5eR4vupPtHva9H/Tjgjuvk8Z2Bafglr4IJKesSlkOebLrNzOhU P33bTk1543efmf30qWamp7pRLR/s5K5/3VLx65qkmlGtmn6zC7/u7norTpdVxnuSs1eVbWQLxNRP 37bTxQJrgzzYAhSuj18rbUz+5uFghP0vhpDb29Yp1RfQzq5+7KJ916p+TqC9X7kLxfqhLwZZuArx m1eN1mKTQQ3Si1Au72MHV032XhpjV03y93FcHJvsQtmOsYMsfNTLCp/0sjbCTXYh78t97WYvqxIn Ta6kzxNqX3NggztXP1oj6cGizZCnokv/umgzUBYF1umiVfGzWq0KnYq62aVtB3MhxjO9y8123nat 2thZfuVmbbadqjV2jke5mbKTTXdqjZ3jS010w0ful+Bpmyun+uL0Zi3qKZizWPRkLrIoG0NqtU2f c1pt0wKLFi86Q5XiIGfcsYz9GUqUYT2nvTweG/3jwVBqJSEGWhY6L9AabdY3lQItC50XaI1xK7uI gZaFzgu0xqVVL8VAy0LnBdq8xTI2KQbabaEWvNjgZwRaNuQZIbEb6P5Ay4qf02pTCATaTQerTnqW d9V5tOVatbGz/KpOti2nqo2d5VF1Rm65U23sLF+qols+cr/EVxf/B18RXXllbmRzdHJlYW0KZW5k b2JqCjYgMCBvYmoKNTc5MAplbmRvYmoKMjcgMCBvYmoKPDwvTGVuZ3RoIDI4IDAgUi9GaWx0ZXIg L0ZsYXRlRGVjb2RlPj4Kc3RyZWFtCnicjZTBbsMgDIbvPIWP2yUD4kC4Tpp2bpU3qNZKk3ro9v7S AGOSLAl1cyiFr39s89sP0J2xoNPDi8tdvZ093H5V3obzZ1n83NRDjV2fPnljub7c4X2KfzQeos50 VSZvG0DsQggD+CF0VhuY7uoFXqdv9TGpkzIhxGMHzmPfuRGMST8NeEvfP1/qOkOjxgTxIcMEQXpi jJa2N5JOL+gZCjZBfLiSrGlb268kLRKF41KyQs4niA8Z/hclHW+iJBGWrNAw5Pf2zSgpiY3kipZB VOFacEQzHFWb0VLILTrHV9FSoIQeVKeilLhIlRLaougxtA2GbhyF7kLf+7a1SEzkK/SxV5qmKmIS R6HXum2nIibxUkEbHnlOcNtaHxnv5mrFSA969hjd9CKjXIw91dI+EtXSFsdodpEglWwQwRvz3QuS yNcq0Uv30YiPL0PrFbE7aBmSDVpr7Pqle4OWIdmgtbEgy7rsDlqGZIPWYlhFuTtoGZINWuvMSnJ3 0LahMry44IJBy4UUjMRaoOeDlhOXqJaEDgZt02BkUpG7qI9a1iIxka+o2VqmIjGRo6gjW3YiMZGX CG155DlxUn/dWvyMZW5kc3RyZWFtCmVuZG9iagoyOCAwIG9iago0NDYKZW5kb2JqCjQgMCBvYmoK PDwvVHlwZS9QYWdlL01lZGlhQm94IFswIDAgNTk1LjIyIDg0Ml0KL1JvdGF0ZSAwL1BhcmVudCAz IDAgUgovUmVzb3VyY2VzPDwvUHJvY1NldFsvUERGIC9JbWFnZUMgL0ltYWdlSSAvVGV4dF0KL0Nv bG9yU3BhY2UgMjIgMCBSCi9FeHRHU3RhdGUgMjMgMCBSCi9YT2JqZWN0IDI0IDAgUgovRm9udCAy NSAwIFIKPj4KL0NvbnRlbnRzIDUgMCBSCj4+CmVuZG9iagoyNiAwIG9iago8PC9UeXBlL1BhZ2Uv TWVkaWFCb3ggWzAgMCA1OTUuMjIgODQyXQovUm90YXRlIDAvUGFyZW50IDMgMCBSCi9SZXNvdXJj ZXM8PC9Qcm9jU2V0Wy9QREYgL1RleHRdCi9FeHRHU3RhdGUgMjkgMCBSCi9Gb250IDMwIDAgUgo+ PgovQ29udGVudHMgMjcgMCBSCj4+CmVuZG9iagozIDAgb2JqCjw8IC9UeXBlIC9QYWdlcyAvS2lk cyBbCjQgMCBSCjI2IDAgUgpdIC9Db3VudCAyCi9Sb3RhdGUgMD4+CmVuZG9iagoxIDAgb2JqCjw8 L1R5cGUgL0NhdGFsb2cgL1BhZ2VzIDMgMCBSCj4+CmVuZG9iago3IDAgb2JqCjw8L1R5cGUvRXh0 R1N0YXRlCi9PUE0gMT4+ZW5kb2JqCjEwIDAgb2JqClsvSW5kZXhlZAovRGV2aWNlUkdCCjI1NQo8 MDAwMDAwRkZGRkZGRkRGQUY2RjZFNjhDRjlFNTZGRkFFMTM5RjlFNDU1RDRDNTQ3OTg4Rjc5QTY5 Qjk5RkFFMQoyQUZDRDcwMEZEREIwMEZEREUwMEVGQ0UwMEI3OTcwMUQ1QkEwQUY2RDc0N0YxQ0E5 MjQ5NDc0N0M4QzhDNzk2Cjk1OTU1QTU5NTU2QzZCNkI4NTdGNzVGOEQ3MThBNjk2MDE3RjY2MDE2 QzYxMDI0RTQ1MkVFNUREQzQwMjAwMDAKMTcxNTE0MzczMDJBRTJFMkUyRkJGMENDRUNENzZFRDZD QzkwRUVFQ0U0RkRGNkU1RjJENjJFRTRDMzAyNUE1NQo0NUZCREYxMEZBREIwNUJGQTQwMzUyMzYw NDZCNUYzQ0QyQzlBREZDRDAwMDM5MjMwNzlEOEIwMjY3NTAyNEY1CkU4QURGQ0UwMDJGNkQ1MDBE RkE2MDFDOUIzMzJGQ0M1MDBGQkRGMDdGQkRGMEJGQURFMDE5MzdFMDAyODEzMDMKQ0Y5NjAyQUVB REFERkJFMDE5ODIzRTAzQUQ5QzZFRTU5QTAwRjJEN0E5RUJCQTAwQkZBQTJCRjlFMjQ4RjdCOQox RDVENTYwMDgzNzMwMTk3NjMwMEVEQTMzREI2ODEwMEFBNzYzM0ZCQjcwMEY5QUEwMEI4OUMyMEQ5 QzQyQTZFCjY5NTdGNjk5MDBDQkFDMDBDREMwNjdGN0EwMDBGN0Q1MEVBMjg4MjhBNTlCM0U5NzhD NDM4RDgyMzRGNDkyMDAKQjFBNzcyOEU4MzE2Nzc2RDMwN0U2RDFCQkNCODhFOTM3QTFDRTdBQTZG ODI3NzM3RUQ5QTM4QUE5NTFGRjZDRQoxOUYwOEIwMDc4NEUwNENBNzEwM0VBODUwMEI2NjUwNEUx N0YwMkVCQUYwNThGNEYwMUQ3NzgwMjgyNUIyMUJCCjcxMEZDMzY3MDNENTg2MzA5QzVGMTdBQTU2 MDI5RjY0MzJCRjdFMjVEMDZFMDI4MTc4MTRBOTkzNTFDMUJEOEYKMDAwMDAwMDAwMDAwMDAwMDAw MDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMAowMDAwMDAwMDAw MDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwCjAw MDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAw MDAwMDAKMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAw MDAwMDAwMDAwMDAwMAowMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAw MDAwMDAwMDAwMDAwMDAwMDAwMDAwCjAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAw MDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAKMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAw MDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMAowMDAwMDAwMDAwMDAwMDAwMDAw MDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwCjAwMDAwMDAwMDAw MDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAKMDAw MDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAw MDAwMAowMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAw MDAwMDAwMDAwMDAwCjAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAw MDAwMDAwMDAwMDAwMDAwMDAwMDA+XWVuZG9iagoyMiAwIG9iago8PC9SMTAKMTAgMCBSPj4KZW5k b2JqCjIzIDAgb2JqCjw8L1I3CjcgMCBSPj4KZW5kb2JqCjI0IDAgb2JqCjw8L1IxMQoxMSAwIFI+ PgplbmRvYmoKMTEgMCBvYmoKPDwvU3VidHlwZS9JbWFnZQovQ29sb3JTcGFjZSAxMCAwIFIKL1dp ZHRoIDE5NwovSGVpZ2h0IDY5Ci9CaXRzUGVyQ29tcG9uZW50IDgKL0ZpbHRlci9GbGF0ZURlY29k ZS9MZW5ndGggMjMxMT4+c3RyZWFtCnic7ZiJVxpLFodFFjO9iBPcYkOqGxONAp10u7dCs2iUCIHG CCEqxOWNkwcCEZ9PjL73r8+tahZBTDQLM3MOv6Pn1HK7uF/de6sXg6Grrrrqqquuuuqqq6666qqr rrrqqquuuvq/Vq/RZLZY+h7947/tyPfLSNEMw4L6rQP/fHxr2qZr8FvLDA2PVFuDo7b2JiP3WOa7 ZHyiExCNcXbH0xYDpGvoa4vYeB5MeGiNCriF2hk5ed4moJF2Uw/WuPHZ84laZ/IFzd7QFHZgerLJ /j4QugmG4PVmOyNCOoJ+PBZGF8O6GbdHfPmKAElNDCzrkeUZcXbul0Dwg7yAhoThH0QwzbMszh3G 7V5YXJozjCs07s4vr6zMzxMIr9un+mT/zcpoCxEYhjIZarFpBzEEdsO6Ie+EyYDtjnq5p8aDDFtN f4YVVTm0qkikH6TMZipYxWCmRF5de/1ViJHqGBKcuCtUPUd8o4WBDE5bzdAGhoKNt/FDKPBAtydN +OBZ33iE83CuegZhXxlvv52X/QoemQ+GLSaLBSiqKcX4kPimUd63IJw1T7GwS43+jRkYD9zo8k7D IOIDI0h4GEKvZZ7RtTm7OjEuQYN1UWbKRTCYSBQCARAus8lkNJnClB4KbL/Irb29G+ImA8J1egeE s8kOQjMowPn0wDAE9Z0Hp/rFWFzBDEEzbLrZRTyVNAzBQCBMxjmjyUKt1CFoRUvUy6IVYqTJN7yz d0DYmg0fcLbOUUGGntqyTUzi7HEFcZIwzMKMtuBm2GWzyWiEPSeeKppCEoyyYAjjs753A25Gp5AA 4m3vHRB8i8N3QrSM8ffefewf9kMi+U5ZINfn8ZBEHHaFMYSFwha0BhAzCEUgEo8ejW5PC4LqqaYf REJL1kLRClHtOw1CbWKktue2RssGJVDz/SvHbhuNg6PeFeIizh7ssylMkkdR6NQA6+p7DnpPIKSE JnkQUq0u6tn29va0Q5BVzzyJEURCSaRrVdEeAnwbbaRJY68bDg9Vcep5FfiG8+EPtLSz+pQmm095 sR/wT1lw8gSxY5oyhSLsgDC9vf1SgOCwtJLQ1lUVIc+8a8dhe7m7t5PJTOnZREM6aens5DcgGl7e D+Krt0uDRU/lmXXYfPDb4gIGyROJvOtb/bi6t++DTEp4ZGSlI7Lj4ODlLOXysmwiocG9Aqkpxnso CDvvNjzvXK5lshCGSKSzrzsFMf56wmCB4qSWWYammf7D0aNnFhfsswc2Wd63TTsgTbyQOzJCKUZU f6NeUMEgFVzxJhMx5BMRmmLcs6osyLIgHLwL4hMKcpFA/KsjEL1mlolwOwxjhvcBL6MwrOzITB+H g15Ws/rcCPUPDCw5hN0VVktwqiqxKlJVWT5+ATfoD9m0KkqABmUEo6oqCA6H40VwhSWHk5ZIZv/d CYhxOPeX+wfoMXn6yAQ8CmNFvk9L+BFiQVMYGXG0d37WsWemvFpCUxGbguDYYcv3wmHz77kYykch XL4UzdlF38qLA8f09B4VhHLpIMSkiwkbLV6J5lOO6TDcwTSag4XUjDls/pBQJDtSaWZGcEClr6wn AYKBumcYjyw4CMRJ1TsOigCeRVxLtunhYzPcV/RsSmbfdgDClOLC73eXFSnl4Q8poNAUDllTyBG2 hJ+kZU7pQd4FUTgOh6nl9WxCROsejoO4qcKsOUw9KRRVfW27lYuIcPzuQCQyEAlyq8MQnYjE7z7R 43hJKYoPRbxU2BJMaynERZAjGKZW0nm4SqY9SN2gKBet5NIy8i/CGC6LTxQ87ZVyybQGp5NfsuLh w1moCdsGtcLiZw6ASCc7EYk3kU1RdFiUz0iV7csUxaQTGsf3yCmvy60k09E891kZiESmlr0LilbK nqKoZpWxvxH38rLXXSpkkxAeZFWkKP4JOKGETBDujJKGAwF/jzsBYRcXUvtP1tN2jpuRaElbyya0 Rb+i4HRYK8Ri/sXP0AFBhpcKybU1TdP8fus6DZIUfzmXTcdOuUVNUawipJZ6tgUHLCQTDgNAJF51 AOJxLBYbi2nJQpHjoqQSS9lQ/pSL4oROlot/REH50/xpFLqFcigUjfk/g8dYsNlr5yWggOiBYERi aLgDwk0SpjDDm895QwcgDH8WktlsLlcq55LQgGYZNjeZJJ1c6QS3wR0gyMJIsXiRF0VRxsoTzOK5 Kp5GYSdifr8eMogPOZfgujU+JnQEojd+Ui6VypV4sVwq5AqF0vklaYHWSsViEZoYh/DlSpVyARoE K004y5VKJQRPHjLkUk9Vp+RYAquoOhvoBMRcX5/1y5ezzaWBrXjx6kQ8nd1d2ilirkJUVMXVy8pJ FSmXK5ROLitXOhURblRgJKfHCyqA5BVBwBBi/tLwCyAG9Q9xIzchQM+NW2NjYxs+7nrDCo1P8Yvi eRmdHW7NjMUr52WMBDEqn//1/OK0eFXGPYKV/6OohnQKnILpZC1AREVVfNoCMTxENHjrUfwhEEMN w7v0fgY0NrP1yUcavt8uKucnJ+WTfCi+E4lELk4JFcGIoigkUggGAKMeHwhQjnRldbS+KmqSrebm jZeiNhBNg/Ur7gPRpH1w27cfj8cv48f7vghWHChO9ODIolwR/wqFKhA3XFC5JhV7hN76Qq0QVd/r r52DN2xsjVfVW6+nN6Jzb4inz3dB+1i4sbu3f22/vqgAxtVVudijhs4i9sgZYJyf61lWaJAUZ52N hVohhpoH+Js2DcTbHwps3wHRqms7VubsslIpVipfdvT+RQi6QKXXjo5SKPzZe+PCW640f7IJNEPU PW/9ZIOz6YchJv7S3bbvX1/vR+w1nZGU0jEABFDgzOY2Nzfth5mPT9tCNH080w8aoT4baEA02nXY H4WoovzdkGHiAJw9JMEADAJSLhUFPFrT4dFtCIOznjN8QF92sDFbmyPDdUP9e+dPgmjSnu6o/QzO LSgWKI6r8/iBfbNJr9pd6Rwlx33jK71zBPcJUgBPDVWLahCP20ad7Rb5SVqtOZw/u4hfXMTjh4eb Lcr8wp//WZo4yhzY7a2e6wGyH++1DcP/rF4dYa1mMh9J42ji25d01Wn1dNVVV036D6cyNf4KZW5k c3RyZWFtCmVuZG9iagoyNSAwIG9iago8PC9SMjEKMjEgMCBSL1IxNQoxNSAwIFIvUjE3CjE3IDAg Ui9SMTkKMTkgMCBSL1IxMwoxMyAwIFIvUjkKOSAwIFI+PgplbmRvYmoKMjkgMCBvYmoKPDwvUjcK NyAwIFI+PgplbmRvYmoKMzAgMCBvYmoKPDwvUjE3CjE3IDAgUj4+CmVuZG9iagozMSAwIG9iago8 PC9MZW5ndGgxIDU0MzIvRmlsdGVyL0ZsYXRlRGVjb2RlL0xlbmd0aCAzMiAwIFI+PnN0cmVhbQp4 nO0YbXBUV/Wc9zabTbKbfZtNIGEJ9y0v4SObdNNQMY1MWbPZRYyVkCz1bcroLtnAkiGyI2mLOHVX G0rYUIuxaiw6Q38wIjPKXYKaIIXoLz/KAEp1/EDriLR0KBMpDIw4rOe+3aQBtR1H/zX3vnPu+br3 3Hvuee/eeYAAYIMUyNCxrsvbBEYp30nosZ7+aCLHOxcC4KqeJwfUx9ybG0jwB+Kvb05s6d8VuFYD IJ0DKDi9ZdtnN+fsS14j9Ei8Nxo7+6eq9QAVQrgyToIKtWQAwLKf+Jp4/8DOvD/qD8q27T3RHG9V Acz/6I/uTJhHpW1kf5yE6qej/b15e50QS2zfMZDnhT818ZnexB8/d/E82V8AkIvM5VBM8zoPbBrP LqYe0ESb/bOBL07Td2PZm/B/KpY8/DcFz0vL/1e/+Dw+iykM4VPYj0/gVvRhD4YJP0PcdvieYXQI rqCKVViKiBo6sBDuYC1WoxNNUEz8VbK5YVh+y8A3sAXeloxowTDBabgAl+AtuIulcIrqFqpH4CXQ QcdFuBQfxo/ANRqdcgdehAxMkM1Pqc/v4TJMoQW78UlM4wuSTVojdZNdJfpxr/SodMdUA4X4lFSG W+QTeAPNWIE1cAJegd/JPPsGHoTX5AZpDHbCx+BX+BD65ENyncyk89IhAF9LqPmDKz/w0IqmBxu9 DzTUe+qWL1u6pLZGW+xW2aLqha4FVZXz51WUO8scir3UZi0pLrIUmgtMsoRQj7zSr2eqCj0ut9sd bsjzC+7luVyrXHdzKLvHyHVfp4X38dX38Ytm+I9zKOdBzd8mBs5A8DIHJ8dyDsILOh8lT/lOgVif FtjKq/yxSIR6tGmKyoNT3vxUjLEzJcV+zd9b3FAPmeISIkuIIttEBoOPoEFIwUBLRgKLraGel3m4 VBsQ0Md9wxEitDYaiTTOdzTj2cl9s1VA3aYpZ45CbvbzQsOvupX7ohyG1Uz9ZHrfuAKbIh5rTItF N1LkojTHDMi1gXhIxDEgIBJXuYkGN5CLJGogrqY1EY5APEJYa6Ne/1ZO4iK/vsc96eJl1Aa4w8PX kMWaXZdccjpQuVUVbDq9R+UH1+uztW6Bw+FwJU04HdBoQBos0NdKS6n0NtTn1pQPQCzSJ3z2RcU8 A31qerjXmOs+Yw6GaSBOGxN9L6t0OhDTArForDU3up/7QkYDoW7dWCCFri2cF+UNSGMyNJG2sDsX 7PZO3S8mpkXbXLltn5FE8hISBKaVqpjBWhqAqz0qh05dI9NmgXqbId3TbCSPO4zUq+OdXrygVtHU 9E3gGNHeunqvJJqXmGuVmyDIoBaMpNNBTQ2mI+noeDa1SVMVLZ1pb08nAhHy2qFTr/HsiWEXD+4L cyUSxxaKvciAYKe+2uV2hKfZjmkWKKUosUqM5VAU6FmbbyjKENLdKgVqgx52UZx0QYeIzrUikShx m2mP82ETMeptngmPP0+63SI7h8d9sIkYnlqv53gVNrmOgc/rof2ICM3ktKZig9CkpjUz3SMaeTkO 4kCt4JYlM49dmecMxFs4znsXdW9Oz51+XXZJ4RwluWRBFXvoTV/F53uIXuZJ0yac07ji4QX6pGtV WFUc9AUQu9elta/v1tVAeiYLcpL8SkUeUKpr0Xg6/yqJpKdPQWtGw6H1GR8OdXXrEwqdpEMh/ZiE kj/SGs7UkE6fUOnTakilGangVMFBu0jAY5LFULkmfAApQ2syBAbfM45gyCzTMoSecSknUwwZFbpa UHwsJXc7AUqbs313ttkOGxGbXUxCIj8Nn4RCiEMBSFR9sIFuIcezWbrHoK8o9MbrNez1yw5GKeCL /dqqrPT9Fn8z4mBnCF4h+AXBzwl+RvBjgu8cqGHfJHjxgMq+cWAZOzDiYn8brWDfHq1iXx+tY18b rWVfJdo3iqNkbr+OL4xUsa+MeNiXR9wMRlA42jhSoqy0n2QnvSdl748QJpQJyU7r/j6qt5O3JeWW est3S07eROWGekNSr3Vck7xXV19dd1VufDXxqjR2bBk7NuZg3rHVYxGe4IkLBX+9VMP+QuC9JByM /YQWIhxljxPxy+QD7DzBuaTKziYdbJLgNMHzp7KnJPvLmH0ZM0cdLHEUlcPqYWl4byNL7/WyvckV bGiwku0heHZwLds96GDPDLawQRpm+5GDR/iRqSMm30uobFQ3+jbKb9OIX0xWsi8kP8pS1H6ePD5N 0JGMJBNJWbG72byKOlZodrOqyjpmkt3MWVbH6hvsdZ7SZcvtS5aW1tTaF2ulqtu+iJW6FlbbKqsW 2CrmzbeVOcttdsVhtdpKrUXFJVZzocUqmwqsgJJVsafsks+cMks+OSVLdlgN6yAJJjt4ifRVbyfm NJyFLFhcH7Iwe4uFyQ9bGDRbWMcK5GXt0B5q5U6ktquVr/C0j1ugkzd52nlRx+N6BvFLYZJyaYi2 J8RNQ5SJITpDuh/Xx7FKqHcbRwpR45ja/dxzrhkqHPZU81h7l84T1WHeJIj91WHwUNkxsGPHDs9/ KJki4T3W2Zq5YhIHTpRf0doyb14xDh/+ptaG+a6zxyCSBp3hcs+sAp4nDPnAv7gzOom3RRKvTyGY 6ZWhpvyHZskEArxnLp4x0IONbofbUUsIyervqQK4I1ogIlfYXJ2rc3WuztW5+r6qdHrmb5/ldHqK /yoLCOgozXHv42KCxQY27uJTSPfuPCbeBLn4IJQZd3Og64QNIBQKNK1sbvrwisaBxkbjno/7oeBd vcwu9/1Gm4Kp7D0CnDbDGZBOwnvvFOYgw4+e+JR91U1w5Rx9d1vXbdH+oHGXnO27+wnbYQuSXZGx Hir/BENqizUKZW5kc3RyZWFtCmVuZG9iagozMiAwIG9iagoyMTE4CmVuZG9iagozMyAwIG9iago8 PC9TdWJ0eXBlL1R5cGUxQy9GaWx0ZXIvRmxhdGVEZWNvZGUvTGVuZ3RoIDM0IDAgUj4+c3RyZWFt CnicbVgJVFPXuj4ROee0ICpprqA24XodsFrnOhZnrRNIFQdQZgIEQgIJkAABwhCmzTwThgQSBiFM MooD1mlV8Vq1avva3t5qW+u9em1vh/ufdPPeejvY++56777FWmednL3P3md///d///fDo6ZPo3g8 noOPJEasfHunXBpm+7mcm8fj5k/j3rRT4pBfJdZ99m9ShxrtZyBHO+Q4vWW+oMWZc5kNQzMhcxZl x+Opy0y75LFJCklEZLyb+7EjJ5YuW7b8X09Wb9q0yS0k6Z8jbrvFSkmEzG0xuUkUS+WxMWJZ/Ba3 XWS2VCoJdYuQJsVGKt2Cw8LEYbbXjgdLxdFueyVSSWysPNHNfddStzWrVq1+m1zWeEliQhKUbkfk McEyNy/5JrdDbp7iMElCzL8PUBS1+tAOWehJz51yX69d4sO7w733KCL2Ko+8Jzm6L8Fn/zFp8EH1 ibfc3l6xctXqNeVr123YuJmi3qYWUIep3dQmagX1B8qb2kytpBZSe6lV1CLqPWo1tZg6Su2jllA+ 1FrKnTpAraOWUsepd6i3qPWUJ7WTWk75UhspB4pHOVEzqVnUbMqZ4lMs9QYloFwoV2oetYaEgJpO naDu8rbxSqdtmdZht9pObfdyuuf0W/bL7DPtf6CP0w8ZH6aXjWE/ey3iNevrua/fd5A5PHXc4Vjs eHPGxhnRM/7LqWjmkpk9s6pm1c6yzLo+64vZm2enz/7G+XfORc63nX/hZ/CHodjJuhKZFZ1Wyuz8 M6yfw4+1OoNKIKOjPFTSyOiQ8KNJWxHrpa4dFnFSBj2qq31Swprx3xXMR7mNKegkO6lm+H3oZJp2 ZyarYPix3CPayRpOlmwHj7PcXoMzv8+6xyzIpcNQitEXXsdKrsVFTmPF5Kh4RbrmVKYrP1YBgWQR M+NdEl+NWlFthdn4gf6sCyTQ3R41ESiB5ceixLSMyHyW35eD0gpSyU0s2awPPqUn80Ar+Avk/oBz 7Z24y8jEvWPgPQQRCEFkB5FcsAAfVdIj2ZUZKJjFYwwKzsoMyGSVoDMx4aXpetTHgoFBJeVVVZXN LaaaFsT2NMQeFeEcBoWnpUhyydR0ExNWnFGDhlgoYR4fv7LpsL/cO1zoxHlzXmd4Zvg9OMMsO24V 8AThwXGpIYhd5/ME5v/YdWu8Xp+lLROVZiKEggv9GuP7EDvUajp7c9/5tXj+kt/jhVj4rTvM+vRq 58ProlfAtVmnm535Rutu60YBLlbQlQUVqLhgIL8mEylRojbx9MGsjPhoReRb8JZLmiG/FNUgY01F bxG5KawrJMF5qWB682ozOgPAD4+44F04LiZcmearc1VAsZnxKlNVITNqNnQ9/L7/kssPeH2NulKJ tK78IJScnGkD2ZiLtAVpBawCnpkZSaGm9tTH+AD0unwOdl+DY2vTw3JXp193I7OVNvN+HuLaYaWd dTP3i2CyjEFHU9O3Z5M3vzIzHlUZ1eg+y3kx6GO9/kEp+bRSBp+elOLTnNQelynoB7r6NHSCnZQz 6HBaukcWeS/NzGwqUzehhywXxUDp5FP7NvoVR7tg4gy0nlGYnX+CdbAKNmEhbJ/D76knOCmYu7lN v9ERndSk78yykTEBHtHgAdMfgf2oejCyUxR5RqLfo2f5icB2Dw9+MvfHFdfwFqEXfiw4Q/PFP148 hqe9FeK7OTC5YUTEFU7nJ/bqa8/e6wp6e0GUHM/MIxHPS+3kPDpJzJfAalhiByVcjADz3Rfg+fgP f10Ic0Dw/CdCPrdlP2GBqFAquN7kgzfiFaoT73urLsFi2N04fvNVnG3gQSest7MetgU6VkHfyWnI QjKkTo1UhGA77OmCHUFTUJxTiZBrqjm3GlWzzfqKQaG50N6Mv1Yw3fmtaoMEDuKvXPASBu3UajfZ EPyeIFiRUUUQhEMMHIUJfUtTdXepK9k1yZhn29f5pxFu18gc/igk27aOUNAG1KA5f4SdlDF8NTqi SbUlM6jMzNbKpCZbCBcz/FHu6OSgfTv3mEbXamtvV5FoRiuYh9lNiWg3i68zZHn1eetrlniD8wXY M4d/17oIlgu8PaOVfmpZepROjvajQz2ycdnl1I/R9yyETsA0cBwbThIPC4fDWqJ6trFmmqR0WiyT GqnVqlEm0pQkVLH8uybvU9Wn5uFTeClegYNwCCzBiyH0/o2GviuiruZmw+hFlqRi6sdwuA/WdfK4 xTBbEOonTQpCYUjWmNiT1JVlyb3LQimt+7NmLLo/6qy/wQexePYGd7wYL/7TMpj1xYWuZ7dFWA/b BMXg2DhwAbWjppQKaam6SF50ktWQ0zWpLVYXC68H9sA/YI8d96b1PQEWk29yx2E4lHzTEgglf0vA nVyD8FJYhv1FBVECmDaBZ5ApwZ54GmYw40mOHQDSW8DCa79lfA/3Uzeh9QtYD1Wwcw6/4cVTAQpU qN7Pt7FYCndpkMJComihEILdYDZOEfEb8HUFPaFr0iB/Fta3CPjSFkPN2Y8vRe3tFrYdr9uJNqDl qneiAyT+p+IPIDYgrXZU5ASPiT6+OQKnOnn3QXQNnOzuc+8I8GMGBWnT/XOI1HWamD1F6TXoNgt3 noysAyfmir65p7E6Oz03L1uXIczU5GpRBittVXb3mc19N7wHN+w7oQyMFSZEZoegd9gjgbQTB+RM HdDWA1d7yLG+J8faPJWpoIerAphX2F080vhF58iH6FOWvx7Y5X8kScjvwY8U9D1dXQraw056Mfxw 5JmmOq1j+T15KCNfS6CAJ2bm3arMSvSA5VKn89e3NNRYvrQEYupQujguXqhURWXvRuxvBQ0udcN0 IyzuntILUtiGuW8J2ScfKuh/SQV/GJ1I1e6YEotk7k/0AITZy2npPpU8QuYf7pOyDbEHbaUPchjg ocGwGsyyrfA0jtZgOiwCU4UsaJj+2spx2HBBgl3NQjM5vTcywTVYFGFy/hJEcwijZbYi9L6SHtZV p6HQqSIUkqENzCJwN5kYr5L0WnSThfeZ8fZhS0Vldkq1kH83tkmf2Tqvr7X97KjEctpXHLvvtIio Dw7pAidwfNIG0suZnc7tX78Lb4ILzAv/bA7/7xSXyC0QXMa/Y/jPqfpAeVnoPOywYf9ySW1Ui1zU rGxKG1O9GxsqRidQSGPSRDzL/1l7KDNKETL3/UcSUsI2fz3+8lPPc4uEB+gtqvL2M+3NA6LRgjFk Lus2uJLlR1tbBi/PRQPpZ+Tt7NbJ4wLy/getyVH+PnHua/1bRozl1e0tIrL3N1ymYKxJvGKFOuL0 qaTeZ39p6hsTTdECrv2fqLzgPrdFZWIqKhpbLSAC/ltMYNLMrCxJqkFXWEhl/p7fF15H8DfDUwWd hOnIyCXIBj//577aqkvf3UiO7BQ2K6pj6w6xJhIF1zRjooXbaEkwOLd9D8Lnc/gWeAafCfjzYVEC o9PoctKRDmUUpZSyfGl1VHxJwrxV+z037B/0/yFI9FFMuwJFs2ExcQF7/EefJArlNN+CN5mZamNZ eSUqRRX5Vblt6Ze17Yh9euvmt/fDht2HRO91yM2om7W0m/tb27SJzUKjqjpJH8hOJTkMdsFF27GB JvlwELbN4X9hDTQLklBaWUYt9od+lxtN5tp2c0db/QAaZfkvgHG/hQ8K+V/YEuNhTp0G7bIlBtqZ keFhk+en/1Nh4WQbA6H4j/YQTE8e5F4IUFmJvriMrFFVVl9qnvddX9CCg5qw6ERRXLIke1sBO6Vm UGjmGJtc9xFJG4dVJBjWXBgQ4OcK+nGOXouOsQuYk7KEfYeCm4ajhIFXEx8jcGRBfAvsYMY5Itwj wmFxi+SVcH/x78L94n8JdwgR7sV4IYT/CDxT/4QI+MQWVNc8qmCdoBMZuY2GR7CSd9fm2sJsCeND XFtmZSYKeZUw2Vn+WpIwWSYmsjS9DvWze7h/CIIiI4MDLFEjI92WkWFJTxAh2WW1hQs2wsZOtcF5 BHigAweCNKfm5AKIiWGy47KykskXaosTKsgX1p+OLhPPw/M3bsXOa8cPgcMR0Z3or4NNceiE616/ gK3v+5oviIWxfamW5HusDKMOusxcUVmPKlB1XnP2xeS+7GFCQvfvvwLnR/6PDnaJ8JIHmwcSzOi8 682h/o9Gh9ThvcLuyHqZ4YANdEKDD8ag08z7hXBAR2o/V0YoiZMZ5JecavMsNsO1qzKlFl1gOZII Fw31H9ls0zMFcy/HkIJ8WRyKdRA6NXLPNvKtgrmja9SQEVvi+GnS9tio8a2Z2VmRXEdWgWSGYMLt N/Fu/9MTP08TDOtNJbfLb5a7tDCBRUR+hlmYxP5K5mJuTToKYnEvExWVqJUjNkRjuCSC77C3khnM qsi0WehSZu3FE1+O99RfvC6c0lruuBHWGXhQRs5DLq8J0A6t9h2bB7SamXVl2ir0CcuZGJiZO+7f gB3YDjof/O0nX9B52N++g24Ah/FzMKuE1HDX31yRvW0l++cC5JuUOmUmfzQz20rVdWh8CpVxY/3D YnL2HxXMxzlGDZFym/e4YOWZeX+DtfBoxM5qx90SoBt19RM2m0Jahf/QGVRop82mYPeje7HDooAb MF8ITxg0UNasr6upqi85X0SmPiVAZ9ekkUzDZGyp0e/DwPORE0kP0EN0r3m4b7S/9RP0bIqwJitr 4t2x4Rli81BHlHRPXn0SCmcny0njkKyS5L1ia1RhShMaIIfL4zw7nc0kBfjgSmq/kDgkLmZCkJGR l5dXkEMcXnZlvQjqmD9vu4ydhPw+bL/t5PYQk2JwoN3UXaerTa8XZVbllaEy1the33O9NdZLuJPB DhF+qnAFMUlBMYmSoLleYwET184ZLl4RVp9sVl9Aw6izvn+QxSeuCWKlqnQlYmMS2wauDw19bhRN 8XEK79dsLrTKLFAUakvT9Pg65+QC3nC81/yigmAypGAaCuoKSgt682uzkAql5GRJpTgGy1wgikH3 9fUPS18hdze38Z8KTiz4rqxXRPyNznCSaTPDAaxqUpJuSOuK0lJzovJJN4dSbY3GN4SmecScrDbx OLVJkFKaXZJdjd1B7fIDODe1GKo7S11N+F0lQ7qhgmLUUlChQxqkyUiWR+t0GrUmUVeeW5ZXEj0U N4xqUU1dmaWQNeFiJdOaW5ZjiIE3sK+LLDw2KjP16A5Vcng+CY6HiYkoSi1HeqRvHeo/V1RcXlFe OXTIpSWqSo3SUXqqTmqbpmOcftVojFZXM+9XiS3UBxT0B3n6DCRFafmanFQ8HT93wRR8k9GVq0eV rsikrxwpLiJFoszWjN1WMIP5hsxqZc8yF/w6/hA7clROfX6FbWZHTeW4rZseUzBDuqrMBj+InHzp 0k6DFxcH2ybjSOsDp7mXtQMtRkuxKwEIvvvl+i/OEPjSu2kO/yvo5GIFFmNNyXCRCVUW1ZQaKuub USPbqjDFJqRnSVXCuPCsjHAZYUdWsX0Tg8ylTQ3ttS3FVUjPWlTN8piEBHlySUZdnNAoLtYWBmfn uYiNujLFQGK3pjoWsTK1ShZRnz6eKzqXW587kMRmM9EJGXnB+exRmTpc3pzQbqyt7DSI2s5WVPS3 l+e4FPjnZ6gkLP+r9IS8zJS5kqb4jo7mlo6avEqtUagayNXnjJaXunRoKrIN8i6/+rQ2xLYbm9st 6rpjJaKAEk1JRANJrmLiU2EbsDwYsJWjXMxCLk0ew2xYJfAXhwWc6g0fO9/fd25MfJa4IuhUG62r iQBdIY58NY21/+ln/y1xAHlp7VbXdp4J1sB8WGMHxdbdggNof2DIrmifhLfRRhaH/IWYdlfSssFS 8IfTbxHLLhB+NBkuQN5ZHpmxWllKajxifdU3wF0ESQwsa7p+/oLBEy8RTpHV6kI29QJ3O7hHlsal mGcfx1QWVxQXF7VVmctrEWuuTjgtmhQwaJMq2b2AVXIOJmZVUbKeGF/uzhTjwZXzBJoHl2G+HaeA CIGSlu1K1y0i7ckym1FY8dmjnjvoE9d/bP0EU76nVBGRwrAwlUS+nW2nO2/2tX+ARpBF3RxRk1gh R/EoUO0T7XfSIzKYNDar2HhuHUOwfAGcANg7F/+IHrN3Q8YO+0ZF+Z3qkI6WlqLCMmFxYREqQmx1 RUa6ODJ4xwmbzUwFd2fohcXbgZ7Dv3ydMwj6P7S0jCP21tnQQ/sjI5fHiJKZL2WW5F4SbGOnrDHg 1NxdYv/DSnV1V5Qw7GxMfQxixbFKybEPI76DLcSh0rD4nGZY1iNKNCXXhDay/MsTZ5s6bsx97HVp kft234U+U5AaYSbYcbZYGmCR7R8qFQJuzxbGeyt+A2/Zv/LwcldsD4XERCyChcSFMDjkd1hMgxgu 2k+9zunhTR4XZ3OU6+hArLA3000TJuMVUqNDaXx6stf+r1PE6LK+0eHcDjPwNBDAJhBhV5tLeAB7 4VPBEXV8WhxR1GcyuUYcOHfRV+8BH0TPPieuO+im20dC/oPn3QODE3O/2XYVO+B5a/Ys2D7g89Jb CI5/E2wY9r+CrrJjvb1jV7sCPE5nxUcFCpdiVsB/Brydpugdc/dFhRzzirj8zZXqzoEbQhKcNPAg PaMHj7RI/w+54e50f7E4wK83/PxYfz951k8IrzJwwQY4YaAtr4Obg8Xi6AhuLY4zKOq/AWJDf9IK ZW5kc3RyZWFtCmVuZG9iagozNCAwIG9iago0OTU3CmVuZG9iagozNSAwIG9iago8PC9TdWJ0eXBl L1R5cGUxQy9GaWx0ZXIvRmxhdGVEZWNvZGUvTGVuZ3RoIDM2IDAgUj4+c3RyZWFtCnicZY+xSgNB EIZ3vUsKvUQjWKk4lSQmSiI20TJBECVCiLGwkNNbz8Xb2/OSKNfZR9NYSywUNJ22Vnai5BGEvIHl LFxjIqhFmmFm/u8bGEr0EUIpNSpcsNpiWQrTHcwLapqqmRE1qzUvVEe1IrNk60aLNQ2taei3anMS 3xP4Oo4vE0Sj9OnjsyC9wOf2cR2SO+XdVDqd+d/k8vk8HAS/CRRZjdsuzPebM+ZITzC3vgaFPu04 /BBsJ/COa2BaFrMGWtV02Amsc4d7njyDZCEFy9lsbrFflteg1BDMlxng7hF3eT0A07VgWzDbBGFa bHCgKHjdD2Aly90/u8TFQaMGPw9DSeZhC8rMbjimP5wQQvQ0LK0SYhBKxsk0iVCqpzLFAFtx9da8 xb0uxauuhlf4PBXKDZSh10UPZTeU6G1E430OW5j4mhL+qePe+Q+d+7uHzum9mItj5wI1vESN4oza HwbCUB+Wztuq2sZKO/o42ht7vDaM3o0RI+QbzHisgQplbmRzdHJlYW0KZW5kb2JqCjM2IDAgb2Jq CjM4MQplbmRvYmoKMzcgMCBvYmoKPDwvTGVuZ3RoMSA3MzY4L0ZpbHRlci9GbGF0ZURlY29kZS9M ZW5ndGggMzggMCBSPj5zdHJlYW0KeJztOQ10VFV633fv+58Z8jI/ySQDOsMYEh0kgRAY6GBGkgnR LBp+lAkQmEAIAQIJEgR/aMi6lDhgRREB15663XPUcmx9cd016lqyqwvscuxq2xV2tUW3nEJcqdQq UpVJv/tmAsTT7fGc9pyes6fvzb33u993f77/d98bQABwwXbg0Hj7gvIpYF+B66i6c+X6lq5sv8gC wIUr7+4O3vSr254mxHvUf6+ta/X6+In2HwOwAQDlz1Z33NOWHe+i+Ua0fVVL67EdH6cAxu4h5LR2 QuQFlW4APUn969rXd2/N7Xc/VZ92dK5syfYLl9KaC9e3bO3iz6k1NF7Qgxta1q/KjZeourarc1N3 tj9W8BnsumtV1xttD/2Cxj8JwHfKF8ANQHUR1Nv1qEt6IYsZPjO6zlhfG5XHb8zCbGgEO/xv8I0u LVf+x9e3oIp2fXH4OMFtUE3wE8M/IY5q2ZOjuF0mLYPq4Z/gL/AVJD3gi/gnuAoRyWb4BJVDeAJp DTwN37UlSmIb7sFKfJdwH2I7a6DWxKXfhCV8laXt1mXzJLiqJj4Fp1XwLtbiePghGvAa7bQeboVW urtgiz3Dh4GrFvoAGoev6PwIHEZG7Udw6/CPaId/xWV8EF6FL2Av9sADDFCHE9gdnzkjOn1a1dTK KZMryifdODFyw/VlpRNKrguPDwWvvWbc2EBxkb+wwOf1uPPNvDEup8PQNVWRJc4QJqLlr0km1lpF NSmrLlwbNoNW3W3n55Zb4A6EwvmVTTdmh1hyxAJPg+VtTPZDPNpkKZFR9NssXmJ+EqJpcwPBhCWV 0C98a0urVTY/GQqb7wQu05toilVckwyFAhYrod8tRKLfrS3BVstsJHwokMXcYkFjUpSB4d9GCWlG Q00BC+YnrWuoOzB8nvpNTf8Fhy+TUwyO4vE2TJv9dUU1tRZ4+6Hutxb4xKDzUbAgZpVFiA2TIFrK b0G5hd5PLPRY6JtLDI9aX8x6P/p16ROta8OJ1jWkxdbUFT2ez2oxFEwH0/OT+ZUE2tz2O4yacM0q 48aJ0G84CHQQRLO6+rHuJrQBVpeY2c9Ac5Gu3IK7hChrrfiuFAHhWlISUTxXKAPDg7uvJgFNG4E8 WQgtpcZS7X2Da6x4iwW7gv0TB9O7B0xYkYo4W8OtLUuTFm8hHvuBlyTaF1pjGxoXE4o2oZJqDwqr 1tqVsFEw0R5MU1+MTVEdrhW2HYVvbV+VEt6AqXAt0fSa5M7QYMByU5uw8iPWHBo2597TAZ5O+NcE RTed3hm0npqXvJoaEjWZ2k+spxNh2o0WS6ydLexVPmIf2+duaRWGaAla21eszbpXy+4R5w6lTavu QojMQIYYmZVTYGtqrWB3bYsQMbE2mN61yhZzty0WuWQwsbZWFDGRHBzuoNmLk4n2cIJ0uSu7IQlN AC/5+txQyCqKiInpdELw19JKnGf5JcIV5oXbByJI/NRY8YV2Awtt/dOO8ZbaphwqN2CxmCYoqdqm plDWxg3zkzVCnnBLbSAr5WVMKochRGKEKLgN30IrWMGVQRFaYRoaFdWqKKRXRm1dhZqQZjVemWXJ JWY4mP4MhFnPfTQa05LDKCXmZyDAunBdKp2uCwfr0ql0y8Dw9hXhoBlO9zc0pLsSqaAd5Ej4V3YF rLrdTZaZaseZZGjhc3XzhYXqgu0t2ZRQHQ4RT/lNI+TG30cG8nTyd4qstEk8WE5KOIFgnRBRJA6R TEQs0u53JMn/V9q+alcUFwto1YCIEN5UklizICc4eWHOWURam5fD0iKhkIidXQNxWEEda/u8ZLYf hBWBFyBeHiG7pQRlcITiu0NQto9QLk9PhcmGDQtG+fJl77rszle7cjo/7A7OKLe5COWyQk2SB1hT FmIBLiAjQhkpZhVGCGYlQgOU9NJksrfClhmx5JrkYCDWFDTzKWMJYy8IN8xbnIyO2Juy41vhn6PI ieA1LYxZWCDwQDnSTtS8MErEyxODiXTO2+yHHID66aUMPf97M29mjmtLBWbUNUtg8HpoAJVgCRiY EICZ9FS+YXiYzmMYH0bAa7AOQUcO7psTW0Px2q2hU+8XFI79h19Rdd/9BYH77i/K6yrvqu66vetU l5y3oXxD9Qae14F5HeUd1R08rxNv7zzVyW5fd2ode/vvaNLdW6ha30VVRydV6zYUBNZt6LmrGDqC HakOq0MSQEUHL+/s7DzcyZev6Vxzag1f3t7Zfqqdd2/2+sauXktV2xqqVrV7A3tW4ar2HRuLizYV 3FtTFLqHSsUBrNiPFY9jxT58dC9EHtkrRR6jdu8+KXJgvxTZT+3jVMz9GN8/uJ+d34/mPozvG9zH zu/D+w7vOvy7w18elsz48IfBDxs/5HAQ4aB5MH4wdbDr4ODBtw6q5TsxOIQwZA4FhyqG4kNdQ9uH 9gwNDr0/dH7IcQXdOJTKkZ4asnLkPDjdeLrzdNfp4dPyw6ex6/Sfnx48feq0ZPameq1eDg/g8zuk yEO7ILJ7lxRJU/sgtbt2qJFv90iRXip/3AORbT1qpIdwfTsgspPGf4faB6g1e9HcieYOHNyBgek+ /zSfr8rnnurLq/Q5p/j0yT6lwsfLfTDJN6E0r6w074ZI3sRI3jXXjglem5dn5judrjFO3XA4FVVz cinul52AzBnAcS6/WuzymYUut+R1NVai5W6AhoWzLQ9Su2C2VRlpGODB+daUSIOlNy5J9iP+aRNh LdY3gJQ9pb4BRo27ZvGS5AAWCfKOwMuACFZDasdDTZFxVqsIwu3jmqwpAtgzrgkio67+sgkJ64ZE izUxkaq9moDdmzdvHj02sql7U/dm+lGzaRNk4RESXRihR4DYaHYkYuO7aWhugA12d9tb6kKexvmz G6zY/AYrr3GJVRymzjHqTKOOMzybJI7QmNb5lhy21UBEOgwpEszuV2H2y2AIyAGzA9gPUEAnIuii 09BNVjwB1s0JyxGxDJroCM+G6mp/xIzh3eVRv+K0FMKq4dlNFJAidFVQ6KWCGgjlh/JLqELCfLFd hi9FCwTQSIpjkNvobUMFAz6KPznTWGykjUPGbwwZ6w3UdFQ1BFWaLjGNRzkz8CS/yBkv1eZoRzXu 18v0ev2YLqGuKoqhkX0MSTGUXlnyynR6RQN7Gfcyxg1OB1pV26Lt1PZrz2gnNKVMQ+1RDfrgJHAH sRKXOJcVZG5aSpJkjqBJzDBk5pS7mIOtZ2S4fPeMWeXuwhn0A391LBarjhWb/7h8WfPcc83NMTOW P2OnPCmyc9sbOyf5RaOasZgokytweXPz8mYaSXUzUIU8hHRzXsmZ+UeZctzEHsZUZmrsq0MLUcp8 JV/4wkCJ3jc2U96j90T+OmnJAd+O36kZfqPMiBr1RtLYavQZBwzd2KvFVa1QE2pZZIuo6iBxbW+s QlP9alLl6l6Jq4Ak6N44dxsGA4W5WCfToIfHnXwdi2Cx35x7KRZBEsM94/5yErB47jniuPpccwSo FfRzzSSKMCjdPipsMj761fd5c2YDFmXOYE+mV76Q6X0t0ws5rn9GXOuwOn6zqhQqpcoiZbWyRVGU vZom+aWkxB3SXhav1w5oTNOS2lmNu0BDxtGtqhI4pE6mQA/GDbyawRH2qs03iK2NxcTWsmZTsHY9 5ueYY5MzVXx5pgNLMu8KljK/G82TDBPiHnlvHN2M9NLFJNpFubzLuYhYnsQfWZL/7Kvv4XF8TL7w 5YPZVeSksAc2xIc1csKozmf57bZeT+rt+gH9WXLLk/pF3aHp/svoY/pZXdP0PqIO2J2LuurQHGWO qIPm2229I+lodxxwPOs45jjpuOggsv8y+pjjrEPTHH1EHbA7Fx00nzyBuUAyDFRkig0N3ZJG/pE0 2g1pppbzlHbbVwaMY8ZZ46LhGHGiEcJZQ99qPEvUkzZdwewAMbPPUAxDRZmDpvm1Mq2eDNWnKTpo PWrcpcaNDUzYyHmV9rINGSe/cgpZq6icHH7jXcuaYyJqRNy4Z5RXTq6gActsfLYWqg7Y3uVAYT98 7ZVXcTBTZT1HFvwp3vSFwb7K/BpLbTsOX5DW277VH++ZLmOUQl+VC+XV8hZZKpRL5enyHHmR3T0q axr4oR22guSHMohCPSTt7jHQoF5HlEHS0a1IlCNkOlhosl8uk6Nk4Xa5T6ZIknu0uEPbwFQohFKY A4voxVwxAHpY3GCXpSZprhLcPYN+QNIecc+YYcc/kA6aN1L42x7lGfFSrP8Uj2e+j7/MTCYxGWa+ EHmxenhIalNMcIIf8+JvHpWQHTMQj7GTjJ1APCMjP0qBEnVh0oVcc/ldZa6oq96VdLW7trr6XAdc z7oGXMdcrrOuiy521HXCxcD1yACJTDl3ugcXeZArHmzzPO552vMjzxHPO54LHrXQU+qZ7pnjWeRZ 7dni2enZ73nG85LnqMepegpzBIE+49E8HygeraCsIFnQV3CgQNbhZMHZAsYLtvG4xtHJinknO2Ge MRk3t+lxVScUVX5epHeQvkhBIwllebPdzcEEAeUboaeNd1U3E8aGbP2RBu0c2oxTIN+E8HjIn+qe 5hmv+LwFlVOmVU2dwFr/Fqdm3v74eOanfZ8cnDgrduOkWEwxM1WZ1zPfzdThX+N3XsOl71364ZJE 3eKmOYklQtMUy9Jm8iQnFMCSeFVpPuY/qusIbtWFrr9BftKLmrfey3TwblPimoJcVTAP/EoXy3P2 QLwQrs5NQrLmESfYKGSpJl8XuakyPzSlwO3zsvD40nxPOL+y0Oa5DV+9+ObDPz+w9h5NU4pm3lwt X0gMw0v/nPn1jku7pOcyL7IlCxbX5fg8YX+jK4Yn4is/l7B0DB4dg2y6hqpeqJfq0/U5+iJ9ta6+ pB/VP9c5CCRjev6jPh86SSDvFi8740XvY8h1tUhz9jmZs4cyTFFfEYlXROE8Vu1kPjdJFRgl1YhA ldvK/eeabcemB161CONzM3LG2SgKNAsxr2E+r+r2hKpMWsOWVYjahsc//outt2z+y2Am3NasaJqz JiZfCG58ff2/dGfOZmaNQRcWXzou/Tjzm/rbH6YYL6Jc/YJ9NpgVn0BPhHqe5H2cq7yQl/LpXHID f0TiCp0yKP3QUaEnVpFj9Yh5BKoviadtrNpO4WGKtjA7nqk6ww+e4Q+xNZf2iyxyJ0XZMtqhAO6N LzzqOUGezbMBwcHzSJVGkeWBMXzE0SW9gELez5KMaayebWUHmDQO/GwDy+txxQtdi1xMdc1xbXHt dxG+0LUu6+obc0nxMiicPFa93HZv+3nqHcPD4ychKxKw7cxsLx73zdv9g+YtH2yfhMcLri3xyxcu Hfv8n/79r1KP/QeOx/EsdmlW87zmR4QcdNSSzpAcGhyML6XHvMrRMUGZphxR3qGDXZkaVZnIiAw1 FfPIf5Ve0LygyKDJBHPJyxWVSxwUTnnGral0dJIFRkOJAXIkqXWDXdEvHXVM0nF+ochw4hREB6Kd 2qSIvM18g1q/DQi/WGYfd5ZTgrezHor8zpdSQG7Gt596Ct9iT1xqY08Q5mlcTOdB3//S/Wz2Js7n XnV3scnse/xOfogfksJ0L5QGsrc89v/vP8zbfsfIfknwUp4QHxqKqVC8QOV1sYqyydP59VOqouXf 5Av+H9gliW8nVEtCP+ddw8NUo6ipL/4vqoTrIAYVdGCaDNNJd9fDFKiiw1O5PcJNehX/OyjgAli4 MDElOrVydm20u6LC/paDe0D+xpx87S+f83B+eBQCR4bh5cIOwf95kVeAdHXhQ9D2e8svoU0+Cm3S 31N5GqqlB6kV5VS28DehSHoR7hxVzoOS9dn/5hK6wafb+q3nX1meF/sMAlllPs9feEa0P7jhW/My b14ytaXqp9TVR3T5n5MYPM4KZW5kc3RyZWFtCmVuZG9iagozOCAwIG9iago0Mzg1CmVuZG9iagoz OSAwIG9iago8PC9TdWJ0eXBlL1R5cGUxQy9GaWx0ZXIvRmxhdGVEZWNvZGUvTGVuZ3RoIDQwIDAg Uj4+c3RyZWFtCnicY2RgYWJgZGTk8kjNKUstyUxOBPGUf0gz/pBh+iHLLMvgM+cZbzcPczcPy6Lv jULfSwW/F/F/zxdgYGZkrOid5ZxfUFmUmZ5RoqARGhSuqa2tgxAxtLS0VEiqhMkouKQWZ6bnKagB GWWpOfkFual5JdYKzkDVOTmZyQrpOZUFGcUKiSkpqSkgbWGJOanZCm6ZOZkFBfllChrOmgpGBgaG ukDCyC8zN6m0WCE4Ma9YwUchKDW9NCexCEWQgYGBUQGIGZgYGVnYv6/iA6KSRT8WzP/uO99nEdtN rnvcNyfw8DyYwsPLwAAADopS9wplbmRzdHJlYW0KZW5kb2JqCjQwIDAgb2JqCjIzNwplbmRvYmoK NDEgMCBvYmoKPDwvU3VidHlwZS9UeXBlMUMvRmlsdGVyL0ZsYXRlRGVjb2RlL0xlbmd0aCA0MiAw IFI+PnN0cmVhbQp4nFVUfVBUVRS/j933docWbFkejqLuQ3SB5UNAkQAtEQU1RQGllEkEWdjFXT5k BRbJlAxGL5OU+DGRslpIg6ZrgMhAiSk2MiaaaRjuOI01UzQZ+NF5O3dtug8spz/ex5z7O/f+7vn9 zmGQ3AMxDDNpmcFcbrCaNueELy4250mhUNGfEad5iNNl2OqqctnY6Wjlx796YZUMq+QnpnEHfaBH DQ2TwPYykjFM5d5DScUltq2mAqNVCF6X/kZIaGjYi0hUXFyckGv7d0VYYigzFRQJOvpTbjAXl1gM RdYEIYmizWbTZqHAbCsxlgk5eXmGPCktM8ds2CIkm8ymkpLiciE4KUSIjoyMCqev6FSTJXdbmZCR U1QmrBQk+v+LIIT4lYmL16cmrV6yZmlyekrGsrXL12UKEQiFo5koEMWjOWgWikSzURTSoSA0FwWj eSgExSA9mo9iEYMmIT9aJSSn8Cw0yCAmlNnDPPJI9Njt0e7hlBXI7snT5DXyFvlXrJwNYcPY16DN WwQMavFeMzMMehlUuSw80T+ziHWgJmOctysf28GHApY3M+IeCOJ7SBALERxpcFlYEs+RQ88sLIRz vUDD1aDmSTo3AmdZKRHUkEafOFD7OEEPxyCQeMOsyZonTj+4x93Hl5q6HF2O1qu4H9/KuxzXc6u7 tRNfxV9u7cs9s8nxZtNCrCR3OdATNQ+xnEa84Shal5lflKwlsRxRyyGF0zz5/mzhqlRD4WItSaFk 52C7a0kzQ4/Sy8RYOM7DdjGbJQs4oiV3SAjcYUkCB7VuGjsOesX47aSEQUqvW8rZ7wd6l4Xo3bs5 0inuZknEMwvoxXqpEFbwET8EX6aLAl2viuV87cCOdnNHxu2oDsJQqrnElwSRnWQHaEgQZMNy8HoC CY1aouNqItZkRlCIMuU3mAsx3/0OnhfO2Yyt2qaK/ZWNeUpv11Er+IoNdPd2epxOojLVZeNn41jT qmUrlubPxvQIwjhmXUseSHEWPsaPsfPUwODA9S+e4kcYuPyxFTdfvxF/ciZW1hNfHibdDSLRJCRe R1TEK2EUwiF8eBS8tBOyiGWgZobonS/Tg4b84CwHMnyzpfdK/2DbQwwsBrn5YdqNDVeSPyVySpuu 692jPE3zVwB7aQGJIfMyFhBWO15xeJeKXEx3/IHueEAyUTkc5mE1B1N/GoFg0L0yTMJo/aVKciIv p1hBMewwLkrMNyaMb/HCKMyPUr4NrvMwJhkQ4kQvULs/oYbspF8tJ9nVTrWaQN+W0FYJfYuib9HY 3xTVOH7ShHXtFJlODUjNDU0QRnwhZLJGhCro5GERd621NPU9vGvvDu3eSmzDO3HKvm1HjUqIOcFZ D9d8hE/WwZa6AxHtzpMdV/DPuL+iz3im8PO3mlZhpdt/4j7Rcrr/csU5fKTuSLVS8+RIdeNW01Rs 3FVcVb397YraTfi5uoId6imZUvD16QT9ZE2lK0ucxpPzRA3xXF3urq3bSpWa89kFa0vT/InXvKeU cNTQCGgudFUXndLabQerDhuUEMDZaEN4j5AlVJd8EkjmkApSCrNIDKz/49vTzktaTeWRUPYXqRms oBLtoGLOQyBchECZ+Fjcx+Pr1V0F7Rsuxn9GvUKSdHFkJmEvr4XgYu1YxZ+VjmK8ZkpqRn7k7PRz d2tnUO82ksi/5kMqVsIakD2geirSviH+LVrCHRPsZjvum/J17+mhkb7NiQ0zJmo+QK/5wURTdUpO 3gf3eXJ/vLFCSQvpgBZpwtDV+4pxOcVhKiW8I0F5cZQHE9XwHi1LqXtU0hHaJpp0QBLb9Hw+TQgs Nc1o87jx2qT0errqrlHM3ZgTHr6x88EMsYa2co0Cxx1L683uyxoqfYAf4KGWvp7ermP92KkUaxT/ DQ0xwA/el4ZaKEfmurvJfLFb+oe9dMBRDhIN0DFQCDpKgOgkAtbjoqMZFjZnHefA0xMCXgLPRpUK Ag6ovBD6B9yaO6EKZW5kc3RyZWFtCmVuZG9iago0MiAwIG9iagoxNDgyCmVuZG9iagoyMSAwIG9i ago8PC9CYXNlRm9udC9PRE5NREcrVFRFMjY4MkEzMHQwMC9Gb250RGVzY3JpcHRvciAyMCAwIFIv VHlwZS9Gb250Ci9GaXJzdENoYXIgMS9MYXN0Q2hhciAxL1dpZHRoc1sgMzQyXQovRW5jb2Rpbmcg NDMgMCBSL1N1YnR5cGUvVHJ1ZVR5cGU+PgplbmRvYmoKNDMgMCBvYmoKPDwvVHlwZS9FbmNvZGlu Zy9CYXNlRW5jb2RpbmcvV2luQW5zaUVuY29kaW5nL0RpZmZlcmVuY2VzWwoxL3NwYWNlXT4+CmVu ZG9iagoxNSAwIG9iago8PC9CYXNlRm9udC9BRk9aTU0rVGltZXMtQm9sZC9Gb250RGVzY3JpcHRv ciAxNCAwIFIvVHlwZS9Gb250Ci9GaXJzdENoYXIgMzIvTGFzdENoYXIgMTQ2L1dpZHRoc1sKMjUw IDAgMCAwIDAgMCAwIDAgMCAwIDUwMCAwIDAgMzMzIDI1MCAyNzgKNTAwIDUwMCA1MDAgNTAwIDUw MCAwIDAgNTAwIDUwMCAwIDMzMyAwIDAgMCAwIDAKMCA3MjIgNjY3IDcyMiA3MjIgNjY3IDYxMSA3 NzggNzc4IDM4OSAwIDc3OCA2NjcgOTQ0IDcyMiA3NzgKNjExIDAgNzIyIDU1NiA2NjcgNzIyIDAg MTAwMCA3MjIgNzIyIDAgMCAwIDAgMCAwCjAgNTAwIDAgNDQ0IDAgNDQ0IDMzMyA1MDAgMCAyNzgg MCAwIDI3OCAwIDU1NiA1MDAKMCAwIDQ0NCAzODkgMCA1NTYgMCAwIDUwMCAwIDAgMCAwIDAgMCAw CjAgMCAwIDAgMCAwIDAgMCAwIDAgMCAwIDAgMCAwIDAKMCAwIDMzM10KL0VuY29kaW5nL1dpbkFu c2lFbmNvZGluZy9TdWJ0eXBlL1R5cGUxPj4KZW5kb2JqCjE3IDAgb2JqCjw8L0Jhc2VGb250L1FF WUhOSytUaW1lcy1Sb21hbi9Gb250RGVzY3JpcHRvciAxNiAwIFIvVHlwZS9Gb250Ci9GaXJzdENo YXIgMzIvTGFzdENoYXIgNTgvV2lkdGhzWwoyNTAgMCAwIDAgMCAwIDAgMCAwIDAgMCA1NjQgMCAw IDI1MCAwCjAgMCAwIDAgMCAwIDAgMCAwIDAgMjc4XQovRW5jb2RpbmcvV2luQW5zaUVuY29kaW5n L1N1YnR5cGUvVHlwZTE+PgplbmRvYmoKMTkgMCBvYmoKPDwvQmFzZUZvbnQvTFlCS1ZQK1RURTI4 NDNCRDh0MDAvRm9udERlc2NyaXB0b3IgMTggMCBSL1R5cGUvRm9udAovRmlyc3RDaGFyIDEvTGFz dENoYXIgMTMvV2lkdGhzWyA1NjcgNjczIDYxNyA3MjIgNDg5IDYwMyA0NzUgMzMzIDQ4OSA1ODEg NjA5IDU4OCA0NzZdCi9FbmNvZGluZyA0NCAwIFIvU3VidHlwZS9UcnVlVHlwZT4+CmVuZG9iago0 NCAwIG9iago8PC9UeXBlL0VuY29kaW5nL0Jhc2VFbmNvZGluZy9XaW5BbnNpRW5jb2RpbmcvRGlm ZmVyZW5jZXNbCjEvUC9BL1kvTS9FL04vVC9zcGFjZS9GL08vUi9VL0xdPj4KZW5kb2JqCjEzIDAg b2JqCjw8L0Jhc2VGb250L1NOREFCTitIZWx2ZXRpY2EvRm9udERlc2NyaXB0b3IgMTIgMCBSL1R5 cGUvRm9udAovRmlyc3RDaGFyIDMyL0xhc3RDaGFyIDMyL1dpZHRoc1sKMjc4XQovRW5jb2Rpbmcv V2luQW5zaUVuY29kaW5nL1N1YnR5cGUvVHlwZTE+PgplbmRvYmoKOSAwIG9iago8PC9CYXNlRm9u dC9WUFJGQ0YrSGVsdmV0aWNhLUJvbGQvRm9udERlc2NyaXB0b3IgOCAwIFIvVHlwZS9Gb250Ci9G aXJzdENoYXIgMzIvTGFzdENoYXIgODkvV2lkdGhzWwoyNzggMCAwIDAgMCAwIDAgMCAwIDAgMCAw IDAgMCAyNzggMAowIDAgMCAwIDAgMCAwIDAgMCAwIDAgMCAwIDAgMCAwCjAgNzIyIDcyMiA3MjIg NzIyIDY2NyA2MTEgNzc4IDcyMiAyNzggMCAwIDYxMSAwIDcyMiA3NzgKNjY3IDAgNzIyIDY2NyA2 MTEgNzIyIDY2NyAwIDAgNjY3XQovRW5jb2RpbmcvV2luQW5zaUVuY29kaW5nL1N1YnR5cGUvVHlw ZTE+PgplbmRvYmoKMjAgMCBvYmoKPDwvVHlwZS9Gb250RGVzY3JpcHRvci9Gb250TmFtZS9PRE5N REcrVFRFMjY4MkEzMHQwMC9Gb250QkJveFswIDAgODc1IDc1MF0vRmxhZ3MgNAovQXNjZW50IDc1 MAovQ2FwSGVpZ2h0IDc1MAovRGVzY2VudCAwCi9JdGFsaWNBbmdsZSAwCi9TdGVtViAxMzEKL01p c3NpbmdXaWR0aCAxMDAwCi9Gb250RmlsZTIgMzEgMCBSPj4KZW5kb2JqCjE0IDAgb2JqCjw8L1R5 cGUvRm9udERlc2NyaXB0b3IvRm9udE5hbWUvQUZPWk1NK1RpbWVzLUJvbGQvRm9udEJCb3hbLTI0 IC0yMDYgOTgxIDY5Ml0vRmxhZ3MgNAovQXNjZW50IDY5MgovQ2FwSGVpZ2h0IDY5MgovRGVzY2Vu dCAtMjA2Ci9JdGFsaWNBbmdsZSAwCi9TdGVtViAxNDcKL01pc3NpbmdXaWR0aCAyNTAKL0NoYXJT ZXQoL2ZvdXIvTC9BL24vYy9YL00vQi9vL1kvTi9DL2Uvc2V2ZW4vTy9EL2YvZWlnaHQvUC9FL3Iv Zy9GL3MvY29sb24vUi9HL2kvUy9IL3UvVC9JL1UvbC9hL0sveC9XL2FzdGVyaXNrL3NwYWNlL2h5 cGhlbi9wZXJpb2Qvc2xhc2gvemVyby9vbmUvdHdvL3F1b3RlcmlnaHQvdGhyZWUpL0ZvbnRGaWxl MyAzMyAwIFI+PgplbmRvYmoKMTYgMCBvYmoKPDwvVHlwZS9Gb250RGVzY3JpcHRvci9Gb250TmFt ZS9RRVlITksrVGltZXMtUm9tYW4vRm9udEJCb3hbMCAtMTEgNTM0IDUwNl0vRmxhZ3MgNAovQXNj ZW50IDUwNgovQ2FwSGVpZ2h0IDUwNgovRGVzY2VudCAtMTEKL0l0YWxpY0FuZ2xlIDAKL1N0ZW1W IDgwCi9NaXNzaW5nV2lkdGggMjUwCi9DaGFyU2V0KC9jb2xvbi9wbHVzL3NwYWNlL3BlcmlvZCkv Rm9udEZpbGUzIDM1IDAgUj4+CmVuZG9iagoxOCAwIG9iago8PC9UeXBlL0ZvbnREZXNjcmlwdG9y L0ZvbnROYW1lL0xZQktWUCtUVEUyODQzQkQ4dDAwL0ZvbnRCQm94WzAgLTExIDg3NSA3NTddL0Zs YWdzIDQKL0FzY2VudCA3NTcKL0NhcEhlaWdodCA3NTcKL0Rlc2NlbnQgLTExCi9JdGFsaWNBbmds ZSAwCi9TdGVtViAxMzEKL01pc3NpbmdXaWR0aCAxMDAwCi9Gb250RmlsZTIgMzcgMCBSPj4KZW5k b2JqCjEyIDAgb2JqCjw8L1R5cGUvRm9udERlc2NyaXB0b3IvRm9udE5hbWUvU05EQUJOK0hlbHZl dGljYS9Gb250QkJveFswIDAgMTAwMCAxMDAwXS9GbGFncyA1Ci9Bc2NlbnQgMAovQ2FwSGVpZ2h0 IDAKL0Rlc2NlbnQgMAovSXRhbGljQW5nbGUgMAovU3RlbVYgMAovQXZnV2lkdGggMjc4Ci9NYXhX aWR0aCAyNzgKL01pc3NpbmdXaWR0aCAyNzgKL0NoYXJTZXQoL3NwYWNlKS9Gb250RmlsZTMgMzkg MCBSPj4KZW5kb2JqCjggMCBvYmoKPDwvVHlwZS9Gb250RGVzY3JpcHRvci9Gb250TmFtZS9WUFJG Q0YrSGVsdmV0aWNhLUJvbGQvRm9udEJCb3hbMCAtMjMgNzQyIDc0MV0vRmxhZ3MgNAovQXNjZW50 IDc0MQovQ2FwSGVpZ2h0IDc0MQovRGVzY2VudCAtMjMKL0l0YWxpY0FuZ2xlIDAKL1N0ZW1WIDEx MQovTWlzc2luZ1dpZHRoIDI3OAovQ2hhclNldCgvTC9BL0IvWS9OL0MvTy9EL1AvRS9GL1IvRy9T L0gvVC9JL1UvVi9zcGFjZS9wZXJpb2QpL0ZvbnRGaWxlMyA0MSAwIFI+PgplbmRvYmoKMiAwIG9i ago8PC9Qcm9kdWNlcihHUEwgR2hvc3RzY3JpcHQgOC4xNSkKL0NyZWF0aW9uRGF0ZShEOjIwMDcx MDIzMDkxNzQ2KQovTW9kRGF0ZShEOjIwMDcxMDIzMDkxNzQ2KQovVGl0bGUoTWljcm9zb2Z0IFdv cmQgLSBERSBMT1RUTy4uZG9jKQovQ3JlYXRvcihQU2NyaXB0NS5kbGwgVmVyc2lvbiA1LjIpCi9B dXRob3IoVVNFUik+PmVuZG9iagp4cmVmCjAgNDUKMDAwMDAwMDAwMCA2NTUzNSBmIAowMDAwMDA2 ODg3IDAwMDAwIG4gCjAwMDAwMjg3MjIgMDAwMDAgbiAKMDAwMDAwNjgxMiAwMDAwMCBuIAowMDAw MDA2NDMzIDAwMDAwIG4gCjAwMDAwMDAwMTUgMDAwMDAgbiAKMDAwMDAwNTg3NSAwMDAwMCBuIAow MDAwMDA2OTM1IDAwMDAwIG4gCjAwMDAwMjg0NTYgMDAwMDAgbiAKMDAwMDAyNjg4NiAwMDAwMCBu IAowMDAwMDA2OTc2IDAwMDAwIG4gCjAwMDAwMDg2NzMgMDAwMDAgbiAKMDAwMDAyODIxOSAwMDAw MCBuIAowMDAwMDI2NzI4IDAwMDAwIG4gCjAwMDAwMjc0MDMgMDAwMDAgbiAKMDAwMDAyNTcxMiAw MDAwMCBuIAowMDAwMDI3Nzc3IDAwMDAwIG4gCjAwMDAwMjYxOTcgMDAwMDAgbiAKMDAwMDAyODAx MyAwMDAwMCBuIAowMDAwMDI2NDE1IDAwMDAwIG4gCjAwMDAwMjcyMDEgMDAwMDAgbiAKMDAwMDAy NTQ3MiAwMDAwMCBuIAowMDAwMDA4NTc5IDAwMDAwIG4gCjAwMDAwMDg2MTEgMDAwMDAgbiAKMDAw MDAwODY0MSAwMDAwMCBuIAowMDAwMDExMTI5IDAwMDAwIG4gCjAwMDAwMDY2NDcgMDAwMDAgbiAK MDAwMDAwNTg5NSAwMDAwMCBuIAowMDAwMDA2NDEzIDAwMDAwIG4gCjAwMDAwMTEyMTQgMDAwMDAg biAKMDAwMDAxMTI0NCAwMDAwMCBuIAowMDAwMDExMjc2IDAwMDAwIG4gCjAwMDAwMTM0NzggMDAw MDAgbiAKMDAwMDAxMzQ5OSAwMDAwMCBuIAowMDAwMDE4NTQyIDAwMDAwIG4gCjAwMDAwMTg1NjMg MDAwMDAgbiAKMDAwMDAxOTAzMCAwMDAwMCBuIAowMDAwMDE5MDUwIDAwMDAwIG4gCjAwMDAwMjM1 MTkgMDAwMDAgbiAKMDAwMDAyMzU0MCAwMDAwMCBuIAowMDAwMDIzODYzIDAwMDAwIG4gCjAwMDAw MjM4ODMgMDAwMDAgbiAKMDAwMDAyNTQ1MSAwMDAwMCBuIAowMDAwMDI1NjI2IDAwMDAwIG4gCjAw MDAwMjY2MTggMDAwMDAgbiAKdHJhaWxlcgo8PCAvU2l6ZSA0NSAvUm9vdCAxIDAgUiAvSW5mbyAy IDAgUgovSUQgWyjMpuX0w5VaIVVQOeMh0ZlIKSjMpuX0w5VaIVVQOeMh0ZlIKV0KPj4Kc3RhcnR4 cmVmCjI4OTE5CiUlRU9GCg== --lnSXiGey81P9nGQ=_Kiw6O53aaHOiHeSwk0-- From owner-freebsd-arch@FreeBSD.ORG Sat Mar 29 07:03:28 2008 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5CF8D1065672 for ; Sat, 29 Mar 2008 07:03:28 +0000 (UTC) (envelope-from jroberson@chesapeake.net) Received: from webaccess-cl.virtdom.com (webaccess-cl.virtdom.com [216.240.101.25]) by mx1.freebsd.org (Postfix) with ESMTP id 24B3E8FC1D for ; Sat, 29 Mar 2008 07:03:28 +0000 (UTC) (envelope-from jroberson@chesapeake.net) Received: from [10.0.1.200] (cpe-24-94-72-120.hawaii.res.rr.com [24.94.72.120]) (authenticated bits=0) by webaccess-cl.virtdom.com (8.13.6/8.13.6) with ESMTP id m2T73Pe7028289 for ; Sat, 29 Mar 2008 03:03:27 -0400 (EDT) (envelope-from jroberson@chesapeake.net) Date: Fri, 28 Mar 2008 21:04:17 -1000 (HST) From: Jeff Roberson X-X-Sender: jroberson@desktop To: arch@freebsd.org In-Reply-To: <20080312152744.I29518@fledge.watson.org> Message-ID: <20080328202602.N72156@desktop> References: <10004.1205307334@critter.freebsd.dk> <200803120908.10403.jhb@freebsd.org> <20080312140422.B29518@fledge.watson.org> <200803121059.45614.jhb@freebsd.org> <20080312152744.I29518@fledge.watson.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: Subject: timeout/callout small step forward X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 29 Mar 2008 07:03:28 -0000 http://people.freebsd.org/~jeff/callout.diff This patch takes the current callout implementation and makes it per-cpu. It also hides callout details from the rest of the kernel by making the callwheel structure private to kern_timeout.c among other things. All callouts default to the first softclock thread to maintain ordering and concurrency guarantees of the older code. The only new interface that is implemented is 'callout_reset_on' which can specify a cpu to run the callout on. This can force a callout which is not presently being dispatched to migrate to a new cpu. The threads are not hard bound to each cpu so that they can't be starved into not processing their queues. They are 'medium' bound by ULE and probably tend to run where they are scheduled on 4BSD but there is no guarantee. The ULE rules will run them on their native CPU if it's running something of lesser priority or idle. If it's running something of greater priority it will look for a more idle cpu. This prevents the softclock thread from being starved by other interrupt thread activity. I believe this helps phk's effort rather than hindering it because: 1) It collects up more implementation details and pushes them into kern_timeout.c 2) It sounds like we're keeping the same current/waiting/cancel algorithm so this packages it up into a per-instances structure as would need to happen in phk's proposal as well. 3) The only new interface, a varient of callout_reset() should be trivial to implement whatever the new scheme is. Also included is changes to kern_event.c and subr_sleepq.c so sleepq() timeouts happen on the cpu that generated them and so do kqueue() poll timeouts. You can see that the additional interface is quite simple. This has been tested and debugged by nokia. Thanks, Jeff From owner-freebsd-arch@FreeBSD.ORG Sat Mar 29 09:57:20 2008 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9D435106566B; Sat, 29 Mar 2008 09:57:20 +0000 (UTC) (envelope-from jroberson@chesapeake.net) Received: from webaccess-cl.virtdom.com (webaccess-cl.virtdom.com [216.240.101.25]) by mx1.freebsd.org (Postfix) with ESMTP id 611958FC14; Sat, 29 Mar 2008 09:57:20 +0000 (UTC) (envelope-from jroberson@chesapeake.net) Received: from [10.0.1.200] (cpe-24-94-72-120.hawaii.res.rr.com [24.94.72.120]) (authenticated bits=0) by webaccess-cl.virtdom.com (8.13.6/8.13.6) with ESMTP id m2T9vA7n040532; Sat, 29 Mar 2008 05:57:14 -0400 (EDT) (envelope-from jroberson@chesapeake.net) Date: Fri, 28 Mar 2008 23:58:02 -1000 (HST) From: Jeff Roberson X-X-Sender: jroberson@desktop To: Hans Petter Selasky In-Reply-To: <200803291013.28797.hselasky@c2i.net> Message-ID: <20080328235600.O72156@desktop> References: <10004.1205307334@critter.freebsd.dk> <20080312152744.I29518@fledge.watson.org> <20080328202602.N72156@desktop> <200803291013.28797.hselasky@c2i.net> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: arch@freebsd.org, freebsd-arch@freebsd.org Subject: Re: timeout/callout small step forward X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 29 Mar 2008 09:57:20 -0000 On Sat, 29 Mar 2008, Hans Petter Selasky wrote: > Hi, > > How does this patch handle when multiple callouts use the same mutex ? Imagine > that two threads are about to lock the same mutex, then the second thread > will get nowhere while waiting for the first thread to execute the callback. You are worried that there will be too much contention or that there is some correctness issue? If you're worried about contention, it would seem that most callers use some per-object mutex for the callout. So there isn't likely to be much contention among callouts. > > I think the solution is that callbacks are put in a tree. Giant locked > callbacks go all into the same tree. Else the user of callouts is responsible > for making up the tree. > > struct callout { > struct thread *exec_td; > struct callout *parent; > }; > > struct callout *c; > > while (c->parent != NULL) { > c = c->parent; > } > > if (c->exec_td != NULL) { > callout should go into the same thread; > } else { > pick the next free thread and set "c->exec_td" > } > > Callouts that belong to the same tree are run from the same thread. This does > not only apply for callouts, but also multithreaded node systems ... > > Yours > --HPS > From owner-freebsd-arch@FreeBSD.ORG Sat Mar 29 09:57:20 2008 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9D435106566B; Sat, 29 Mar 2008 09:57:20 +0000 (UTC) (envelope-from jroberson@chesapeake.net) Received: from webaccess-cl.virtdom.com (webaccess-cl.virtdom.com [216.240.101.25]) by mx1.freebsd.org (Postfix) with ESMTP id 611958FC14; Sat, 29 Mar 2008 09:57:20 +0000 (UTC) (envelope-from jroberson@chesapeake.net) Received: from [10.0.1.200] (cpe-24-94-72-120.hawaii.res.rr.com [24.94.72.120]) (authenticated bits=0) by webaccess-cl.virtdom.com (8.13.6/8.13.6) with ESMTP id m2T9vA7n040532; Sat, 29 Mar 2008 05:57:14 -0400 (EDT) (envelope-from jroberson@chesapeake.net) Date: Fri, 28 Mar 2008 23:58:02 -1000 (HST) From: Jeff Roberson X-X-Sender: jroberson@desktop To: Hans Petter Selasky In-Reply-To: <200803291013.28797.hselasky@c2i.net> Message-ID: <20080328235600.O72156@desktop> References: <10004.1205307334@critter.freebsd.dk> <20080312152744.I29518@fledge.watson.org> <20080328202602.N72156@desktop> <200803291013.28797.hselasky@c2i.net> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: arch@freebsd.org, freebsd-arch@freebsd.org Subject: Re: timeout/callout small step forward X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 29 Mar 2008 09:57:20 -0000 On Sat, 29 Mar 2008, Hans Petter Selasky wrote: > Hi, > > How does this patch handle when multiple callouts use the same mutex ? Imagine > that two threads are about to lock the same mutex, then the second thread > will get nowhere while waiting for the first thread to execute the callback. You are worried that there will be too much contention or that there is some correctness issue? If you're worried about contention, it would seem that most callers use some per-object mutex for the callout. So there isn't likely to be much contention among callouts. > > I think the solution is that callbacks are put in a tree. Giant locked > callbacks go all into the same tree. Else the user of callouts is responsible > for making up the tree. > > struct callout { > struct thread *exec_td; > struct callout *parent; > }; > > struct callout *c; > > while (c->parent != NULL) { > c = c->parent; > } > > if (c->exec_td != NULL) { > callout should go into the same thread; > } else { > pick the next free thread and set "c->exec_td" > } > > Callouts that belong to the same tree are run from the same thread. This does > not only apply for callouts, but also multithreaded node systems ... > > Yours > --HPS > From owner-freebsd-arch@FreeBSD.ORG Sat Mar 29 10:12:32 2008 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 53F501065676; Sat, 29 Mar 2008 10:12:32 +0000 (UTC) (envelope-from hselasky@c2i.net) Received: from swip.net (mailfe09.swipnet.se [212.247.155.1]) by mx1.freebsd.org (Postfix) with ESMTP id EB87D8FC1C; Sat, 29 Mar 2008 10:12:30 +0000 (UTC) (envelope-from hselasky@c2i.net) X-Cloudmark-Score: 0.000000 [] Received: from [62.113.132.89] (account mc467741@c2i.net [62.113.132.89] verified) by mailfe09.swip.net (CommuniGate Pro SMTP 5.1.13) with ESMTPA id 702449625; Sat, 29 Mar 2008 10:12:18 +0100 From: Hans Petter Selasky To: freebsd-arch@freebsd.org Date: Sat, 29 Mar 2008 10:13:28 +0100 User-Agent: KMail/1.9.7 References: <10004.1205307334@critter.freebsd.dk> <20080312152744.I29518@fledge.watson.org> <20080328202602.N72156@desktop> In-Reply-To: <20080328202602.N72156@desktop> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200803291013.28797.hselasky@c2i.net> Cc: arch@freebsd.org Subject: Re: timeout/callout small step forward X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 29 Mar 2008 10:12:32 -0000 Hi, How does this patch handle when multiple callouts use the same mutex ? Imagine that two threads are about to lock the same mutex, then the second thread will get nowhere while waiting for the first thread to execute the callback. I think the solution is that callbacks are put in a tree. Giant locked callbacks go all into the same tree. Else the user of callouts is responsible for making up the tree. struct callout { struct thread *exec_td; struct callout *parent; }; struct callout *c; while (c->parent != NULL) { c = c->parent; } if (c->exec_td != NULL) { callout should go into the same thread; } else { pick the next free thread and set "c->exec_td" } Callouts that belong to the same tree are run from the same thread. This does not only apply for callouts, but also multithreaded node systems ... Yours --HPS From owner-freebsd-arch@FreeBSD.ORG Sat Mar 29 10:12:32 2008 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 53F501065676; Sat, 29 Mar 2008 10:12:32 +0000 (UTC) (envelope-from hselasky@c2i.net) Received: from swip.net (mailfe09.swipnet.se [212.247.155.1]) by mx1.freebsd.org (Postfix) with ESMTP id EB87D8FC1C; Sat, 29 Mar 2008 10:12:30 +0000 (UTC) (envelope-from hselasky@c2i.net) X-Cloudmark-Score: 0.000000 [] Received: from [62.113.132.89] (account mc467741@c2i.net [62.113.132.89] verified) by mailfe09.swip.net (CommuniGate Pro SMTP 5.1.13) with ESMTPA id 702449625; Sat, 29 Mar 2008 10:12:18 +0100 From: Hans Petter Selasky To: freebsd-arch@freebsd.org Date: Sat, 29 Mar 2008 10:13:28 +0100 User-Agent: KMail/1.9.7 References: <10004.1205307334@critter.freebsd.dk> <20080312152744.I29518@fledge.watson.org> <20080328202602.N72156@desktop> In-Reply-To: <20080328202602.N72156@desktop> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200803291013.28797.hselasky@c2i.net> Cc: arch@freebsd.org Subject: Re: timeout/callout small step forward X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 29 Mar 2008 10:12:32 -0000 Hi, How does this patch handle when multiple callouts use the same mutex ? Imagine that two threads are about to lock the same mutex, then the second thread will get nowhere while waiting for the first thread to execute the callback. I think the solution is that callbacks are put in a tree. Giant locked callbacks go all into the same tree. Else the user of callouts is responsible for making up the tree. struct callout { struct thread *exec_td; struct callout *parent; }; struct callout *c; while (c->parent != NULL) { c = c->parent; } if (c->exec_td != NULL) { callout should go into the same thread; } else { pick the next free thread and set "c->exec_td" } Callouts that belong to the same tree are run from the same thread. This does not only apply for callouts, but also multithreaded node systems ... Yours --HPS From owner-freebsd-arch@FreeBSD.ORG Sat Mar 29 13:20:13 2008 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id F07DE106566C; Sat, 29 Mar 2008 13:20:13 +0000 (UTC) (envelope-from hselasky@c2i.net) Received: from swip.net (mailfe15.tele2.se [212.247.155.193]) by mx1.freebsd.org (Postfix) with ESMTP id 29DB48FC1B; Sat, 29 Mar 2008 13:20:12 +0000 (UTC) (envelope-from hselasky@c2i.net) X-Cloudmark-Score: 0.000000 [] Received: from [62.113.132.89] (account mc467741@c2i.net [62.113.132.89] verified) by mailfe15.swip.net (CommuniGate Pro SMTP 5.1.13) with ESMTPA id 134377604; Sat, 29 Mar 2008 13:20:09 +0100 From: Hans Petter Selasky To: freebsd-arch@freebsd.org Date: Sat, 29 Mar 2008 13:21:13 +0100 User-Agent: KMail/1.9.7 References: <10004.1205307334@critter.freebsd.dk> <200803291013.28797.hselasky@c2i.net> <20080328235600.O72156@desktop> In-Reply-To: <20080328235600.O72156@desktop> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200803291321.15896.hselasky@c2i.net> Cc: arch@freebsd.org Subject: Re: timeout/callout small step forward X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 29 Mar 2008 13:20:14 -0000 Hi Jeff, Probably you're right that the chance of congestion is not very high, hence most clients will use a per-object lock, except for Giant locked systems. Though it would be nice if you could get this part right while you are first working at it. I get the feeling that you understand my point: Multiple worker threads will not help if the functions called by the worker threads are protected by the same mutex. Then it is better to run these in series on the same thread. --HPS On Saturday 29 March 2008, Jeff Roberson wrote: > On Sat, 29 Mar 2008, Hans Petter Selasky wrote: > > Hi, > > > > How does this patch handle when multiple callouts use the same mutex ? > > Imagine that two threads are about to lock the same mutex, then the > > second thread will get nowhere while waiting for the first thread to > > execute the callback. > > You are worried that there will be too much contention or that there is > some correctness issue? > > If you're worried about contention, it would seem that most callers use > some per-object mutex for the callout. So there isn't likely to be > much contention among callouts. > > > I think the solution is that callbacks are put in a tree. Giant locked > > callbacks go all into the same tree. Else the user of callouts is > > responsible for making up the tree. > > > > struct callout { > > struct thread *exec_td; > > struct callout *parent; > > }; > > > > struct callout *c; > > > > while (c->parent != NULL) { > > c = c->parent; > > } > > > > if (c->exec_td != NULL) { > > callout should go into the same thread; > > } else { > > pick the next free thread and set "c->exec_td" > > } > > > > Callouts that belong to the same tree are run from the same thread. This > > does not only apply for callouts, but also multithreaded node systems ... > > > > Yours > > --HPS > > _______________________________________________ > freebsd-arch@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-arch > To unsubscribe, send any mail to "freebsd-arch-unsubscribe@freebsd.org" From owner-freebsd-arch@FreeBSD.ORG Sat Mar 29 13:20:13 2008 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id F07DE106566C; Sat, 29 Mar 2008 13:20:13 +0000 (UTC) (envelope-from hselasky@c2i.net) Received: from swip.net (mailfe15.tele2.se [212.247.155.193]) by mx1.freebsd.org (Postfix) with ESMTP id 29DB48FC1B; Sat, 29 Mar 2008 13:20:12 +0000 (UTC) (envelope-from hselasky@c2i.net) X-Cloudmark-Score: 0.000000 [] Received: from [62.113.132.89] (account mc467741@c2i.net [62.113.132.89] verified) by mailfe15.swip.net (CommuniGate Pro SMTP 5.1.13) with ESMTPA id 134377604; Sat, 29 Mar 2008 13:20:09 +0100 From: Hans Petter Selasky To: freebsd-arch@freebsd.org Date: Sat, 29 Mar 2008 13:21:13 +0100 User-Agent: KMail/1.9.7 References: <10004.1205307334@critter.freebsd.dk> <200803291013.28797.hselasky@c2i.net> <20080328235600.O72156@desktop> In-Reply-To: <20080328235600.O72156@desktop> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200803291321.15896.hselasky@c2i.net> Cc: arch@freebsd.org Subject: Re: timeout/callout small step forward X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 29 Mar 2008 13:20:14 -0000 Hi Jeff, Probably you're right that the chance of congestion is not very high, hence most clients will use a per-object lock, except for Giant locked systems. Though it would be nice if you could get this part right while you are first working at it. I get the feeling that you understand my point: Multiple worker threads will not help if the functions called by the worker threads are protected by the same mutex. Then it is better to run these in series on the same thread. --HPS On Saturday 29 March 2008, Jeff Roberson wrote: > On Sat, 29 Mar 2008, Hans Petter Selasky wrote: > > Hi, > > > > How does this patch handle when multiple callouts use the same mutex ? > > Imagine that two threads are about to lock the same mutex, then the > > second thread will get nowhere while waiting for the first thread to > > execute the callback. > > You are worried that there will be too much contention or that there is > some correctness issue? > > If you're worried about contention, it would seem that most callers use > some per-object mutex for the callout. So there isn't likely to be > much contention among callouts. > > > I think the solution is that callbacks are put in a tree. Giant locked > > callbacks go all into the same tree. Else the user of callouts is > > responsible for making up the tree. > > > > struct callout { > > struct thread *exec_td; > > struct callout *parent; > > }; > > > > struct callout *c; > > > > while (c->parent != NULL) { > > c = c->parent; > > } > > > > if (c->exec_td != NULL) { > > callout should go into the same thread; > > } else { > > pick the next free thread and set "c->exec_td" > > } > > > > Callouts that belong to the same tree are run from the same thread. This > > does not only apply for callouts, but also multithreaded node systems ... > > > > Yours > > --HPS > > _______________________________________________ > freebsd-arch@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-arch > To unsubscribe, send any mail to "freebsd-arch-unsubscribe@freebsd.org" From owner-freebsd-arch@FreeBSD.ORG Sat Mar 29 21:33:21 2008 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 158D91065673 for ; Sat, 29 Mar 2008 21:33:21 +0000 (UTC) (envelope-from bakul@bitblocks.com) Received: from mail.bitblocks.com (bitblocks.com [64.142.15.60]) by mx1.freebsd.org (Postfix) with ESMTP id EFB0E8FC20 for ; Sat, 29 Mar 2008 21:33:20 +0000 (UTC) (envelope-from bakul@bitblocks.com) Received: from bitblocks.com (localhost.bitblocks.com [127.0.0.1]) by mail.bitblocks.com (Postfix) with ESMTP id 387805B42; Sat, 29 Mar 2008 14:33:20 -0700 (PDT) To: Poul-Henning Kamp In-reply-to: Your message of "Fri, 28 Mar 2008 19:09:28 -0000." <6472.1206731368@critter.freebsd.dk> Date: Sat, 29 Mar 2008 14:33:20 -0700 From: Bakul Shah Message-Id: <20080329213320.387805B42@mail.bitblocks.com> Cc: arch@freebsd.org Subject: Re: Flash disks and FFS layout heuristics X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 29 Mar 2008 21:33:21 -0000 On Fri, 28 Mar 2008 19:09:28 -0000 Poul-Henning Kamp wrote: > > I've laid my hands on a "M-Tron MOBI3000 32GB" flash disk (2.5" format, > it'll be in my laptop before long :-) > > Here is a naive benchmark sequence, comparing it to a WD Raptor > () > > Flash Disk > --------------------------------------------------------------- > Empty fsck: 0.83 2.47 -66% > restore -rf 839 1251 -33% > loaded fsck: 10.34 78.81 -87% > dump 0f /dev/null: 563.21 1094.91 -49% > --------------------------------------------------------------- > > So far so good, it's clearly the seektime that dominates the > flash-advantage. > > But this reproducible observation by fsck concerns me a bit: > > Flash: (205727 frags, 896270 blocks, 1.4% fragmentation) > > Disk: (197095 frags, 1193944 blocks, 1.1% fragmentation) > > I might indicate that the flash is fast enough to confuse some of > FFS's layout heuristics. > > Any aspiring filesystems hackers should start to consider the > implications for filesystemlayout, if there is in essence no > seek-time penalty for reads and a fair seek pentalty for writes. On a flash "disk" the write penalty has to do with the large erase block size. We can confirm this by looking at the MOBI disk's datasheet: it can do 130 IOPS (IO ops/sec) for random writes for 512B or 4KB blocksize but 16500 IOPs for 4KB blocksize sequential writes. Presumably it can coalesces sequential writes to bigger blocks but not random writes. Given this, does it even makes sense to use the FFS layout? For best performance ideally all writes happen sequentially, with occasional fix ups of the super block etc. Even inodes that changed should be laid out sequentially. Basically you just write the journal and fix up its roots so that on reboot you can quickly discover the filesystem structure! From owner-freebsd-arch@FreeBSD.ORG Sat Mar 29 21:56:17 2008 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 632AF1065672 for ; Sat, 29 Mar 2008 21:56:17 +0000 (UTC) (envelope-from phk@critter.freebsd.dk) Received: from phk.freebsd.dk (phk.freebsd.dk [130.225.244.222]) by mx1.freebsd.org (Postfix) with ESMTP id 292248FC1C for ; Sat, 29 Mar 2008 21:56:17 +0000 (UTC) (envelope-from phk@critter.freebsd.dk) Received: from critter.freebsd.dk (unknown [192.168.61.3]) by phk.freebsd.dk (Postfix) with ESMTP id 59DCC17104; Sat, 29 Mar 2008 21:56:15 +0000 (UTC) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.14.2/8.14.2) with ESMTP id m2TLuDKZ008264; Sat, 29 Mar 2008 21:56:14 GMT (envelope-from phk@critter.freebsd.dk) To: Bakul Shah From: "Poul-Henning Kamp" In-Reply-To: Your message of "Sat, 29 Mar 2008 14:33:20 MST." <20080329213320.387805B42@mail.bitblocks.com> Date: Sat, 29 Mar 2008 21:56:13 +0000 Message-ID: <8263.1206827773@critter.freebsd.dk> Sender: phk@critter.freebsd.dk Cc: arch@freebsd.org Subject: Re: Flash disks and FFS layout heuristics X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 29 Mar 2008 21:56:17 -0000 In message <20080329213320.387805B42@mail.bitblocks.com>, Bakul Shah writes: >Given this, does it even makes sense to use the FFS layout? Certainly not, but until somebody writes us a filesystem that is optimized for SSD access times, that's what people will be using. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From owner-freebsd-arch@FreeBSD.ORG Sat Mar 29 23:53:09 2008 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 71FE3106566C for ; Sat, 29 Mar 2008 23:53:09 +0000 (UTC) (envelope-from mckusick@mckusick.com) Received: from chez.mckusick.com (chez.mckusick.com [64.81.247.49]) by mx1.freebsd.org (Postfix) with ESMTP id 40E5B8FC15 for ; Sat, 29 Mar 2008 23:53:09 +0000 (UTC) (envelope-from mckusick@mckusick.com) Received: from chez.mckusick.com (localhost.mckusick.com [127.0.0.1]) by chez.mckusick.com (8.13.8/8.13.8) with ESMTP id m2TNrCOW094875; Sat, 29 Mar 2008 16:53:12 -0700 (PDT) (envelope-from mckusick@chez.mckusick.com) Message-Id: <200803292353.m2TNrCOW094875@chez.mckusick.com> To: "Poul-Henning Kamp" Date: Sat, 29 Mar 2008 16:53:12 -0700 From: Kirk McKusick Cc: arch@freebsd.org Subject: Re: Flash disks and FFS layout heuristics X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 29 Mar 2008 23:53:09 -0000 You should try running your experiment using ZFS. Because it is a non-overwriting filesystem, it might work better with flash. Kirk McKusick