From owner-freebsd-current@FreeBSD.ORG Thu Aug 23 20:22:15 2012 Return-Path: Delivered-To: freebsd-current@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 40C3F106566B; Thu, 23 Aug 2012 20:22:15 +0000 (UTC) (envelope-from bapt@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 205A28FC12; Thu, 23 Aug 2012 20:22:15 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q7NKMFRc061065; Thu, 23 Aug 2012 20:22:15 GMT (envelope-from bapt@FreeBSD.org) Received: (from bapt@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q7NKMEo7061064; Thu, 23 Aug 2012 20:22:14 GMT (envelope-from bapt@FreeBSD.org) X-Authentication-Warning: freefall.freebsd.org: bapt set sender to bapt@FreeBSD.org using -f Date: Thu, 23 Aug 2012 22:22:12 +0200 From: Baptiste Daroussin To: Kris Moore Message-ID: <20120823202212.GD18372@ithaqua.etoilebsd.net> References: <1345739186.30848.YahooMailClassic@web111307.mail.gq1.yahoo.com> <50365F37.7040601@pcbsd.org> <50368973.5040202@pcbsd.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="MAH+hnPXVZWQ5cD/" Content-Disposition: inline In-Reply-To: <50368973.5040202@pcbsd.org> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: Jeremy Messenger , freebsd-current@FreeBSD.org, FreeBSD Ports Subject: Re: pkgng default schedule... registering a few reasons for rethinking the final implementation... X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Aug 2012 20:22:15 -0000 --MAH+hnPXVZWQ5cD/ Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Aug 23, 2012 at 03:50:11PM -0400, Kris Moore wrote: > On 08/23/2012 13:10, Jeremy Messenger wrote: > > On Thu, Aug 23, 2012 at 11:49 AM, Kris Moore wrote: > >> On 08/23/2012 12:26, Jeffrey Bouquet wrote: > >>> I am following with dread the planned implementation of the deprecati= on of /var/db/pkg as a package registry... I use each /var/db/pkg directory= as a database into the port installation/status, using sed/grep/portmaster= /portmanager/.sh scripts/find/pipes etc... to fix stuff. For instance, an = upgrade py26 > py27. > >>> cd /var/db/pkg > >>> ls -lac | grep py26 > >>> ls -lac | grep python > >>> as the more simple example. > >>> .... > >>> With due respect to its developers and the persons who agree that > >>> the package tools could be upgraded, the mandatory > >>> usage of a front-end database to a file directory one > >>> is here viewd as mutt-only-mbox, registry-and-bsod rather > >>> than /etc/local/rc files, deprecation of sed/grep/find/locate/.sh/por= tmaster/portmanager as tools to fixup/upgrade the ports that are registered; > >>> ... > >>> I see concurrently too few tests on lower-end p2, p3 as to whether > >>> pkg can run with lesser memory machines (routers...) (pfsense) > >>> ... > >>> I suspect stalling of successful frontends to bsd (pc-bsd, ghostbsd, > >>> pfsense..) due to less-reliability, more-possibility of bugs.. > >>> > >> This is of some concern to me as well. A number of our utilities / > >> scripts rely on checking /var/db/pkg as a means to test if a particular > >> package is installed. This is often much faster than running the pkg_* > >> commands, especially when we may be checking thousands of packages in a > >> single run. It will be some work to adjust our utilities to using the > >> various "pkg" commands now, but it can be done. What worries me is > >> performance. If this is significantly slower, it may cause some issues > >> on our end. > > Guys, please test it before you say anything. Otherwise it's going to > > be moved forward without you. > > > > >=20 > Well, it was about time I got to doing a benchmark of this anyway :) >=20 > I did quick benchmark of how one of our utilities parses through a list > of 1k packages on a newer i5 system: >=20 > First test, using /var/db/pkg/ check we have been doing: >=20 > 0.178s 0:00.31 54.8% > 0.123s 0:00.26 61.5% > 0.099s 0:00.15 60.0% > =20 > Second test, using "pkg info ": >=20 > 5.347s 0:11.41 91.7%=20 > 5.444s 0:11.52 91.3% > 5.878s 0:11.32 91.4% >=20 > The pkg info command is quite a bit slower in this case, but 5 seconds > isn't horrible. >=20 > Now I ran the same benchmark on a slower 1.66gz Atom system, checking > about 1200~ packages: >=20 > First test, using /var/db/pkg/ check we have been doing: >=20 > 0.604s 0:00.76 86.8% > 0.622s 0:00.77 84.4% > 0.614s 0:00.73 90.4% > =20 > Second test, using "pkg info ": >=20 > 28.507s 0:54.80 99.1% > 28.282s 0:54.60 99.4% > 28.302s 0:54.52 99.4% >=20 > Now this is what concerns me a bit. It took closer to 30 seconds, which > is quite a while to wait, especially if a utility like ours has to run > these checks when it starts up, to show the user whats installed / not > installed on the system. >=20 > The only way around It I've found is to do a quick "pkg info" on the > entire DB, dump that to a list, then begin to grep through that list for > each item, but it still takes 10~ seconds on the atom. That may be what > I end up having to do, but it still stinks to go from a half a second > startup, to 10 seconds each time. Any other ideas on how to do this > faster with the new pkgng? >=20 > --=20 > Kris Moore > PC-BSD Software > iXsystems >=20 > _______________________________________________ > freebsd-ports@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-ports > To unsubscribe, send any mail to "freebsd-ports-unsubscribe@freebsd.org" Do you have an example of what information you are trying to get out of that list of packages? Or even better do you have a sample of your script so that I can convert it= to pkgng? (For your informations most of the scripts people gave me, that I converted= to pkgng are faster with pkgng) regards, Bapt --MAH+hnPXVZWQ5cD/ Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) iEYEARECAAYFAlA2kPQACgkQ8kTtMUmk6EzfCgCguBvXiooNgle2GvBC+ZmLeDgn mM4AoKDRM4tFrRI+IumrrazwCvgsOklK =YTFV -----END PGP SIGNATURE----- --MAH+hnPXVZWQ5cD/--