From owner-freebsd-pkg@FreeBSD.ORG Tue Jul 23 10:09:22 2013 Return-Path: Delivered-To: freebsd-pkg@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 664104A2 for ; Tue, 23 Jul 2013 10:09:22 +0000 (UTC) (envelope-from kpaasial@gmail.com) Received: from mail-qc0-x234.google.com (mail-qc0-x234.google.com [IPv6:2607:f8b0:400d:c01::234]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 3216D24E0 for ; Tue, 23 Jul 2013 10:09:22 +0000 (UTC) Received: by mail-qc0-f180.google.com with SMTP id a1so4167313qcx.25 for ; Tue, 23 Jul 2013 03:09:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=eZFzB+HxdxPwq2krp+/2aTdlmiRtkjGkfPq+atn+nRg=; b=aPBTEv1Ru41Hu+RlvdfGmpCx1AEP8RLobeEfBq00dmsyZV6WGPpeXasyNIlHDMfq1s 9qNwBnpRdJxQEcrdzKh2tPrwnBj93zS2C7sjVu0EXs2c0r8WdCmIqIhDTbn9aJPvigtY 4XgVGEkmdk9s6zWV3rDY9NYwIbsZkkUzEJQBCeBtCWFITSrcyGk13wZPaKMAafdT2G5Y cvyuuUsfDlV1bZ3ebX8+R6pGc2YjuAiozCuvnBrES16TbGRatJ7Z91amHshbw+h2xqjt fG7HQZxW9Kl3+YHtEfxMo/UdBfU5HArcz8YQ8bjWsaxP3k7rAbZ8VLO7cxGDmx3Zvg4+ gLYg== MIME-Version: 1.0 X-Received: by 10.49.127.4 with SMTP id nc4mr37511762qeb.41.1374574161337; Tue, 23 Jul 2013 03:09:21 -0700 (PDT) Received: by 10.224.78.194 with HTTP; Tue, 23 Jul 2013 03:09:21 -0700 (PDT) Date: Tue, 23 Jul 2013 13:09:21 +0300 Message-ID: Subject: Poudriere jails. From: Kimmo Paasiala To: freebsd-pkg@freebsd.org Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-pkg@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Binary package management and package tools discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Jul 2013 10:09:22 -0000 Is there anything special about the jails that poudriere installs with "poudriere jail -c"? Should I be able to use a pre-installed jail that i have installed with "make installworld DESTDIR=/usr/local/poudriere/jails/somejail"? I'm asking this because if I could install and update the jails without using "poudriere jail" command it would save lots of time because I could just use the world I already have built for updating the build host. -Kimmo From owner-freebsd-pkg@FreeBSD.ORG Tue Jul 23 11:28:37 2013 Return-Path: Delivered-To: freebsd-pkg@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id BC0C1AA8 for ; Tue, 23 Jul 2013 11:28:37 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id A423728C3 for ; Tue, 23 Jul 2013 11:28:37 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.7/8.14.7) with ESMTP id r6NBSbhn077518 for ; Tue, 23 Jul 2013 11:28:37 GMT (envelope-from bdrewery@freefall.freebsd.org) Received: (from bdrewery@localhost) by freefall.freebsd.org (8.14.7/8.14.7/Submit) id r6NBSbxR077516 for freebsd-pkg@freebsd.org; Tue, 23 Jul 2013 11:28:37 GMT (envelope-from bdrewery) Received: (qmail 2768 invoked from network); 23 Jul 2013 06:28:35 -0500 Received: from unknown (HELO ?10.10.0.24?) (freebsd@shatow.net@10.10.0.24) by sweb.xzibition.com with ESMTPA; 23 Jul 2013 06:28:35 -0500 Message-ID: <51EE68E0.6020601@FreeBSD.org> Date: Tue, 23 Jul 2013 06:28:32 -0500 From: Bryan Drewery Organization: FreeBSD User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/20130620 Thunderbird/17.0.7 MIME-Version: 1.0 To: Kimmo Paasiala Subject: Re: Poudriere jails. References: In-Reply-To: X-Enigmail-Version: 1.5.1 OpenPGP: id=3C9B0CF9; url=http://www.shatow.net/bryan/bryan.asc Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="----enig2KKJCDXOHXVUJONOUUVBF" Cc: freebsd-pkg@freebsd.org X-BeenThere: freebsd-pkg@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Binary package management and package tools discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Jul 2013 11:28:37 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) ------enig2KKJCDXOHXVUJONOUUVBF Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable On 7/23/2013 5:09 AM, Kimmo Paasiala wrote: > Is there anything special about the jails that poudriere installs with > "poudriere jail -c"? Should I be able to use a pre-installed jail that > i have installed with "make installworld > DESTDIR=3D/usr/local/poudriere/jails/somejail"? >=20 > I'm asking this because if I could install and update the jails > without using "poudriere jail" command it would save lots of time > because I could just use the world I already have built for updating > the build host. >=20 > -Kimmo Yes, they have more done to them than just an install. I do plan to add a way to import an existing directory into poudriere and to have it finish initialization of it. 1. It sets up /etc/make.conf If host is amd64 and jail is i386, this is added: ARCH=3Di386 MACHINE=3Di386 MACHINE_ARCH=3Di386 This is always added: USE_PACKAGE_DEPENDS=3Dyes BATCH=3Dyes WRKDIRPREFIX=3D/wrkdirs 2. It updates /etc/login.conf for default class so `uname` is correct: amd64 host i386 jail: :setenv=3DMAIL=3D/var/mail/$,BLOCKSIZE=3DK,FTP_PASSIVE_MODE=3DYES,UNAME_r= =3D8.3-RELEASE,UNAME_v=3DFreeBSD 8.3-RELEASE,OSVERSION=3D803000,UNAME_p=3Di386,UNAME_m=3Di386:\ amd64 jail: :setenv=3DMAIL=3D/var/mail/$,BLOCKSIZE=3DK,FTP_PASSIVE_MODE=3DYES,UNAME_r= =3D8.3-RELEASE,UNAME_v=3DFreeBSD 8.3-RELEASE,OSVERSION=3D803000:\ 3. cap_mkdb /etc/login.conf is ran 4. ld paths updated in jail: /sbin/ldconfig -m /lib /usr/lib /usr/lib/com= pat 5. /etc/master.passwd generated in jail: pwd_mkdb -d ${JAILMNT}/etc/ -p ${JAILMNT}/etc/master.passwd 6. All of /usr/local/etc/poudriere.d/jails/{method,fs,version,mnt,arch} are added 7. If the jail is on ZFS, a @clean snapshot is required for clone/rollbac= k --=20 Regards, Bryan Drewery ------enig2KKJCDXOHXVUJONOUUVBF Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBAgAGBQJR7mjgAAoJEG54KsA8mwz5Gz4P/0Sm3mXpJ3hu54t2sjVRKHYs l73nFVy88qozgv6YESFkuAmzzj7n8mbKRehc5LE8a3013caoos+uW8godA30jSqk ehiQe/8fT6y92+MpapPCxK47ngJRNeqeHr2uK1Dg95qao24tHpTNc7DUvpsqVfWi 5UmxztUg9MyG+zkV45+me675YADC4dKicTEZasiMR1ko7zQy7oSXVm6WjVy5dcwF ZpMgkcK4bMc5LLyWgO2KQLulJXBnc2gdrb36uco7JowtooJr6cvhb3TR3qf41Dvi T/t4PoBUsT+jw1Tea9P3Jvn8M/KqRFtXZLfypUS2PCfC3yeomLJCDuyfyQtoXgg+ JC3bx211kxz+7yVSCSitGpPLsrkkdIb3vUlwfEXBsJbsPbbwcQUDW013ZHDe/5hd wYVttJCKBWMtU72mJrt+OAlBh7P/UeuklHDx618G8RNZDOO2u5rOXL8TdZBywYC+ 74saLMQZ5ImGxYrfrLNShUy7gFJKekn0yEHOGaIi2YmFz/rJ9LrXPFwauxk3YxaE JzpUhoP4GDqiM8GlbqFDH466tWsA45kgMuJatWoVtkLaAgvL/IEAVA01cTUIKK1F ivBWf0wxvoTD3BdaYiTMNhgZ+1UvHgsKBCtD9LndUzN+5ahgo1HRJq3Ikk+tAy+j BGRjfwefTkrpjL1vWFQ0 =ugvh -----END PGP SIGNATURE----- ------enig2KKJCDXOHXVUJONOUUVBF-- From owner-freebsd-pkg@FreeBSD.ORG Tue Jul 23 11:34:07 2013 Return-Path: Delivered-To: freebsd-pkg@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 30733C41 for ; Tue, 23 Jul 2013 11:34:07 +0000 (UTC) (envelope-from baptiste.daroussin@gmail.com) Received: from mail-wi0-x229.google.com (mail-wi0-x229.google.com [IPv6:2a00:1450:400c:c05::229]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id BC0582915 for ; Tue, 23 Jul 2013 11:34:06 +0000 (UTC) Received: by mail-wi0-f169.google.com with SMTP id c10so3718928wiw.0 for ; Tue, 23 Jul 2013 04:34:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=AibahFU9X/xNpboFN6d7/UZUMZwUalf0jJ9QN3OZypY=; b=IIPsycfGb6FX4BKI4GGOAXfbFfvZwn7ztn4Fjq5GY4X445UcvyDusOJJEo55O1aH1j Ru2aKmUQUdGZkQxiKp2Pz2qR7Q1OBicXFPAZQdh0m0ookShxtBy20ae1QsXhK1eWcpx0 GJROnlqPJgatu9dQ4ZHKHbI7IV2dvLxWZkgw6YYrhITLWEwdHcYBtv9dJRdx0OaxNfpt 5oOVPvriwvQ2zeru1SXAsljMUUx/+QFFNW+LYc4vj9DQY08QUW4UYV/+e7qbESFDo1i0 m4qrXO473kS3V3XVDCJcqpR2Ymla2IdPELg/tWd3AFcLdFBUwrfTJDcVRsTGSPTtoI1u RiLw== X-Received: by 10.180.208.17 with SMTP id ma17mr21433781wic.7.1374579245112; Tue, 23 Jul 2013 04:34:05 -0700 (PDT) Received: from ithaqua.etoilebsd.net (ithaqua.etoilebsd.net. [37.59.37.188]) by mx.google.com with ESMTPSA id j20sm5214976wie.7.2013.07.23.04.34.03 for (version=TLSv1 cipher=RC4-SHA bits=128/128); Tue, 23 Jul 2013 04:34:03 -0700 (PDT) Sender: Baptiste Daroussin Date: Tue, 23 Jul 2013 13:34:01 +0200 From: Baptiste Daroussin To: Kimmo Paasiala Subject: Re: Poudriere jails. Message-ID: <20130723113401.GK26964@ithaqua.etoilebsd.net> References: MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="QxIEt88oQPsT6QmF" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-pkg@freebsd.org X-BeenThere: freebsd-pkg@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Binary package management and package tools discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Jul 2013 11:34:07 -0000 --QxIEt88oQPsT6QmF Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Jul 23, 2013 at 01:09:21PM +0300, Kimmo Paasiala wrote: > Is there anything special about the jails that poudriere installs with > "poudriere jail -c"? Should I be able to use a pre-installed jail that > i have installed with "make installworld > DESTDIR=3D/usr/local/poudriere/jails/somejail"? >=20 > I'm asking this because if I could install and update the jails > without using "poudriere jail" command it would save lots of time > because I could just use the world I already have built for updating > the build host. >=20 There are a bit of some special things yes, best is probably for you to make release after make buildworld to create the sets and then point poudriere j= ail to those binary sets. regards, Bapt --QxIEt88oQPsT6QmF Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) iEYEARECAAYFAlHuaikACgkQ8kTtMUmk6EwdwgCeJgpBfoEfSwNEpJ2qCknbiXdG /OwAoLorVhz+XBt9l+fdd/KN5kzwvcAA =3wpi -----END PGP SIGNATURE----- --QxIEt88oQPsT6QmF-- From owner-freebsd-pkg@FreeBSD.ORG Tue Jul 23 11:42:06 2013 Return-Path: Delivered-To: freebsd-pkg@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 4358BDDC; Tue, 23 Jul 2013 11:42:06 +0000 (UTC) (envelope-from kpaasial@gmail.com) Received: from mail-qc0-x234.google.com (mail-qc0-x234.google.com [IPv6:2607:f8b0:400d:c01::234]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id EF038297A; Tue, 23 Jul 2013 11:42:05 +0000 (UTC) Received: by mail-qc0-f180.google.com with SMTP id a1so4206380qcx.25 for ; Tue, 23 Jul 2013 04:42:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=kOXuEWS8HhxFKpEFD6TWcH4lCxI3uMHF46MzzCq31qk=; b=RTZwfTS9ScHE//XoXeJQukOd6708JIYYuKThlQ0I9OGZ7dyw6r7zwVTiQKyRpSfqBS 9d1cbjRNyzrObLsZDVEyqnibuIBAqSlbkSkmQROo6qLKRcMXy8I7MEjtieTHp9J6pvnJ /eF9JuVajwNpIkyk9bgCGocAcY+HVnpnBqqdoMJn9F3/PTBGuAz7sOECTbc9QxKZNE75 aYTWJFNWTGbNCxUjkqIEB9hajIdid+0u9lgm3V9/iSBxVmWIXqH63Y+9WHkva6bqZdgT u0LvwdQ7jqRovKltA1OD04jkSRl7OE6BIE+gruEGNFUP6cvFXgZr2RodFHOkODS1ChHi Ulpg== MIME-Version: 1.0 X-Received: by 10.224.74.1 with SMTP id s1mr39011909qaj.25.1374579725158; Tue, 23 Jul 2013 04:42:05 -0700 (PDT) Received: by 10.224.78.194 with HTTP; Tue, 23 Jul 2013 04:42:05 -0700 (PDT) In-Reply-To: <20130723113401.GK26964@ithaqua.etoilebsd.net> References: <20130723113401.GK26964@ithaqua.etoilebsd.net> Date: Tue, 23 Jul 2013 14:42:05 +0300 Message-ID: Subject: Re: Poudriere jails. From: Kimmo Paasiala To: Baptiste Daroussin Content-Type: text/plain; charset=UTF-8 Cc: freebsd-pkg@freebsd.org X-BeenThere: freebsd-pkg@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Binary package management and package tools discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Jul 2013 11:42:06 -0000 On Tue, Jul 23, 2013 at 2:34 PM, Baptiste Daroussin wrote: > On Tue, Jul 23, 2013 at 01:09:21PM +0300, Kimmo Paasiala wrote: >> Is there anything special about the jails that poudriere installs with >> "poudriere jail -c"? Should I be able to use a pre-installed jail that >> i have installed with "make installworld >> DESTDIR=/usr/local/poudriere/jails/somejail"? >> >> I'm asking this because if I could install and update the jails >> without using "poudriere jail" command it would save lots of time >> because I could just use the world I already have built for updating >> the build host. >> > > There are a bit of some special things yes, best is probably for you to make > release after make buildworld to create the sets and then point poudriere jail > to those binary sets. > > regards, > Bapt Thank you. You should update the help text of "poudriere jail" command to make a mention of the url= method, the manual page does already mention it. -Kimmo From owner-freebsd-pkg@FreeBSD.ORG Thu Jul 25 15:02:16 2013 Return-Path: Delivered-To: pkg@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 0F09EB6A; Thu, 25 Jul 2013 15:02:16 +0000 (UTC) (envelope-from vsevolod@FreeBSD.org) Received: from n.highsecure.ru (unknown [IPv6:2001:41d0:8:dd9a::99]) (using TLSv1.1 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 9D3372D4F; Thu, 25 Jul 2013 15:02:12 +0000 (UTC) Received: from medway.cl.cam.ac.uk (medway.cl.cam.ac.uk [128.232.64.24]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: vsevolod@highsecure.ru) by n.highsecure.ru (Postfix) with ESMTPSA id 46ACD221279; Thu, 25 Jul 2013 16:01:12 +0100 (BST) Message-ID: <51F13DF0.5040209@FreeBSD.org> Date: Thu, 25 Jul 2013 16:02:08 +0100 From: Vsevolod Stakhov User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130623 Thunderbird/17.0.7 MIME-Version: 1.0 To: Baptiste Daroussin Subject: The new architecture of pkgng solver Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: pkg@FreeBSD.org X-BeenThere: freebsd-pkg@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Binary package management and package tools discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Jul 2013 15:02:16 -0000 Hello, At the moment I plan to do the following: 1) - introduce 'the universe' structure, which contains all deps (for install/fetch/upgrade), rdeps (for deinstall), conflicts and their rdeps (for install); - add request list - a list of packages to install, remove or upgrade - write universal solver code that takes universe and a request list and then either produces cudf output or uses an internal SAT solver; - results are written into two lists: packages to add/upgrade and packages to remove (e.g. due to conflicts). 2) Plan 'provides' logic that means that a dependency can be provided by a number of (potentially conflicting) packages. It is implemented as pkg field but it is not used by solver. 3) Internal solver should work based on the following procedure: 1. Iterate over the universe and create a boolean problem: * A depends on B that is provided by B1, B2 or B3 is transferred to a set of rules: (!A | B1 | B2 | B3) * A conflicts between B1, B2 and B3 is transferred to the following: (!B1 | !B2) & (!B1 | !B2) & (!B1 | !B3) or in case of conflict to a provided feature if A conflicts with B provided by B1, B2 or B3: (!A | !B1) & (!A | !B2) & (!A | !B3) 2. Explicit requests for install or remove are written as: (A) - install A (!A) - remove A 3. For all explicit request (e.g. install A) set the value of these requests either to true (for packages to install) or false (for packages to deinstall). 4. Iterate over the record and set all unary rules to true, for example (A) & (!A | B1 | B2 | B3) & (!A | !C) which means that A depends on (B1 or B2 or B3) and conflicts with C. After this step we set A to TRUE as it is unary rule. 5. For all other rules perform propagation procedure, substitute variables to make a rule as true. For example in the previous example: * (A) & (!A | B1 | B2 | B3) & (!A | !C) - A is TRUE; * (!A | !C) - !C should be TRUE as !A is FALSE and C thus is FALSE; * (!A | B1 | B2 | B3) - !A is FALSE, so one of B1/B2/B3 should be TRUE. 6. For the last step there is a choice, what to select. I'd suggest to follow this logic: * try installed packages first; * do not install any packages that are not needed (e.g. set all available alternatives to FALSE); 7. Check the overall expression and if it is FALSE try to reassign the last assigned variable from the previous step. 8. 6 and 7 till the expression is either TRUE or there is no way to reassign variables (unsolvable solution). With this logic we have only a single problematic point - how to convert the packages universe and request to a SAT problem. On the other hand, with this architecture we have a *universal* solution for all procedures (install/remove/upgrade). Moreover, it is compatible with CUDF format, therefore I suggest to rework the current jobs architecture to this one. I'll do it if there are no objections. (And, well, we need to think eventually what to do with share libraries dependencies and flexible dependencies, e.g. Depends: libsomething (>1.4)) -- Vsevolod Stakhov From owner-freebsd-pkg@FreeBSD.ORG Thu Jul 25 15:33:49 2013 Return-Path: Delivered-To: freebsd-pkg@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 967EFCC1 for ; Thu, 25 Jul 2013 15:33:49 +0000 (UTC) (envelope-from matthew@freebsd.org) Received: from smtp.infracaninophile.co.uk (smtp6.infracaninophile.co.uk [IPv6:2001:8b0:151:1:3cd3:cd67:fafa:3d78]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id E1F6D2F82 for ; Thu, 25 Jul 2013 15:33:48 +0000 (UTC) Received: from rufus.webfusion.com (mail.heartinternet.co.uk [79.170.40.31]) (authenticated bits=0) by smtp.infracaninophile.co.uk (8.14.7/8.14.7) with ESMTP id r6PFXg7O083974 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO) for ; Thu, 25 Jul 2013 16:33:43 +0100 (BST) (envelope-from matthew@freebsd.org) DKIM-Filter: OpenDKIM Filter v2.8.3 smtp.infracaninophile.co.uk r6PFXg7O083974 Authentication-Results: smtp.infracaninophile.co.uk/r6PFXg7O083974; dkim=none reason="no signature"; dkim-adsp=none (unprotected policy) X-Authentication-Warning: lucid-nonsense.infracaninophile.co.uk: Host mail.heartinternet.co.uk [79.170.40.31] claimed to be rufus.webfusion.com Message-ID: <51F14556.9040605@freebsd.org> Date: Thu, 25 Jul 2013 16:33:42 +0100 From: Matthew Seaman User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/20130715 Thunderbird/17.0.7 MIME-Version: 1.0 To: freebsd-pkg@freebsd.org Subject: Re: The new architecture of pkgng solver References: <51F13DF0.5040209@FreeBSD.org> In-Reply-To: <51F13DF0.5040209@FreeBSD.org> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Virus-Scanned: clamav-milter 0.97.8 at lucid-nonsense.infracaninophile.co.uk X-Virus-Status: Clean X-Spam-Status: No, score=-1.6 required=5.0 tests=AWL,BAYES_00, RCVD_IN_DNSWL_NONE,SPF_SOFTFAIL autolearn=no version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on lucid-nonsense.infracaninophile.co.uk X-BeenThere: freebsd-pkg@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Binary package management and package tools discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Jul 2013 15:33:49 -0000 On 25/07/2013 16:02, Vsevolod Stakhov wrote: > Hello, > > At the moment I plan to do the following: > > 1) > - introduce 'the universe' structure, which contains all deps (for > install/fetch/upgrade), rdeps (for deinstall), conflicts and their rdeps > (for install); > - add request list - a list of packages to install, remove or upgrade > - write universal solver code that takes universe and a request list and > then either produces cudf output or uses an internal SAT solver; > - results are written into two lists: packages to add/upgrade and > packages to remove (e.g. due to conflicts). > > 2) > Plan 'provides' logic that means that a dependency can be provided by a > number of (potentially conflicting) packages. It is implemented as pkg > field but it is not used by solver. > > 3) > Internal solver should work based on the following procedure: > > 1. Iterate over the universe and create a boolean problem: > * A depends on B that is provided by B1, B2 or B3 is transferred to a > set of rules: (!A | B1 | B2 | B3) > * A conflicts between B1, B2 and B3 is transferred to the following: > (!B1 | !B2) & (!B1 | !B2) & (!B1 | !B3) > or in case of conflict to a provided feature if A conflicts with B > provided by B1, B2 or B3: > (!A | !B1) & (!A | !B2) & (!A | !B3) > > 2. Explicit requests for install or remove are written as: > (A) - install A > (!A) - remove A > > 3. For all explicit request (e.g. install A) set the value of these > requests either to true (for packages to install) or false (for packages > to deinstall). > > 4. Iterate over the record and set all unary rules to true, for example > (A) & (!A | B1 | B2 | B3) & (!A | !C) which means that A depends on (B1 > or B2 or B3) and conflicts with C. After this step we set A to TRUE as > it is unary rule. > > 5. For all other rules perform propagation procedure, substitute > variables to make a rule as true. For example in the previous example: > * (A) & (!A | B1 | B2 | B3) & (!A | !C) - A is TRUE; > * (!A | !C) - !C should be TRUE as !A is FALSE and C thus is FALSE; > * (!A | B1 | B2 | B3) - !A is FALSE, so one of B1/B2/B3 should be TRUE. > > 6. For the last step there is a choice, what to select. I'd suggest to > follow this logic: > * try installed packages first; > * do not install any packages that are not needed (e.g. set all > available alternatives to FALSE); > > 7. Check the overall expression and if it is FALSE try to reassign the > last assigned variable from the previous step. > > 8. 6 and 7 till the expression is either TRUE or there is no way to > reassign variables (unsolvable solution). > > With this logic we have only a single problematic point - how to convert > the packages universe and request to a SAT problem. On the other hand, > with this architecture we have a *universal* solution for all procedures > (install/remove/upgrade). Moreover, it is compatible with CUDF format, > therefore I suggest to rework the current jobs architecture to this one. > > I'll do it if there are no objections. > > (And, well, we need to think eventually what to do with share libraries > dependencies and flexible dependencies, e.g. Depends: libsomething (>1.4)) > In (1) you say: * A depends on B that is provided by B1, B2 or B3 At the moment, strictly the only data we have about 'B that is provided by B1, B2 or B3' *is* the shlib provides/requires data. All the dependency relationships between pkgs other than that are based on 'package X-N.NN.NN requires package Y-M.MM.MM' where Y-M.MM.MM was the version of the package for Y on the system at the point when package X was built. ie. there are no alternatives visible for the solver to work with. shlibs provides/requires is handy because it can be automatically generated. There are a few other similar sorts of relationships that could be extracted automatically -- for instance dependencies between perl modules -- and we can generate 'conflicts' data based on packages wanting to install files to the same place. But I think a lot of this is going to need some sort of manually specified 'PROVIDES/REQUIRES/CONFLICTS' data within each port. Cheers, Matthew From owner-freebsd-pkg@FreeBSD.ORG Thu Jul 25 15:54:59 2013 Return-Path: Delivered-To: freebsd-pkg@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 138B6628 for ; Thu, 25 Jul 2013 15:54:59 +0000 (UTC) (envelope-from vsevolod@FreeBSD.org) Received: from n.highsecure.ru (n.highsecure.ru [178.32.219.154]) (using TLSv1.1 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 9D90120AF for ; Thu, 25 Jul 2013 15:54:58 +0000 (UTC) Received: from medway.cl.cam.ac.uk (medway.cl.cam.ac.uk [128.232.64.24]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: vsevolod@highsecure.ru) by n.highsecure.ru (Postfix) with ESMTPSA id D0D32221279 for ; Thu, 25 Jul 2013 16:53:52 +0100 (BST) Message-ID: <51F14A48.3030305@FreeBSD.org> Date: Thu, 25 Jul 2013 16:54:48 +0100 From: Vsevolod Stakhov User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130623 Thunderbird/17.0.7 MIME-Version: 1.0 To: freebsd-pkg@freebsd.org Subject: Re: The new architecture of pkgng solver References: <51F13DF0.5040209@FreeBSD.org> <51F14556.9040605@freebsd.org> In-Reply-To: <51F14556.9040605@freebsd.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-pkg@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Binary package management and package tools discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Jul 2013 15:54:59 -0000 Matthew, On 25/07/13 16:33, Matthew Seaman wrote: > On 25/07/2013 16:02, Vsevolod Stakhov wrote: >> Hello, >> >> At the moment I plan to do the following: >> >> Internal solver should work based on the following procedure: >> >> 1. Iterate over the universe and create a boolean problem: >> * A depends on B that is provided by B1, B2 or B3 is transferred to a >> set of rules: (!A | B1 | B2 | B3) >> * A conflicts between B1, B2 and B3 is transferred to the following: >> (!B1 | !B2) & (!B1 | !B2) & (!B1 | !B3) >> or in case of conflict to a provided feature if A conflicts with B >> provided by B1, B2 or B3: >> (!A | !B1) & (!A | !B2) & (!A | !B3) >> >> > > In (1) you say: > * A depends on B that is provided by B1, B2 or B3 > > At the moment, strictly the only data we have about 'B that is provided > by B1, B2 or B3' *is* the shlib provides/requires data. All the > dependency relationships between pkgs other than that are based on > 'package X-N.NN.NN requires package Y-M.MM.MM' where Y-M.MM.MM was the > version of the package for Y on the system at the point when package X > was built. ie. there are no alternatives visible for the solver to work > with. With my GSoC project I've introduced both provides and conflicts. Conflicts are generated automatically during repository creation (by pkg repo -l) and exported during 'pkg update'. Provides are not generated but they could be specified in a manifest and therefore generated by ports. Here is my idea about provides: - if a package A provides virtual package B (for instance perl5.10 provides perl5 package) then it provides B of version x.x. - if a package C depends on B of any compatible version (e.g. version >= x.x) than package A installed satisfies this dependency. > shlibs provides/requires is handy because it can be automatically > generated. Not really, as these dependencies are plain as well, and I think we should follow the cudf scenario for this purposes: package formulae, boolean formulae over versioned package predicates: python-minimal: any package called python-minimal libedac1 = 1: version 1 of package libedac1 libxft2 > 1: any version of libxft2 greater than 1 haskell-doc <= 2: any version of haskell-doc less than or equal to 2 libz-dev != 3: any version of libz-dev other than 3 postfix > 2 | exim4-base: either a version of postfix greater than 2, or any version of exim4-base ocaml-nox, libc6 >= 6: both (any version of) ocaml-nox and a version of libc6 greater than 6 php4, apache | httpd: both (any version of) php4 and one among: (any version of) apache and (any version of) httpd With this abstraction we won't need explicit 'requires' - just an advanced package dependencies: dependency ::= | dep_rules ::= "|" | dep_rule ::= name ::= type ::= origin (for direct deps) | shlib | virtual (for provides) operation ::= compassion operation: < | > | = | != | >= | <= | any version ::= However, this change is not literally compatible with ports. Especially this is true for conflicts as from my point of view it is nearly impossible to convert from CONFLICTS= somepkg1.* to a reasonable formulae presented above. > There are a few other similar sorts of relationships that > could be extracted automatically -- for instance dependencies between > perl modules -- and we can generate 'conflicts' data based on packages > wanting to install files to the same place. Yes, but now these conflicts are generated when a repo is created, so we have this information already without any requirement of files rescanning (as it is quite an expensive operation). > But I think a lot of this > is going to need some sort of manually specified > 'PROVIDES/REQUIRES/CONFLICTS' data within each port. Yes, but remove requires, they are not needed as we already have a dependency definition :) -- Vsevolod Stakhov From owner-freebsd-pkg@FreeBSD.ORG Thu Jul 25 17:10:41 2013 Return-Path: Delivered-To: pkg@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id D6F6926C; Thu, 25 Jul 2013 17:10:41 +0000 (UTC) (envelope-from baptiste.daroussin@gmail.com) Received: from mail-wi0-x232.google.com (mail-wi0-x232.google.com [IPv6:2a00:1450:400c:c05::232]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 438CB24B2; Thu, 25 Jul 2013 17:10:41 +0000 (UTC) Received: by mail-wi0-f178.google.com with SMTP id hi5so2066919wib.11 for ; Thu, 25 Jul 2013 10:10:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=v8mLcWpsSum49jIz+h3++YDP62u2W+jfykNzB5/ZzDg=; b=E9PKeoy11VQ2t/uECwLRKGK94D0CLG9kwH28Y/4E/2v3eiPuSsjS+7CIdqF6CE9vPb wK3b2O5iydcoWQaa684aMxSmuLJlW5q3Blks68d32ddFzZN2VVDPnJYvICwNSQqlhl+L XzR9JzgkN2brQbFxinC9YM8ueBX+ZFwa3r23o/8tLmAFojW9+pkSPkLZHRLBnVWzytaV UHvemTuuYElHRX8FcO7nFGdhpshdBRW9FYOa5SqAA15O8rQLbbDIbYc/QFV/qbnkzJFS a7feZ5+JqT6Xf6tJT0DUD1KgeNZEivWCyLebJKJ8GQsadizslA1oSwC1pAflrQcKETJ9 4vbw== X-Received: by 10.194.238.199 with SMTP id vm7mr30767809wjc.37.1374772239603; Thu, 25 Jul 2013 10:10:39 -0700 (PDT) Received: from ithaqua.etoilebsd.net (ithaqua.etoilebsd.net. [37.59.37.188]) by mx.google.com with ESMTPSA id i1sm5128528wiz.6.2013.07.25.10.10.38 for (version=TLSv1 cipher=RC4-SHA bits=128/128); Thu, 25 Jul 2013 10:10:38 -0700 (PDT) Sender: Baptiste Daroussin Date: Thu, 25 Jul 2013 19:10:37 +0200 From: Baptiste Daroussin To: Vsevolod Stakhov Subject: Re: The new architecture of pkgng solver Message-ID: <20130725171037.GQ61207@ithaqua.etoilebsd.net> References: <51F13DF0.5040209@FreeBSD.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="6tPipYVl+OcoAvSh" Content-Disposition: inline In-Reply-To: <51F13DF0.5040209@FreeBSD.org> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: pkg@FreeBSD.org X-BeenThere: freebsd-pkg@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Binary package management and package tools discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Jul 2013 17:10:41 -0000 --6tPipYVl+OcoAvSh Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Jul 25, 2013 at 04:02:08PM +0100, Vsevolod Stakhov wrote: > Hello, >=20 > At the moment I plan to do the following: >=20 [...] > I'll do it if there are no objections. I bet there won't be any objecttions here :) Please implement what you said= that is all we need ! >=20 > (And, well, we need to think eventually what to do with share libraries= =20 > dependencies and flexible dependencies, e.g. Depends: libsomething (>1.4)) That is the more problematic thing, pkg add and the ports tree package buil= ding rely on the old form aka full direct dependency So what I do propose here is: to provide the 2 kind of dependency old and new format. in the manifest. Let's say: Depends: { name: ..., formulae: ... } Deps: {....} for the shared library we already have the needed We also need requires: provides: In the long temps Deps will just be dropped but we need to be able in the p= orts tree to match properly the depends and the shared libraries, I don't know y= et how to do that, but I'll dig :) Would be nice if you solver could take in account local files (aka being us= ed for pkg add) :) regards, Bapt --6tPipYVl+OcoAvSh Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) iEYEARECAAYFAlHxXA0ACgkQ8kTtMUmk6ExUYgCfcFoymhxDGa9sAq7P49YvEH7z nDIAnR87yD/RJXQWzX1FuwrVYYWkFCru =LicN -----END PGP SIGNATURE----- --6tPipYVl+OcoAvSh-- From owner-freebsd-pkg@FreeBSD.ORG Fri Jul 26 17:47:46 2013 Return-Path: Delivered-To: pkg@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 2287444F for ; Fri, 26 Jul 2013 17:47:46 +0000 (UTC) (envelope-from mwlucas@bewilderbeast.blackhelicopters.org) Received: from bewilderbeast.blackhelicopters.org (mwlucas-2-pt.tunnel.tserv9.chi1.ipv6.he.net [IPv6:2001:470:1f10:b9c::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id E75892898 for ; Fri, 26 Jul 2013 17:47:45 +0000 (UTC) Received: from bewilderbeast.blackhelicopters.org (localhost [127.0.0.1]) by bewilderbeast.blackhelicopters.org (8.14.5/8.14.5) with ESMTP id r6QHlhoM037962 for ; Fri, 26 Jul 2013 13:47:44 -0400 (EDT) (envelope-from mwlucas@bewilderbeast.blackhelicopters.org) Received: (from mwlucas@localhost) by bewilderbeast.blackhelicopters.org (8.14.5/8.14.5/Submit) id r6QHlh0D037961 for pkg@freebsd.org; Fri, 26 Jul 2013 13:47:43 -0400 (EDT) (envelope-from mwlucas) Date: Fri, 26 Jul 2013 13:47:43 -0400 From: "Michael W. Lucas" To: pkg@freebsd.org Subject: "pkg upgrade -y" behavior differs in script vs command line Message-ID: <20130726174743.GA37930@bewilderbeast.blackhelicopters.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.21 (2010-09-15) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.7 (bewilderbeast.blackhelicopters.org [127.0.0.1]); Fri, 26 Jul 2013 13:47:44 -0400 (EDT) X-BeenThere: freebsd-pkg@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Binary package management and package tools discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Jul 2013 17:47:46 -0000 Hi, I'm testing a 9.2-beta box, and want to deploy pkgng immediately upon installation. I'm writing a script to do this automatically, before any packages get installed. pkg upgrade -y behaves differently in a script versus the command line. Do I need to do something special to make pkg upgrade work in a script? The script and output follow. -- #!/bin/sh FTP_PROXY=blah HTTP_PROXY=blah ftp_proxy=blah http_proxy=blah pkg_add -r pkg echo "WITH_PKGNG=YES" >> /etc/make.conf pkg2ng cp pkg.conf /usr/local/etc/ chown root:wheel /usr/local/etc/pkg.conf echo "" echo "basic pkg config done, now let pkg upgrade itself" pkg upgrade -y pkg upgrade -y echo "pkgng now installed, so installing packages" -- (My setup script has more, but I've stripped the rest out while debugging this one problem.) When I run the script I get: ... basic pkg config done, now let pkg upgrade itself Updating repository catalogue digests.txz 100% 1040KB 1.0MB/s 1.0MB/s 00:00 packagesite.txz 100% 5689KB 1.9MB/s 2.2MB/s 00:03 pkg: Invalid manifest format: mapping values are not allowed in this context Incremental update completed, 0 packages processed: 0 packages updated, 0 removed and 23554 added. pkg: No digest falling back on legacy catalog format packagesite repository catalogue is up-to-date, no need to fetch fresh copy Nothing to do pkg install can't find any packages, even though 23554 should be there. I run the command interactively and get: # pkg upgrade -y Updating repository catalogue digests.txz 100% 1011KB 1.0MB/s 1.0MB/s 00:00 packagesite.txz 100% 5689KB 5.6MB/s 3.3MB/s 00:01 (long pause as pkg eats ~90% of my cpu) Incremental update completed, 0 packages processed: 0 packages updated, 0 removed and 22912 added. New version of pkg detected; it needs to be installed first. After this upgrade it is recommended that you do a full upgrade using: 'pkg upgrade' Uprgades have been requested for the following 1 packages: Reinstalling pkg-1.1.4_1 (needed shared library changed) The upgrade will free 10 kB 1 MB to be downloaded pkg-1.1.4_1.txz 100% 1598KB 1.6MB/s 1.6MB/s 00:01 Checking integrity... done [1/1] Reinstalling pkg-1.1.4_1... done # Do I need to do something special to script pkg commands? Thanks, ==ml -- Michael W. Lucas - mwlucas@michaelwlucas.com, Twitter @mwlauthor http://www.MichaelWLucas.com/, http://blather.MichaelWLucas.com/ Absolute OpenBSD 2/e - http://www.nostarch.com/openbsd2e coupon code "ILUVMICHAEL" gets you 30% off & helps me. From owner-freebsd-pkg@FreeBSD.ORG Fri Jul 26 19:29:19 2013 Return-Path: Delivered-To: pkg@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id BCF78796 for ; Fri, 26 Jul 2013 19:29:19 +0000 (UTC) (envelope-from m.seaman@infracaninophile.co.uk) Received: from smtp.infracaninophile.co.uk (smtp6.infracaninophile.co.uk [IPv6:2001:8b0:151:1:3cd3:cd67:fafa:3d78]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 09EDB2CD1 for ; Fri, 26 Jul 2013 19:29:18 +0000 (UTC) Received: from seedling.black-earth.co.uk (seedling.black-earth.co.uk [81.2.117.99]) (authenticated bits=0) by smtp.infracaninophile.co.uk (8.14.7/8.14.7) with ESMTP id r6QJTD2Z046076 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Fri, 26 Jul 2013 20:29:13 +0100 (BST) (envelope-from m.seaman@infracaninophile.co.uk) DKIM-Filter: OpenDKIM Filter v2.8.3 smtp.infracaninophile.co.uk r6QJTD2Z046076 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=infracaninophile.co.uk; s=201001-infracaninophile; t=1374866953; bh=y8fr+F4bzDJDtfm//m47tDUwY+0Fk1HoDva2pSCziog=; h=Date:From:To:CC:Subject:References:In-Reply-To; z=Date:=20Fri,=2026=20Jul=202013=2020:29:04=20+0100|From:=20Matthew =20Seaman=20|To:=20"Michael=20W.= 20Lucas"=20|CC:=20pkg@freebsd.org|Subje ct:=20Re:=20"pkg=20upgrade=20-y"=20behavior=20differs=20in=20scrip t=20vs=20command=20line|References:=20<20130726174743.GA37930@bewi lderbeast.blackhelicopters.org>|In-Reply-To:=20<20130726174743.GA3 7930@bewilderbeast.blackhelicopters.org>; b=AlXfBuTCQdPN5GBf464Evg64vFjqKKFFeiahEG95ZfTpkB+fpPNzWTX13mvKA37Ct 2eq7RS/pwOY6ygJLCkQcs9+rbroC2fuEcRnzBEoXNddgdAy9ImyQbvTMFA9WvIbrho q97xP8v5LaVMxytv/EhVcZUhPRfHDQWARIrS90UQ= Message-ID: <51F2CE00.5060901@infracaninophile.co.uk> Date: Fri, 26 Jul 2013 20:29:04 +0100 From: Matthew Seaman User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:17.0) Gecko/20130620 Thunderbird/17.0.7 MIME-Version: 1.0 To: "Michael W. Lucas" Subject: Re: "pkg upgrade -y" behavior differs in script vs command line References: <20130726174743.GA37930@bewilderbeast.blackhelicopters.org> In-Reply-To: <20130726174743.GA37930@bewilderbeast.blackhelicopters.org> X-Enigmail-Version: 1.5.2 OpenPGP: id=60AE908C Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="f5uxUuFnoPaBw7KUfEW00vmS65F8Edrsg" X-Virus-Scanned: clamav-milter 0.97.8 at lucid-nonsense.infracaninophile.co.uk X-Virus-Status: Clean X-Spam-Status: No, score=-2.0 required=5.0 tests=AWL,BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,SPF_FAIL,SPF_HELO_FAIL autolearn=no version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on lucid-nonsense.infracaninophile.co.uk Cc: pkg@freebsd.org X-BeenThere: freebsd-pkg@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Binary package management and package tools discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Jul 2013 19:29:19 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --f5uxUuFnoPaBw7KUfEW00vmS65F8Edrsg Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable On 26/07/2013 18:47, Michael W. Lucas wrote: > Hi, >=20 > I'm testing a 9.2-beta box, and want to deploy pkgng immediately upon > installation. I'm writing a script to do this automatically, before > any packages get installed. >=20 > pkg upgrade -y behaves differently in a script versus the command > line. Do I need to do something special to make pkg upgrade work in a > script? The script and output follow. >=20 > -- > #!/bin/sh > FTP_PROXY=3Dblah > HTTP_PROXY=3Dblah > ftp_proxy=3Dblah > http_proxy=3Dblah >=20 > pkg_add -r pkg > echo "WITH_PKGNG=3DYES" >> /etc/make.conf > pkg2ng >=20 > cp pkg.conf /usr/local/etc/ > chown root:wheel /usr/local/etc/pkg.conf >=20 > echo "" > echo "basic pkg config done, now let pkg upgrade itself" > pkg upgrade -y > pkg upgrade -y >=20 > echo "pkgng now installed, so installing packages" > -- >=20 > (My setup script has more, but I've stripped the rest out while > debugging this one problem.) >=20 > When I run the script I get: >=20 > ... > basic pkg config done, now let pkg upgrade itself > Updating repository catalogue > digests.txz 100% 1040KB 1.0MB/s 1.0MB/s 0= 0:00 =20 > packagesite.txz 100% 5689KB 1.9MB/s 2.2MB/s 0= 0:03 =20 > pkg: Invalid manifest format: mapping values are not allowed in this co= ntext > Incremental update completed, 0 packages processed: > 0 packages updated, 0 removed and 23554 added. > pkg: No digest falling back on legacy catalog format > packagesite repository catalogue is up-to-date, no need to fetch fresh = copy > Nothing to do >=20 > pkg install can't find any packages, even though 23554 should be there.= >=20 > I run the command interactively and get: >=20 > # pkg upgrade -y > Updating repository catalogue > digests.txz 100% 1011KB 1.0MB/s 1.0M= B/s 00:00 =20 > packagesite.txz 100% 5689KB 5.6MB/s 3.3M= B/s 00:01=20 > (long pause as pkg eats ~90% of my cpu) > Incremental update completed, 0 packages processed: > 0 packages updated, 0 removed and 22912 added. > New version of pkg detected; it needs to be installed first. > After this upgrade it is recommended that you do a full upgrade using: = 'pkg upgrade' >=20 > Uprgades have been requested for the following 1 packages: >=20 > Reinstalling pkg-1.1.4_1 (needed shared library changed) >=20 > The upgrade will free 10 kB >=20 > 1 MB to be downloaded > pkg-1.1.4_1.txz 100% 1598KB 1.6MB/s 1.6M= B/s 00:01 =20 > Checking integrity... done > [1/1] Reinstalling pkg-1.1.4_1... done > # >=20 > Do I need to do something special to script pkg commands? There's nothing in the pkg(8) code designed to make it act differently when run from a script rather than interactively. At least, nothing intentional. Two possibilities: - there's something significantly different about the environment within your script compared to your command line session, and that's causing different behaviour. Any of the variables in pkg.conf(5) can be set in the environment and should override the settings in pkg.conf. - pkg(8) actually modified your package database the first time you ran it in the script, and subsequently re-running it would have had a different outcome whether run from the script or the command line. Or I could be completely wrong, and you've discovered some rather unfortunate unintentional misbehaviour. Hmmm... Cheers, Matthew --=20 Dr Matthew J Seaman MA, D.Phil. PGP: http://www.infracaninophile.co.uk/pgpkey JID: matthew@infracaninophile.co.uk --f5uxUuFnoPaBw7KUfEW00vmS65F8Edrsg Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.16 (Darwin) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlHyzggACgkQ8Mjk52CukIwAaQCeKmIWXXGOtv5flqJeIGpckklA qpQAniVrft/79IKa/97xM3YB1xwTiiKR =tp5f -----END PGP SIGNATURE----- --f5uxUuFnoPaBw7KUfEW00vmS65F8Edrsg--