From owner-freebsd-hackers@FreeBSD.ORG Fri Mar 5 19:00:16 2010 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3548B106566C for ; Fri, 5 Mar 2010 19:00:16 +0000 (UTC) (envelope-from xorquewasp@googlemail.com) Received: from mail-wy0-f182.google.com (mail-wy0-f182.google.com [74.125.82.182]) by mx1.freebsd.org (Postfix) with ESMTP id B0BBD8FC6D for ; Fri, 5 Mar 2010 19:00:15 +0000 (UTC) Received: by wyb32 with SMTP id 32so2319857wyb.13 for ; Fri, 05 Mar 2010 11:00:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=gamma; h=domainkey-signature:received:received:received:received :x-authentication-warning:date:from:to:cc:subject:message-id :references:mime-version:content-type:content-disposition :in-reply-to; bh=FCQcTTE+fKKiHbUdkNG4XtKSitK0qaPL9UUACm2dXC4=; b=F8gANVlQ/AVg1cdhNG1cWt4N9yF+tWaZdl2DAJ5sACb1IwzJH8YRtEDYJPFv/TSuG6 wRCB798EfNioMPEH/kwm3DBmtjq5Ek+c3lnUKtVs60yJQZbtnru2npak7aqreDTpXCzF yCBHbklXAvIiqP52f3WSC6r5pbF8jG1u/Sxto= DomainKey-Signature: a=rsa-sha1; c=nofws; d=googlemail.com; s=gamma; h=x-authentication-warning:date:from:to:cc:subject:message-id :references:mime-version:content-type:content-disposition :in-reply-to; b=uqRTL1RCwkhn41IcPK/H8x+plLrux7CsbiInJ+He1Ysx4/SqOV0KB1zHuSdzdiBkxO 4lC7EG+YlqfoCPlwK14ir1Llfd7dPAllAsDo2GaKjJpTx0mJ7JPfHn78/M2mqjt6laBK x+xmmoGoF6fXKnL8NNPiy1I9FfZNGloXUVx8o= Received: by 10.216.88.136 with SMTP id a8mr419544wef.77.1267815609060; Fri, 05 Mar 2010 11:00:09 -0800 (PST) Received: from viper.internal.network (dsl78-143-196-85.in-addr.fast.co.uk [78.143.196.85]) by mx.google.com with ESMTPS id t12sm6322664gvd.7.2010.03.05.11.00.06 (version=TLSv1/SSLv3 cipher=OTHER); Fri, 05 Mar 2010 11:00:07 -0800 (PST) Received: from viper.internal.network (localhost [127.0.0.1]) by viper.internal.network (Postfix) with ESMTP id 827A24AC08; Fri, 5 Mar 2010 19:00:03 +0000 (UTC) Received: (from m0@localhost) by viper.internal.network (8.14.3/8.14.3/Submit) id o25IxwH8010552; Fri, 5 Mar 2010 18:59:58 GMT (envelope-from xorquewasp@googlemail.com) X-Authentication-Warning: viper.internal.network: m0 set sender to xorquewasp@googlemail.com using -f Date: Fri, 5 Mar 2010 18:59:58 +0000 From: xorquewasp@googlemail.com To: jhell Message-ID: <20100305185958.GB82765@logik.internal.network> References: <20100226163227.GA15162@logik.internal.network> <4B88074E.7050007@FreeBSD.org> <20100226222113.GA14592@logik.internal.network> <4B884D48.90509@FreeBSD.org> <20100227093409.GA40858@logik.internal.network> <864ol0w4g5.fsf@ds4.des.no> <20100304175819.GC31036@logik.internal.network> <867hpr56ek.fsf@ds4.des.no> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Cc: Dag-Erling =?iso-8859-1?Q?Sm=F8rgrav?= , freebsd-hackers@freebsd.org Subject: Re: package building failure irritation X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Mar 2010 19:00:16 -0000 'Lo, On 2010-03-05 09:47:11, jhell wrote: > Adding on to this. There were reports in various cases dating back to ~1 > year with bad results, possible data loss, hard and soft dead locks when > nullfs was used with ZFS. nullfs at one point that I do remember was not > recommended to be used at all with ZFS and there exist quite a bit of > other functionality in ZFS "cloning, snapshots, mountpoint=" etc... that > serve well enough to not use nullfs at all. Except, as far as I can see (not far), there's no way to emulate one directory/fs being mounted in multiple places. I'll come to "why" in a second... > Surely in the case above you are talking about packages and in which you > really should not need to mount this multiple times as a writable FS, > correct me if you feel that it does and lets see why (please provide your > process if you do). Basically, what I'm doing is trying to share a writable distfile directory across multiple package-building jails as a sort of cache. I don't have unlimited bandwidth usage per-month and would like to avoid multiple jails downloading 150mb distfiles (I've already exceeded monthly bandwidth limits due to this exact problem). I'd also like to avoid having multiple copies of distfiles laying around (as the number of jails increases, the space required becomes not insignificant). A possible solution to the above might be to use "make fetch-recursive" on the "host" and then just mount the distfile directory read-only inside the actual build jail to eliminate one of the read/write mounts. Of course, what distfiles are actually required might change based on the settings in make.conf in each jail... I'm also sharing the ports tree read-only across jails to ensure that each jail has the exact same tree. This isn't a problem as far as I know (due to the "read-only" part). It might be the case that the only real problem (the one that causes the package building errors) is the read/write nullfs mount of the /work directory so I'm about to build a test jail now and see if this is so (and I'll use DES' method of setting the ZFS mountpoint for this directory if it is as there's specifically no need for shared state here). > This should suffice mounting a packages type collection in multiple > places: > > # Mount one dataset wherever you need it. > zfs set mountpoint=/path/to/wherever pool/packages > > # Create a snapshot and clone it for further mountpoints. > zfs snapshot pool/packages@20100305 > zfs clone pool/packages@20100305 pool/packages2 > zfs set mountpoint=/path/to/other/dir pool/packages2 > > If none of this would suffice then your package management needs to be > re-thought out. Central FTP, NFS, SMB, RSYNC?. In this setup, jails build packages for other jails that specifically don't have network access (see the example given before about sandboxing a pdf reader): 8.0-amd64-mistrust_pkg mounts /storage/jails/8.0/x86_64/mistrust/pkg at /pkg and writes built packages to it whilst 8.0-amd64-mistrust mounts the same directory read-only at /pkg and, of course, installs packages from it. In this case, the only sane way to do things seems to be to mount a read-only view of a directory of packages built by another jail with nullfs. I don't particularly want to be cloning and managing snapshots - I'd be exchanging the simplicity of a direct solution (a direct, read-only view of the current state of a given package directory) with a solution where I have to endlessly shuffle and clone snapshots. > You have plenty of options available but the only one that will be > suitable to you & your process will be the one that is planned for > accordingly and thoroughly. I'm surprised that something as simple as the above requires this much thought to be honest. xw