From owner-freebsd-questions@FreeBSD.ORG Tue Apr 15 11:14:28 2014 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 6CD8F7B6 for ; Tue, 15 Apr 2014 11:14:28 +0000 (UTC) Received: from sola.nimnet.asn.au (paqi.nimnet.asn.au [115.70.110.159]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id CD76B1C0A for ; Tue, 15 Apr 2014 11:14:27 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by sola.nimnet.asn.au (8.14.2/8.14.2) with ESMTP id s3FBE8qU005582; Tue, 15 Apr 2014 21:14:09 +1000 (EST) (envelope-from smithi@nimnet.asn.au) Date: Tue, 15 Apr 2014 21:14:08 +1000 (EST) From: Ian Smith To: Arthur Chance Subject: Re: size of source tree In-Reply-To: Message-ID: <20140415194933.Y55844@sola.nimnet.asn.au> References: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Cc: John Mehr , Erich Dollansky , freebsd-questions@freebsd.org X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Apr 2014 11:14:28 -0000 In freebsd-questions Digest, Vol 514, Issue 11, Message: 1 On Sat, 12 Apr 2014 13:04:09 +0100 Arthur Chance wrote: > On 12/04/2014 12:50, Matthew Seaman wrote: > > On 12/04/2014 10:09, Erich Dollansky wrote: > >> Hi, > >> > >> On Sat, 12 Apr 2014 09:26:23 +0100 > >> Matthew Seaman wrote: > >> > >>> On 12/04/2014 04:54, Erich Dollansky wrote: > >>>> The source tree dated 26.03.14 has a size of some 899GM while > >>>> today's source tree has some 1.8GB. > >>>> > >>>> Why is it suddenly so huge? > >>>> > >>> > >>> How were these source trees obtained? A checkout from SVN will have a > >>> .svn directory containing pristine copies of all of the files, which > >>> will pretty much double the space requirement. > >>> > >> both have been obtained with svn. The smaller one was started last year > >> with the then current version of svn. > >> > >> Is there an option to turn this behaviour of keeping a copy off? > > > > Not if you want to use SVN. You can use freebsd-update to get system > > sources -- but only from a release branch. > > Wasn't svnup (${PORTS}/net/svnup) designed for this? It fetches from svn > respositories but doesn't use the .svn directory. Indeed. I did some early testing (c. v0.56 IIRC) of svnup and sent John Mehr (cc'd) some observations and a few suggestions on a then 9.1 system that subsequently died. Early versions only used svn: protocol and were pretty slow (and CPU-intensive) from here, before John rewrote the tree indexing code and added http: and https: protocols. So the other night I installed it from 9.2-R ports, getting v1.0 (since upgraded to fix a bug with 10.0 I see). I fetched the 'release' target to a work area so I could compare it with my virgin 9.2-R sources but silly me hadn't noticed that target fetched base/releng/9.2 rather than base/release/9.2.0, so all the ID headers were different of course :) So I blew that away and did it again with release/9.2.0, after which diff -r /usr/src /tmp/release returned nothing, well passing that test. Both times took about two hours to pull the sources (using https:) which still seemed kinda slow, with svnup never using over ~7% CPU, mostly half that, so I don't really understand why it takes tat long, unless the RTT to svn0.us-west.freebsd.org of ~215ms might be a factor? I haven't tried svn for comparison but would expect to get more than ~170KB/s out of my 8-10Mbps otherwise idle link. Maybe someone could comment on that, though I notice in the very next digest message: > From: Lena@lena.kiev.ua > [..] > rm -rf /usr/src > svn export svn://svn0.us-east.FreeBSD.org/base/releng/8.4 /usr/src > > Takes about 30 minutes. To which Erich later responded: > At your place, yes, not where I am most of the time. So Erich, how does an initial two hours sound to you, to save 900MB? You might do better from Europe to us-east than from here, dunno. However a subsequent repeat run (updating nothing) took only a bit over 2 minutes, confirming earlier experience that the initial whole-tree fetch is slow but null or small incremental updates are quite fast enough for its intended use - ie a cvsup replacement for non-developers who want to upgrade their sources, track stable and such without the need for pushing patches (ie committing); for designed use I think it's a useful and very lightweight tool; I'd recommend having a play with it. As for overheads, svnup just keeps a full index of the src repository (which for 9.2.0 is 800MiB) in -rw-r--r-- 1 root wheel 3.5M Apr 14 02:37 /var/tmp/svnup/release which is seriously light-weight vs the size doubling of /usr/src/.svn I haven't tried svnup for ports; portsnap works well and quickly for me. cheers, Ian