From owner-freebsd-questions@FreeBSD.ORG Fri Jan 25 21:52:23 2013 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 8349BA01 for ; Fri, 25 Jan 2013 21:52:23 +0000 (UTC) (envelope-from glarkin@FreeBSD.org) Received: from mail1.sourcehosting.net (mail1.sourcehosting.net [74.205.51.45]) by mx1.freebsd.org (Postfix) with ESMTP id 5D11ECB4 for ; Fri, 25 Jan 2013 21:52:23 +0000 (UTC) Received: from 24-181-237-39.dhcp.oxfr.ma.charter.com ([24.181.237.39] helo=Gregory-Larkins-iMac.local) by mail1.sourcehosting.net with esmtp (Exim 4.73 (FreeBSD)) (envelope-from ) id 1TyrBz-0004fa-Sa; Fri, 25 Jan 2013 16:52:16 -0500 Received: from Gregory-Larkins-iMac.local (localhost [127.0.0.1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by Gregory-Larkins-iMac.local (Postfix) with ESMTPS id 6696E19AF9D1; Fri, 25 Jan 2013 16:52:11 -0500 (EST) Message-ID: <5102FE8A.109@FreeBSD.org> Date: Fri, 25 Jan 2013 16:52:10 -0500 From: Greg Larkin Organization: The FreeBSD Project User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:17.0) Gecko/20130107 Thunderbird/17.0.2 MIME-Version: 1.0 To: Xyne References: <20130116185832.GB8524@external.screwed.box> <50F6FD37.5060309@gmail.com> <20130116193534.GC8524@external.screwed.box> <50F704CE.3010904@gmail.com> <20130116200754.GD8524@external.screwed.box> <50F70C60.4000108@FreeBSD.org> <20130116205533.GE8524@external.screwed.box> <20130117045431.GB5392@external.screwed.box> <20130119052245.004da49b@archlinux.ca> <51004B62.5090305@FreeBSD.org> <20130124221636.307dca09@archlinux.ca> <5102A771.1080704@FreeBSD.org> <20130125185303.1396d2a4@archlinux.ca> In-Reply-To: <20130125185303.1396d2a4@archlinux.ca> X-Enigmail-Version: 1.5 X-SA-Exim-Connect-IP: 24.181.237.39 X-SA-Exim-Mail-From: glarkin@FreeBSD.org X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail1.sourcehosting.net X-Spam-Level: X-Spam-Status: No, score=-1.1 required=5.0 tests=AWL,BAYES_00, RCVD_IN_SORBS_DUL,RDNS_DYNAMIC,TVD_RCVD_IP autolearn=no version=3.3.1 Subject: Re: svn-export Re: svn bdb checkout? Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-SA-Exim-Version: 4.2 X-SA-Exim-Scanned: Yes (on mail1.sourcehosting.net) Cc: Peter Vereshagin , freebsd-questions@freebsd.org X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: glarkin@FreeBSD.org List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Jan 2013 21:52:23 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 1/25/13 1:53 PM, Xyne wrote: > Greg Larkin wrote: > >> # python3 svn-export -r 310000 >> http://svn.freebsd.org/ports/head/ports-mgmt ports-mgmt Exporting >> new repository. Exported revision 310000 # python3 svn-export >> --revision-file rev.dat >> http://svn.freebsd.org/ports/head/ports-mgmt ports-mgmt >> Determining current revision. Current revision is 310981. >> Determining changes [310000:310981]. 16 new/modified files, 5 >> deletions Traceback (most recent call last): File "svn-export", >> line 459, in svn.main() File "svn-export", line 432, in >> main self.remove_old_files(deleted) File "svn-export", line 310, >> in remove_old_files os.unlink(path) PermissionError: [Errno 1] >> Operation not permitted: 'ports-mgmt/pkg/files' # exit >> >> As far as I can tell, svn-export attempts to remove a directory >> before the file contained within that directory. It seems like >> PermissionError isn't the best error code for Python to throw in >> that case, since it appears to mask the root problem. > > On Arch Linux, Python raises the expected "IsADirectoryError" > exception. Perhaps "PermissionError" is due to a bug in Python 3 on > FreeBSD. It would not be the first exception bug that I have > encountered (e.g. os.makedirs(..., exist_ok=True) raises > "FileExists" when permissions differ on existing directories... > very confusing the first time it shows up). > > Regardless, svn-export now tries to remove a directory in both > cases. Please updated to version 2013.1.25 and let me know if it > works on FreeBSD. Yes, it does work on FreeBSD now, but I suggest investigating whether it's possible to reorder the removal list with files first, then directories. That way, you won't have to rely on an exception to determine if a directory to be removed is not empty yet. The os.walk() method looks like it would be useful here: http://docs.python.org/3/library/os.html?highlight=os.walk#os.walk > > >>> Incidentally, if anyone is looking at the code for svn-export, >>> feel free to suggest a better way to incrementally export new >>> subdirectories (see the "get_new_files()" method). >> >> I'll take a look at that soon to see if there are any possible >> optimizations. > > Ok, thanks. I looked at the method, and my first idea was to "svn cat" as many files at once as possible, up to the maximum command line length. You will avoid spawning a lot of svn processes and opening new network connections each time. If the threads option is >1, you could divide the maximum command line length by that number and keep filling in filenames up to that limit in each child process. I expect you would also get a nice performance boost if you changed from spawning svn commands to the Python SVN bindings. Here's an example of how it works: http://svn.apache.org/repos/asf/subversion/trunk/tools/examples/getfile.py > > >> Once we create a port for svn-export, we typically point to the >> original download site in the port Makefile. It's a good idea >> for the port maintainer to find a mirror site and include that >> too, but if not, all distro files are eventually mirrored to the >> FreeBSD cluster. >> >> Ideally, the source distros remain in place forever or at least >> until the FreeBSD port is updated to the latest version, but if >> that's not possible, I can mirror the files too. > > I have changed the time-to-live to 6 months for old archives. Excellent, thank you. Hope that all helps, Greg -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.13 (Darwin) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlEC/ooACgkQ0sRouByUApCa0wCfQr4rIT9oQrI3LkYPlMGbfCnj KqoAniabxFC2xeNAzvdLHLYhXhxYs1xi =cfUS -----END PGP SIGNATURE-----