Date: Fri, 25 Jan 2013 16:52:10 -0500 From: Greg Larkin <glarkin@FreeBSD.org> To: Xyne <xyne@archlinux.ca> Cc: Peter Vereshagin <peter@vereshagin.org>, freebsd-questions@freebsd.org Subject: Re: svn-export Re: svn bdb checkout? Message-ID: <5102FE8A.109@FreeBSD.org> In-Reply-To: <20130125185303.1396d2a4@archlinux.ca> References: <20130116185832.GB8524@external.screwed.box> <50F6FD37.5060309@gmail.com> <20130116193534.GC8524@external.screwed.box> <50F704CE.3010904@gmail.com> <20130116200754.GD8524@external.screwed.box> <50F70C60.4000108@FreeBSD.org> <20130116205533.GE8524@external.screwed.box> <20130117045431.GB5392@external.screwed.box> <20130119052245.004da49b@archlinux.ca> <51004B62.5090305@FreeBSD.org> <20130124221636.307dca09@archlinux.ca> <5102A771.1080704@FreeBSD.org> <20130125185303.1396d2a4@archlinux.ca>
next in thread | previous in thread | raw e-mail | index | archive | help
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 1/25/13 1:53 PM, Xyne wrote: > Greg Larkin wrote: > >> # python3 svn-export -r 310000 >> http://svn.freebsd.org/ports/head/ports-mgmt ports-mgmt Exporting >> new repository. Exported revision 310000 # python3 svn-export >> --revision-file rev.dat >> http://svn.freebsd.org/ports/head/ports-mgmt ports-mgmt >> Determining current revision. Current revision is 310981. >> Determining changes [310000:310981]. 16 new/modified files, 5 >> deletions Traceback (most recent call last): File "svn-export", >> line 459, in <module> svn.main() File "svn-export", line 432, in >> main self.remove_old_files(deleted) File "svn-export", line 310, >> in remove_old_files os.unlink(path) PermissionError: [Errno 1] >> Operation not permitted: 'ports-mgmt/pkg/files' # exit >> >> As far as I can tell, svn-export attempts to remove a directory >> before the file contained within that directory. It seems like >> PermissionError isn't the best error code for Python to throw in >> that case, since it appears to mask the root problem. > > On Arch Linux, Python raises the expected "IsADirectoryError" > exception. Perhaps "PermissionError" is due to a bug in Python 3 on > FreeBSD. It would not be the first exception bug that I have > encountered (e.g. os.makedirs(..., exist_ok=True) raises > "FileExists" when permissions differ on existing directories... > very confusing the first time it shows up). > > Regardless, svn-export now tries to remove a directory in both > cases. Please updated to version 2013.1.25 and let me know if it > works on FreeBSD. Yes, it does work on FreeBSD now, but I suggest investigating whether it's possible to reorder the removal list with files first, then directories. That way, you won't have to rely on an exception to determine if a directory to be removed is not empty yet. The os.walk() method looks like it would be useful here: http://docs.python.org/3/library/os.html?highlight=os.walk#os.walk > > >>> Incidentally, if anyone is looking at the code for svn-export, >>> feel free to suggest a better way to incrementally export new >>> subdirectories (see the "get_new_files()" method). >> >> I'll take a look at that soon to see if there are any possible >> optimizations. > > Ok, thanks. I looked at the method, and my first idea was to "svn cat" as many files at once as possible, up to the maximum command line length. You will avoid spawning a lot of svn processes and opening new network connections each time. If the threads option is >1, you could divide the maximum command line length by that number and keep filling in filenames up to that limit in each child process. I expect you would also get a nice performance boost if you changed from spawning svn commands to the Python SVN bindings. Here's an example of how it works: http://svn.apache.org/repos/asf/subversion/trunk/tools/examples/getfile.py > > >> Once we create a port for svn-export, we typically point to the >> original download site in the port Makefile. It's a good idea >> for the port maintainer to find a mirror site and include that >> too, but if not, all distro files are eventually mirrored to the >> FreeBSD cluster. >> >> Ideally, the source distros remain in place forever or at least >> until the FreeBSD port is updated to the latest version, but if >> that's not possible, I can mirror the files too. > > I have changed the time-to-live to 6 months for old archives. Excellent, thank you. Hope that all helps, Greg -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.13 (Darwin) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlEC/ooACgkQ0sRouByUApCa0wCfQr4rIT9oQrI3LkYPlMGbfCnj KqoAniabxFC2xeNAzvdLHLYhXhxYs1xi =cfUS -----END PGP SIGNATURE-----
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?5102FE8A.109>