Date: Wed, 26 Oct 2011 17:07:53 +0200 From: "C. P. Ghost" <cpghost@cordula.ws> To: Peter Kryszkiewicz <tundra2bear@gmail.com> Cc: freebsd-questions@freebsd.org Subject: Re: rsync and the ports tree Message-ID: <CADGWnjUeud0RUH3YivZA1H4R3wtscp8-fCBFvdgrP6nZNkECig@mail.gmail.com> In-Reply-To: <CAEGUhakDj=bGxyy-qhya0ccKL7SrqKQkvaKSxZ778xfQgY7K1w@mail.gmail.com> References: <CAEGUhakDj=bGxyy-qhya0ccKL7SrqKQkvaKSxZ778xfQgY7K1w@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Wed, Oct 26, 2011 at 12:39 PM, Peter Kryszkiewicz
<tundra2bear@gmail.com> wrote:
> I have several machines installed in my temporary location and only my
> laptop gets the internet through wireless. So far I've been building ports
> on the other machines by rsync'ing the distfiles from the laptop as I need
> them (all machines have the same FreeBSD 8.2 installed).
>
> The problem comes after I did a 'portupgrade -a' on the laptop. To ensure
> the other ports trees are in sync, can I rsync the /usr/ports directory to
> the other machines? Since some of them are different architectures (amd64
> multicore for instance) I ran into situations where the distfiles are
> different (for gcc for example).
First of all, rsync is working perfectly if you want to
distribute /usr/ports/distfiles, /usr/ports to your internal
machines, even when they are not of the same architecture.
I'm doing this with a BIG farm of servers running i386, amd64,
and sparc64 for a long, long time.
You only need to make sure to rsync the *union* of your
/usr/ports/distfiles directories, or else it won't work.
Say, on amd64 you have
/usr/ports/distfiles/some-distfile-for-amd64-only.tar.bz2
and on i386 you have
/usr/ports/distfiles/some-distfile-for-i386-only.tar.bz2
Yes, that happens every now and then.
So you have to rsync both ways, so that you end up with
/usr/ports/distfiles/some-distfile-for-amd64-only.tar.bz2
/usr/ports/distfiles/some-distfile-for-i386-only.tar.bz2
on both i386 and amd64 machines.
The catch is: look out for rsync's --delete flag! When some
port managers delete old/stale distfiles, they may also delete
distfiles for the *other* arches because they (rightly) think
they are not needed here... and when you then rsync with --delete,
that would (wrongly) propagate such deletes to those arches,
and you end up with missing distfiles on the targets.
Since I have more than just two arches, I use a slightly different
2-layer workflow:
0. I have 3 servers that are allowed to fetch files from the outside:
i386-master, amd64-master, sparc64-master.
and a whole bunch of i386-slave-NNN, amd64-slave-NNN and
sparc64-slave-NNN machines that would duplicate from their
relative masters via rsync.
On all -master(s), I keep $DISTFILES outside of /usr/ports
(on /usr/local/distfiles, with a symbolic link in /usr/ports
/usr/ports/distfiles -> /usr/local/distfiles)
Initial update of i386-master, as usual:
1. On i386-master, csup /usr/ports.
Run portmaster as usual to upgrade everything.
This may delete old stale distfiles and non-i386-distfiles.
This may fetch additional generic and i386-specific distfiles.
Copy the new /usr/ports (without distfiles) to the other
arch masters:
2. rsync -av --delete i386-master:/usr/ports to amd64-master
and sparc64-master. CAUTION: Use --delete is okay, but only
because distfiles are not under /usr/ports, so as not to nuke
non-i386-specific distfiles of the other arches.
Copy i386-master's NEW distfiles to the other arch masters:
3. rsync -av i386-master:/usr/local/distfiles to amd64-master
and sparc64-master. BEWARE: Don't use --delete here!
Do this to copy new generic distfiles (and i386) from
the i386-master build to amd64-master and sparc64-master.
Update amd64-master and sparc64-master's ports as usual:
4. On amd64-master, run portmaster as usual to upgrade everything.
This may delete old stale distfiles and non-amd64-distfiles.
This may fetch additional (generic and) amd64-specific-distfiles.
5. On sparc64-master, run portmaster as usual to upgrade everything.
This may delete old stale distfiles and non-sparc64-distfiles.
This may fetch additional (generic and) sparc64-specific-distfiles.
At this point, i386-master, amd64-master and sparc64-master are
fully updated, and their /usr/local/distfiles directories are up
to date w.r.t. their specific architectures. Now, copy everything
from the masters to the slaves:
6. On every i386-slave-NNN, rsync -av --delete:
/usr/ports, /usr/local (including /usr/local/distfiles),
/var/db/pkg, /var/db/ports
from i386-master.
7. On every amd64-slave-NNN, rsync -av --delete:
/usr/ports, /usr/local (including /usr/local/distfiles)
/var/db/pkg, /var/db/ports
from amd64-master.
8. On every sparc64-slave-NNN, rsync -av --delete:
/usr/ports, /usr/local (including /usr/local/distfiles)
/var/db/pkg, /var/db/ports
from sparc64-master.
You may also need to update entries in /etc and /usr/local/etc
on the slaves.
> If not rsync, what is the best way to keep multiple ports trees on different
> hardware in sync, assuming everything runs FreeBSD 8.2?
>
> regards,
> Peter Kryszkiewicz
-cpghost.
--
Cordula's Web. http://www.cordula.ws/
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CADGWnjUeud0RUH3YivZA1H4R3wtscp8-fCBFvdgrP6nZNkECig>