Date: Wed, 7 Aug 2002 11:16:25 +0000 From: Josh Paetzel <friar_josh@webwarrior.net> To: BSD Freak <bsd-freak@mbox.com.au> Cc: Roger 'Rocky' Vetterberg <listsub@401.cx>, FreeBSD Questions <freebsd-questions@FreeBSD.ORG> Subject: Re: There must be a better way to maintain older systems Message-ID: <20020807111625.B207@twincat.vladsempire.net> In-Reply-To: <e5fe64e5c22e.e5c22ee5fe64@mbox.com.au>; from bsd-freak@mbox.com.au on Wed, Aug 07, 2002 at 09:18:05PM %2B1000 References: <e5fe64e5c22e.e5c22ee5fe64@mbox.com.au>
next in thread | previous in thread | raw e-mail | index | archive | help
On Wed, Aug 07, 2002 at 09:18:05PM +1000, BSD Freak wrote: > I think what many don't realise is how much testing, backup, > documentation etc. goes into our upgrade process before we roll out a > new version of FreeBSD. We only perform binary upgrades (I see building > from source, cvsup etc. far too risky). Many of you will no doubt > disagree very strongly but we are completely reliant on our FreeBSD > boxes being in tip top shape - mistakes cannot be tolerated. These are > not machines we run as a hobby, if they are down - our enterprise is > down. We also run some complex configurations involving multiple jails > etc. Hence our upgrade procedure is as follows for each server (with > slight variations): > > 1. We start off by testing the application on a test server with > identical hardware and the new version of FreeBSD we wish to migrate to. > > 2. We review any new features of the new version we should be > implementing including new/improved kernel options, sysctl paramters, > optimisations etc. and test these. > > 3. We do a "clean" install of the new version of FreeBSD and our > application on the test server. Thoroughly documenting the whole build > process). > > > 4. We copy all the application data from the production to the test > server. > > 5. (This is the tricky bit) We either freeze data (where possible) or > take the production server offline and plug into a seperate network > with the test server and synchronise the data (usually with rsync). > This step is very time critical and we have to do this as fast as > possible because the production server is offline. > > 6. We switch the ip addresses of the production and test server and the > test server now becomes our new production server. > > 7. We keep the old production server around (but off) for about a week > to make sure all is well with the new setup. > > 8. We wipe the old production server and it becomes the new test server > for the next box we need to upgrade. > > * Voila, upgrade complete with minimal possible downtime, minimal risk > and a "clean install" of the new version of the OS. * > > The process now starts again for the next (of our 14) FreeBSD servers. > Total time for this process is 2-3 weeks sometimes a month depending on > my sysadmin and other workload. Because of the length of this process I > have started skipping releases and upgrading servers every second > release. > > Any suggestions on how I can improve this process without introducing > higher risks would be *GREATLY* appreciated...... > > ----- Original Message ----- > From: Roger 'Rocky' Vetterberg <listsub@401.cx> > Date: Wednesday, August 7, 2002 1:23 pm > Subject: Re: There must be a better way to maintain older systems > > > BSD Freak wrote: > > > Hi all, > > > > > > I am responsible for maintaining 14 FreeBSD, 1 Windows 2000 and > > 1 > > > Solaris servers at three sites. While I am certianly no fan of > > Windows > > > 2000 or the commercial UNIX distributions I have to say they > > take up a > > > lot less of my time to maintain. For example I can download > > (binary > > > packages) patches and "Service Packs"/hotfixes to patch bugs and > > > vulnerabilities and then I forget about it. Upgrades of OS > > happen once > > > every 3-4 years (and usually accomany a hardware upgrade which > > makes it > > > a bit neater and less risky). > > > > > > With FreeBSD however I find myself upgrading every six months or > > so > > > when a new version is released. I spend half my time upgrading > > the 14 > > > production servers (in the middle of the night usually!), then > > by the > > > time I have gotten around to the last system, I'm usually only a > > month > > > or so away from the next -RELEASE and I I have to do it all > > again if I > > > am to keep my systems secure and current. > > > > > > I find myself thinking there *MUST* be a better way. I am quite > > happy > > > with the stability/features of older versions (ie 4.4-R 4.5-R > > etc). > > > Surely I don't have go through this upgrade cycle every six > > months! It > > > would be great to just run a pkg_add which would overwrite any > > insecure > > > binaries with newer patched ones (and do an actual binary > > upgrade only > > > when absolutely required - e.g. every 2-3 years). I am even > > thinking of > > > starting such a project myself. > > > > > > Am I missing something? (i.e. is there a better way?) > > > (If someone tells me to cvsup and do a makeworld on my busy > > production > > > servers I will scream!) > > > > I understand that you do not wish to run make buildworld on a lot > > of production machines, but there is another way. > > I have a machine whichs only task in life is to run make > > buildworld. It does nothing but cvsup its sources and build > > binaries to share with other machines. Doing a make installworld > > takes only a few minutes, reboot included, which is acceptable or > > atleast unavoidable even on production machines. Im sure a lot of > > the binary patches for your win2k server requires you to reboot > > too, dont they? > > With 14 machines, I would dedicate one of them as a 'builder'. > > Let it buildworld, share /usr/src and /usr/obj via NFS, mount > > them on the other machines and I would guess you could upgrade > > all 14 machines with 40-50 minutes of work. A few simple scripts > > and you could do it in 10. > > Well, in looking at that procedure, the actual upgrade itself is a 2 hour process tops. I don't see where binary patches would save you any time, unless the procedure you use when doing a patch is radically different from the one you use to upgrade. It's already been suggested that you build a machine specifically for doing buildworlds on and then installing all the other machines off the first one. Total downtime for production server == one reboot to load the new kernel. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-questions" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20020807111625.B207>