Date: Wed, 07 Aug 2002 13:59:18 +0200 From: "Roger 'Rocky' Vetterberg" <listsub@rambo.simx.org> To: BSD Freak <bsd-freak@mbox.com.au> Cc: "Roger 'Rocky' Vetterberg" <listsub@401.cx>, FreeBSD Questions <freebsd-questions@FreeBSD.ORG> Subject: Re: There must be a better way to maintain older systems Message-ID: <3D510B96.8010105@rambo.simx.org> References: <e5fe64e5c22e.e5c22ee5fe64@mbox.com.au>
next in thread | previous in thread | raw e-mail | index | archive | help
I do realise what you are going through. I work at a company that are 100% dependent of its internet connection. Every system we use need internet, most of our customers are someway or another internet (dot.com) businesses, and even parts of our phonesystem must have internet access to function properly. If we lose connectivity, we loose money as every second goes by. Just as you, we use duplicate servers. One is in constant use, the other is standing by. We usually upgrade the stand-by server using cvsup and make buildworld, test it to see that everything is working, and if it doesnt, we restore from tapes and try again. When we see that it works as expected, we syncronize the data from the production server over to the stand-by machine, pull the cat-5 cables from the production server and plug them in the stand-by. This means that the former production server is now the backup, and when its time for the next upgrade it will be the one testing it out. This is for the really mission critical machines, the ones that are not vital is updated from a central "builder", that builds world and exports /usr/src and /usr/obj via nfs to the others. This way a upgrade takes only a few minutes, reboot included. I do understand the amount of work that goes into every upgrade. We only upgrade when we find it absolutely necessary, due to the extreme amount of work and manhours that it requires each time. I fail to see how this could be different, a upgrade of a mission-critical machine will always require a lot of testing, backing up and even more testing. If you have a lot of critical machines, it means a lot of work. If you find an easier and still secure way to accomplish this, please share it, sysadmins all over the world will thank you. -- R BSD Freak wrote: > I think what many don't realise is how much testing, backup, > documentation etc. goes into our upgrade process before we roll out a > new version of FreeBSD. We only perform binary upgrades (I see building > from source, cvsup etc. far too risky). Many of you will no doubt > disagree very strongly but we are completely reliant on our FreeBSD > boxes being in tip top shape - mistakes cannot be tolerated. These are > not machines we run as a hobby, if they are down - our enterprise is > down. We also run some complex configurations involving multiple jails > etc. Hence our upgrade procedure is as follows for each server (with > slight variations): > > 1. We start off by testing the application on a test server with > identical hardware and the new version of FreeBSD we wish to migrate to. > > 2. We review any new features of the new version we should be > implementing including new/improved kernel options, sysctl paramters, > optimisations etc. and test these. > > 3. We do a "clean" install of the new version of FreeBSD and our > application on the test server. Thoroughly documenting the whole build > process). > > > 4. We copy all the application data from the production to the test > server. > > 5. (This is the tricky bit) We either freeze data (where possible) or > take the production server offline and plug into a seperate network > with the test server and synchronise the data (usually with rsync). > This step is very time critical and we have to do this as fast as > possible because the production server is offline. > > 6. We switch the ip addresses of the production and test server and the > test server now becomes our new production server. > > 7. We keep the old production server around (but off) for about a week > to make sure all is well with the new setup. > > 8. We wipe the old production server and it becomes the new test server > for the next box we need to upgrade. > > * Voila, upgrade complete with minimal possible downtime, minimal risk > and a "clean install" of the new version of the OS. * > > The process now starts again for the next (of our 14) FreeBSD servers. > Total time for this process is 2-3 weeks sometimes a month depending on > my sysadmin and other workload. Because of the length of this process I > have started skipping releases and upgrading servers every second > release. > > Any suggestions on how I can improve this process without introducing > higher risks would be *GREATLY* appreciated...... > > ----- Original Message ----- > From: Roger 'Rocky' Vetterberg <listsub@401.cx> > Date: Wednesday, August 7, 2002 1:23 pm > Subject: Re: There must be a better way to maintain older systems > > >>BSD Freak wrote: >> >>>Hi all, >>> >>>I am responsible for maintaining 14 FreeBSD, 1 Windows 2000 and >> >>1 >> >>>Solaris servers at three sites. While I am certianly no fan of >> >>Windows >> >>>2000 or the commercial UNIX distributions I have to say they >> >>take up a >> >>>lot less of my time to maintain. For example I can download >> >>(binary >> >>>packages) patches and "Service Packs"/hotfixes to patch bugs and >>>vulnerabilities and then I forget about it. Upgrades of OS >> >>happen once >> >>>every 3-4 years (and usually accomany a hardware upgrade which >> >>makes it >> >>>a bit neater and less risky). >>> >>>With FreeBSD however I find myself upgrading every six months or >> >>so >> >>>when a new version is released. I spend half my time upgrading >> >>the 14 >> >>>production servers (in the middle of the night usually!), then >> >>by the >> >>>time I have gotten around to the last system, I'm usually only a >> >>month >> >>>or so away from the next -RELEASE and I I have to do it all >> >>again if I >> >>>am to keep my systems secure and current. >>> >>>I find myself thinking there *MUST* be a better way. I am quite >> >>happy >> >>>with the stability/features of older versions (ie 4.4-R 4.5-R >> >>etc). >> >>>Surely I don't have go through this upgrade cycle every six >> >>months! It >> >>>would be great to just run a pkg_add which would overwrite any >> >>insecure >> >>>binaries with newer patched ones (and do an actual binary >> >>upgrade only >> >>>when absolutely required - e.g. every 2-3 years). I am even >> >>thinking of >> >>>starting such a project myself. >>> >>>Am I missing something? (i.e. is there a better way?) >>>(If someone tells me to cvsup and do a makeworld on my busy >> >>production >> >>>servers I will scream!) >> >>I understand that you do not wish to run make buildworld on a lot >>of production machines, but there is another way. >>I have a machine whichs only task in life is to run make >>buildworld. It does nothing but cvsup its sources and build >>binaries to share with other machines. Doing a make installworld >>takes only a few minutes, reboot included, which is acceptable or >>atleast unavoidable even on production machines. Im sure a lot of >>the binary patches for your win2k server requires you to reboot >>too, dont they? >>With 14 machines, I would dedicate one of them as a 'builder'. >>Let it buildworld, share /usr/src and /usr/obj via NFS, mount >>them on the other machines and I would guess you could upgrade >>all 14 machines with 40-50 minutes of work. A few simple scripts >>and you could do it in 10. >> >>-- >>R >> >> >> >> >> >> > > > --------------------------------------------------------------------- > Never lose a fax again, receive faxes to your personal email account! > Visit http://www.mbox.com.au/fax To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-questions" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?3D510B96.8010105>