Date: Wed, 23 Apr 2014 12:06:33 -0600 From: markham breitbach <markham_breitbach@ssimicro.com> To: freebsd-questions@freebsd.org Subject: Re: FBSD jail versus VMWare? What services do YOU run in a jail? Message-ID: <53580129.5010909@ssimicro.com> In-Reply-To: <CAFS4T6apJ30_WPrV3-azuwr5LHFE8htEk5a_xqe7DRZ7Wy5XqQ@mail.gmail.com> References: <CAFS4T6apJ30_WPrV3-azuwr5LHFE8htEk5a_xqe7DRZ7Wy5XqQ@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
I work for an ISP delivering services to most of the remote communities across Canada's arctic via satellite. To mask some of the effects of high latencies across satellite links we pushed our services out to the edge, so things like email, dns, and authentication all happen nearly instantly locally instead of having to wait for several seconds to pass data back and forth across a satellite link. We have been using jails for nearly a decade now quite happily. Jails simplified maintenance tasks, improved our uptime and reduced costs by creating a logical separation of services and reducing the cost of redundancy. By planning our jails carefully, we were able to isolate related services within separate jails, so when there was a requirement to update one service we could easily create a patch on our development jail, and push it out everywhere, and because services were isolated regression testing became much simpler. I knew, for example, that updates to sendmail and its dependencies could not impact my LDAP installation in any way because they were essentially two completely separate servers. This logical separation also allowed us to maintain a simple rsync job between two jail host servers at each remote site for each jail. In the event of a hardware failure we could recover in a matter of minutes by simply starting the jails on the backup server and flushing the arp-cache on the switch. Because the jail is just another part of the file system rsync was a very efficient way of maintaining synchronization versus the need to copy a whole binary blob that represents a typical VM disk image. Our original design involved severely stripped down jails with nothing more than the bare minimum of binaries, libraries and config to provide a service. I think our original mail server jail packed down to about 25MB including sendmail, dovecot, and bind. This works because the host system shares its kernel. We have since expanded the jails to include a minimal installation, as we found troubleshooting to be somewhat awkward in an environment without basic command line tools like grep and tail, but this still keeps our jail systems to around 85MB for a complete system image. More recently, I have setup a smallish web hosting environmet using a small cluster of FreeBSD servers and a jail environment. Currently hosting about 60 small business and personal website, although we have had as many as 100 running with an average throughput of about 15Mbps and a peak of about 60Mbps. Five of the servers are running diskless with a netboot from the Master and mounting a common "jail" partition via NFS. This allows for any particular jail to be launched on any given client, although the default is for the server to arbitrarily choose. The hosting environment itself, is nullfs mounted into the jail as a read-only partition, so each web-host can only write to their own home directory and some local configuration files, allowing updates and patches across the entire cluster in a single operation. This has allowed me to repurpose some older hardware with some minor upgrades instead of investing thousands of dollars into a brand-new machine and gives me the advantage of redundancy (if one server dies for some reason, I can restart those jails on another server within seconds.) It also allows some isolation without having to enforce strict limitations on everyone. In the event that one host becomes heavily loaded it will only affect the hosts on that machine and not all of my web hosts. If the problem cannot be quickly resolved, the other hosts can easily be moved to another server with less than 30s downtime. While jails constrain you to operating within the FreeBSD environment, and do not have all the advantages of a full virtualization solution or clustering/cloud system, you also gain the advantage of minimized overhead. Systems like VMWare can incur significant performance overhead. I have run a large database installation (> 40M records) at more than 4 times the speed on bare metal versus ESXi. The jail does not incur any performance hits because there are not extra abstraction layers. Your application is interfacing directly with the kernel the same way it would if it were running outside the jail. It is certainly possible to run any service within a jail, although there are often some security implications (do you use sysVIPC, does the application need access to /dev/kmem?) you really will need to look at your own situation specifically to see if the advantages of jails are suitable to your environment and applications. I like to keep an open mind and try to apply the best tool for the job. -Markham --- [1]Markham Breitbach Network Operations SSi People, Ideas, Technology - - - - - - - - - - - - - - - - - - - - - +1 867 669 7500 work +1 867 669 7510 fax [2]markham_breitbach@ssimicro.com [3]www.ssimicro.com 356B Old Airport Road Yellowknife , NT X1A 3T4 Canada - - - - - - - - - - - - - - - - - - - - - Visit some of our other networks [4]www.qiniq.com & [5]www.airware.ca On 14-04-22 3:47 PM, edflecko . wrote: I'm really interested in the comparison of using a FBSD jail rather than VMWare in the context of virtualization. At my business, we heavily use VMWare - you might say we consider ourselves a VMWare "shop". 99% of our servers are virtualized. I've heard that it's possible to run hundreds, if not thousands, of services in FBSD jails on a given host server because of the sharing of resources that all of your jails take advantage of. If I understand that correctly, that's one of the HUGE advantages of running services in jails as opposed to creating VM after VM after VM - each VM eats up disk space on the SAN as well as memory resources, etc. Additionally, the jailed service is far better from a security perspective? Having said all of that, I'm curious to hear from some of you who may be doing just this - are you running a FBSD server with some of your mission critical services (Apache, Bind, DHCP, etc., etc.) within jails and how do you like it versus running hundreds of VMs and VMWare? What type of services CAN be run from within a jail? Thank you, Ed _______________________________________________ [6]freebsd-questions@freebsd.org mailing list [7]http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [8]"freebsd-questions-unsubscribe@freebsd.org" References 1. http://www.ssimicro.com/ 2. mailto:markham_breitbach@ssimicro.com 3. http://www.ssimicro.com/ 4. http://www.qiniq.com/ 5. http://www.airware.ca/ 6. mailto:freebsd-questions@freebsd.org 7. http://lists.freebsd.org/mailman/listinfo/freebsd-questions 8. mailto:freebsd-questions-unsubscribe@freebsd.org
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?53580129.5010909>