Date: Thu, 25 Feb 2016 16:14:13 +0100 From: "Martin \"eto\" Misuth" <eto.freebsd@ethome.sk > To: freebsd-jail@freebsd.org Subject: Re: Jail management Message-ID: <20160225161413.25f17811@eto-mona.office.smartweb.sk> In-Reply-To: <0eaf61d4-43e6-265a-f773-820244fc8931@ish.com.au> References: <ff8307f6-1264-30ec-1ef8-ed3b0a18dd84@ish.com.au> <DFFE2BFC-1D53-457D-A4C3-633418D3690D@erdgeist.org> <0f5cae7e-7de3-2617-fcf6-3423d4caf13a@ish.com.au> <56CAE974.4050508@quip.cz> <0eaf61d4-43e6-265a-f773-820244fc8931@ish.com.au>
next in thread | previous in thread | raw e-mail | index | archive | help
On Mon, 22 Feb 2016 22:26:02 +1100
Aristedes Maniatis <ari@ish.com.au> wrote:
>
> You are right, and perhaps I should just bite the bullet. I am afraid of only
> two things.
>
Hi I am jails newcomer as well, and this is kinda report of how far I got.
But depending on how much you want bet on this technology, I strongly suggest
killing some serious time with jails. It's really worth it, if you intend to
use them more extensively on FreeBSD, eg. besides "standard" add few lines to
rc.conf using ezjail or whatever.
For all intent and purposes jail is actually just glorified chroot (although
VNET ones are running separate TCP/IP stacks and I have not have had time to
look at them). And thus in actual usage jails are pretty "simple" as most things
on FreeBSD are (which is benefit). I considered them magical too in the past.
In the end it seems same base rules as for managing chroots apply. Although
chroot is with us for long time, I don't see much easily accessible
information on how to efficiently build and manage chroots themselves and large
numbers of chroots concurently. As such the rule is, if there are mentions of
chroot parameter in manpages of tool it can be used as both jail and chroot
(operating directly on jail tree) management tool.
Seems like at certain point most admins doing this kind of work absorb enough
knowledge to build chroots on their own, either by some custom scripts stack or
with help of package manager and other tools.
Several observations:
- you can assemble your jail tree at any location, although most setups pick
certain path and use that as jail "hive"
- if you have your given jail tree mounted in place, you can always chroot to
it, without spawning it as jail (this might have security implications!)
to test programs are dynamically linking together or whatever
- if jail is offline, you can pkg "do" (install whatever) into it with chroot
parameter (offline here is just to avoid locking and general clashing
with "running" jail)
- you can pkg "do" (install, whatever) into running jail as well, with jail
parameter
- you can use literally any mounted fs under jail's root, and unless you allow
given jail to do mounts and stat for fs information, it sees it as one
single / it cannot modify
- not sure about Miroslav's problems with freebsd-update, but it seems to work
pretty well with -basedir /jail/tree parameter nowadays (there might be
corner cases)
- you can have older jail-base run on newest kernel (other way around is not
possible)
- you can kill many files in given jail to get bare minimal running setup
(this seems completely driven by gut, from what I gathered, as some things
might have un-obvious dependencies)
- you can mount many things into jail read-only (this makes them more rigid
and harder to "manage" "live")
- jails can have limits on number of procs living in them and can be
allowed to be nested(!) (jail-in-jail)
- with rctl you can cap resources per jail
- you don't need to run all daemons usually run on host jail
- depending on your setup you can get rid of native syslog and use something
lighter
- jails have their own user databases, but if you do this, uids don't map
to usernames when viewed from host (ps, htop etc)
- imho simplest possible setup with "live" uid database (eg not copying
files around) was for me to use nss ldap extension, that way all jails
and host see same synced uids (if you store them in single ldap
space, this might have security implications as users in other jails
can see others in the system)
- as ports tree uses same uids on every machine it made sense to me to import
those into main ldap setup, then have per jail subtree with jail local
users this requires from you to adjust nss ldap search rules
As the idea of jail is having chroot like tree with more protections, once you
add ZFS into the mix, number possibilities starts exploding :
- by various ways of nesting your jailtree ZFS datasets
- you can get to instant "full jails snapshots". Once snapshotted, given
your connection limitations and zfs skillset, jails can be sent over to
other zfs pools (within same machine or between multiple machines) by
virtues of 'zfs send'
- similarly you can implement automatic incremental snapshots
- you can instruct zfs to attach given dataset to jail (from there jail can
control it's own snapshots or other zfs operations from inside)
- taking advantage of FreeBSD's / and /usr/local split you can reuse (by zfs
clone) same "root" dataset in multiple jails and treat is as easily
discardable
- this allows you to keep /usr/local subtree and swap roots around
after freebsd-update-ing template (by cloning)
With last most interesting option you can insert nullfs into the mix
this allows you to share some part of host tree within multiple jails.
Keep in mind nullfs-es don't mount recursively and this becomes managemnt
issue with deeper mounts.
So now because it's so easy, when you made 60+ jails starts the fun
- all these jails need love to keep them up-to date
- packages in these jails need to be kept up-to date
- fs partitioning or zfs nesting you picked might turn out to not be
exactly optimal
- if you use fstab option of jail.conf, unmounting and re-mounting
might sometimes fail
- some daemons (I suspect native syslog being one of them for me) might
leave jail in dying state because of what I guess, are TCP/IP and unix
socket timeouts, this might, make "remount" with custom fstab
impossible in quick succession (after teardown)
- as mentioned nullfs mounts don't nest so you might need some kind of
script glue if you want to use that
So with all tools mentioned in this thread, you get many of those decisions made
for you and you can mostly use your hive only in the way tool's author
envisioned.
The more you use it directly more convoluted and flexible setups are
possible provided you can manage them.
Generally, the more static (all jail mounts at system boot + persistent jails
with fixed jail ids) setup is, more reliably it operates, but more rigid it
becomes to change.
I personally don't run heavy load hosts. My main use currently is to test
various combinations of http servers with different versions of php-s
with various versions of our php stuff.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20160225161413.25f17811>