Date: Tue, 16 Sep 2014 15:00:52 +0200 From: Niklaas Baudet von Gersdorff <niklaas@kulturflatrate.net> To: freebsd-questions@freebsd.org Subject: Re: ZFS, Jails, network, routing, domains and IP addresses Message-ID: <20140916130052.GA28361@len-x61s.klaas> In-Reply-To: <540F0A3D.4070209@freebsd.org> <540F1F1F.2040600@ssimicro.com> <20140909200327.GD36353@slackbox.erewhon.home>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi,
thank your very much for all the replies. Unfortunately, it took me some
time to reply since I wanted to finish setting up my virtual test system
and check whether I understood everything correctly.
Roland Smith [2014-09-09 22:03 +0200] :
>If you mount the ports tree with nullfs, you only get the "recipes" for
>installing software.
OK, thank you for clarifying this.
>One way to "automatically" update every jail is to mount /usr/local with a
>nullfs in every jail. And then use a unionfs in every jail for the
>configuration files in /usr/local/etc.
>
>This does have limitations;
>
> * Every jail then has access to *everything* in /usr/local. That might not
> be what you want.
> * Every jail needs its own /usr/local/etc, hence the need for unionfs.
> * You could run into a situation where /usr/local is updated but not a
> jail's configuration files in /usr/local/etc. That might mean that you
> e.g. cannot restart a service until a config file is updated as well.
>
>You could also use the host to built packages, and make a repository available
>to the jails. In the jails you can then use pkg(8) to keep the packages
>updated. This is a good combination of only building a piece of software once
>yet being able to keep different packages in different jails.
OK, went for the ezjails solution as described below.
markham_breitbach@ssimicro.com [2014-09-09 09:39 -0600] :
>Jails are a great solution for this. I have used jails to roll out
>identical systems across a large geographic area, so that each jail uses
>a local configuration file to set some local variables (IP address,
>hostname, etc.) that are inserted at jail startup time. In this way, I
>can maintain a single "image" that can be deployed and installed to
>quickly upgrade dozens of servers. Another trick I have used is to
>setup a base "instance" that is nullfs mounted for individual jails
>which things like /var and /home remain unique and RW. I am currently
>running about 100 apache AMP hosts this way, so I can quickly upgrade
>apache once in the single instance and all 100 hosts are instantly upgraded.
So, do you use ezjails for this or did you set up everything on your own
as described on the following link?
https://www.freebsd.org/doc/en/books/handbook/jails-application.html
I haven't had a closer look on the explanations there yet since I
already had quite some success with ezjail. I am only wondering whether
ezjail will restrict me at some point since it is "only" a wrapper
script.
>Technically what you are asking is not possible. The server doesn't
>know what lookup the client has done, although http can send a header to
>identify the domain requested. This will allow you to proxy requests to
>the appropriate jail, but I suspect the money you save in IP addresses
>will quickly be lost double in support time if you try to
>over-complicate things with layer7 proxies and clever routing tricks,
>double again when something goes wrong.
Yes, of course, I could use proxies for web and mail. I think I won't go
for this approach since I want the "end-user" (who gets a jail at our
server) to have free hand at his system -- and I don't want to always
set up proxies for different services.
Matthew Seaman [2014-09-09 15:10 +0100] :
>If you're going to be building a lot of jails with much the same
>software load-out, then you can make it all quite space efficient by
>building a template jail and then using ZFS cloning. Either as a DiY
>setup or try ezjail.
Thank you for your hint on ezjail. I am currently using it in the test
environment and it works great. Are there advantages in doing the DiY
setup?
>Note: don't turn on ZFS deduplication. It sounds attractive, but you
>will need a lot more RAM than you have in order for it to be effective,
>and it does entail trading off performance for storage efficiency.
OK, thank you!
>Read about the -c and -j flags in pkg(8). Also, I recommend managing
>your jails entirely through binary packages, rather than mounting ports
>trees everywhere. You can either use the standard FreeBSD pkg repos, or
>build your own with poudriere or indeed a combination of the two.
This is amazing. Great. Since I got in touch with FreeBSD I don't want
to stay away from it. It's much much better from what I've seen before.
So, concerning pkg -c/-j, I would only need -j wouldn't I? Or do I need
to indicate where the jail is with -c ?
>This might be possible, but it's not something that is usually done.
>Given you've said the applications you'll be supplying are postfix and
>apache, then you should be able to have a small instance of either of
>those acting as a reverse proxy in front of your jailed environments
>(which can just use some private address space). You can then decide
>how to route the traffic to the appropriate jail based on the SMTP or
>HTTP protocol headers involved. This is bog standard webserver stuff,
>and I think it's not uncommon for mail servers either.
OK. I think I'll got for IPv6 (thanks to your hint!)...
>Yes, this is certainly possible. The technique is called 'thin jails'
>-- however, each jail will need a distinct IP, and the idea of the
>jailed applications being able to bind to different ports on the same IP
>doesn't work. You can do fancy firewall redirects and stuff to make
>this sort of thing work, but honestly, I think you'ld be better off
>doing the proxying etc. at Layer 7 rather than Layer 3.
OK, I would rather go for more IP addresses then...
>> I would like to understand this and the technical limitations better to
>> get an idea about how many fixed public IP addresses I have to buy. So I
>> can eventually save some money. :-)
>
>Go with IPv6. You'll have more IP numbers than you can possibly consume
>thrust upon you...
This really made my day. I checked at the page of the provider which I
have chose for the new system and I am going to get a IPv6 /64 subnet,
so quite an amount of IP addresses...
So will there be a problem if I assign the IPv4 address I have to the
jails' host and the jails themselves only get IPv6 addresses?[^1] Are
there drawbacks if a machine only has a IPv6 address? Only machines that
are capable of IPv6 can communicate with it, can't they? This would be
important for the jails running a mail server since only mail servers
that support IPv6 can communicate with them, am I right?
1: Of course, the host itself also needs the IPv6 addresses attached
to it but every connection is forwarded to the appropriate host.
This was quite easy to do with ezjails in my test environment.
Thank you for your help!
Niklaas
--
Niklaas Baudet von Gersdorff
niklaas@kulturflatrate.net
http://www.twitter.com/NBvGersdorff
http://www.kulturflatrate.net/niklaas
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20140916130052.GA28361>