Date: Wed, 10 Sep 2008 00:15:07 +0100 From: "Peter Maxwell" <peter@allicient.co.uk> To: freebsd-pf@freebsd.org Subject: pf not creating state on cloned local interface (with FreeBSD jail) Message-ID: <7731938b0809091615i6a9624fape21e0711cbbde447@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
Hi, Looking for anyone's help on this: I'm not sure if pf's behaviour is correct, if its a bug, if it is working correctly, or if I'm just trying to do something that really shouldn't be done. Anyway, my setup and issue is as follows: Kernel is GENERIC 7.0-STABLE #1, amd64 with IPSEC and ALTQ options added. There is one external interface and I've created a series of cloned interfaces; lo1, lo2, lo3,,,, lox; each assigned IP 10.0.x.1/24 where the x is the same as the cloned interface number. There are no other 10. addresses in the routing table. I felt it was preferable to create cloned interfaces rather than aliased IPs on lo0 for the jails (i.e. I didn't want any possibilty of the jailed services being able to access a host service on lo0). I don't have, and wouldn't be able to justify enough public addresses for this. The salient parts of my pf.conf (very much condensed) are set state-policy if-bound scrub all antispoof on re0 (external interface) set skip on lo0 There are processes (e.g. apache) in each of the jails binding to each of the 10.0.x.1 IP addresses. Now here's the crunch... when I add a "set skip on lo?" for each of the cloned interfaces all is good and well, each of the services can talk to each other and I can add appropriate NAT rules to allow access out to the internet. However, I want to prevent general network access across the cloned interfaces (and hence jails), i.e I want to give apache access to mysql, say, but not to the smtp server. So the extra "set skip lo?" lines are removed and only "set skip lo0" remains. Which then requires creation of appropriate rules. If I try and setup rules to allow access from the main host into a jailed service (keeping it simple to start with), it seems to require two rules (one to pass in, the other to pass out - i.e. its not keeping state). If I take the example of apache running on 10.0.3.1 (lo3) in a jail, the necessary rules to allow access from the host system (i.e. not from another jail) is: pass in log on lo3 proto tcp from any to any flags S/SA modulate state pass out log on lo3 proto tcp from any to any flags S/SA modulate state I know the rules are a bit general, but that's shouldn't be important just now. If I take a look at the logs for apache the source IP for the incoming packets are 10.0.3.1 (the jail IP, the same IP as apache is running on), which I'm guessing is the standard behaviour (have done a rdr test with the "set skip" lines in place from an external IP and it takes the appropriate source address in that scenario). If the "pass out" is removed, an "Operation not permitted" error is generated (which is sort of what I expected). If the "pass in" rule is omitted then it just stalls (which really is not what I expected). If tcpdump is ran on pflog0, you cannot see the expected initial SYN packets and all you can see is a load of SYN/ACKs getting blocked. When both rules are in place and I check the output of "pfctl -s r" there is something like (and it seems to work): lo3 tcp 10.0.3.1:63167 -> 10.0.3.1:80 ESTABLISHED:ESTABLISHED lo3 tcp 10.0.3.1:80 <- 10.0.3.1:63167 ESTABLISHED:ESTABLISHED which says to me there's two state entries been created where there should only be one, that being the first line (and that I should also expect some nasty problems down the line if I leave it like that). Should the first "pass out" rule not be enough to create a state entry from which the return packets get matched? If I add a "set skip" for every cloned interface except lo3, then similar behaviour is seen - two rules are needed. Any help or advice is appreciated. Cheers, Peter
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?7731938b0809091615i6a9624fape21e0711cbbde447>