Date: Sat, 18 May 2013 19:38:25 +1000 (EST) From: Ian Smith <smithi@nimnet.asn.au> To: freebsd-ipfw@freebsd.org Cc: Anders Hagman <anders.hagman@netplex.se> Subject: Re: vnet jail with ipfw having logging problem (fwd) Message-ID: <20130518192057.S86776@sola.nimnet.asn.au>
next in thread | raw e-mail | index | archive | help
I'm forwarding this from freebsd-jail as it's clearly an ipfw issue. Sorry, been totally tied up with $otherstuff since then. The below conversation is a bit drawn out and in my case, kinda tetchy, but the bottom line is that Anders here confirms a bug reported by Joe Barbish, that ipfw in a vimage jail is logging its messages to the host's /var/log/security and /var/log/messages and not the jail's, although other things that log (including logger(1)) do so correctly. As mentioned, I hunted the logging down to /sys/netpfil/ipfw/ip_fw_log.c on 9.1, but in the time available haven't found where log() was defined. Am I right assuming something's missed being VNET-ed here somewhere? cheers, Ian ---------- Forwarded message ---------- Date: Thu, 2 May 2013 22:05:49 +0200 From: Anders Hagman <anders.hagman@netplex.se> To: Ian Smith <smithi@nimnet.asn.au> Cc: freebsd-jail <freebsd-jail@freebsd.org> Subject: Re: vnet jail with ipfw having logging problem 2 maj 2013 kl. 18:46 skrev Ian Smith <smithi@nimnet.asn.au>: > On Thu, 2 May 2013 12:09:08 +0200, Anders Hagman wrote: >> Hi > Yo >> 2 maj 2013 kl. 07:42 skrev Ian Smith <smithi@nimnet.asn.au>: >> >>> On Wed, 1 May 2013 17:43:03 -0400, Joe wrote: >>>>>> I have ipfw running inside of a vnet jail on a 9.1-RELEASE host using >>>>> the >>>>>> jail(8) definition statements for starting and stopping the vnet jail. >>>>> As a >>>>>> side note non-vnet jails are working as expected. >>>>>>> The host is running a custom kernel with modules and with >>>>>> options VIMAGE >>>>>> nooptions SCTP >>>>>> options IPFIREWALL >>>>>> options IPFIREWALL_VERBOSE >>>>>> options IPFIREWALL_VERBOSE_LIMIT=10 >>> >>> Please maintain attributions for the archives. I wrote: >>> >>>>> What steps have you taken during testing to override this ridiculously low >>>>> limit on logging? Otherwise, after e.g. just 5 pings and 5 ping responses >>>>> are logged, all logging ceases until issuing 'ipfw resetlog'. >>>> >>>> /usr/src/sys/conf/NOTES says IPFIREWALL_VERBOSE_LIMIT; limits the number of >>>> times a matching entry can be logged. Says nothing about this limit being the >>>> maximum number of log records allowed after which the log file is closed for >>>> business. Are you saying the /usr/src/sys/conf/NOTES info is no longer true? >>> >>> You showed one (1) 'log' rule for each of the host's and jail's ruleset. >>> Once that one rule has been logged 'logamount' times (default as per >>> NOTES is 100, but in your case is 10) then logging for THAT rule stops, >>> therefore with only one 'log' rule, ALL logging stops. Understand? >>> >>> If you take the time to properly study the correct reference, ipfw(8), >>> all of this will become clear. See especially section SYSCTL VARIABLES, >>> and read thoroughly 'log [logamount number]', at the very least. Ignore >>> the Handbook section on ipfw, it's full of errors and misunderstandings. >>> >>>> Without IPFIREWALL_VERBOSE and IPFIREWALL_VERBOSE_LIMIT where does the logged >>>> packets get written to? /var/log/security >>> >>> See above. Both of these options merely set defaults for the sysctls. >>> >>>> I have not used ipfw since it's ipfw2 rewrite so my knowledge is dated. >>> >>> Indeed it is; that's a very long time ago. >>> >>>>>> options IPFIREWALL_DEFAULT_TO_ACCEPT >>>>>> options IPFIREWALL_IPDIVERT >>>>> >>>>> You'd likely do better using in-kernel NAT; natd doesn't get much love. >>>>> >>>> >>>> I kept getting kernel compile errors using "options IPFIREWALL_NAT". I >>>> thought the error was caused by vimage. Now I know "options LIBALIAS" is >>>> required. Could not find info on internet search for IPFIREWALL_NAT with >>>> vimage kernel. >>> >>> Apart from FIREWALL_FORWARD (not even that in 10.x), none of that needs >>> to be in the kernel, it's all loadable as modules; see /etc/rc.d/ipfw. >>> >>> If you're doing NAT in the vimage jail, you must have at least two >>> interfaces assigned to the jail. Care to show your config for that? >>> >>>> Do you have first hand experience getting "ipfw kernel nat" to work in a >>>> vimage jail or having logging work on the host and within the vnet jail? >>> >>> No, but I have just on 15 years experience managing ipfw firewalls :) >> >> When you are new at things you do mistakes, remember. > > I still do mistakes. Trying to teach fishing rather than just tossing > another fish is often one of mine :) I'm glad you had some to spare. I know the game. ;-> > >> To try to answer Joes question: >> >> You don't need to compile anything into the kernel regarding ipfw. >> >> Just load the ipfw module in the host system with: >> >> kldload ipfw >> >> By default a deny all rule is added, so add a allow rule to the host system. >> >> ipfw add 10 allow ip from any to any >> >> To log things you change the sysctl value net.inet.ip.fw.verbose to 1 >> >> sysctl net.inet.ip.fw.verbose=1 >> >> If you keep net.inet.ip.fw.verbose_limit=0 you don't have a log limit, and for tests thats fine. > > Sure, though the default of 100 is plenty for such tests; it's > surprisingly easy to DoS syslogd with e.g. a logged flood ping .. > >> log in to the jail system. Change the sysctl value net.inet.ip.fw.verbose to 1 >> >> sysctl net.inet.ip.fw.verbose=1 >> >> Add a logging firewall rule >> >> ipfw add 10 allow log ip from any to any >> >> Do a ping to an external system. >> Look inside /var/log/security in the jail system and its empty. > > But it does exist, rw for root, with 0 or more bytes, right? And does > the vimage jail's /etc/syslog.conf contain: > security.* /var/log/security > Yes > That is, I'm checking that the jail's syslogd should be handling these. > What happens if you run in the jail, say: > # logger -p security.info Syslog, wherefore art thou, Syslog? > Does that go to the jail's /var/log/security? or the host's? In jail system webben: logger -p security.info Syslog, wherefore art thou, Syslog? tail /var/log/security May 2 21:24:48 webben root: Syslog, wherefore art thou, Syslog? > >> Go to the main host and look at the /var/log/security file and you will find log entries. > > Showing the host's hostname, or the jail's? Can you post some examples? In host system dator5: tail /var/log/security May 2 21:29:15 dator5 kernel: ipfw: 10 Accept TCP 10.2.0.101:80 94.153.64.32:3085 out via vlan101 May 2 21:29:15 dator5 kernel: ipfw: 10 Accept TCP 94.153.64.32:3085 10.2.0.101:80 in via vlan101 > >> I can confirm Joes bug. I don't have a log rule in the main host but still get log messages. >> All log messages are from the log rule in the jail system. >> >> System used: 9.1-RELEASE-p2 >> >> BR >> /Anders > > Ok, before determining that this is an ipfw-only issue - in which case > we need to move it over to freebsd-ipfw@ - can you confirm that normal > syslogging in the jail to /var/log/messages and such is working? > In jail system login anders password ***** tail /var/log/messages May 2 21:41:57 webben login: login_getclass: unknown class 'svensk' May 2 21:42:00 webben last message repeated 3 times > In particular I'm wondering what happens when you do set (say) > net.inet.ip.fw.verbose_limit=10 and then ping from the jail until > logging stops .. you should then see a message such as > > Apr 23 23:42:05 sola kernel: ipfw: limit 500 reached on entry 26400 > > both in /var/log/security and in /var/log/messages since it's logged > as security.notice and default syslog.conf is for *.notice to log to > /var/log/messages .. see the tail of /sys/netpfil/ipfw/ip_fw_log.c > > Yes sure, I'm flying blind, don't have a system with jails here yet, and > am making assumptions about how syslogd(8) should work in jails that I > really don't have time to properly research currently, nor am I properly > across all the security implications of (particularly vimage) jails. > On jail system: sysctl net.inet.ip.fw.verbose_limit=10 Pinging repeatedly. Just continue to log to host system. Add new ipfw log role will use the new limit: ipfw add 5 allow log ip from any to any 00005 allow log logamount 10 ip from any to any New ping test. /var/log/security in host system : May 2 21:52:28 dator5 kernel: ipfw: 5 Accept ICMP:8.0 10.2.0.101 195.49.241.132 out via vlan101 May 2 21:52:28 dator5 kernel: ipfw: 5 Accept ICMP:0.0 195.49.241.132 10.2.0.101 in via vlan101 May 2 21:52:28 dator5 kernel: ipfw: limit 10 reached on entry 5 /var/log/messages in host system : May 2 21:52:28 dator5 kernel: ipfw: limit 10 reached on entry 5 Nothing at all is logged to the jail syslog. BR /Anders
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20130518192057.S86776>