From owner-freebsd-jail@FreeBSD.ORG Thu May 2 16:46:20 2013 Return-Path: Delivered-To: freebsd-jail@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 521936DD for ; Thu, 2 May 2013 16:46:20 +0000 (UTC) (envelope-from smithi@nimnet.asn.au) Received: from sola.nimnet.asn.au (paqi.nimnet.asn.au [115.70.110.159]) by mx1.freebsd.org (Postfix) with ESMTP id 691DA18BF for ; Thu, 2 May 2013 16:46:18 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by sola.nimnet.asn.au (8.14.2/8.14.2) with ESMTP id r42GkBpv081883; Fri, 3 May 2013 02:46:11 +1000 (EST) (envelope-from smithi@nimnet.asn.au) Date: Fri, 3 May 2013 02:46:10 +1000 (EST) From: Ian Smith To: Anders Hagman Subject: Re: vnet jail with ipfw having logging problem In-Reply-To: Message-ID: <20130503010007.C30818@sola.nimnet.asn.au> References: <44AC45947DA14449AEDFB13B9F6C5F7DAF3E1FA5@ltcfiswmsgmb25> <517A7BCB.8060604@a1poweruser.com> <13CA24D6AB415D428143D44749F57D7201F22068@ltcfiswmsgmb21> <517D3426.1090703@a1poweruser.com> <51805EFB.6050806@a1poweruser.com> <20130502021830.O30818@sola.nimnet.asn.au> <51818C67.7070708@a1poweruser.com> <20130502142443.V30818@sola.nimnet.asn.au> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Cc: freebsd-jail X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 May 2013 16:46:20 -0000 On Thu, 2 May 2013 12:09:08 +0200, Anders Hagman wrote: > Hi Yo > 2 maj 2013 kl. 07:42 skrev Ian Smith : > > > On Wed, 1 May 2013 17:43:03 -0400, Joe wrote: > >>>> I have ipfw running inside of a vnet jail on a 9.1-RELEASE host using > >>> the > >>>> jail(8) definition statements for starting and stopping the vnet jail. > >>> As a > >>>> side note non-vnet jails are working as expected. > >>>>> The host is running a custom kernel with modules and with > >>>> options VIMAGE > >>>> nooptions SCTP > >>>> options IPFIREWALL > >>>> options IPFIREWALL_VERBOSE > >>>> options IPFIREWALL_VERBOSE_LIMIT=10 > > > > Please maintain attributions for the archives. I wrote: > > > >>> What steps have you taken during testing to override this ridiculously low > >>> limit on logging? Otherwise, after e.g. just 5 pings and 5 ping responses > >>> are logged, all logging ceases until issuing 'ipfw resetlog'. > >> > >> /usr/src/sys/conf/NOTES says IPFIREWALL_VERBOSE_LIMIT; limits the number of > >> times a matching entry can be logged. Says nothing about this limit being the > >> maximum number of log records allowed after which the log file is closed for > >> business. Are you saying the /usr/src/sys/conf/NOTES info is no longer true? > > > > You showed one (1) 'log' rule for each of the host's and jail's ruleset. > > Once that one rule has been logged 'logamount' times (default as per > > NOTES is 100, but in your case is 10) then logging for THAT rule stops, > > therefore with only one 'log' rule, ALL logging stops. Understand? > > > > If you take the time to properly study the correct reference, ipfw(8), > > all of this will become clear. See especially section SYSCTL VARIABLES, > > and read thoroughly 'log [logamount number]', at the very least. Ignore > > the Handbook section on ipfw, it's full of errors and misunderstandings. > > > >> Without IPFIREWALL_VERBOSE and IPFIREWALL_VERBOSE_LIMIT where does the logged > >> packets get written to? /var/log/security > > > > See above. Both of these options merely set defaults for the sysctls. > > > >> I have not used ipfw since it's ipfw2 rewrite so my knowledge is dated. > > > > Indeed it is; that's a very long time ago. > > > >>>> options IPFIREWALL_DEFAULT_TO_ACCEPT > >>>> options IPFIREWALL_IPDIVERT > >>> > >>> You'd likely do better using in-kernel NAT; natd doesn't get much love. > >>> > >> > >> I kept getting kernel compile errors using "options IPFIREWALL_NAT". I > >> thought the error was caused by vimage. Now I know "options LIBALIAS" is > >> required. Could not find info on internet search for IPFIREWALL_NAT with > >> vimage kernel. > > > > Apart from FIREWALL_FORWARD (not even that in 10.x), none of that needs > > to be in the kernel, it's all loadable as modules; see /etc/rc.d/ipfw. > > > > If you're doing NAT in the vimage jail, you must have at least two > > interfaces assigned to the jail. Care to show your config for that? > > > >> Do you have first hand experience getting "ipfw kernel nat" to work in a > >> vimage jail or having logging work on the host and within the vnet jail? > > > > No, but I have just on 15 years experience managing ipfw firewalls :) > > When you are new at things you do mistakes, remember. I still do mistakes. Trying to teach fishing rather than just tossing another fish is often one of mine :) I'm glad you had some to spare. > To try to answer Joes question: > > You don't need to compile anything into the kernel regarding ipfw. > > Just load the ipfw module in the host system with: > > kldload ipfw > > By default a deny all rule is added, so add a allow rule to the host system. > > ipfw add 10 allow ip from any to any > > To log things you change the sysctl value net.inet.ip.fw.verbose to 1 > > sysctl net.inet.ip.fw.verbose=1 > > If you keep net.inet.ip.fw.verbose_limit=0 you don't have a log limit, and for tests thats fine. Sure, though the default of 100 is plenty for such tests; it's surprisingly easy to DoS syslogd with e.g. a logged flood ping .. > log in to the jail system. Change the sysctl value net.inet.ip.fw.verbose to 1 > > sysctl net.inet.ip.fw.verbose=1 > > Add a logging firewall rule > > ipfw add 10 allow log ip from any to any > > Do a ping to an external system. > Look inside /var/log/security in the jail system and its empty. But it does exist, rw for root, with 0 or more bytes, right? And does the vimage jail's /etc/syslog.conf contain: security.* /var/log/security That is, I'm checking that the jail's syslogd should be handling these. What happens if you run in the jail, say: # logger -p security.info Syslog, wherefore art thou, Syslog? Does that go to the jail's /var/log/security? or the host's? > Go to the main host and look at the /var/log/security file and you will find log entries. Showing the host's hostname, or the jail's? Can you post some examples? > I can confirm Joes bug. I don't have a log rule in the main host but still get log messages. > All log messages are from the log rule in the jail system. > > System used: 9.1-RELEASE-p2 > > BR > /Anders Ok, before determining that this is an ipfw-only issue - in which case we need to move it over to freebsd-ipfw@ - can you confirm that normal syslogging in the jail to /var/log/messages and such is working? In particular I'm wondering what happens when you do set (say) net.inet.ip.fw.verbose_limit=10 and then ping from the jail until logging stops .. you should then see a message such as Apr 23 23:42:05 sola kernel: ipfw: limit 500 reached on entry 26400 both in /var/log/security and in /var/log/messages since it's logged as security.notice and default syslog.conf is for *.notice to log to /var/log/messages .. see the tail of /sys/netpfil/ipfw/ip_fw_log.c Yes sure, I'm flying blind, don't have a system with jails here yet, and am making assumptions about how syslogd(8) should work in jails that I really don't have time to properly research currently, nor am I properly across all the security implications of (particularly vimage) jails. cheers, Ian