Date: Wed, 14 Sep 2011 13:50:43 -0700 From: Julian Elischer <julian@freebsd.org> To: Vladimir Budnev <vladimir.budnev@gmail.com> Cc: freebsd-net@freebsd.org, freebsd-ipfw@freebsd.org Subject: Re: IPFW hidden/broken rule? (Free 7.2) Message-ID: <4E7113A3.80605@freebsd.org> In-Reply-To: <4E7066CE.3070702@gmail.com> References: <4E7066CE.3070702@gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 9/14/11 1:33 AM, Vladimir Budnev wrote: > Hello list > > I am not sure which list this question must go to, so I am sending > to -net and -ipfw lists. > > We have faced some strange problem with ipfw behavior, which we > can't understand ourselves. An it really hurts:( > > We are running 7.2-RELEASE. > > I'll try to describe the problem as we observe it and our steps to > figure out what is happening. An questions will be at the end. > > In short there is a situation which looks like some old rule keep on > playing but we cant see this in ipfw outputs. I'd say 'hidden' rule > > So we have ipfw using tables for pipes, and we have e.g. following > ipfw output.The pipes configuration does mean nothing, it will be > clear at the end of mail, where key moment will be described. > > [Out1] > <...> > 04701 pipe tablearg ip from table(2) to any in via em0 > 04801 pipe tablearg ip from any to table(3) out via em0 > 04901 allow ip from table(4) to any in via em0 > 05001 allow ip from any to table(4) out via em0 > 05101 fwd <example.server> tcp from table(5) to any dst-port > 80,443,8828 in via em0 > <...> > > > We'v noticed that no packets from specific ip(10.121.241.23) reache > 5101 rule with fwd to example.server. > > This ip is in table 5: > # ipfw table 5 list > 10.10.122.23/32 0 > 10.10.122.167/32 0 > > We’ve parsed all tables and no matches. > OK, we started placing debug rules to realize which rule accepts > packets. > > And we found out following.If we place the following debug rule(we > used the same fwd rule) such way(added 4602 rule): > [Out2] > <...> > 04602 113 75 fwd <example.server> tcp from table(5) to any dst-port > 80,443,8828 in via em0 <-- OK her > 04701 107971113 85095893815 pipe tablearg ip from table(2) to any in > via em0 > 04801 102517924 83276945675 pipe tablearg ip from any to table(3) > out via em0 > 04901 4413338 991348968 allow ip from table(4) to any in via em0 > 05001 7146323 8293221022 allow ip from any to table(4) out via em0 > 05101 0 0 fwd <example.server> tcp from table(5) to any dst-port > 80,443,8828 in via em0 > <...> > > Then packets match the rule.But if we delete 4602 and place debug > rule at 4702: > > [Out3] > <...> > 04701 108458823 85372134891 pipe tablearg ip from table(2) to any in > via em0 > 04702 0 0 fwd <example.server> tcp from table(5) to any dst-port > 80,443,8828 in via em0 <-- NOT WORKING > 04801 - - pipe tablearg ip from any to table(3) out via em0 > 04901 - - allow ip from table(4) to any in via em0 > 05001 - - allow ip from any to table(4) out via em0 > 05101 0 0 fwd <example.server> tcp from table(5) to any dst-port > 80,443,8828 in via em0 > <...> > > So placing rule after 4701 gives zero counters, that means that our > ip packets match 4701 rule :) > > But we DO NOT HAVE target ip in table 2. It resides only in table 5 > as shown before.But it must be noticed that such rule WAS in table 2 > before.But then was removed from table 2 and added to table 6. > > Records in table 2 looks like that: > # ipfw table 2 list > 10.10.122.20/32 9 > 10.10.122.21/32 4 > 10.10.122.25/32 9 > 10.10.122.28/32 6 > 10.10.122.30/32 6 > > We also thought that mb we added missconfigured rule e.g. > 10.10.122.0/24 rule in table 2 but no. > > > > Here goes the KEY moment! > OK crawling through the logs, different tests etc we decide to clear > table 2. > At first step, we’ve deleted all records manually, with: > ipfw table 2 delete <addr> > > and NO effect at the time table was clear.So there were no records > in table 2, but rule 4701 was still catching packets from the target > ip. > > Second, we did flush with > ipfw table 2 flush > And voila, everything went fine from that moment! Miracle flush? > > So the problem seems like the rule resides deep in the kernel heart, > but we can NOT see this rule with ipfw output. > > > So i think there are at least to questions: > > 1. Have anyone ever met such situation? Or may be something close to > this one with 'hidden' ipfw rules? I have never seen it.. > 2. Is there a way how to get more info about rules and firewall > matching decisions? Mb some sysctl tuning or something else? Cause > its the first time we met such situation and don’t even imagine how > to diagnose such 'hidden' rule:( turn on logging and add 'log' to each rule. (we probably should have an option to turn on logging by default but I don't know of such). > > > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4E7113A3.80605>