Date: Fri, 21 Jul 2006 18:15:22 +0200 From: Martin Beran <mb@tns.cz> To: freebsd-stable@freebsd.org Subject: Re: Kernel panic with PF Message-ID: <20060721161522.GA10111@mb.tns.cz> In-Reply-To: <20060721010559.GB23227@insomnia.benzedrine.cx> <200607210205.51614.max@love2party.net> <1153433881.1173.3.camel@genius.i.cz> <44BFA8F9.8010403@jellydonut.org> <1153410809.1126.66.camel@genius.i.cz> References: <200607210205.51614.max@love2party.net> <20060721010559.GB23227@insomnia.benzedrine.cx> <1153410809.1126.66.camel@genius.i.cz> <200607210205.51614.max@love2party.net> <1153410809.1126.66.camel@genius.i.cz> <44BFA8F9.8010403@jellydonut.org> <1153433881.1173.3.camel@genius.i.cz> <1153410809.1126.66.camel@genius.i.cz> <44BFA8F9.8010403@jellydonut.org> <1153410809.1126.66.camel@genius.i.cz>
next in thread | previous in thread | raw e-mail | index | archive | help
Hello, I am the author of the proxy code you have been discussing here. I have done some investigation today and I will try to summarize the results. > Thank you. No, I am not using it and I am quite sure the proxies aren't > doing it behind my back either. In fact there isn't a single entry in I can confirm that the proxies do not use any user/group rules. > the rules tables - there are only rdr rules generated on the fly by the > proxies. Depending on configuration, the proxies can install rdr and/or map rules. The rdr rules are used for "incoming transparency", i.e., to catch clients' connections going through the firewall, but with the target server's destination address, not the firewall's. The map rules are used for "outgoing transparency", i.e., for changing the source address of connections to servers (for example, the original client source IP can be used instead of the firewall's IP). > Which proxies are you using? The "pool_ticket: 1429 != 1430" messages you > quote below indicate a synchronization problem within the app talking to pf > via ioctl's. Tickets are used to ensure atomic commits for operations that This should be only a transient error. The proxies detect this situation and retry the failed operation. The idea behind this behavior was that the ruleset manipulation is done by a small number of ioctls called quickly one after another by a single function, hence the collisions should not occur too often. But maybe I can add exclusive locking of the PF device, which should remove all collisions among several proxy processes trying to change PF rules at the same time. > If the proxy is using DIOCCHANGERULE (it must be the proxy, pfctl isn't > using it at all), AND is trying to add/update a rule that requires at > least one replacement address but contains an empty list, then this > would cause the panic seen when that rule later matches a packet. I think this is not the case. The proxy uses either DIOCXBEGIN + DIOCBEGINADDRS + DIOCADDADDR + DIOCADDRULE + DIOCXCOMMIT or DIOCCHANGERULE(PF_CHANGE_GET_TICKET) + DIOCBEGINADDRS + DIOCADDADDR + DIOCCHANGERULE(PF_CHANGE_ADD_TAIL). The first method is used in the first call to create the ruleset. In the subsequent call, the second method is used to modify the ruleset. But the list is never empty. If it was, the panics would occur always, which is not happening - there are other installations (but probably not 64bit SMP) working well. I can imagine the list becoming empty only if some other process deletes it by DIOCBEGINADDRS during pfioctl processing, after the "pcr->pool_ticket != ticket_pabuf" check. But this should be guarded by PF_LOCK. Of course, I could make some mistake in the calling sequence of PF ioctl. I wrote this piece of code by trial and error, using pfctl as a source of ideas, because I have not found a detailed manual for the PF API. > Michal, can you please confirm that the patch above fixes the panic? > The proxy will still misbehave and cause the log messages (one more > EINVAL in this case ;), but the kernel shouldn't crash anymore. Yes, the patch should fix the panics, but it does not solve the problem. > This functionality of the software (using PF with anchors) is quite new It is not so new, it is now about 9 months in production use. > Anchors were introduced for this purpose, i.e. splitting the ruleset > into separate pieces, over each of which a single process can have > authority, so different processes don't stomp on each other's toes with > ruleset modifications. In fact, the possibility to split the ruleset into anchors owned by individual processes was one our major reasons to move from IPF to PF. > Ask them if they really need to still use DIOCCHANGERULE, as the idea > with anchors is generally to only operate within one anchor, and usually > flush or replace the (smaller) ruleset within. DIOCCHANGERULE is useful for us, because each proxy process can have several redirections or mappings and it creates/deletes them incrementally, as it opens/closes individual network connections. It seems to me unnecessary to always replace the whole ruleset. > Each anchor has its own ticket, so if you're seeing ticket mismatches, > that means there are concurrent operations on the same anchor, even. But the conflicts are on the pool_ticket which is, as I understand it, only one for all operations. > They (the Kernun authors) run multiple processes for each proxy. > Originally they used slightly modified Apached core for their proxies I > believe. Thus there are probably more processes using the same anchor. No, there are not. The anchors are even named by the owner process ID. > I don't really understand what they do inside - I would think that when > there are no traffic blocking rules, there's no point in doing anything > with PF except initial setting of the rdr rule to the proxy. As I have mentioned above, there are dynamicaly created rules for outgoing transparent connections (source-address in the configuration) and for some special cases, e.g., handling FTP data connections. -- Martin Beran Senior Developer Trusted Network Solutions, a.s. [ www.tns.cz ]
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20060721161522.GA10111>