From owner-freebsd-net@FreeBSD.ORG Thu Mar 6 14:07:18 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1C280106566B for ; Thu, 6 Mar 2008 14:07:18 +0000 (UTC) (envelope-from bra@fsn.hu) Received: from people.fsn.hu (people.fsn.hu [195.228.252.137]) by mx1.freebsd.org (Postfix) with ESMTP id B46588FC21 for ; Thu, 6 Mar 2008 14:07:17 +0000 (UTC) (envelope-from bra@fsn.hu) Received: from japan.t-online.private (people [192.168.2.4]) by people.fsn.hu (Postfix) with ESMTP id 2C8A493D3C; Thu, 6 Mar 2008 15:07:09 +0100 (CET) Message-ID: <47CFFA8D.4060203@fsn.hu> Date: Thu, 06 Mar 2008 15:07:09 +0100 From: Attila Nagy User-Agent: Thunderbird 2.0.0.9 (X11/20080213) MIME-Version: 1.0 To: Max Laier References: <47CFAD07.6020008@fsn.hu> <49906.192.168.4.151.1204805824.squirrel@router> In-Reply-To: <49906.192.168.4.151.1204805824.squirrel@router> Content-Type: text/plain; charset=ISO-8859-2; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-net@freebsd.org Subject: Re: pf reply-to broken in RELENG_7 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 06 Mar 2008 14:07:18 -0000 On 03/06/08 13:17, Max Laier wrote: > Am Do, 6.03.2008, 09:36, schrieb Attila Nagy: > >> Hello, >> >> I've just upgraded some of our 6-STABLE servers to 7-STABLE to notice >> that pf reply-to for directly connected IPs seems to be broken. >> >> I have the following relevant rule in pf.conf: >> pass in on $ext_if reply-to ( $ext_if csmvip ) proto tcp from any to any >> port 25 label "mxtraffic-tcp" keep state >> >> which routes incoming SMTP connections (to be exact, the replies to >> them) to the csmvip host, which is a load balancer. This is needed >> because the LB doesn't do source NAT (it does destination NAT however to >> direct traffic addressed to its virtual IP to the real servers' IPs), >> and the servers have a different default route than the LB. This way the >> servers reply to the LB, so it can rewrite the replies' source address >> to its virtual IP, so the client will see the correct IP (the LB's >> virtual IP) in the address, instead of the host's real address. >> >> It seems that this still works in 7-STABLE for the internet (not >> directly connected) hosts, but not for directly connected hosts, for >> example the ones, which are in the same subnet as my servers. >> To overcome this, I've had to add static ARP entries to the servers, to >> tell that the clients' hardware address is the address of the load >> balancer, but it would be better if the previous behaviour (as in >> 6-STABLE) could be restored. >> >> Could anybody help to resolve this? >> > > Might be the lack of sleep and coffee, but I can't quite figure out the > network layout you are talking about. Could you draw up a small example > setup so I can follow? Or at least (pseudo-)IP addresses for client, > load-balancer, pf-box and servers? > Of course, see: http://people.fsn.hu/~bra/freebsd/route-to-20080306/lb.png 10.0.0.1 is a normal router, which connects the servers to the internet (they can be reached at their real IPs through it and they can reach the internet via them). This is the default GW for the hosts. 10.0.0.2 is a load balancer, which has another address too (10.1.1.2), which is what I've called virtual IP. We do the service on this virtual IP, so the clients on the internet connect to this. 10.0.0.3 and 10.0.0.4 are the real servers behind the virtual IP. The load balancer does this: - a client -say 10.2.2.2 on the internet- connects to the load balancer's virtual IP, 10.1.1.2, for example with TCP/25 - the load balancer then takes a machine out of its configured pool for this service, for example 10.0.0.3 - to reach 10.0.0.3:25, the load balancer changes the packet's destination address from 10.1.1.2 to 10.0.0.3 and sends it out on its 10.0.0.2 interface - the host 10.0.0.3 gets this and sees that the source address is not directly connected, so it tries to send it via the default GW (10.0.0.1), which is of course not good, so we have a reply-to rule, which forces these replies back to 10.0.0.2 - the load balancer gets the reply, then changes its source IP to 10.1.1.2 (the virtual IP for this service) and sends out to the internet We can't send the reply to 10.0.0.1 (the default GW), because it won't do the source NAT and the load balancer will see only one way of the traffic, and it won't let it through (it has a state table, so for example a TCP session must be built up through it, it has to follow the states). We also can't use the load balancer as a default GW, because it's not a router, it's pointless to route normal outgoing traffic through it (which is originated at the hosts), and the load balancer wouldn't allow this anyway, because it only handles configured services and everything must be in its state table. This part works OK both with FreeBSD 6 and 7, the replies to the servers go to the load balancer. What does not work is the scenario, when 10.0.0.4 tries to connect to 10.1.1.2. You can easily see, that this is a problem, because 10.0.0.4 sends the first packet through 10.0.0.1, which in turn forwards it to 10.1.1.2 (the virtual IP of the load balancer). The load balancer then changes the destination IP from 10.1.1.2 to 10.0.0.3 (I really should draw three or more boxes, but please imagine that 10.0.0.4 won't connect to a service in which it is placed, so it won't happen that both the source and the destination is the same machine) and sends it through its 10.0.0.2 interface. The host gets the packet and sees that the source is 10.0.0.4, which is directly connected to it, so it sends the answer directly and 10.0.0.3 will stand confused (sent a request to 10.1.1.2, got the reply from 10.0.0.3). This is the second thing, which is the above reply-to is there: the machine must send the reply to 10.0.0.2, not to 10.0.0.4. The load balancer then will change the source address to 10.1.1.2 and send the reply to 10.0.0.4, so everything will be OK. This is what works in 6.x, but not in 7. I hope I could make this clear enough, if not, please ask. Thanks, -- Attila Nagy e-mail: Attila.Nagy@fsn.hu Free Software Network (FSN.HU) phone: +3630 306 6758 http://www.fsn.hu/