Date: Wed, 6 Jun 2001 01:30:07 -0700 (PDT) From: Ruslan Ermilov <ru@FreeBSD.org> To: freebsd-bugs@FreeBSD.org Subject: Re: kern/27890: FreeBSD not always seems to take the best route Message-ID: <200106060830.f568U7C75363@freefall.freebsd.org>
next in thread | raw e-mail | index | archive | help
The following reply was made to PR kern/27890; it has been noted by GNATS. From: Ruslan Ermilov <ru@FreeBSD.org> To: Andre Albsmeier <andre.albsmeier@mchp.siemens.de> Cc: bug-followup@FreeBSD.org Subject: Re: kern/27890: FreeBSD not always seems to take the best route Date: Wed, 6 Jun 2001 11:24:19 +0300 On Tue, Jun 05, 2001 at 06:30:14PM +0200, Andre Albsmeier wrote: > > I have observed this behaviour for a long time now but finally had > time to dig into it... I reference syslogd as an example here > but I think the problem lies in the network code of the kernel... > > > Simple network: > - two routers (1 and 2) > - host C with IP 192.168.1.3 > - host S with IP 192.168.2.1 > > All machines are FreeBSD 4.3-STABLE. > > Router 1 routes pkts between the Internet and 192.168.1.0 > Router 2 routes pkts between 192.168.1.0 and 192.168.2.0 > > > +-----+ +-----+ > default | | 192.168.1.0 | | 192.168.2.0 > -----------| 1 |--------+--------| 2 |--------+-------- more hosts > | | | | | | > +-----+ | +-----+ | > | | > +-----+ +-----+ > | | | | > 192.168.1.3 | C | | S | 192.168.2.1 > | | | | > +-----+ +-----+ > > > Relevant parts of netstat -rn on C during normal operation: > ------------------------------------------------------------- > Destination Gateway Flags Netif Expire > default 192.168.1.1 UGSc fxp0 > 127.0.0.1 127.0.0.1 UH lo0 > 192.168.1 link#1 UC fxp0 => > 192.168.1.1 0:e0:18:90:91:bb UHLW fxp0 1182 > 192.168.1.2 0:e0:18:90:94:c8 UHLW fxp0 1058 > 192.168.1.3 0:e0:18:90:45:dc UHLW lo0 > 192.168.1.255 ff:ff:ff:ff:ff:ff UHLWb fxp0 > 192.168.2 192.168.1.2 UGc fxp0 > > > The syslogd on host C is configured to log messages > to syslogd running on host S. This works perfectly, > all messages appear on host S. > > Now we delete the route to net 192.168.2.0 on host C (this > can appear automatically if router 2 and/or its routed go > down for a while). If syslogd now wants to send a message > to S, the kernel uses the default route which is obvious > because the route to net 192.168.2.0 is gone. We can see the > packets go into router 1. I consider this as the correct > behaviour as well. > > Now we bring back the route to net 192.168.2.0 again on host > C exactly as it was before (e.g. by restarting router 2 and/or > its routed). We can verify this with netstat -rn on C. We can > also ping host S or telnet to it or do other stuff which all > work perfectly. > > The problem is that each time when syslogd on C wants to send > a packet to S, the kernel still uses 1 as router even though > it should send them through 2. After HUPing or restarting > syslogd on C (which means that the UDP socket is closed and > opened again) things are back to normal. > > It seems that as long as packets can be send somewhere, the > kernel doesn't bother if there is a better route to the > destination until the socket is closed and opened again. > > >How-To-Repeat: > > See above. > I can't reproduce this problem on my 4.3-STABLE box. Yes, the UDP socket has the reference to the protocol-cloned route to the destination host S through the router 1 initially, and UDP packets go through that router. In my tests, router 1 (192.168.1.1) was the host *not* configured to act as the router, so all "foreign" packets sent to it got silently ignored. I used the ports/net/netcat utility to connect to the UDP `echo' port of the destination S (192.168.2.1): Fig.1: Initial state, before UDP socket is open. : # netstat -arn : Destination Gateway Flags Refs Use Netif Expire : default 192.168.1.1 UGSc 0 2 rl0 : 127.0.0.1 127.0.0.1 UH 1 6 lo0 : 192.168.1 link#1 UC 3 0 rl0 => Fig.2: We connect(2) UDP socket to the "echo" port on S (192.168.2.1). : # nc -u 192.168.2.1 echo : ping1 : ping2 : ping3 [...] As you can see, we receive no echos back. Fig.3: Routing table after UDP socket is open. : # netstat -arn : Destination Gateway Flags Refs Use Netif Expire : default 192.168.1.1 UGSc 1 2 rl0 : 127.0.0.1 127.0.0.1 UH 1 6 lo0 : 192.168.1 link#1 UC 4 0 rl0 => : 192.168.2.1 192.168.1.1 UGHW 1 14 rl0 The route to S (192.168.2.1) was cloned (W) from the `default' route. refcnt=1 on the 192.168.2.1 route indicates that the UDP socket holds a reference to this route. Fig.4: I manually add the route to the 192.168.2 network. : # route add -net 192.168.2 192.168.1.2 : add net 192.168.2: gateway 192.168.1.2 Fig.5: Routing table after the route to the 192.168.2 network was added. : # netstat -arn : Destination Gateway Flags Refs Use Netif Expire : default 192.168.1.1 UGSc 1 2 rl0 : 127.0.0.1 127.0.0.1 UH 1 6 lo0 : 192.168.1 link#1 UC 4 0 rl0 => : 192.168.2 192.168.1.2 UGSc 0 0 rl0 As you can see, the route to the 192.168.2.1 host is deleted from the routing table. It actually doesn't get freed completely, as it had non-zero reference count (UDP socket still holds on it), but instead it gets marked as DOWN, and will be freed and reallocated in ip_output() on the next use. Fig.6: We continue to send UDP datagrams. : # nc -u 192.168.2.1 echo (continued) : ping4 : ping4 : ping5 : ping5 : ping6 : ping6 As you can see, this time we get the echos back. Fig.7: Routing table after we sent more UDP datagrams. : # netstat -arn -finet : Destination Gateway Flags Refs Use Netif Expire : default 192.168.1.1 UGSc 0 2 rl0 : 127.0.0.1 127.0.0.1 UH 1 6 lo0 : 192.168.1 link#1 UC 4 0 rl0 => : 192.168.2 192.168.1.2 UGSc 1 3 rl0 The refcount on 192.168.2 route has grown to 1, indicating that the UDP socket now holds on this route. The `Use' count of 3 corresponds to our three UDP datagrams (ping4, ping5, and ping6). Could you please repeat these steps in your environment, and try to detect where it behaved differently in your case. Cheers, -- Ruslan Ermilov Oracle Developer/DBA, ru@sunbay.com Sunbay Software AG, ru@FreeBSD.org FreeBSD committer, +380.652.512.251 Simferopol, Ukraine http://www.FreeBSD.org The Power To Serve http://www.oracle.com Enabling The Information Age To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-bugs" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200106060830.f568U7C75363>