From owner-freebsd-stable@FreeBSD.ORG Tue Mar 15 13:44:09 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 815D61065678 for ; Tue, 15 Mar 2011 13:44:09 +0000 (UTC) (envelope-from korvus@comcast.net) Received: from qmta14.westchester.pa.mail.comcast.net (qmta14.westchester.pa.mail.comcast.net [76.96.59.212]) by mx1.freebsd.org (Postfix) with ESMTP id 264F38FC27 for ; Tue, 15 Mar 2011 13:44:08 +0000 (UTC) Received: from omta01.westchester.pa.mail.comcast.net ([76.96.62.11]) by qmta14.westchester.pa.mail.comcast.net with comcast id KRT91g0030EZKEL5ERWs9Z; Tue, 15 Mar 2011 13:30:52 +0000 Received: from [192.168.2.164] ([206.210.89.202]) by omta01.westchester.pa.mail.comcast.net with comcast id KRWh1g00g4Mx3R23MRWjVz; Tue, 15 Mar 2011 13:30:50 +0000 Message-ID: <4D7F69FF.2020608@comcast.net> Date: Tue, 15 Mar 2011 09:30:39 -0400 From: Steve Polyack User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-US; rv:1.9.2.13) Gecko/20110125 Lightning/1.0b2 Thunderbird/3.1.7 MIME-Version: 1.0 To: freebsd-net@freebsd.org, freebsd-stable Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Subject: MAC address / per-proto ARP caching in 8.1-RELEASE X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Mar 2011 13:44:12 -0000 Is anyone aware of some sort of facility in either FreeBSD 8.1-RELEASE or the em(4) driver which would cause it to cache MAC addresses / ARP entries for hosts on a per-protocol basis? We've been doing some testing with new routers, and almost every time we switch them in or out our FreeBSD machines get stuck sending UDP DNS requests to the MAC address of the old default gateway in complete disregard for the current contents of the ARP table. New TCP connections do not seem to be affected: [spolyack@web01 ~]$ uname -a FreeBSD web01 8.1-RELEASE-p2 FreeBSD 8.1-RELEASE-p2 #3: Mon Dec 6 08:58:21 EST 2010 root@web01:/usr/obj/usr/src/sys/WEB-AMD64 amd64 Current ARP table after the router replacement, the default gateway is 10.0.1.254, which it has the correct new MAC address for already: [spolyack@web01 ~]$ arp -an ? (10.0.1.17) at 00:0c:29:47:74:3a on em2 permanent [ethernet] ? (10.0.1.130) at 00:0c:29:47:74:26 on em0 permanent [ethernet] ? (10.0.1.254) at 00:a0:c9:00:01:01 on em0 expires in 915 seconds [ethernet] ? (10.0.0.17) at 00:0c:29:47:74:30 on em1 permanent [ethernet] ? (10.0.0.15) at 00:0c:29:47:74:30 on em1 permanent [ethernet] ? (10.0.0.11) at 00:0c:29:47:74:30 on em1 permanent [ethernet] ? (10.0.0.2) at 00:1f:a0:10:28:70 on em1 expires in 1162 seconds [ethernet] ? (10.0.0.1) at 00:1f:a0:10:28:70 on em1 expires in 943 seconds [ethernet] tcpdump shows the DNS requests heading to the *old* router's MAC address despite the contents of the ARP table: [spolyack@web01 ~]$ sudo tcpdump -i em0 -s 256 -vvv -e -n -ttt 'port 53' tcpdump: listening on em0, link-type EN10MB (Ethernet), capture size 256 bytes 00:00:00.000000 00:0c:29:47:74:26 > 54:75:d0:a3:7c:8c, ethertype IPv4 (0x0800), length 86: (tos 0x0, ttl 64, id 55590, offset 0, flags [none], proto UDP (17), length 72) 10.0.1.130.52419 > 10.0.2.80.53: [bad udp cksum fdc5!] 52051+ A? db-testing-lab. (44) [spolyack@web01 ~]$ arp -an | grep 54:75:d0 [spolyack@web01 ~]$ Using telnet to form a TCP connection to port 53 on the same DNS server which the above UDP requests are headed to works just fine: [spolyack@web01 ~]$ telnet 10.0.2.80 53 Trying 10.0.2.80... Connected to 10.0.2.80. Escape character is '^]'. 00:03:43.272134 00:0c:29:47:74:26 > 00:a0:c9:00:01:01, ethertype IPv4 (0x0800), length 74: (tos 0x10, ttl 64, id 24383, offset 0, flags [DF], proto TCP (6), length 60) 10.0.2.130.20130 > 10.0.2.80.53: Flags [S], cksum 0x0353 (incorrect -> 0xf74d), seq 2674341615, win 65535, options [mss 1460,nop,wscale 3,sackOK,TS val 60433610 ecr 0], length ... Even after the above, standard UDP DNS requests are still headed to the old default gateway MAC address. tcpdumping and looking at ARP requests/responses doesn't show any traces of the old MAC address either. The switch which connects the server and router doesn't have any entries referencing the old router's MAC address anywhere either. I'm assuming this isn't related to the new flowtable features included in 8.x, since they appear to function on a 4-tuple of src addr, src port, dst addr, and dst port, which isn't going to match for new DNS requests. If anyone has any ideas or suggestions on what additional data I can grab, I'd be happy to investigate further, but I've run out of ideas here.