Date: Mon, 27 Mar 2006 09:51:02 -0500 From: "Michael W. Lucas" <mwlucas@blackhelicopters.org> To: Charles Swiger <cswiger@mac.com> Cc: questions@freebsd.org Subject: Re: filling up UDP socket buffers like mad Message-ID: <20060327145102.GA57216@bewilderbeast.blackhelicopters.org> In-Reply-To: <83E0BC22-BFFA-47EE-88DA-D6A5D1862081@mac.com> References: <20060324211741.GA40819@bewilderbeast.blackhelicopters.org> <83E0BC22-BFFA-47EE-88DA-D6A5D1862081@mac.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--ew6BAiZeqk4r7MaW Content-Type: text/plain; charset=us-ascii Content-Disposition: inline On Fri, Mar 24, 2006 at 06:03:47PM -0500, Charles Swiger wrote: > On Mar 24, 2006, at 4:17 PM, Michael W. Lucas wrote: > >Running FreeBSD 6.1-PRERELEASE as a DNS, dhcp, and syslog server. > > > >I'm having trouble with DNS, DHCP, and syslogd locking up, and I think > >I've found what they all share in common. > > > >During the lockups, the box starts dropping UDP due to full socket > >buffers. I have a dumb little script to capture the rate of drops > >over 5 seconds, and it's about 45 a second. > > > >168725 dropped due to full socket buffers > >168958 dropped due to full socket buffers > > There is generally a cause behind the socket buffers filling up, > whether that is some form of livelock due to an OS problem or a > misconfiguration with a firewall/dummynet setup. You could look at > the output of "netstat -a(n)" for insight as to where the packets are > being queued up, but "netstat -s" would be useful to show to us as well. Thanks. I think you've shown me how to find the problem: # netstat -na ... udp4 0 0 127.0.0.1.57058 127.0.0.1.53 udp4 0 0 127.0.0.1.61259 127.0.0.1.53 udp4 0 0 127.0.0.1.54240 127.0.0.1.53 udp4 0 0 127.0.0.1.52997 127.0.0.1.53 udp4 0 0 *.67 *.* udp4 43414 0 *.514 *.* udp4 0 0 *.49661 *.* ... We have no firewall on this machine; it's buried behind three layers of firewall. I've tried running syslogd in debug mode, but not found anything particularly useful yet. Syslogd is now set to restart every 15 minutes, and run in debug mode, so hopefully the next time this happens I'll have the debugging output. The problem happens even within fifteen minutes, but because of my timeouts nobody notices. I'm attaching the output of netstat -na and netstat -s for general informative purposes; if anyone has any further suggestions, I'm all ears. Thanks, ==ml -- Michael W. Lucas mwlucas@FreeBSD.org, mwlucas@BlackHelicopters.org http://www.BlackHelicopters.org/~mwlucas/ "The cloak of anonymity protects me from the nuisance of caring." -Non Sequitur --ew6BAiZeqk4r7MaW Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="debug.txt" Active Internet connections (including servers) Proto Recv-Q Send-Q Local Address Foreign Address (state) tcp4 96 0 10.184.1.11.53 10.184.1.8.49838 ESTABLISHED tcp4 104 0 10.184.1.11.53 10.184.1.8.61719 ESTABLISHED tcp4 0 0 127.0.0.1.6011 *.* LISTEN tcp4 0 48 10.184.1.11.22 10.184.184.108.52392 ESTABLISHED tcp4 0 0 127.0.0.1.6010 *.* LISTEN tcp4 0 0 10.184.1.11.22 10.184.184.108.64567 ESTABLISHED tcp4 0 0 *.199 *.* LISTEN tcp4 0 0 *.80 *.* LISTEN tcp4 0 0 127.0.0.1.25 *.* LISTEN tcp4 0 0 *.22 *.* LISTEN tcp4 0 0 127.0.0.1.953 *.* LISTEN tcp4 0 0 127.0.0.1.53 *.* LISTEN tcp4 0 0 10.184.3.248.53 *.* LISTEN tcp4 0 0 10.184.1.11.53 *.* LISTEN udp4 0 0 127.0.0.1.59578 127.0.0.1.53 udp4 0 0 127.0.0.1.57058 127.0.0.1.53 udp4 0 0 127.0.0.1.61259 127.0.0.1.53 udp4 0 0 127.0.0.1.54240 127.0.0.1.53 udp4 0 0 127.0.0.1.52997 127.0.0.1.53 udp4 0 0 *.67 *.* udp4 43414 0 *.514 *.* udp4 0 0 *.49661 *.* udp4 0 0 *.52767 *.* udp4 0 0 *.63357 *.* udp4 0 0 127.0.0.1.58915 127.0.0.1.5680 udp4 0 0 127.0.0.1.62275 127.0.0.1.5679 udp4 0 0 127.0.0.1.57431 127.0.0.1.5678 udp4 0 0 *.63077 *.* udp4 0 0 *.161 *.* udp4 0 0 *.57589 *.* udp4 0 0 *.5681 *.* udp4 0 0 127.0.0.1.5680 *.* udp4 0 0 127.0.0.1.5679 *.* udp4 0 0 127.0.0.1.5678 *.* udp4 0 0 *.64486 *.* udp4 0 0 *.53558 *.* udp4 0 0 127.0.0.1.123 *.* udp4 0 0 10.184.3.248.123 *.* udp4 0 0 10.184.1.11.123 *.* udp4 0 0 *.123 *.* udp4 909 0 *.58684 *.* udp4 4129 0 127.0.0.1.53 *.* udp4 278 0 10.184.3.248.53 *.* udp4 8436 0 10.184.1.11.53 *.* icm4 0 0 *.* *.* Active UNIX domain sockets Address Type Recv-Q Send-Q Inode Conn Refs Nextref Addr c332d690 stream 0 0 c31c4bb0 0 0 0 /tmp/ssh-GceIKkNhTt/agent.1161 c332d71c stream 0 0 0 c332d460 0 0 c332d460 stream 0 0 0 c332d71c 0 0 c332dc08 stream 0 0 c31aa770 0 0 0 /tmp/ssh-YFEQdkNEm5/agent.1148 c332d8c0 stream 0 0 0 c332de38 0 0 c332de38 stream 0 0 0 c332d8c0 0 0 c332d08c stream 0 0 c3667330 0 0 0 /var/run/services.softflowd.ctl c332d578 stream 0 0 c3667660 0 0 0 /var/run/mpls.softflowd.ctl c332d604 stream 0 0 c3667990 0 0 0 /var/run/anx.softflowd.ctl c332e000 stream 0 0 c3146cc0 0 0 0 /var/run/devd.pipe c332d834 dgram 0 0 0 c332dc94 0 c332d7a8 c332d7a8 dgram 0 0 0 c332dc94 0 c332d118 c332d118 dgram 0 0 0 c332dc94 0 c332db7c c332d2bc dgram 0 0 0 c332d348 0 0 c332db7c dgram 0 0 0 c332dc94 0 0 c332d348 dgram 4048 0 c32a5660 0 c332d2bc 0 /var/named/var/run/log c332dd20 dgram 0 0 c32a5770 0 0 0 /var/run/log c332dc94 dgram 4048 0 c32a5880 0 c332d834 0 /var/run/logpriv c332d000 dgram 0 0 c32a5990 0 0 0 /var/run/log c332dec4 dgram 0 0 0 0 0 c332d7a8 c332d4ec dgram 0 0 0 0 0 c332d8c0 c332d9d8 dgram 0 0 0 0 0 0 c332da64 dgram 0 0 0 0 0 c332db7c tcp: 1986 packets sent 1254 data packets (124262 bytes) 0 data packets (0 bytes) retransmitted 0 data packets unnecessarily retransmitted 0 resends initiated by MTU discovery 486 ack-only packets (35 delayed) 0 URG only packets 0 window probe packets 0 window update packets 246 control packets 2428 packets received 1587 acks (for 124643 bytes) 23 duplicate acks 0 acks for unsent data 1142 packets (79497 bytes) received in-sequence 0 completely duplicate packets (0 bytes) 0 old duplicate packets 0 packets with some dup. data (0 bytes duped) 0 out-of-order packets (0 bytes) 0 packets (0 bytes) of data after window 0 window probes 8 window update packets 0 packets received after close 0 discarded for bad checksums 0 discarded for bad header offset fields 0 discarded because packet too short 25 connection requests 208 connection accepts 2 bad connection attempts 0 listen queue overflows 0 ignored RSTs in the windows 225 connections established (including accepts) 232 connections closed (including 0 drops) 26 connections updated cached RTT on close 26 connections updated cached RTT variance on close 0 connections updated cached ssthresh on close 8 embryonic connections dropped 1392 segments updated rtt (of 1373 attempts) 0 retransmit timeouts 0 connections dropped by rexmit timeout 0 persist timeouts 0 connections dropped by persist timeout 0 keepalive timeouts 0 keepalive probes sent 0 connections dropped by keepalive 187 correct ACK header predictions 525 correct data packet header predictions 208 syncache entries added 0 retransmitted 0 dupsyn 0 dropped 208 completed 0 bucket overflow 0 cache overflow 0 reset 0 stale 0 aborted 0 badack 0 unreach 0 zone failures 0 cookies sent 0 cookies received 0 SACK recovery episodes 0 segment rexmits in SACK recovery episodes 0 byte rexmits in SACK recovery episodes 0 SACK options (SACK blocks) received 0 SACK options (SACK blocks) sent 0 SACK scoreboard overflow udp: 550908 datagrams received 0 with incomplete header 0 with bad data length field 0 with bad checksum 0 with no checksum 33451 dropped due to no socket 63890 broadcast/multicast datagrams dropped due to no socket 63147 dropped due to full socket buffers 0 not for hashed pcb 390420 delivered 276481 datagrams output ip: 560670 total packets received 0 bad header checksums 0 with size smaller than minimum 0 with data size < data length 0 with ip length > max ip packet size 0 with header length < data size 0 with data length < header length 0 with bad options 0 with incorrect version number 0 fragments received 0 fragments dropped (dup or out of space) 0 fragments dropped after timeout 0 packets reassembled ok 554968 packets for this host 42 packets for unknown/unsupported protocol 0 packets forwarded (0 packets fast forwarded) 3059 packets not forwardable 2601 packets received for unknown multicast group 0 redirects sent 312928 packets sent from this host 0 packets sent with fabricated ip header 0 output packets dropped due to no bufs, etc. 0 output packets discarded due to no route 0 output datagrams fragmented 0 fragments created 0 datagrams that can't be fragmented 0 tunneling packets that can't find gif 0 datagrams with bad address in header icmp: 33294 calls to icmp_error 0 errors not generated in response to an icmp message Output histogram: echo reply: 1088 destination unreachable: 33294 0 messages with bad code fields 0 messages < minimum length 0 bad checksums 0 messages with bad length 0 multicast echo requests ignored 0 multicast timestamp requests ignored Input histogram: echo reply: 16 destination unreachable: 261 routing redirect: 309 echo: 1088 1088 message responses generated 0 invalid return addresses 0 no return routes ICMP address mask responses are disabled igmp: 0 messages received 0 messages received with too few bytes 0 messages received with bad checksum 0 membership queries received 0 membership queries received with invalid field(s) 0 membership reports received 0 membership reports received with invalid field(s) 0 membership reports received for groups to which we belong 0 membership reports sent --ew6BAiZeqk4r7MaW--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20060327145102.GA57216>