Date: Sun, 29 Nov 2009 08:45:54 +0100 From: =?iso-8859-1?Q?Eirik_=D8verby?= <ltning@anduin.net> To: pyunyh@gmail.com, freebsd-current@freebsd.org Cc: weldon@excelsusphoto.com, Gavin Atkinson <gavin@freebsd.org> Subject: Re: FreeBSD 8.0 - network stack crashes? Message-ID: <74BFE523-4BB3-4748-98BA-71FBD9829CD5@anduin.net> In-Reply-To: <20091129013026.GA1355@michelle.cdnetworks.com> References: <A1648B95-F36D-459D-BBC4-FFCA63FC1E4C@anduin.net> <20091129013026.GA1355@michelle.cdnetworks.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 29. nov. 2009, at 02.30, Pyun YongHyeon wrote: > On Sat, Nov 28, 2009 at 08:46:12AM +0100, Eirik ??verby wrote: >> Hi, >>=20 >> Gavin Atkinson wrote: >>> On Tue, 2009-11-03 at 08:32 -0500, Weldon S Godfrey 3 wrote: >>>>=20 >>>> If memory serves me right, sometime around Yesterday, Gavin = Atkinson told me: >>>>=20 >>>> Gavin, thank you A LOT for helping us with this, I have answered as = much=20 >>>> as I can from the most recent crash below. We did hit max mbufs. = It is=20 >>>> at 25Kclusters, which is the default. I have upped it to 32K = because a=20 >>>> rather old article mentioned that as the top end and I need to get = into=20 >>>> work so I am not trying to do this with a remote console to go = higher. I=20 >>>> have already set it to reboot next with 64K clusters. I already = have kmem=20 >>>> maxed to what is bootable (or at least at one time) in 8.0, 4GB, = how high=20 >>>> can I safely go? This is a NFS server running ZFS with sustained 5 = min=20 >>>> averages of 120-200Mb/s running as a store for a mail system. >>>>=20 >>>>> Some things that would be useful: >>>>>=20 >>>>> - Does "arp -da" fix things? >>>>=20 >>>> no, it hangs like ssh, route add, etc >>>>=20 >>>>> - What's the output of "netstat -m" while the networking is = broken? >>>> Tue Nov 3 07:02:11 CST 2009 >>>> 36971/2033/39004 mbufs in use (current/cache/total) >>>> 24869/731/25600/25600 mbuf clusters in use = (current/cache/total/max) >>>> 24314/731 mbuf+clusters out of packet secondary zone in use=20 >>>> (current/cache) >>>> 0/35/35/12800 4k (page size) jumbo clusters in use=20 >>>> (current/cache/total/max) >>>> 0/0/0/6400 9k jumbo clusters in use (current/cache/total/max) >>>> 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max) >>>> 58980K/2110K/61091K bytes allocated to network = (current/cache/total) >>>> 0/201276/90662 requests for mbufs denied = (mbufs/clusters/mbuf+clusters) >>>> 0/0/0 requests for jumbo clusters denied (4k/9k/16k) >>>> 0/0/0 sfbufs in use (current/peak/max) >>>> 0 requests for sfbufs denied >>>> 0 requests for sfbufs delayed >>>> 0 requests for I/O initiated by sendfile >>>> 0 calls to protocol drain routines >>>=20 >>> OK, at least we've figured out what is going wrong then. As a >>> workaround to get the machine to stay up longer, you should be able = to >>> set kern.ipc.nmbclusters=3D256000 in /boot/loader.conf -but = hopefully we >>> can resolve this soon. >>=20 >> I'll chip in with a report of exactly the same situation, and I'm on = 8.0-RELEASE. >> We've been struggling with this for some time, and latest yesterday = the box was rebooted, and already last night it wedged again. We're at a = whopping=20 >> kern.ipc.nmbclusters: 524288 >> and I've just doubled it once more, which means we're allocating 2GB = to networking.. >>=20 >> Much like the original poster, we're seeing this on a amd64 storage = server with a large ZFS array shared through NFS, and network interfaces = are two em(4) combined in a lagg(4) interface (lacp). Using either of = the two em interfaces without lagg shows the same problem, just lower = performance.. >>=20 >>=20 >>> Firstly, what kernel was the above output from? And what network = card >>> are you using? In your initial post you mentioned testing both = bce(4) >>> and em(4) cards, be aware that em(4) had an issue that would cause >>> exactly this issue, which was fixed with a commit on September 11th >>> (r197093). Make sure your kernel is from after that date if you are >>> using em(4). I guess it is also possible that bce(4) has the same >>> issue, I'm not aware of any fixes to it recently. >>=20 >> We're on GENERIC . >>=20 >>=20 >>> So, from here, I think the best thing would be to just use the em(4) = NIC >>> and an up-to-date kernel, and see if you can reproduce the issue. >>=20 >> em(4) and 8.0-RELEASE still shows this problem. >>=20 >>=20 >>> How important is this machine? If em(4) works, are you able to help >>> debug the issues with the bce(4) driver? >>=20 >> We have no bce(4), but we have the problem on em(4) so can help debug = there. The server is important, but making it stable is more important.. = See below the sig for some debug info. >>=20 >=20 > How about disabling TSO/Tx checksum offloading of em(4)? > Last time I checked the driver, em(4) seems to assume it can access > IP/TCP header in mbuf chains without computing required header size. Hi, I just did that (-rxcsum -txcsum -tso), but the numbers still keep = rising. I'll wait and see if it goes down again, then reboot with those = values to see how it behaves. But right away it doesn't look too good .. /Eirik=
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?74BFE523-4BB3-4748-98BA-71FBD9829CD5>