Date: Wed, 16 May 2012 00:43:02 +0400 From: Dmitry Sivachenko <mitya@yandex-team.ru> To: freebsd-net@freebsd.org Subject: network stops working Message-ID: <4FB2BFD6.8040607@yandex-team.ru>
next in thread | raw e-mail | index | archive | help
Hello! I am using FreeBSD-9-stable. I use Intel network cards (em, igb) with mtu=9000 (it's important, the problem disappears if I switch to mtu=1500). I have a number of servers running a few web-services written in our company. After a few days of working network suddenly stops functioning. There is nothing in log/messages. It just stops working. If I execute ifconfig down/up, I get the following error message: # ifconfig net0 down # ifconfig net0 up em1: Could not setup receive structures # and network still does not work. If I stop mentioned programs serving web requests (even not all of them, just random one), ifconfig net0 up command succeeds and network resumes its operations... for more few days and then the same problems comes back. Consider netstat -m output: > netstat -m 1025/4945/5970 mbufs in use (current/cache/total) 0/3446/3446/262144 mbuf clusters in use (current/cache/total/max) 0/2090 mbuf+clusters out of packet secondary zone in use (current/cache) 0/1230/1230/65536 4k (page size) jumbo clusters in use (current/cache/total/max) 1023/4907/5930/65536 9k jumbo clusters in use (current/cache/total/max) ^^^^^^^^^^^^^^^^^^^^^^^ Note that max is much higher that total. 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max) 9463K/57211K/66674K bytes allocated to network (current/cache/total) 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/2542746/0 requests for jumbo clusters denied (4k/9k/16k) ^^^^^^^^^^^^ There are 9k jumbo clusters allocations denied. 0/0/0 sfbufs in use (current/peak/max) 0 requests for sfbufs denied 0 requests for sfbufs delayed 641 requests for I/O initiated by sendfile 0 calls to protocol drain routines I am attaching vmstat -z output below for reference. What can be the cause of 9k jumbo closters allocation denies? What additional information cat I provide to help trach this down? Thanks in advance! > vmstat -z ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP UMA Kegs: 208, 0, 92, 10, 92, 0, 0 UMA Zones: 3456, 0, 92, 0, 92, 0, 0 UMA Slabs: 568, 0, 6270, 702, 2222844, 0, 0 UMA RCntSlabs: 568, 0, 8883, 581, 7364017, 0, 0 UMA Hash: 256, 0, 1, 14, 4, 0, 0 16 Bucket: 152, 0, 39, 136, 185, 0, 0 32 Bucket: 280, 0, 130, 206, 401, 0, 0 64 Bucket: 536, 0, 138, 149, 470, 84, 0 128 Bucket: 1048, 0, 1812, 639, 1736739,16793, 0 VM OBJECT: 232, 0, 52210, 7454,20006697, 0, 0 MAP: 232, 0, 7, 25, 7, 0, 0 KMAP ENTRY: 120, 1549690, 3102, 2602, 6160517, 0, 0 MAP ENTRY: 120, 0, 7131, 3409,54473178, 0, 0 fakepg: 120, 0, 0, 589, 53928, 0, 0 mt_zone: 4112, 0, 276, 3, 276, 0, 0 16: 16, 0, 4047, 4353,3946273892, 0, 0 32: 32, 0, 3955, 2408,13600755, 0, 0 64: 64, 0, 11979, 7509,116304518, 0, 0 128: 128, 0, 20979, 36673,90194687, 0, 0 256: 256, 0, 5569, 16886,137956814, 0, 0 512: 512, 0, 4368, 5194, 8638401, 0, 0 1024: 1024, 0, 84, 1196, 4033660, 0, 0 2048: 2048, 0, 199, 1751, 1203566, 0, 0 4096: 4096, 0, 318, 2704, 1142726, 0, 0 Files: 80, 0, 5613, 5097,2006566966, 0, 0 TURNSTILE: 136, 0, 5605, 855, 6379, 0, 0 umtx pi: 96, 0, 0, 0, 0, 0, 0 PROC: 1160, 0, 63, 3657, 709134, 0, 0 THREAD: 1112, 0, 3969, 1635, 524885, 0, 0 SLEEPQUEUE: 80, 0, 5605, 1094, 6379, 0, 0 VMSPACE: 392, 0, 43, 2387, 709117, 0, 0 cpuset: 72, 0, 2, 98, 2, 0, 0 mbuf_packet: 256, 0, 0, 1765,4356065566, 0, 0 mbuf: 256, 0, 1024, 3181,26760389876, 0, 0 mbuf_cluster: 2048, 262144, 1765, 1681,1318169467, 0, 0 mbuf_jumbo_page: 4096, 65536, 0, 1230,61648445, 0, 0 mbuf_jumbo_9k: 9216, 65536, 1023, 4907,8926430021,2542746, 0 mbuf_jumbo_16k: 16384, 3200, 0, 0, 0, 0, 0 mbuf_ext_refcnt: 4, 0, 0, 2352, 5422000, 0, 0 NetGraph items: 72, 4118, 0, 58, 5, 0, 0 NetGraph data items: 72, 522, 0, 58, 2, 0, 0 g_bio: 232, 0, 0, 4544,300858503, 0, 0 ttyinq: 160, 0, 180, 252, 615, 0, 0 ttyoutq: 256, 0, 95, 175, 327, 0, 0 ata_request: 328, 0, 0, 0, 0, 0, 0 ata_composite: 336, 0, 0, 0, 0, 0, 0 VNODE: 480, 0, 78944, 3840, 952605, 0, 0 VNODEPOLL: 112, 0, 0, 0, 1, 0, 0 NAMEI: 1024, 0, 0, 1632,139785175, 0, 0 S VFS Cache: 108, 0, 87356, 1579, 556273, 0, 0 L VFS Cache: 328, 0, 2924, 1768, 15274, 0, 0 DIRHASH: 1024, 0, 9, 1259, 20893, 0, 0 NCLNODE: 560, 0, 0, 0, 0, 0, 0 Mountpoints: 768, 0, 5, 10, 5, 0, 0 AIO: 208, 0, 0, 0, 0, 0, 0 AIOP: 32, 0, 0, 0, 0, 0, 0 AIOCB: 480, 0, 0, 0, 0, 0, 0 AIOL: 128, 0, 0, 0, 0, 0, 0 AIOLIO: 272, 0, 0, 0, 0, 0, 0 pipe: 728, 0, 7, 2928, 515298, 0, 0 ksiginfo: 112, 0, 3680, 3250, 192170, 0, 0 itimer: 344, 0, 1, 21, 6, 0, 0 KNOTE: 128, 0, 31, 4406,11850035546, 0, 0 socket: 680, 262140, 66, 4986,1949420563, 0, 0 ipq: 56, 8253, 0, 0, 0, 0, 0 udp_inpcb: 392, 262140, 20, 2170, 1133408, 0, 0 udpcb: 16, 262248, 20, 3676, 1133408, 0, 0 tcp_inpcb: 392, 262140, 3373, 24417,1947917187, 0, 0 tcpcb: 976, 262140, 26, 5010,1947917187, 0, 0 tcptw: 72, 41000, 3347, 24153,1352095629, 0, 0 syncache: 152, 15375, 0, 2375,2029926070, 0, 0 hostcache: 136, 153608, 5, 667, 4953, 0, 0 tcpreass: 40, 16464, 0, 2436,27465901, 0, 0 sackhole: 32, 0, 0, 909, 41329, 0, 0 sctp_ep: 1368, 25600, 0, 0, 0, 0, 0 sctp_asoc: 2288, 40000, 0, 0, 0, 0, 0 sctp_laddr: 48, 80064, 0, 216, 13, 0, 0 sctp_raddr: 704, 80000, 0, 0, 0, 0, 0 sctp_chunk: 136, 400008, 0, 0, 0, 0, 0 sctp_readq: 104, 400032, 0, 0, 0, 0, 0 sctp_stream_msg_out: 112, 400026, 0, 0, 0, 0, 0 sctp_asconf: 40, 400008, 0, 0, 0, 0, 0 sctp_asconf_ack: 48, 400032, 0, 0, 0, 0, 0 ripcb: 392, 262140, 0, 70, 138, 0, 0 unpcb: 240, 262144, 14, 1986, 369808, 0, 0 rtentry: 200, 0, 19, 38, 37, 0, 0 IPFW dynamic rule: 120, 0, 0, 0, 0, 0, 0 divcb: 392, 262140, 0, 0, 0, 0, 0 g_stripe_zone: 131072, 100, 0, 0, 0, 0, 0 selfd: 56, 0, 6286, 2534,62657688, 0, 0 SWAPMETA: 288, 116519, 1109, 2817,14987719, 0, 0 FFS inode: 168, 0, 78896, 5232, 952545, 0, 0 FFS1 dinode: 128, 0, 0, 0, 0, 0, 0 FFS2 dinode: 256, 0, 78896, 5014, 952545, 0, 0 > cat /boot/loader.conf userconfig_script_load="YES" vm.exec_map_entries="48" hw.igb.rxd="256" hw.igb.txd="256" hw.usb.no_pf="1"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4FB2BFD6.8040607>