From owner-freebsd-bugs@FreeBSD.ORG Sat Dec 4 18:38:11 2010 Return-Path: Delivered-To: freebsd-bugs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3564C1065674 for ; Sat, 4 Dec 2010 18:38:11 +0000 (UTC) (envelope-from kes-kes@yandex.ru) Received: from forward15.mail.yandex.net (forward15.mail.yandex.net [95.108.130.119]) by mx1.freebsd.org (Postfix) with ESMTP id D18278FC0A for ; Sat, 4 Dec 2010 18:38:09 +0000 (UTC) Received: from smtp13.mail.yandex.net (smtp13.mail.yandex.net [95.108.130.68]) by forward15.mail.yandex.net (Yandex) with ESMTP id 77BB64458868 for ; Sat, 4 Dec 2010 21:22:49 +0300 (MSK) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yandex.ru; s=mail; t=1291486969; bh=q0A9M7ewwFiGn9Obs0rru9YTapVR75Zgg5fiMYvFpKI=; h=Date:From:Reply-To:Message-ID:To:Subject:MIME-Version: Content-Type; b=fkp4K0WhiJCEss5U/8fI/qrF/AKdmr48LIkD2x4ZNcHqfetcuUoDuSnNipOtPacBO Hb0Gyh/n60kc+WH8aRiZ85FUdeu5MrNyAVcabPytbTU7RkdnFf1ELsTQqscD+ckrfZ rBRZZ85LM146YnvEc886fo4FB9+07OcOS2NnGPx0= Received: from kes.in (unknown [77.93.42.18]) by smtp13.mail.yandex.net (Yandex) with ESMTPA id 9D92F41580B0 for ; Sat, 4 Dec 2010 21:22:48 +0300 (MSK) Date: Sat, 4 Dec 2010 20:22:50 +0200 From: =?windows-1251?B?yu7t/Oru4iDF4uPl7ejp?= X-Mailer: The Bat! (v4.0.24) Professional Organization: =?windows-1251?B?188gyu7t/Oru4iwgRnJlZUxpbmU=?= X-Priority: 3 (Normal) Message-ID: <1356153913.20101204202250@yandex.ru> To: freebsd-bugs@freebsd.org MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----------121D0EF2EC7CA44" X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: Something wrong in network subsystem X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: =?windows-1251?B?yu7t/Oru4iDF4uPl7ejp?= List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 04 Dec 2010 18:38:11 -0000 ------------121D0EF2EC7CA44 Content-Type: text/plain; charset=windows-1251 Content-Transfer-Encoding: 8bit Hi, Freebsd-bugs. server is under usual load, but in one momenth something is broke and it has 80% interrupt. (see graph) does this relate to BUG? detail are below # uname -a FreeBSD 9.0-CURRENT FreeBSD 9.0-CURRENT #7: Wed Nov 24 11:39:13 EET 2010 @:/usr/obj/usr/src/sys/KES_KERN_v9 i386 # uptime 20:19 up 9 days, 2:17, 1 user, load averages: 1,42 1,40 1,37 # vmstat -i interrupt total rate irq1: atkbd0 7804 0 irq16: ehci0 2902776774 3694 irq17: atapci0 6825719 8 irq23: ehci1 1572029 2 cpu0:timer 3240929814 4124 irq256: re0 154845556 197 Total 6306957696 8026 #top -SIHP last pid: 13539; load averages: 1.37, 1.38, 1.34 up 9+02:06:10 20:08:55 82 processes: 3 running, 64 sleeping, 15 waiting CPU: 0.0% user, 0.0% nice, 9.8% system, 76.9% interrupt, 13.3% idle Mem: 58M Active, 758M Inact, 210M Wired, 208K Cache, 112M Buf, 901M Free Swap: 32G Total, 32G Free PID USERNAME PRI NICE SIZE RES STATE TIME WCPU COMMAND 12 root -64 - 0K 120K WAIT 170:43 41.85% {irq16: ehci0} 12 root -28 - 0K 120K WAIT 32.5H 34.13% {swi5: +} 11 root 171 ki31 0K 8K RUN 168.7H 17.63% idle 13 root 53 - 0K 8K RUN 531:33 7.91% ng_queue systat -v 1 users Load 1,27 1,36 1,33 4 дек 20:09 Mem:KB REAL VIRTUAL VN PAGER SWAP PAGER Tot Share Tot Share Free in out in out Act 95292 4968 382456 6368 922588 count All 261588 7652 2540372 16716 pages Proc: Interrupts r p d s w Csw Trp Sys Int Sof Flt cow 243k total 32 484k 64 594 238k 33k 8 zfod atkbd0 1 ozfod 222k ehci0 16 9,4%Sys 78,5%Intr 0,3%User 0,0%Nice 11,8%Idle %ozfod 11 atapci0 17 | | | | | | | | | | | daefr 2 ehci1 23 =====+++++++++++++++++++++++++++++++++++++++ prcfr 4121 cpu0:timer 31 dtbuf 2 totfr 16750 re0 256 Namei Name-cache Dir-cache 110738 desvn react Calls hits % hits % 92681 numvn pdwak 3 3 100 27684 frevn pdpgs intrn Disks ad4 214984 wire KB/t 16,00 59440 act tps 11 776476 inact MB/s 0,17 204 cache %busy 0 922384 free # netstat -w 1 -h -d -I re0 input (re0) output packets errs idrops bytes packets errs bytes colls drops 14K 0 0 8.5M 14K 0 8.0M 0 0 15K 0 0 9.4M 15K 0 8.8M 0 0 15K 0 0 9.1M 15K 0 8.5M 0 0 15K 0 0 9.7M 16K 0 9.0M 0 0 16K 0 0 10M 16K 0 9.3M 0 0 16K 0 0 10M 17K 0 9.3M 0 0 16K 0 0 10M 16K 0 9.3M 0 0 16K 0 0 11M 17K 0 9.9M 0 0 16K 0 0 10M 17K 0 9.6M 0 0 13K 0 0 7.8M 13K 0 7.3M 0 0 13K 0 0 8.0M 14K 0 7.5M 0 0 13K 0 0 7.8M 14K 0 7.4M 0 0 # netstat -s -p tcp tcp: 3423432 packets sent 1858123 data packets (48135556 bytes) 26011 data packets (609532 bytes) retransmitted 1372 data packets unnecessarily retransmitted 0 resends initiated by MTU discovery 1497899 ack-only packets (1435187 delayed) 0 URG only packets 0 window probe packets 2136 window update packets 39450 control packets 3831803 packets received 1842790 acks (for 48124547 bytes) 22529 duplicate acks 0 acks for unsent data 1900424 packets (226227761 bytes) received in-sequence 1490 completely duplicate packets (38592 bytes) 0 old duplicate packets 6 packets with some dup. data (136 bytes duped) 2322 out-of-order packets (2982496 bytes) 9 packets (0 bytes) of data after window 0 window probes 69 window update packets 38 packets received after close 0 discarded for bad checksums 0 discarded for bad header offset fields 0 discarded because packet too short 0 discarded due to memory problems 15318 connection requests 8441 connection accepts 0 bad connection attempts 0 listen queue overflows 128 ignored RSTs in the windows 23736 connections established (including accepts) 23656 connections closed (including 2121 drops) 21705 connections updated cached RTT on close 23094 connections updated cached RTT variance on close 16478 connections updated cached ssthresh on close 21 embryonic connections dropped 1821810 segments updated rtt (of 1815670 attempts) 26393 retransmit timeouts 1140 connections dropped by rexmit timeout 0 persist timeouts 0 connections dropped by persist timeout 0 Connections (fin_wait_2) dropped because of timeout 10 keepalive timeouts 9 keepalive probes sent 1 connection dropped by keepalive 379 correct ACK header predictions 178731 correct data packet header predictions 9149 syncache entries added 337 retransmitted 171 dupsyn 0 dropped 8441 completed 0 bucket overflow 0 cache overflow 629 reset 79 stale 0 aborted 0 badack 0 unreach 0 zone failures 9149 cookies sent 0 cookies received 41 SACK recovery episodes 36 segment rexmits in SACK recovery episodes 10768 byte rexmits in SACK recovery episodes 8493 SACK options (SACK blocks) received 2157 SACK options (SACK blocks) sent 0 SACK scoreboard overflow 0 packets with ECN CE bit set 0 packets with ECN ECT(0) bit set 0 packets with ECN ECT(1) bit set 0 successful ECN handshakes 0 times ECN reduced the congestion window # netstat -Q Configuration: Setting Current Limit Thread count 1 1 Default queue limit 256 10240 Direct dispatch enabled n/a Forced direct dispatch enabled n/a Threads bound to CPUs disabled n/a Protocols: Name Proto QLimit Policy Flags ip 1 256 flow --- igmp 2 256 source --- rtsock 3 256 source --- arp 7 256 source --- ip6 10 256 flow --- Workstreams: WSID CPU Name Len WMark Disp'd HDisp'd QDrops Queued Handled 0 0 ip 0 1 9280761530 0 0 85388 9280846918 igmp 0 0 0 0 0 0 0 rtsock 0 1 0 0 0 10 10 arp 0 0 2341888 0 0 0 2341888 ip6 0 0 107430 0 0 0 107430 # ifconfig -m re0 re0: flags=8843 metric 0 mtu 1500 options=389b capabilities=389b ether 00:30:67:5a:44:72 inet I.N.E.T netmask 0xfffffc00 broadcast 10.X.X.X inet6 fe80::I.N.E.T%re0 prefixlen 64 scopeid 0x1 nd6 options=29 media: Ethernet autoselect (1000baseT ) status: active supported media: media autoselect media 1000baseT mediaopt full-duplex media 1000baseT media 100baseTX mediaopt full-duplex media 100baseTX media 10baseT/UTP mediaopt full-duplex media 10baseT/UTP media none # vmstat -z ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP UMA Kegs: 128, 0, 90, 0, 90, 0, 0 UMA Zones: 896, 0, 90, 2, 90, 0, 0 UMA Slabs: 284, 0, 1800, 6, 3785559, 0, 0 UMA RCntSlabs: 544, 0, 2252, 2, 2252, 0, 0 UMA Hash: 128, 0, 3, 27, 4, 0, 0 16 Bucket: 76, 0, 37, 13, 58, 0, 0 32 Bucket: 140, 0, 61, 23, 81, 0, 0 64 Bucket: 268, 0, 91, 7, 142, 13, 0 128 Bucket: 524, 0, 2816, 5, 472941,8127, 0 VM OBJECT: 136, 0, 65515, 8145, 2255395, 0, 0 MAP: 136, 0, 7, 22, 7, 0, 0 KMAP ENTRY: 72, 57505, 30, 129, 8723924, 0, 0 MAP ENTRY: 72, 0, 764, 561, 5406137, 0, 0 DP fakepg: 72, 0, 0, 0, 0, 0, 0 SG fakepg: 72, 0, 0, 0, 0, 0, 0 mt_zone: 2060, 0, 301, 121, 301, 0, 0 16: 16, 0, 3967, 296,51165091, 0, 0 32: 32, 0, 3925, 27715,3398801221, 0, 0 64: 64, 0, 8928, 90015,22166775, 0, 0 128: 128, 0, 107721, 11679,183686782, 0, 0 256: 256, 0, 2619, 46161, 732044, 0, 0 512: 512, 0, 1000, 64,10976314, 0, 0 1024: 1024, 0, 362, 18,13018880, 0, 0 2048: 2048, 0, 784, 960,18518183, 0, 0 4096: 4096, 0, 100, 29, 608770, 0, 0 Files: 56, 0, 320, 149,22304433, 0, 0 TURNSTILE: 72, 0, 156, 24, 156, 0, 0 umtx pi: 52, 0, 0, 0, 0, 0, 0 MAC labels: 20, 0, 0, 0, 0, 0, 0 PROC: 680, 0, 49, 23, 113477, 0, 0 THREAD: 716, 0, 104, 51, 319099, 0, 0 SLEEPQUEUE: 44, 0, 156, 80, 156, 0, 0 VMSPACE: 236, 0, 30, 34, 113447, 0, 0 cpuset: 40, 0, 2, 182, 2, 0, 0 audit_record: 816, 0, 0, 0, 0, 0, 0 mbuf_packet: 256, 0, 336, 4144,6244701687, 0, 0 mbuf: 256, 0, 74, 3906,13729996172, 0, 0 mbuf_cluster: 2048, 25600, 4480, 6, 4480, 0, 0 mbuf_jumbo_page: 4096, 12800, 0, 9, 845, 0, 0 mbuf_jumbo_9k: 9216, 6400, 0, 0, 0, 0, 0 mbuf_jumbo_16k: 16384, 3200, 0, 0, 0, 0, 0 mbuf_ext_refcnt: 4, 0, 0, 0, 0, 0, 0 NetGraph items: 36, 4130, 41, 195,3783340178, 0, 0 NetGraph data items: 36, 531, 57, 474,6923698050,469071, 0 g_bio: 140, 0, 0, 5432,27292578, 0, 0 ttyinq: 152, 0, 210, 206, 1695, 0, 0 ttyoutq: 256, 0, 111, 99, 882, 0, 0 ata_request: 208, 0, 0, 1387, 6823108, 0, 0 ata_composite: 180, 0, 0, 0, 0, 0, 0 VNODE: 272, 0, 92682, 7418, 2581805, 0, 0 VNODEPOLL: 60, 0, 1, 125, 1, 0, 0 S VFS Cache: 72, 0, 99871, 6924,10213652, 0, 0 L VFS Cache: 292, 0, 326, 532, 24017, 0, 0 NAMEI: 1024, 0, 0, 12,16782213, 0, 0 NFSMOUNT: 524, 0, 0, 0, 0, 0, 0 NFSNODE: 460, 0, 0, 0, 0, 0, 0 DIRHASH: 1024, 0, 894, 2, 1138, 0, 0 Mountpoints: 648, 0, 7, 5, 7, 0, 0 pipe: 392, 0, 7, 23, 55266, 0, 0 ksiginfo: 80, 0, 59, 997, 40546, 0, 0 itimer: 220, 0, 0, 0, 0, 0, 0 KNOTE: 72, 0, 8, 151,21515268, 0, 0 socket: 412, 25605, 448, 137,15228499, 0, 0 ipq: 32, 904, 0, 226, 28329, 0, 0 udp_inpcb: 220, 25614, 5, 103,15173014, 0, 0 udpcb: 8, 25781, 5, 198,15173014, 0, 0 tcp_inpcb: 220, 25614, 218, 52, 23882, 0, 0 tcpcb: 632, 25602, 217, 47, 23882, 0, 0 tcptw: 52, 5184, 1, 215, 21093, 0, 0 syncache: 112, 15365, 0, 70, 9152, 0, 0 hostcache: 76, 15400, 47, 153, 5899, 0, 0 tcpreass: 20, 1690, 0, 169, 2322, 0, 0 sackhole: 20, 0, 0, 169, 3943, 0, 0 sctp_ep: 864, 25600, 0, 0, 0, 0, 0 sctp_asoc: 1488, 40000, 0, 0, 0, 0, 0 sctp_laddr: 24, 80040, 0, 145, 4, 0, 0 sctp_raddr: 432, 80001, 0, 0, 0, 0, 0 sctp_chunk: 92, 400008, 0, 0, 0, 0, 0 sctp_readq: 76, 400000, 0, 0, 0, 0, 0 sctp_stream_msg_out: 64, 400020, 0, 0, 0, 0, 0 sctp_asconf: 24, 400055, 0, 0, 0, 0, 0 sctp_asconf_ack: 24, 400055, 0, 0, 0, 0, 0 ripcb: 220, 25614, 203, 49, 8503, 0, 0 unpcb: 172, 25622, 16, 30, 6556, 0, 0 rtentry: 108, 0, 224, 64, 7644, 0, 0 IPFW dynamic rule: 108, 0, 0, 0, 0, 0, 0 divcb: 220, 25614, 0, 0, 0, 0, 0 selfd: 28, 0, 330, 178,5185648874, 0, 0 SWAPMETA: 276, 121576, 0, 0, 0, 0, 0 FFS inode: 116, 0, 92544, 7413, 2577255, 0, 0 FFS1 dinode: 128, 0, 0, 0, 0, 0, 0 FFS2 dinode: 256, 0, 92544, 7416, 2577255, 0, 0 # vmstat -i interrupt total rate irq1: atkbd0 7804 0 irq16: ehci0 2860299092 3640 irq17: atapci0 6823504 8 irq23: ehci1 1571645 2 cpu0:timer 3240147978 4124 irq256: re0 151751242 193 Total 6260601265 7969 # vmstat -m Type InUse MemUse HighUse Requests Size(s) sigio 1 1K - 1 32 filedesc 63 37K - 143649 16,32,64,128,256,512,1024,2048,4096 kenv 81 7K - 85 16,32,64,128,4096 kqueue 2 5K - 7477 128,1024,4096 acpitask 1 1K - 1 1024 proc-args 30 2K - 251543 16,32,64,128,256 CAM XPT 12 2K - 26 16,32,64,1024 ithread 80 7K - 80 16,64,128 acpisem 16 2K - 16 64,128 KTRACE 100 13K - 100 128 linker 226 255K - 2535 16,32,256,1024,2048,4096 lockf 20 2K - 1648 32,64 ip6ndp 274 18K - 275 64,128 ip6opt 0 0K - 36006 128 temp 560 239K - 42233884 16,32,64,128,256,512,1024,2048,4096 devbuf 2085 2863K - 4666 16,32,64,128,256,512,1024,2048,4096 module 480 31K - 480 64,128 mtx_pool 2 8K - 2 4096 entropy 1024 64K - 1024 64 subproc 123 219K - 113562 256,4096 proc 2 8K - 2 4096 session 25 2K - 4389 64 pgrp 29 2K - 4934 64 cred 48 5K - 2604364 64,128 uidinfo 5 2K - 3962 64,1024 plimit 17 5K - 56385 256 sysctltmp 0 0K - 366073 16,32,64,128,4096 sysctloid 2844 88K - 2942 16,32,64,128 sysctl 0 0K - 242565 16,32,64 tidhash 1 8K - 1 umtx 310 30K - 310 64,128 p1003.1b 1 1K - 1 16 SWAP 2 4373K - 2 64 bus-sc 55 71K - 3758 16,32,64,128,256,512,1024,2048,4096 bus 1194 58K - 9260 16,32,64,128,256,512,1024 devstat 6 13K - 6 16,4096 eventhandler 78 4K - 78 32,64,128 kobj 329 658K - 385 2048 Per-cpu 1 1K - 1 16 rman 191 12K - 685 32,64 CAM dev queue 1 1K - 1 128 sbuf 1 1K - 7398453 16,32,64,128,256,512,1024,2048,4096 CAM queue 3 1K - 7 16 CAM SIM 1 1K - 1 128 stack 0 0K - 18 128 taskqueue 13 1K - 13 16,64 Unitno 15 1K - 638235 16,64 iov 0 0K - 26364929 16,64,128,256,4096 select 90 6K - 90 64 ioctlops 0 0K - 13608191 16,32,64,128,256,512,1024,2048,4096 msg 4 25K - 4 1024,4096 sem 4 101K - 4 1024,4096 shm 1 12K - 1 tty 22 11K - 62 512,1024,2048 pts 3 1K - 25 128 mbuf_tag 63 2K - 3394221816 32,64 shmfd 1 4K - 1 4096 pcb 32 79K - 2254279 16,32,64,512,1024,2048,4096 soname 6 1K - 27671763 16,32,128 biobuf 0 0K - 5590 2048 vfscache 1 512K - 1 cl_savebuf 0 0K - 4137 32,64 vfs_hash 1 256K - 1 vnodes 4 1K - 16 32,128 vnodemarker 0 0K - 252520 512 mount 99 4K - 225 16,32,64,128,256 BPF 272 17K - 1959 16,64,256 ether_multi 445 11K - 15290 16,32,64 ifaddr 1861 359K - 9281 16,32,64,128,256,512,2048 ifnet 273 274K - 278 64,128,256,512,1024,2048 clone 6 24K - 6 4096 arpcom 2 1K - 2 16 lltable 710 158K - 10689 128,256 ata_generic 1 1K - 1 1024 USBdev 16 5K - 16 32,128,1024 USB 32 15K - 36 16,32,64,1024,4096 ad_driver 1 1K - 1 32 ar_driver 0 0K - 6 512,2048 routetbl 378 63K - 122164 16,32,64,128,256 netgraph_msg 0 0K - 13367718 64,128,256,512,1024 netgraph_node 1989 249K - 72985 128 netgraph_hook 4740 593K - 193060 128 netgraph 1450 1194K - 56081 64,256,512,2048 igmp 272 34K - 272 128 kbdmux 6 18K - 6 16,256,1024,2048 in_multi 208 26K - 7628 128 dummynet 31 16K - 31 512 dummynet 170 86K - 170 256,1024 IpFw/IpAcct 116 13K - 254 16,32,64,128,256,512,1024 ipfw_tbl 66 9K - 80 128 sctp_iter 0 0K - 2 256 sctp_ifn 2 1K - 2 128 sctp_ifa 5 1K - 5 128 sctp_vrf 1 1K - 1 64 sctp_a_it 0 0K - 2 16 hostcache 1 16K - 1 syncache 1 72K - 1 libalias 95892 12051K - 161463174 128 sctpnat 6 40K - 6 4096 DEVFS1 78 20K - 105 256 in6_multi 20 3K - 20 16,256 DEVFS3 181 23K - 223 128 DEVFS2 77 2K - 85 16 mld 272 34K - 272 128 NFS FHA 1 1K - 1 1024 rpc 2 5K - 2 128,4096 audit_evclass 172 3K - 211 16 savedino 0 0K - 55901 256 sbdep 0 0K - 1 32 freework 0 0K - 82090 64 newdirblk 0 0K - 5704 32 dirrem 0 0K - 117641 64 mkdir 0 0K - 11180 64 diradd 0 0K - 119904 64 freefile 0 0K - 59555 32 freeblks 0 0K - 58942 64 freefrag 0 0K - 32692 64 indirdep 0 0K - 163100 64 newblk 1 64K - 247271 128 bmsafemap 1 4K - 103589 128,4096 inodedep 1 256K - 233414 256 pagedep 1 64K - 13225 128 ufs_dirhash 711 131K - 1026 16,32,64,128,256,512 ufs_mount 13 34K - 22 64,256,512,2048,4096 UMAHash 1 1K - 3 256,512,1024 DEVFS_RULE 53 13K - 53 32,256 DEVFS 26 1K - 27 16,64 vm_pgdata 2 65K - 2 64 DEVFSP 0 0K - 421 32 atkbddev 2 1K - 2 32 pfs_nodes 21 3K - 21 128 pfs_vncache 100 4K - 4396 32 apmdev 1 1K - 1 64 GEOM 62 14K - 402 16,32,64,128,512,1024 CAM periph 2 1K - 12 16,32,64,128 acpidev 111 4K - 111 32 memdesc 1 4K - 1 4096 nexusdev 4 1K - 4 16 pci_link 16 2K - 16 16,64,128 isadev 8 1K - 8 64 cdev 9 2K - 9 128 io_apic 1 1K - 1 1024 acpica 3895 205K - 1875978 16,32,64,128,256,512,1024,2048 agp 1 1K - 1 16 MCA 33 5K - 33 128 msi 9 1K - 9 64 netgraph_sock 4 1K - 16534 64 netgraph_path 0 0K - 6852803 16,32 netgraph_mppc 0 0K - 1 1024 netgraph_iface 269 17K - 269 64 netgraph_ppp 269 2152K - 269 netgraph_ksock 203 13K - 8195 64 netgraph_bpf 2472 232K - 122016 64,128 -- С уважением, Коньков mailto:kes-kes@yandex.ru ------------121D0EF2EC7CA44--