Date: Fri, 25 May 2012 20:33:59 +0300 From: Eugen Konkov <kes-kes@yandex.ru> To: freebsd-bugs@FreeBSD.org Subject: re0 eat all CPU, but only 300-1000 Interrupts occured Message-ID: <727504748.20120525203359@yandex.ru>
next in thread | raw e-mail | index | archive | help
Tell me please how to debug why re0 take all CPU time?
last pid: 74164; load averages: 17.97, 13.81, 14.40 up 47+09:31:54 21:50:58
110 processes: 9 running, 87 sleeping, 14 waiting
CPU: 0.5% user, 0.0% nice, 1.6% system, 97.9% interrupt, 0.0% idle
Mem: 194M Active, 449M Inact, 204M Wired, 20K Cache, 112M Buf, 1143M Free
Swap: 512M Total, 512M Free
PID USERNAME PRI NICE SIZE RES STATE TIME WCPU COMMAND
12 root -76 - 0K 112K WAIT 65.3H 100.00% {irq256: re0}
65795 root 39 0 13420K 5804K select 19:58 0.63% snmpd
2664 root 36 0 14608K 6812K RUN 75:49 0.54% bgpd
0 root -76 0 0K 80K - 31.6H 0.44% {dummynet}
7758 bind 36 0 75268K 62880K RUN 178:21 0.24% {named}
12 root -44 - 0K 112K WAIT 29.4H 0.10% {swi4: clock}
# systat -v
2 users Load 13.61 12.24 13.96 May 24 21:49
Mem:KB REAL VIRTUAL VN PAGER SWAP PAGER
Tot Share Tot Share Free in out in out
Act 256100 6368 626872 8116 1171172 count
All 353960 9500 2792812 26356 pages
Proc: Interrupts
r p d s w Csw Trp Sys Int Sof Flt 3 cow 4404 total
12 32 2207 106 4216 4404 3983 48 28 zfod 1 ata0 14
ozfod 4119 hpet0 20
4.1%Sys 93.4%Intr 2.6%User 0.0%Nice 0.0%Idle %ozfod uhci0 ehci
| | | | | | | | | | | daefr 284 re0 256
==+++++++++++++++++++++++++++++++++++++++++++++++> 7 prcfr
40 dtbuf 44 totfr
Namei Name-cache Dir-cache 111725 desvn react
Calls hits % hits % 64097 numvn pdwak
3177 3177 100 27926 frevn pdpgs
intrn
Disks ad1 da0 pass0 209000 wire
KB/t 14.85 0.00 0.00 198616 act
tps 1 0 0 459200 inact
MB/s 0.01 0.00 0.00 20 cache
%busy 2 0 0 1169956 free
# vmstat -i
interrupt total rate
irq14: ata0 5946124 1
irq20: hpet0 3979660990 971
irq23: uhci0 ehci0 85 0
irq256: re0 3847013663 939
Total 7832620862 1912
# uptime
9:44PM up 47 days, 9:25, 2 users, load averages: 9.05, 14.17, 15.61
#bwm-ng -I re0
bwm-ng v0.6 (probing every 0.500s), press 'h' for help
input: getifaddrs type: rate
- iface Rx Tx Total
==============================================================================
re0: 13.32 Mb/s 12.42 Mb/s 25.74 Mb/s
------------------------------------------------------------------------------
total: 14.95 Mb/s 13.94 Mb/s 28.89 Mb/s
# vmstat -z
ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP
UMA Kegs: 128, 0, 92, 28, 92, 0, 0
UMA Zones: 224, 0, 92, 10, 92, 0, 0
UMA Slabs: 284, 0, 1158, 4, 29780, 0, 0
UMA RCntSlabs: 544, 0, 919, 5, 919, 0, 0
UMA Hash: 128, 0, 2, 28, 3, 0, 0
16 Bucket: 76, 0, 36, 14, 36, 0, 0
32 Bucket: 140, 0, 43, 13, 44, 0, 0
64 Bucket: 268, 0, 59, 11, 67, 90, 0
128 Bucket: 524, 0, 895, 1, 27120, 585, 0
VM OBJECT: 136, 0, 37976, 130,40520105, 0, 0
MAP: 140, 0, 7, 49, 7, 0, 0
KMAP ENTRY: 72, 57505, 32, 127, 79899, 0, 0
MAP ENTRY: 72, 0, 2094, 1192,100480394, 0, 0
DP fakepg: 72, 0, 0, 0, 0, 0, 0
SG fakepg: 72, 0, 0, 0, 0, 0, 0
mt_zone: 2060, 0, 268, 11, 268, 0, 0
16: 16, 0, 5106, 375,251057215, 0, 0
32: 32, 0, 26924, 196,1931894354, 0, 0
64: 64, 0, 6372, 413,172309547, 0, 0
128: 128, 0, 23931, 41019,195195229, 0, 0
256: 256, 0, 2786, 289,2302309652, 0, 0
512: 512, 0, 638, 58, 4189195, 0, 0
1024: 1024, 0, 533, 15, 1586826, 0, 0
2048: 2048, 0, 251, 59, 17903, 0, 0
4096: 4096, 0, 154, 29, 2738963, 0, 0
Files: 56, 0, 180, 289,224690270, 0, 0
TURNSTILE: 72, 0, 156, 24, 156, 0, 0
umtx pi: 52, 0, 0, 0, 0, 0, 0
MAC labels: 20, 0, 0, 0, 0, 0, 0
PROC: 688, 0, 57, 23, 1272718, 0, 0
THREAD: 728, 0, 124, 31, 206859, 0, 0
SLEEPQUEUE: 44, 0, 156, 80, 156, 0, 0
VMSPACE: 232, 0, 38, 30, 1272743, 0, 0
cpuset: 40, 0, 2, 182, 2, 0, 0
audit_record: 816, 0, 0, 0, 0, 0, 0
mbuf_packet: 256, 0, 256, 1536,5400971809, 0, 0
mbuf: 256, 0, 6, 1412,9019029846, 0, 0
mbuf_cluster: 2048, 25600, 1792, 6, 1792, 0, 0
mbuf_jumbo_page: 4096, 12800, 0, 20, 828696, 0, 0
mbuf_jumbo_9k: 9216, 6400, 0, 0, 0, 0, 0
mbuf_jumbo_16k: 16384, 3200, 0, 0, 0, 0, 0
mbuf_ext_refcnt: 4, 0, 0, 0, 0, 0, 0
NetGraph items: 36, 4130, 2, 175,1535407748, 0, 0
NetGraph data items: 36, 531, 0, 531,3322859915,26160, 0
g_bio: 140, 0, 0, 4144,23787066, 0, 0
ttyinq: 152, 0, 300, 12, 675, 0, 0
ttyoutq: 256, 0, 157, 8, 353, 0, 0
ata_request: 204, 0, 0, 1045, 5946685, 0, 0
ata_composite: 180, 0, 0, 0, 0, 0, 0
VNODE: 272, 0, 64112, 134, 7873012, 0, 0
VNODEPOLL: 60, 0, 0, 0, 0, 0, 0
S VFS Cache: 72, 0, 66367, 466, 8051562, 0, 0
L VFS Cache: 292, 0, 595, 198, 109305, 0, 0
NAMEI: 1024, 0, 0, 12,171046456, 0, 0
NFSMOUNT: 532, 0, 0, 0, 0, 0, 0
NFSNODE: 460, 0, 0, 0, 0, 0, 0
DIRHASH: 1024, 0, 485, 11, 497, 0, 0
pipe: 392, 0, 10, 30, 784482, 0, 0
Mountpoints: 648, 0, 6, 6, 6, 0, 0
ksiginfo: 80, 0, 66, 990, 210783, 0, 0
itimer: 220, 0, 0, 0, 0, 0, 0
KNOTE: 72, 0, 8, 310,92381338, 0, 0
socket: 416, 25605, 73, 242,131092823, 0, 0
unpcb: 172, 25622, 26, 43, 3331350, 0, 0
ipq: 32, 904, 0, 226, 8756, 0, 0
udp_inpcb: 220, 25614, 7, 245,127020097, 0, 0
udpcb: 8, 25781, 7, 399,127020097, 0, 0
tcp_inpcb: 220, 25614, 33, 57, 644727, 0, 0
tcpcb: 692, 25600, 26, 39, 644727, 0, 0
tcptw: 52, 5184, 7, 137, 92058, 0, 0
syncache: 112, 15365, 0, 70, 360404, 0, 0
hostcache: 76, 15400, 30, 70, 3515, 0, 0
tcpreass: 20, 1690, 0, 169, 1927, 0, 0
sackhole: 20, 0, 0, 169, 620, 0, 0
sctp_ep: 880, 25600, 0, 0, 0, 0, 0
sctp_asoc: 1524, 40000, 0, 0, 0, 0, 0
sctp_laddr: 24, 80040, 0, 145, 45, 0, 0
sctp_raddr: 464, 80000, 0, 0, 0, 0, 0
sctp_chunk: 96, 400000, 0, 0, 0, 0, 0
sctp_readq: 76, 400000, 0, 0, 0, 0, 0
sctp_stream_msg_out: 72, 400044, 0, 0, 0, 0, 0
sctp_asconf: 24, 400055, 0, 0, 0, 0, 0
sctp_asconf_ack: 24, 400055, 0, 0, 0, 0, 0
ripcb: 220, 25614, 6, 48, 78019, 0, 0
rtentry: 108, 0, 24066, 18, 288480, 0, 0
IPFW dynamic rule: 108, 0, 0, 0, 0, 0, 0
divcb: 220, 25614, 0, 0, 0, 0, 0
selfd: 28, 0, 158, 223,1786944834, 0, 0
ip4flow: 40, 25668, 680, 24160,36376221, 0, 0
ip6flow: 64, 25636, 0, 174, 20, 0, 0
SWAPMETA: 276, 121576, 0, 0, 0, 0, 0
FFS inode: 116, 0, 64073, 46, 7872685, 0, 0
FFS1 dinode: 128, 0, 0, 0, 0, 0, 0
FFS2 dinode: 256, 0, 64073, 22, 7872685, 0, 0
# uname -a
FreeBSD bor 9.0-CURRENT FreeBSD 9.0-CURRENT #1: Wed Feb 23 09:39:22 UTC 2011 @:/usr/obj/usr/src/sys/KES_KERN_v9 i386
#tail /var/log/messages
May 24 21:58:57 bor kernel: Limiting icmp unreach response from 273 to 200 packets/sec
May 24 21:58:58 bor kernel: Limiting icmp unreach response from 223 to 200 packets/sec
May 24 21:58:59 bor kernel: Limiting icmp unreach response from 235 to 200 packets/sec
May 24 21:59:00 bor kernel: Limiting icmp unreach response from 278 to 200 packets/sec
May 24 21:59:01 bor kernel: Limiting icmp unreach response from 283 to 200 packets/sec
May 24 21:59:02 bor kernel: Limiting icmp unreach response from 255 to 200 packets/sec
May 24 21:59:03 bor kernel: Limiting icmp unreach response from 269 to 200 packets/sec
May 24 21:59:04 bor kernel: Limiting icmp unreach response from 286 to 200 packets/sec
May 24 21:59:05 bor kernel: Limiting icmp unreach response from 225 to 200 packets/sec
May 24 21:59:06 bor kernel: Limiting icmp unreach response from 328 to 200 packets/sec
May 24 21:59:07 bor kernel: Limiting icmp unreach response from 285 to 200 packets/sec
May 24 21:59:08 bor kernel: Limiting icmp unreach response from 286 to 200 packets/sec
May 24 21:59:09 bor kernel: Limiting icmp unreach response from 304 to 200 packets/sec
May 24 21:59:10 bor kernel: Limiting icmp unreach response from 304 to 200 packets/sec
May 24 21:59:11 bor kernel: Limiting icmp unreach response from 311 to 200 packets/sec
Thank you.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?727504748.20120525203359>