Date: Sat, 4 Dec 2010 20:22:50 +0200 From: =?windows-1251?B?yu7t/Oru4iDF4uPl7ejp?= <kes-kes@yandex.ru> To: freebsd-bugs@freebsd.org Subject: Something wrong in network subsystem Message-ID: <1356153913.20101204202250@yandex.ru>
next in thread | raw e-mail | index | archive | help
------------121D0EF2EC7CA44
Content-Type: text/plain; charset=windows-1251
Content-Transfer-Encoding: 8bit
Hi, Freebsd-bugs.
server is under usual load, but in one momenth something is broke and
it has 80% interrupt. (see graph)
does this relate to BUG?
detail are below
# uname -a
FreeBSD 9.0-CURRENT FreeBSD 9.0-CURRENT #7: Wed Nov 24 11:39:13 EET 2010 @:/usr/obj/usr/src/sys/KES_KERN_v9 i386
# uptime
20:19 up 9 days, 2:17, 1 user, load averages: 1,42 1,40 1,37
# vmstat -i
interrupt total rate
irq1: atkbd0 7804 0
irq16: ehci0 2902776774 3694
irq17: atapci0 6825719 8
irq23: ehci1 1572029 2
cpu0:timer 3240929814 4124
irq256: re0 154845556 197
Total 6306957696 8026
#top -SIHP
last pid: 13539; load averages: 1.37, 1.38, 1.34 up 9+02:06:10 20:08:55
82 processes: 3 running, 64 sleeping, 15 waiting
CPU: 0.0% user, 0.0% nice, 9.8% system, 76.9% interrupt, 13.3% idle
Mem: 58M Active, 758M Inact, 210M Wired, 208K Cache, 112M Buf, 901M Free
Swap: 32G Total, 32G Free
PID USERNAME PRI NICE SIZE RES STATE TIME WCPU COMMAND
12 root -64 - 0K 120K WAIT 170:43 41.85% {irq16: ehci0}
12 root -28 - 0K 120K WAIT 32.5H 34.13% {swi5: +}
11 root 171 ki31 0K 8K RUN 168.7H 17.63% idle
13 root 53 - 0K 8K RUN 531:33 7.91% ng_queue
systat -v
1 users Load 1,27 1,36 1,33 4 дек 20:09
Mem:KB REAL VIRTUAL VN PAGER SWAP PAGER
Tot Share Tot Share Free in out in out
Act 95292 4968 382456 6368 922588 count
All 261588 7652 2540372 16716 pages
Proc: Interrupts
r p d s w Csw Trp Sys Int Sof Flt cow 243k total
32 484k 64 594 238k 33k 8 zfod atkbd0 1
ozfod 222k ehci0 16
9,4%Sys 78,5%Intr 0,3%User 0,0%Nice 11,8%Idle %ozfod 11 atapci0 17
| | | | | | | | | | | daefr 2 ehci1 23
=====+++++++++++++++++++++++++++++++++++++++ prcfr 4121 cpu0:timer
31 dtbuf 2 totfr 16750 re0 256
Namei Name-cache Dir-cache 110738 desvn react
Calls hits % hits % 92681 numvn pdwak
3 3 100 27684 frevn pdpgs
intrn
Disks ad4 214984 wire
KB/t 16,00 59440 act
tps 11 776476 inact
MB/s 0,17 204 cache
%busy 0 922384 free
# netstat -w 1 -h -d -I re0
input (re0) output
packets errs idrops bytes packets errs bytes colls drops
14K 0 0 8.5M 14K 0 8.0M 0 0
15K 0 0 9.4M 15K 0 8.8M 0 0
15K 0 0 9.1M 15K 0 8.5M 0 0
15K 0 0 9.7M 16K 0 9.0M 0 0
16K 0 0 10M 16K 0 9.3M 0 0
16K 0 0 10M 17K 0 9.3M 0 0
16K 0 0 10M 16K 0 9.3M 0 0
16K 0 0 11M 17K 0 9.9M 0 0
16K 0 0 10M 17K 0 9.6M 0 0
13K 0 0 7.8M 13K 0 7.3M 0 0
13K 0 0 8.0M 14K 0 7.5M 0 0
13K 0 0 7.8M 14K 0 7.4M 0 0
# netstat -s -p tcp
tcp:
3423432 packets sent
1858123 data packets (48135556 bytes)
26011 data packets (609532 bytes) retransmitted
1372 data packets unnecessarily retransmitted
0 resends initiated by MTU discovery
1497899 ack-only packets (1435187 delayed)
0 URG only packets
0 window probe packets
2136 window update packets
39450 control packets
3831803 packets received
1842790 acks (for 48124547 bytes)
22529 duplicate acks
0 acks for unsent data
1900424 packets (226227761 bytes) received in-sequence
1490 completely duplicate packets (38592 bytes)
0 old duplicate packets
6 packets with some dup. data (136 bytes duped)
2322 out-of-order packets (2982496 bytes)
9 packets (0 bytes) of data after window
0 window probes
69 window update packets
38 packets received after close
0 discarded for bad checksums
0 discarded for bad header offset fields
0 discarded because packet too short
0 discarded due to memory problems
15318 connection requests
8441 connection accepts
0 bad connection attempts
0 listen queue overflows
128 ignored RSTs in the windows
23736 connections established (including accepts)
23656 connections closed (including 2121 drops)
21705 connections updated cached RTT on close
23094 connections updated cached RTT variance on close
16478 connections updated cached ssthresh on close
21 embryonic connections dropped
1821810 segments updated rtt (of 1815670 attempts)
26393 retransmit timeouts
1140 connections dropped by rexmit timeout
0 persist timeouts
0 connections dropped by persist timeout
0 Connections (fin_wait_2) dropped because of timeout
10 keepalive timeouts
9 keepalive probes sent
1 connection dropped by keepalive
379 correct ACK header predictions
178731 correct data packet header predictions
9149 syncache entries added
337 retransmitted
171 dupsyn
0 dropped
8441 completed
0 bucket overflow
0 cache overflow
629 reset
79 stale
0 aborted
0 badack
0 unreach
0 zone failures
9149 cookies sent
0 cookies received
41 SACK recovery episodes
36 segment rexmits in SACK recovery episodes
10768 byte rexmits in SACK recovery episodes
8493 SACK options (SACK blocks) received
2157 SACK options (SACK blocks) sent
0 SACK scoreboard overflow
0 packets with ECN CE bit set
0 packets with ECN ECT(0) bit set
0 packets with ECN ECT(1) bit set
0 successful ECN handshakes
0 times ECN reduced the congestion window
# netstat -Q
Configuration:
Setting Current Limit
Thread count 1 1
Default queue limit 256 10240
Direct dispatch enabled n/a
Forced direct dispatch enabled n/a
Threads bound to CPUs disabled n/a
Protocols:
Name Proto QLimit Policy Flags
ip 1 256 flow ---
igmp 2 256 source ---
rtsock 3 256 source ---
arp 7 256 source ---
ip6 10 256 flow ---
Workstreams:
WSID CPU Name Len WMark Disp'd HDisp'd QDrops Queued Handled
0 0 ip 0 1 9280761530 0 0 85388 9280846918
igmp 0 0 0 0 0 0 0
rtsock 0 1 0 0 0 10 10
arp 0 0 2341888 0 0 0 2341888
ip6 0 0 107430 0 0 0 107430
# ifconfig -m re0
re0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=389b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC>
capabilities=389b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC>
ether 00:30:67:5a:44:72
inet I.N.E.T netmask 0xfffffc00 broadcast 10.X.X.X
inet6 fe80::I.N.E.T%re0 prefixlen 64 scopeid 0x1
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
supported media:
media autoselect
media 1000baseT mediaopt full-duplex
media 1000baseT
media 100baseTX mediaopt full-duplex
media 100baseTX
media 10baseT/UTP mediaopt full-duplex
media 10baseT/UTP
media none
# vmstat -z
ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP
UMA Kegs: 128, 0, 90, 0, 90, 0, 0
UMA Zones: 896, 0, 90, 2, 90, 0, 0
UMA Slabs: 284, 0, 1800, 6, 3785559, 0, 0
UMA RCntSlabs: 544, 0, 2252, 2, 2252, 0, 0
UMA Hash: 128, 0, 3, 27, 4, 0, 0
16 Bucket: 76, 0, 37, 13, 58, 0, 0
32 Bucket: 140, 0, 61, 23, 81, 0, 0
64 Bucket: 268, 0, 91, 7, 142, 13, 0
128 Bucket: 524, 0, 2816, 5, 472941,8127, 0
VM OBJECT: 136, 0, 65515, 8145, 2255395, 0, 0
MAP: 136, 0, 7, 22, 7, 0, 0
KMAP ENTRY: 72, 57505, 30, 129, 8723924, 0, 0
MAP ENTRY: 72, 0, 764, 561, 5406137, 0, 0
DP fakepg: 72, 0, 0, 0, 0, 0, 0
SG fakepg: 72, 0, 0, 0, 0, 0, 0
mt_zone: 2060, 0, 301, 121, 301, 0, 0
16: 16, 0, 3967, 296,51165091, 0, 0
32: 32, 0, 3925, 27715,3398801221, 0, 0
64: 64, 0, 8928, 90015,22166775, 0, 0
128: 128, 0, 107721, 11679,183686782, 0, 0
256: 256, 0, 2619, 46161, 732044, 0, 0
512: 512, 0, 1000, 64,10976314, 0, 0
1024: 1024, 0, 362, 18,13018880, 0, 0
2048: 2048, 0, 784, 960,18518183, 0, 0
4096: 4096, 0, 100, 29, 608770, 0, 0
Files: 56, 0, 320, 149,22304433, 0, 0
TURNSTILE: 72, 0, 156, 24, 156, 0, 0
umtx pi: 52, 0, 0, 0, 0, 0, 0
MAC labels: 20, 0, 0, 0, 0, 0, 0
PROC: 680, 0, 49, 23, 113477, 0, 0
THREAD: 716, 0, 104, 51, 319099, 0, 0
SLEEPQUEUE: 44, 0, 156, 80, 156, 0, 0
VMSPACE: 236, 0, 30, 34, 113447, 0, 0
cpuset: 40, 0, 2, 182, 2, 0, 0
audit_record: 816, 0, 0, 0, 0, 0, 0
mbuf_packet: 256, 0, 336, 4144,6244701687, 0, 0
mbuf: 256, 0, 74, 3906,13729996172, 0, 0
mbuf_cluster: 2048, 25600, 4480, 6, 4480, 0, 0
mbuf_jumbo_page: 4096, 12800, 0, 9, 845, 0, 0
mbuf_jumbo_9k: 9216, 6400, 0, 0, 0, 0, 0
mbuf_jumbo_16k: 16384, 3200, 0, 0, 0, 0, 0
mbuf_ext_refcnt: 4, 0, 0, 0, 0, 0, 0
NetGraph items: 36, 4130, 41, 195,3783340178, 0, 0
NetGraph data items: 36, 531, 57, 474,6923698050,469071, 0
g_bio: 140, 0, 0, 5432,27292578, 0, 0
ttyinq: 152, 0, 210, 206, 1695, 0, 0
ttyoutq: 256, 0, 111, 99, 882, 0, 0
ata_request: 208, 0, 0, 1387, 6823108, 0, 0
ata_composite: 180, 0, 0, 0, 0, 0, 0
VNODE: 272, 0, 92682, 7418, 2581805, 0, 0
VNODEPOLL: 60, 0, 1, 125, 1, 0, 0
S VFS Cache: 72, 0, 99871, 6924,10213652, 0, 0
L VFS Cache: 292, 0, 326, 532, 24017, 0, 0
NAMEI: 1024, 0, 0, 12,16782213, 0, 0
NFSMOUNT: 524, 0, 0, 0, 0, 0, 0
NFSNODE: 460, 0, 0, 0, 0, 0, 0
DIRHASH: 1024, 0, 894, 2, 1138, 0, 0
Mountpoints: 648, 0, 7, 5, 7, 0, 0
pipe: 392, 0, 7, 23, 55266, 0, 0
ksiginfo: 80, 0, 59, 997, 40546, 0, 0
itimer: 220, 0, 0, 0, 0, 0, 0
KNOTE: 72, 0, 8, 151,21515268, 0, 0
socket: 412, 25605, 448, 137,15228499, 0, 0
ipq: 32, 904, 0, 226, 28329, 0, 0
udp_inpcb: 220, 25614, 5, 103,15173014, 0, 0
udpcb: 8, 25781, 5, 198,15173014, 0, 0
tcp_inpcb: 220, 25614, 218, 52, 23882, 0, 0
tcpcb: 632, 25602, 217, 47, 23882, 0, 0
tcptw: 52, 5184, 1, 215, 21093, 0, 0
syncache: 112, 15365, 0, 70, 9152, 0, 0
hostcache: 76, 15400, 47, 153, 5899, 0, 0
tcpreass: 20, 1690, 0, 169, 2322, 0, 0
sackhole: 20, 0, 0, 169, 3943, 0, 0
sctp_ep: 864, 25600, 0, 0, 0, 0, 0
sctp_asoc: 1488, 40000, 0, 0, 0, 0, 0
sctp_laddr: 24, 80040, 0, 145, 4, 0, 0
sctp_raddr: 432, 80001, 0, 0, 0, 0, 0
sctp_chunk: 92, 400008, 0, 0, 0, 0, 0
sctp_readq: 76, 400000, 0, 0, 0, 0, 0
sctp_stream_msg_out: 64, 400020, 0, 0, 0, 0, 0
sctp_asconf: 24, 400055, 0, 0, 0, 0, 0
sctp_asconf_ack: 24, 400055, 0, 0, 0, 0, 0
ripcb: 220, 25614, 203, 49, 8503, 0, 0
unpcb: 172, 25622, 16, 30, 6556, 0, 0
rtentry: 108, 0, 224, 64, 7644, 0, 0
IPFW dynamic rule: 108, 0, 0, 0, 0, 0, 0
divcb: 220, 25614, 0, 0, 0, 0, 0
selfd: 28, 0, 330, 178,5185648874, 0, 0
SWAPMETA: 276, 121576, 0, 0, 0, 0, 0
FFS inode: 116, 0, 92544, 7413, 2577255, 0, 0
FFS1 dinode: 128, 0, 0, 0, 0, 0, 0
FFS2 dinode: 256, 0, 92544, 7416, 2577255, 0, 0
# vmstat -i
interrupt total rate
irq1: atkbd0 7804 0
irq16: ehci0 2860299092 3640
irq17: atapci0 6823504 8
irq23: ehci1 1571645 2
cpu0:timer 3240147978 4124
irq256: re0 151751242 193
Total 6260601265 7969
# vmstat -m
Type InUse MemUse HighUse Requests Size(s)
sigio 1 1K - 1 32
filedesc 63 37K - 143649 16,32,64,128,256,512,1024,2048,4096
kenv 81 7K - 85 16,32,64,128,4096
kqueue 2 5K - 7477 128,1024,4096
acpitask 1 1K - 1 1024
proc-args 30 2K - 251543 16,32,64,128,256
CAM XPT 12 2K - 26 16,32,64,1024
ithread 80 7K - 80 16,64,128
acpisem 16 2K - 16 64,128
KTRACE 100 13K - 100 128
linker 226 255K - 2535 16,32,256,1024,2048,4096
lockf 20 2K - 1648 32,64
ip6ndp 274 18K - 275 64,128
ip6opt 0 0K - 36006 128
temp 560 239K - 42233884 16,32,64,128,256,512,1024,2048,4096
devbuf 2085 2863K - 4666 16,32,64,128,256,512,1024,2048,4096
module 480 31K - 480 64,128
mtx_pool 2 8K - 2 4096
entropy 1024 64K - 1024 64
subproc 123 219K - 113562 256,4096
proc 2 8K - 2 4096
session 25 2K - 4389 64
pgrp 29 2K - 4934 64
cred 48 5K - 2604364 64,128
uidinfo 5 2K - 3962 64,1024
plimit 17 5K - 56385 256
sysctltmp 0 0K - 366073 16,32,64,128,4096
sysctloid 2844 88K - 2942 16,32,64,128
sysctl 0 0K - 242565 16,32,64
tidhash 1 8K - 1
umtx 310 30K - 310 64,128
p1003.1b 1 1K - 1 16
SWAP 2 4373K - 2 64
bus-sc 55 71K - 3758 16,32,64,128,256,512,1024,2048,4096
bus 1194 58K - 9260 16,32,64,128,256,512,1024
devstat 6 13K - 6 16,4096
eventhandler 78 4K - 78 32,64,128
kobj 329 658K - 385 2048
Per-cpu 1 1K - 1 16
rman 191 12K - 685 32,64
CAM dev queue 1 1K - 1 128
sbuf 1 1K - 7398453 16,32,64,128,256,512,1024,2048,4096
CAM queue 3 1K - 7 16
CAM SIM 1 1K - 1 128
stack 0 0K - 18 128
taskqueue 13 1K - 13 16,64
Unitno 15 1K - 638235 16,64
iov 0 0K - 26364929 16,64,128,256,4096
select 90 6K - 90 64
ioctlops 0 0K - 13608191 16,32,64,128,256,512,1024,2048,4096
msg 4 25K - 4 1024,4096
sem 4 101K - 4 1024,4096
shm 1 12K - 1
tty 22 11K - 62 512,1024,2048
pts 3 1K - 25 128
mbuf_tag 63 2K - 3394221816 32,64
shmfd 1 4K - 1 4096
pcb 32 79K - 2254279 16,32,64,512,1024,2048,4096
soname 6 1K - 27671763 16,32,128
biobuf 0 0K - 5590 2048
vfscache 1 512K - 1
cl_savebuf 0 0K - 4137 32,64
vfs_hash 1 256K - 1
vnodes 4 1K - 16 32,128
vnodemarker 0 0K - 252520 512
mount 99 4K - 225 16,32,64,128,256
BPF 272 17K - 1959 16,64,256
ether_multi 445 11K - 15290 16,32,64
ifaddr 1861 359K - 9281 16,32,64,128,256,512,2048
ifnet 273 274K - 278 64,128,256,512,1024,2048
clone 6 24K - 6 4096
arpcom 2 1K - 2 16
lltable 710 158K - 10689 128,256
ata_generic 1 1K - 1 1024
USBdev 16 5K - 16 32,128,1024
USB 32 15K - 36 16,32,64,1024,4096
ad_driver 1 1K - 1 32
ar_driver 0 0K - 6 512,2048
routetbl 378 63K - 122164 16,32,64,128,256
netgraph_msg 0 0K - 13367718 64,128,256,512,1024
netgraph_node 1989 249K - 72985 128
netgraph_hook 4740 593K - 193060 128
netgraph 1450 1194K - 56081 64,256,512,2048
igmp 272 34K - 272 128
kbdmux 6 18K - 6 16,256,1024,2048
in_multi 208 26K - 7628 128
dummynet 31 16K - 31 512
dummynet 170 86K - 170 256,1024
IpFw/IpAcct 116 13K - 254 16,32,64,128,256,512,1024
ipfw_tbl 66 9K - 80 128
sctp_iter 0 0K - 2 256
sctp_ifn 2 1K - 2 128
sctp_ifa 5 1K - 5 128
sctp_vrf 1 1K - 1 64
sctp_a_it 0 0K - 2 16
hostcache 1 16K - 1
syncache 1 72K - 1
libalias 95892 12051K - 161463174 128
sctpnat 6 40K - 6 4096
DEVFS1 78 20K - 105 256
in6_multi 20 3K - 20 16,256
DEVFS3 181 23K - 223 128
DEVFS2 77 2K - 85 16
mld 272 34K - 272 128
NFS FHA 1 1K - 1 1024
rpc 2 5K - 2 128,4096
audit_evclass 172 3K - 211 16
savedino 0 0K - 55901 256
sbdep 0 0K - 1 32
freework 0 0K - 82090 64
newdirblk 0 0K - 5704 32
dirrem 0 0K - 117641 64
mkdir 0 0K - 11180 64
diradd 0 0K - 119904 64
freefile 0 0K - 59555 32
freeblks 0 0K - 58942 64
freefrag 0 0K - 32692 64
indirdep 0 0K - 163100 64
newblk 1 64K - 247271 128
bmsafemap 1 4K - 103589 128,4096
inodedep 1 256K - 233414 256
pagedep 1 64K - 13225 128
ufs_dirhash 711 131K - 1026 16,32,64,128,256,512
ufs_mount 13 34K - 22 64,256,512,2048,4096
UMAHash 1 1K - 3 256,512,1024
DEVFS_RULE 53 13K - 53 32,256
DEVFS 26 1K - 27 16,64
vm_pgdata 2 65K - 2 64
DEVFSP 0 0K - 421 32
atkbddev 2 1K - 2 32
pfs_nodes 21 3K - 21 128
pfs_vncache 100 4K - 4396 32
apmdev 1 1K - 1 64
GEOM 62 14K - 402 16,32,64,128,512,1024
CAM periph 2 1K - 12 16,32,64,128
acpidev 111 4K - 111 32
memdesc 1 4K - 1 4096
nexusdev 4 1K - 4 16
pci_link 16 2K - 16 16,64,128
isadev 8 1K - 8 64
cdev 9 2K - 9 128
io_apic 1 1K - 1 1024
acpica 3895 205K - 1875978 16,32,64,128,256,512,1024,2048
agp 1 1K - 1 16
MCA 33 5K - 33 128
msi 9 1K - 9 64
netgraph_sock 4 1K - 16534 64
netgraph_path 0 0K - 6852803 16,32
netgraph_mppc 0 0K - 1 1024
netgraph_iface 269 17K - 269 64
netgraph_ppp 269 2152K - 269
netgraph_ksock 203 13K - 8195 64
netgraph_bpf 2472 232K - 122016 64,128
--
С уважением,
Коньков mailto:kes-kes@yandex.ru
------------121D0EF2EC7CA44--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1356153913.20101204202250>