Skip site navigation (1)Skip section navigation (2)
Date:      Wed, 5 May 2004 13:27:50 -0500
From:      "adp" <dap99@i-55.com>
To:        <questions@freebsd.org>
Cc:        performance@freebsd.org
Subject:   Abnormal  network errors?
Message-ID:  <01a501c432ce$adf20e30$4b0a000a@yourqqh4336axf>

next in thread | raw e-mail | index | archive | help
I am working on some network performance issues. One of the first things I
inspected was netstat-s. This is for a FreeBSD 4.9-REL NFS and MySQL server
that has been up for 33 days. It seems to me that I have a lot of errors,
specifically with UDP (NFS related I would guess). The server is a dual P3
with 512MB of RAM, software RAID-1 (one of the Promise hybrid
hardware/kernel RAID cards), and FastEthernet. We are using just a
consumer'ish 10/100 switch. (Hope to rectify that soon enough.)

On this server I'm thinking I need two things:

1. More sockets available.
2. Larger sockbufs for send and recv.

Is this an accurate assessment?

What is "2432320 packets for unknown/unsupported protocol"? What
specifically does this mean? (In other words, what should I do to resolve
this?)

What about "921363 calls to icmp_error"?

Under tcp I have "481930 embryonic connections dropped". I assume that means
I don't have enough sockets available for when this server gets loaded.
Correct?

I am including 'ifconfig', 'netstat -m', 'netstat -s', 'nfsstat -s', 'sysctl
net' and
'sysctl kern.ipc' below.

Is there a way to see the peak sockets I have had opened? It doesn't look
like it.

P.S. I am Cc'ing performance@ since this is both a general administration
and performance-related question.

# ifconfig rl0
rl0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        inet 192.168.42.70 netmask 0xffffffe0 broadcast 192.168.42.95
        ..multiple aliases..
        ether 00:50:ba:60:4d:e5
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active

# netstat -m
34/736/10112 mbufs in use (current/peak/max):
        34 mbufs allocated to data
0/504/2528 mbuf clusters in use (current/peak/max)
1192 Kbytes allocated to network (15% of mb_map in use)
0 requests for memory denied
0 requests for memory delayed
0 calls to protocol drain routines

# netstat -s
tcp:
        332918288 packets sent
                279842528 data packets (1533896593 bytes)
                184127 data packets (216536937 bytes) retransmitted
                2 resends initiated by MTU discovery
                41010478 ack-only packets (14887613 delayed)
                0 URG only packets
                12197 window probe packets
                4899618 window update packets
                6969381 control packets
        298154198 packets received
                237883532 acks (for 1547823911 bytes)
                5548105 duplicate acks
                0 acks for unsent data
                194397219 packets (612913715 bytes) received in-sequence
                175092 completely duplicate packets (40526143 bytes)
                386 old duplicate packets
                7879 packets with some dup. data (1540154 bytes duped)
                1113196 out-of-order packets (1061944624 bytes)
                5645 packets (6794550 bytes) of data after window
                209 window probes
                3540787 window update packets
                217721 packets received after close
                2925 discarded for bad checksums
                0 discarded for bad header offset fields
                0 discarded because packet too short
        1450821 connection requests
        5141899 connection accepts
        1739 bad connection attempts
        0 listen queue overflows
        5676325 connections established (including accepts)
        6592771 connections closed (including 153553 drops)
                2325245 connections updated cached RTT on close
                2325245 connections updated cached RTT variance on close
                2124129 connections updated cached ssthresh on close
        481930 embryonic connections dropped
        232282251 segments updated rtt (of 188115355 attempts)
        2843613 retransmit timeouts
                829 connections dropped by rexmit timeout
        25922 persist timeouts
                0 connections dropped by persist timeout
        528 keepalive timeouts
                111 keepalive probes sent
                417 connections dropped by keepalive
        10677309 correct ACK header predictions
        40631044 correct data packet header predictions
        5143661 syncache entries added
                10004 retransmitted
                6951 dupsyn
                3 dropped
                5141899 completed
                0 bucket overflow
                0 cache overflow
                382 reset
                1327 stale
                0 aborted
                0 badack
                53 unreach
                0 zone failures
        0 cookies sent
        0 cookies received
udp:
        272987897 datagrams received
        0 with incomplete header
        0 with bad data length field
        870 with bad checksum
        682 with no checksum
        921363 dropped due to no socket
        0 broadcast/multicast datagrams dropped due to no socket
        19976574 dropped due to full socket buffers
        0 not for hashed pcb
        252089090 delivered
        275246074 datagrams output
ip:
        578001924 total packets received
        0 bad header checksums
        0 with size smaller than minimum
        0 with data size < data length
        0 with ip length > max ip packet size
        0 with header length < data size
        0 with data length < header length
        0 with bad options
        0 with incorrect version number
        4899083 fragments received
        4 fragments dropped (dup or out of space)
        750 fragments dropped after timeout
        842689 packets reassembled ok
        571513013 packets for this host
        2432320 packets for unknown/unsupported protocol
        0 packets forwarded (0 packets fast forwarded)
        197 packets not forwardable
        0 packets received for unknown multicast group
        0 redirects sent
        609745425 packets sent from this host
        1 packet sent with fabricated ip header
        22 output packets dropped due to no bufs, etc.
        0 output packets discarded due to no route
        1914687 output datagrams fragmented
        10496350 fragments created
        0 datagrams that can't be fragmented
        0 tunneling packets that can't find gif
        0 datagrams with bad address in header
icmp:
        921363 calls to icmp_error
        0 errors not generated 'cuz old message was icmp
        Output histogram:
                echo reply: 347036
                destination unreachable: 921363
        0 messages with bad code fields
        0 messages < minimum length
        1183 bad checksums
        0 messages with bad length
        2 multicast echo requests ignored
        0 multicast timestamp requests ignored
        Input histogram:
                echo reply: 10453
                destination unreachable: 2443161
                source quench: 47
                echo: 347038
                time exceeded: 1354
        347036 message responses generated
        0 invalid return addresses
        0 no return routes
        ICMP address mask responses are disabled
igmp:
        0 messages received
        0 messages received with too few bytes
        0 messages received with bad checksum
        0 membership queries received
        0 membership queries received with invalid field(s)
        0 membership reports received
        0 membership reports received with invalid field(s)
        0 membership reports received for groups to which we belong
        0 membership reports sent
netstat: sysctl: net.inet.pim.stats: No such file or directory

# nfsstat -s

Server Info:
  Getattr   Setattr    Lookup  Readlink      Read     Write    Create
Remove
      501    199150  80001872       280   4426274   2330054    267945
458716
   Rename      Link   Symlink     Mkdir     Rmdir   Readdir  RdirPlus
Access
   361427    190173         0      1977       451   1803697         0
114714817
    Mknod    Fsstat    Fsinfo  PathConf    Commit    GLease    Vacate
Evict
        0   4846498       234         0    398335         0         0
0
Server Ret-Failed
         76071546
Server Faults
            0
Server Cache Stats:
   Inprog      Idem  Non-idem    Misses
   341182     97307      1569 209905447
Server Lease Stats:
   Leases     PeakL   GLeases
        0         0         0
Server Write Gathering:
 WriteOps  WriteRPC   Opsaved
  2330050   2330054         4

# sysctl net
net.local.stream.sendspace: 8192
net.local.stream.recvspace: 8192
net.local.dgram.maxdgram: 2048
net.local.dgram.recvspace: 4096
net.local.inflight: 0
net.inet.ip.portrange.lowfirst: 1023
net.inet.ip.portrange.lowlast: 600
net.inet.ip.portrange.first: 1024
net.inet.ip.portrange.last: 5000
net.inet.ip.portrange.hifirst: 49152
net.inet.ip.portrange.hilast: 65535
net.inet.ip.forwarding: 0
net.inet.ip.redirect: 1
net.inet.ip.ttl: 64
net.inet.ip.rtexpire: 27
net.inet.ip.rtminexpire: 10
net.inet.ip.rtmaxcache: 128
net.inet.ip.sourceroute: 0
net.inet.ip.intr_queue_maxlen: 50
net.inet.ip.intr_queue_drops: 121
net.inet.ip.accept_sourceroute: 0
net.inet.ip.fastforwarding: 0
net.inet.ip.keepfaith: 0
net.inet.ip.subnets_are_local: 0
net.inet.ip.maxfragpackets: 79
net.inet.ip.maxfragsperpacket: 16
net.inet.ip.sendsourcequench: 0
net.inet.ip.check_interface: 0
net.inet.ip.fw.enable: 1
net.inet.ip.fw.one_pass: 1
net.inet.ip.fw.debug: 1
net.inet.ip.fw.verbose: 0
net.inet.ip.fw.verbose_limit: 0
net.inet.ip.fw.dyn_buckets: 256
net.inet.ip.fw.curr_dyn_buckets: 256
net.inet.ip.fw.dyn_count: 0
net.inet.ip.fw.dyn_max: 1000
net.inet.ip.fw.static_count: 5
net.inet.ip.fw.dyn_ack_lifetime: 300
net.inet.ip.fw.dyn_syn_lifetime: 20
net.inet.ip.fw.dyn_fin_lifetime: 1
net.inet.ip.fw.dyn_rst_lifetime: 1
net.inet.ip.fw.dyn_udp_lifetime: 10
net.inet.ip.fw.dyn_short_lifetime: 5
net.inet.ip.fw.dyn_grace_time: 10
net.inet.icmp.maskrepl: 0
net.inet.icmp.icmplim: 200
net.inet.icmp.drop_redirect: 0
net.inet.icmp.log_redirect: 0
net.inet.icmp.icmplim_output: 1
net.inet.icmp.bmcastecho: 0
net.inet.tcp.rfc1323: 1
net.inet.tcp.rfc1644: 0
net.inet.tcp.mssdflt: 512
net.inet.tcp.keepidle: 7200000
net.inet.tcp.keepintvl: 75000
net.inet.tcp.sendspace: 32768
net.inet.tcp.recvspace: 57344
net.inet.tcp.keepinit: 75000
net.inet.tcp.delacktime: 100
net.inet.tcp.log_in_vain: 0
net.inet.tcp.blackhole: 0
net.inet.tcp.delayed_ack: 1
net.inet.tcp.reass.maxsegments: 158
net.inet.tcp.reass.cursegments: 0
net.inet.tcp.reass.overflows: 0
net.inet.tcp.path_mtu_discovery: 1
net.inet.tcp.slowstart_flightsize: 1
net.inet.tcp.local_slowstart_flightsize: 4
net.inet.tcp.newreno: 1
net.inet.tcp.tcbhashsize: 512
net.inet.tcp.do_tcpdrain: 1
net.inet.tcp.pcbcount: 100
net.inet.tcp.icmp_may_rst: 1
net.inet.tcp.isn_reseed_interval: 0
net.inet.tcp.inflight_enable: 0
net.inet.tcp.inflight_debug: 0
net.inet.tcp.inflight_min: 6144
net.inet.tcp.inflight_max: 1073725440
net.inet.tcp.inflight_stab: 20
net.inet.tcp.syncookies: 1
net.inet.tcp.syncache.bucketlimit: 30
net.inet.tcp.syncache.cachelimit: 15359
net.inet.tcp.syncache.count: 0
net.inet.tcp.syncache.hashsize: 512
net.inet.tcp.syncache.rexmtlimit: 3
net.inet.tcp.msl: 30000
net.inet.tcp.rexmit_min: 1000
net.inet.tcp.rexmit_slop: 200
net.inet.tcp.always_keepalive: 1
net.inet.udp.checksum: 1
net.inet.udp.maxdgram: 9216
net.inet.udp.recvspace: 41600
net.inet.udp.log_in_vain: 0
net.inet.udp.blackhole: 0
net.inet.accf.unloadable: 0
net.inet.raw.maxdgram: 8192
net.inet.raw.recvspace: 8192
net.link.generic.system.ifcount: 3
net.link.ether.inet.prune_intvl: 300
net.link.ether.inet.max_age: 1200
net.link.ether.inet.host_down_time: 20
net.link.ether.inet.maxtries: 5
net.link.ether.inet.useloopback: 1
net.link.ether.inet.proxyall: 0
net.link.ether.inet.log_arp_wrong_iface: 1
net.link.ether.inet.log_arp_movements: 1
net.link.ether.ipfw: 0

# sysctl kern.ipc
kern.ipc.maxsockbuf: 262144
kern.ipc.sockbuf_waste_factor: 8
kern.ipc.somaxconn: 128
kern.ipc.max_linkhdr: 16
kern.ipc.max_protohdr: 40
kern.ipc.max_hdr: 56
kern.ipc.max_datalen: 156
kern.ipc.nmbclusters: 2528
kern.ipc.msgmax: 16384
kern.ipc.msgmni: 40
kern.ipc.msgmnb: 2048
kern.ipc.msgtql: 40
kern.ipc.msgssz: 8
kern.ipc.msgseg: 2048
kern.ipc.semmap: 30
kern.ipc.semmni: 10
kern.ipc.semmns: 60
kern.ipc.semmnu: 30
kern.ipc.semmsl: 60
kern.ipc.semopm: 100
kern.ipc.semume: 10
kern.ipc.semusz: 92
kern.ipc.semvmx: 32767
kern.ipc.semaem: 16384
kern.ipc.shmmax: 33554432
kern.ipc.shmmin: 1
kern.ipc.shmmni: 192
kern.ipc.shmseg: 128
kern.ipc.shmall: 8192
kern.ipc.shm_use_phys: 0
kern.ipc.shm_allow_removed: 0
kern.ipc.mbuf_wait: 32
kern.ipc.mbtypes: 689 47 0 0 0 0 0 0 0 0 0 0 0 0 0 0
kern.ipc.nmbufs: 10112
kern.ipc.m_clreflimithits: 0
kern.ipc.mcl_pool_max: 0
kern.ipc.mcl_pool_now: 0
kern.ipc.maxsockets: 4072







Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?01a501c432ce$adf20e30$4b0a000a>