From owner-freebsd-net@freebsd.org Mon Jun 3 18:15:38 2019 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B32DD15B95DE; Mon, 3 Jun 2019 18:15:38 +0000 (UTC) (envelope-from ler@lerctr.org) Received: from thebighonker.lerctr.org (ns-b.lerctr.org [IPv6:2001:470:1f0f:3ad::53:2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "*.lerctr.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 976CC8E425; Mon, 3 Jun 2019 18:15:37 +0000 (UTC) (envelope-from ler@lerctr.org) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lerctr.org; s=ler2019; h=Message-ID:References:In-Reply-To:Subject:Cc:To:From:Date: Content-Transfer-Encoding:Content-Type:MIME-Version:Sender:Reply-To: Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender: Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe: List-Subscribe:List-Post:List-Owner:List-Archive; bh=8/JByQtNoY3MI8e+9HMu8RqWmM2csS4ej2fCZaFT4Yw=; b=LUi+451zFoYUBPwH4XTE+a6NH7 nEvmwnNtaybzAuO76H1h2sLI//PBkwh31ynU9aU58H4/Lk68dQrjwW6o/AIlEEAlaQFHyeaoHb6E/ 2X8xPYzhh9ZkMFmGTgX+C6tsND0tUx+qysb6S3VxwB2ulVwC9jfAGaPtyRpyqz+pdI8x79Ur9y7Ej lpIdFlcoWifaE1V8HvegQsan7oNrawjrAxHcOXGrQ7RaOLQJYToZaxEJpSEEmbxYJ8GTDWh0CaSM4 YlKv4kx5ZRgNl4yJ5wc3XMAfIMb/er909n4Z2xIwR6ma1d2B+pYpVMZ9zF357EBsERHRvX5RA8sgm 9+E8+T8A==; Received: from thebighonker.lerctr.org ([2001:470:1f0f:3ad:bb:dcff:fe50:d900]:49379 helo=webmail.lerctr.org) by thebighonker.lerctr.org with esmtpsa (TLSv1.3:TLS_AES_256_GCM_SHA384:256) (Exim 4.92 (FreeBSD)) (envelope-from ) id 1hXrUe-000LFv-I2; Mon, 03 Jun 2019 13:15:36 -0500 Received: from 2600:1700:210:b180:c89b:801e:ddd7:a6bb by webmail.lerctr.org with HTTP (HTTP/1.1 POST); Mon, 03 Jun 2019 13:15:36 -0500 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Date: Mon, 03 Jun 2019 13:15:36 -0500 From: Larry Rosenman To: Eugene Grosbein Cc: Freebsd net , owner-freebsd-net@freebsd.org Subject: Re: SNMP/No Bufferspace In-Reply-To: References: <8dba5eaea800356d279eb81d7d8b2a30@lerctr.org> Message-ID: <03051c226dadc9907dcd0839d5025e18@lerctr.org> X-Sender: ler@lerctr.org User-Agent: Roundcube Webmail/1.3.9 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jun 2019 18:15:38 -0000 On 06/03/2019 1:09 pm, Eugene Grosbein wrote: > 04.06.2019 0:49, Larry Rosenman wrote: > >> On 06/03/2019 12:23 pm, Eugene Grosbein wrote: >>> 03.06.2019 22:56, Larry Rosenman wrote: >>> >>>> I have a mrtg job that runs every 5 minutes on a FreeBSD-CURRENT box >>>> and gets this randomly: >>>> >>>> SNMP Error: >>>> send_query: No buffer space available >>> >>> [skip] >>> >>>> How can I debug this? >>> >>> Your outgoing network interface stalls for some reason. >>> Where is this traffic directed to? Some kind of VPN tunnel? Physical >>> NIC that looses link? >>> Some Wifi or mobile network? >>> >> physical NIC on the same physical network (all Ubiquiti gear). >> >> >> ⌂66% [ler@borg.lerctr.org:~] $ ifconfig >> lo0: flags=8049 metric 0 mtu 16384 >> options=680003 >> inet6 ::1 prefixlen 128 >> inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 >> inet 127.0.0.1 netmask 0xff000000 >> groups: lo >> nd6 options=21 >> bce0: flags=8943 >> metric 0 mtu 1500 >> >> options=c01bb >> ether a4:ba:db:29:66:95 >> media: Ethernet autoselect (1000baseT ) >> status: active >> nd6 options=29 >> bce1: flags=8802 metric 0 mtu 1500 >> >> options=c01bb >> ether a4:ba:db:29:66:97 >> media: Ethernet autoselect >> nd6 options=29 >> bce2: flags=8802 metric 0 mtu 1500 >> >> options=c01bb >> ether a4:ba:db:29:66:99 >> media: Ethernet autoselect >> nd6 options=29 >> bce3: flags=8802 metric 0 mtu 1500 >> >> options=c01bb >> ether a4:ba:db:29:66:9b >> media: Ethernet autoselect >> nd6 options=29 >> bridge0: flags=8843 metric 0 >> mtu 1500 >> ether 02:d7:b8:51:f2:00 >> inet6 fe80::d7:b8ff:fe51:f200%bridge0 prefixlen 64 scopeid 0x6 >> inet6 2600:1700:210:b180:d7:b8ff:fe51:f200 prefixlen 64 autoconf >> inet 192.168.200.4 netmask 0xfffffc00 broadcast 192.168.203.255 >> id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 >> maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 >> root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 >> member: bce0 flags=143 >> ifmaxaddr 0 port 2 priority 128 path cost 55 >> groups: bridge >> nd6 options=23 >> ⌂68% [ler@borg.lerctr.org:~] $ >> >> bridge0 via bce0 to the "main-switch" . and all the >> AP's,switches, etc. > > You should search system logs for possible link problems > and check counters shown by "netstat -idnh" (errs/drop/coll). > > Also check counters by "netstat -m" for mbufs/mbuf clusters. ⌂79% [ler@borg.lerctr.org:~] $ netstat -idnh Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs Coll Drop lo0 16K lo0 27M 0 0 27M 0 0 0 lo0 - ::1/128 ::1 582k - - 582k - - - lo0 - fe80::%lo0/64 fe80::1%lo0 0 - - 0 - - - lo0 - 127.0.0.0/8 127.0.0.1 190k - - 26M - - - bce0 1.5K a4:ba:db:29:66:95 715M 186k 0 533M 0 0 15M bce1* 1.5K a4:ba:db:29:66:97 0 0 0 0 0 0 0 bce2* 1.5K a4:ba:db:29:66:99 0 0 0 0 0 0 0 bce3* 1.5K a4:ba:db:29:66:9b 0 0 0 0 0 0 0 bridg 1.5K 02:d7:b8:51:f2:00 715M 0 0 533M 15M 0 0 bridg - fe80::%bridge fe80::d7:b8ff:fe5 7.2k - - 7.2k - - - bridg - 2600:1700:210 2600:1700:210:b18 578M - - 476M - - - bridg - 192.168.200.0 192.168.200.4 161M - - 72M - - - ⌂86% [ler@borg.lerctr.org:~] $ netstat -m 3212/18193/21405 mbufs in use (current/cache/total) 2078/11604/13682/8175421 mbuf clusters in use (current/cache/total/max) 2045/8075 mbuf+clusters out of packet secondary zone in use (current/cache) 6/9355/9361/4087710 4k (page size) jumbo clusters in use (current/cache/total/max) 0/0/0/1211173 9k jumbo clusters in use (current/cache/total/max) 0/0/0/681285 16k jumbo clusters in use (current/cache/total/max) 4983K/65176K/70159K bytes allocated to network (current/cache/total) 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters delayed (4k/9k/16k) 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 0 sendfile syscalls 0 sendfile syscalls completed without I/O request 0 requests for I/O initiated by sendfile 0 pages read by sendfile as part of a request 0 pages were valid at time of a sendfile request 0 pages were valid and substituted to bogus page 0 pages were requested for read ahead by applications 0 pages were read ahead by sendfile 0 times sendfile encountered an already busy page 0 requests for sfbufs denied 0 requests for sfbufs delayed ⌂83% [ler@borg.lerctr.org:~] $ Hrm. I may be able to move directly away from the bridge..... ⌂83% [ler@borg.lerctr.org:~] $ grep bce0 /var/log/messages ⌂80% [ler@borg.lerctr.org:~] 1 $ grep bridge /var/log/messages Jun 3 02:48:30 borg kernel: arp: 192.168.200.54 moved from 04:c9:d9:63:22:87 to 04:c9:d9:63:22:89 on bridge0 Jun 3 02:49:49 borg kernel: arp: 192.168.200.54 moved from 04:c9:d9:63:22:89 to 04:c9:d9:63:22:87 on bridge0 ⌂83% [ler@borg.lerctr.org:~] $ uptime 1:15PM up 1 day, 18:11, 1 user, load averages: 20.08, 18.82, 18.13 ⌂83% [ler@borg.lerctr.org:~] $ -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 E-Mail: ler@lerctr.org US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106