From owner-freebsd-net@freebsd.org Tue Nov 26 06:36:47 2019 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 7F5EC1AB114 for ; Tue, 26 Nov 2019 06:36:47 +0000 (UTC) (envelope-from bsd-lists@BSDforge.com) Received: from udns.ultimatedns.net (static-24-113-41-81.wavecable.com [24.113.41.81]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "ultimatedns.net", Issuer "Let's Encrypt Authority X3" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 47MZ1p5PVkz4S70 for ; Tue, 26 Nov 2019 06:36:46 +0000 (UTC) (envelope-from bsd-lists@BSDforge.com) Received: from udns.ultimatedns.net (localhost [IPv6:0:0:0:0:0:0:0:1]) by udns.ultimatedns.net (8.15.2/8.15.2) with ESMTPS id xAQ6bX5h051005 (version=TLSv1.2 cipher=DHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO); Mon, 25 Nov 2019 22:37:39 -0800 (PST) (envelope-from bsd-lists@BSDforge.com) X-Mailer: Cypht MIME-Version: 1.0 Cc: In-Reply-To: <8daf2859-ee6a-9aa9-7420-1e1c259ad02c@viklenko.net> From: Chris Reply-To: bsd-lists@BSDforge.com To: Artem Viklenko Subject: Re: How to remove watchdog? Date: Mon, 25 Nov 2019 22:37:39 -0800 Message-Id: <08ef44b1f9698307f0384c7bf18d91af@udns.ultimatedns.net> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 47MZ1p5PVkz4S70 X-Spamd-Bar: +++ Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of bsd-lists@BSDforge.com has no SPF policy when checking 24.113.41.81) smtp.mailfrom=bsd-lists@BSDforge.com X-Spamd-Result: default: False [3.05 / 15.00]; ARC_NA(0.00)[]; HAS_REPLYTO(0.00)[bsd-lists@BSDforge.com]; XM_UA_NO_VERSION(0.01)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; IP_SCORE(-0.14)[ip: (-1.27), ipnet: 24.113.0.0/16(-0.63), asn: 11404(1.24), country: US(-0.05)]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[BSDforge.com]; REPLYTO_ADDR_EQ_FROM(0.00)[]; AUTH_NA(1.00)[]; NEURAL_SPAM_MEDIUM(0.74)[0.735,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_SPAM_LONG(0.54)[0.545,0]; R_SPF_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; SUBJECT_ENDS_QUESTION(1.00)[]; ASN(0.00)[asn:11404, ipnet:24.113.0.0/16, country:US]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Nov 2019 06:36:47 -0000 On Tue, 26 Nov 2019 08:06:37 +0200 Artem Viklenko artem@viklenko=2Enet said > Sorry, small update=2E >=20 > Just re-cheked=2E It was not final change=2E=2E=2E wrong place=2E I've set it even= =20 > smaller than 4096=2E Now it 3072=2E Bummer=2E :( >=20 > Sorry=2E No problem=2E Thanks for trying! :) --Chris >=20 > 26=2E11=2E19 07:55, Artem Viklenko via freebsd-net =D0=BF=D0=B8=D1=88=D0= =B5: > > Hi! > >=20 > > I have several small boxes with realtek nics acting as a router/firewal= l=2E > > Also had same issues=2E FreeBSD driver didn't work at least for me so I= =20 > > switched to Realtek's driver=2E But after some time traffic stops passing= =20 > > my routers=2E Did some investigation and found that the issue is 9k mbufs= =2E > > As far as I understand more traffic you push more issues with 9k mbufs > > appears due to memory fragmentation=2E > > You can check it with 'vmstat -z | grep mbuf'=2E > >=20 > > So I decided to do wery dirty hack - I've changed Jumbo_Frame_9k > > to Jumbo_Frame_4k in the if_re=2Ec from Realtek's latest 1=2E95 driver=2E > > It comiles and work on FreeBSD 10=2Ex and 11=2Ex and now it just works > > (vendor says that driver is for older versions of the FreeBSD OS)=2E > > And no more issues=2E > >=20 > > ITEM=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 SIZE=C2=A0 LIMIT=C2=A0=C2= =A0=C2=A0=C2=A0 USED=C2=A0=C2=A0=C2=A0=C2=A0 > > FREE=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 REQ FAIL SLEEP > >=20 > > mbuf_packet:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 256, 2362080,=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 2,=C2=A0= =C2=A0=C2=A0 1263, > > 2054916,=C2=A0=C2=A0 0,=C2=A0=C2=A0 0 > > mbuf:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 256, 2362080,=C2=A0=C2=A0= =C2=A0=C2=A0 514,=C2=A0=C2=A0=C2=A0 > > 1776,3460790080,=C2=A0=C2=A0 0,=C2=A0=C2=A0 0 > > mbuf_cluster:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 204= 8, 369076,=C2=A0=C2=A0=C2=A0 1265,=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 31,=C2= =A0 > > 154081,=C2=A0=C2=A0 0,=C2=A0=C2=A0 0 > > mbuf_jumbo_page:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 4096, 184537,= =C2=A0=C2=A0=C2=A0=C2=A0 513,=C2=A0=C2=A0=C2=A0=C2=A0 > > 294,1592339809,=C2=A0=C2=A0 0,=C2=A0=C2=A0 0 > > mbuf_jumbo_9k:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 9216,= =C2=A0 54677,=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0,=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 > > 0,=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0,=C2=A0=C2=A0 0,=C2=A0=C2=A0 0 > > mbuf_jumbo_16k:=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 16384,=C2=A0 30756,= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0,=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 > > 0,=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0,=C2=A0=C2=A0 0,=C2=A0=C2=A0 0 > >=20 > > Now driver use mbuf_jumbo_page not mbuf_jumbo_9k and no fails=2E > >=20 > > I'm ok with mtu 1500 in my environment and I don't know if mtu 9000 wil= l > > work with this change=2E But at least it is stable now even after 100 day= s > > of uptime (just rebooted after upgraded to 11=2E3-RELEASE-p5)=2E > >=20 > > Hope this helps=2E > >=20 > >=20 > > 26=2E11=2E19 02:44, Chris =D0=BF=D0=B8=D1=88=D0=B5: > >> Or at least make it non fatal=2E > >> OK here's the story; I'm experimenting with a multiport NIC (re(4)) > >> as we hope to start using multiport 10G NICs=2E > >> Any of the re's we've used in the past have been very stable, which > >> is why I picked the one I did for this experiment=2E This one has been > >> performing rock solid for some 4 to 6 mos, under full time use=2E That > >> is until the last week=2E Where we're seeing: > >> watchdog timeout > >> repeated frequently=2E Which is ultimately fatal=2E ifconfig up/down will > >> not resuscitate it=2E Nor will service ifconfig restart, or plugging/ > >> unplugging the cable(s)=2E Bouncing the server is the only cure=2E Which > >> is unacceptable=2E Any, and All suggestions, or insight into the matter > >> GREATLY appreciated=2E Note; while this is an old 11=2E1, we're not planni= ng > >> to up this box until we can confirm this can be cured=2E :) > >> > >> Details follow: > >> 11=2E1-STABLE r327867 amd64 > >> > >> watchdog timeout > >> watchdog timeout > >> watchdog timeout > >> watchdog timeout > >> watchdog timeout > >> watchdog timeout > >> watchdog timeout > >> watchdog timeout > >> watchdog timeout > >> watchdog timeout > >> watchdog timeout > >> watchdog timeout > >> > >> rc=2Econf(5) > >> ifconfig_re0=3D"inet AA=2EBBB=2ECC=2EXX netmask 255=2E255=2E255=2E0 rxcsum txcsum = tso4" > >> ifconfig_re1=3D"inet AA=2EBBB=2ECC=2EWW netmask 255=2E255=2E255=2E0 rxcsum txcsum = tso4" > >> ifconfig_re1_alias0=3D"inet AA=2EBBB=2ECC=2EZZ netmask 255=2E255=2E255=2E0" > >> > >> ifconfig(8) > >> re0: flags=3D8843 metric 0 mtu= 1500 > >> > > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0options=3D8219b > >=20 > >> > >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0ether 00:13:3b:0f:13:44 > >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0hwaddr 00:13:3b:0f:13:44 > >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0inet6 fe80::213:3bff:fe0f:1344%re0 prefi= xlen 64 scopeid 0x1=20 > >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0inet AA=2EBBB=2ECC=2EXX netmask 0xffffff00 bro= adcast 24=2E113=2E41=2E255=20 > >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0nd6 options=3D23 > >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0media: Ethernet autoselect (1000baseT ) > >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0status: active > >> re1: flags=3D8843 metric 0 mtu= 1500 > >> > > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0options=3D8219b > >=20 > >> > >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0ether 00:13:3b:0f:13:45 > >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0hwaddr 00:13:3b:0f:13:45 > >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0inet AA=2EBBB=2ECC=2EWW netmask 0xffffff00 bro= adcast 24=2E113=2E41=2E255=20 > >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0inet AA=2EBBB=2ECC=2EZZ netmask 0xffffff00 bro= adcast 24=2E113=2E41=2E255=20 > >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0inet6 fe80::213:3bff:fe0f:1345%re1 prefi= xlen 64 scopeid 0x2=20 > >> =C2=A0=C2=A0=C2=A0=C2=A0nd6 options=3D23 > >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0media: Ethernet autoselect (1000baseT ) > >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0status: active > >> lo0: flags=3D8049 metric 0 mtu 16384 > >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0options=3D600003 > >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0inet6 ::1 prefixlen 128 =C2=A0=C2=A0= =C2=A0=C2=A0inet6 fe80::1%lo0 prefixlen 64=20 > >> scopeid 0x3 =C2=A0=C2=A0=C2=A0=C2=A0inet 127=2E0=2E0=2E1 netmask 0xff000000 = =C2=A0=C2=A0=C2=A0=C2=A0nd6=20 > >> options=3D21 > >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0groups: lo > >> pciconf(8) > >> re0@pci0:5:0:0:=C2=A0=C2=A0=C2=A0 class=3D0x020000 card=3D0x012310ec c= hip=3D0x816810ec=20 > >> rev=3D0x07 hdr=3D0x00 > >> =C2=A0=C2=A0=C2=A0 vendor=C2=A0=C2=A0=C2=A0=C2=A0 =3D 'Realtek Semicon= ductor Co=2E, Ltd=2E' > >> =C2=A0=C2=A0=C2=A0 device=C2=A0=C2=A0=C2=A0=C2=A0 =3D 'RTL8111/8168/84= 11 PCI Express Gigabit Ethernet=20 > >> Controller' > >> =C2=A0=C2=A0=C2=A0 class=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =3D network > >> =C2=A0=C2=A0=C2=A0 subclass=C2=A0=C2=A0 =3D ethernet > >> re1@pci0:6:0:0:=C2=A0=C2=A0=C2=A0 class=3D0x020000 card=3D0x012310ec c= hip=3D0x816810ec=20 > >> rev=3D0x07 hdr=3D0x00 > >> =C2=A0=C2=A0=C2=A0 vendor=C2=A0=C2=A0=C2=A0=C2=A0 =3D 'Realtek Semicon= ductor Co=2E, Ltd=2E' > >> =C2=A0=C2=A0=C2=A0 device=C2=A0=C2=A0=C2=A0=C2=A0 =3D 'RTL8111/8168/84= 11 PCI Express Gigabit Ethernet=20 > >> Controller' > >> =C2=A0=C2=A0=C2=A0 class=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =3D network > >> =C2=A0=C2=A0=C2=A0 subclass=C2=A0=C2=A0 =3D ethernet > >> > >> Thanks again! > >> > >> --Chris > >> > >> > >> _______________________________________________ > >> freebsd-net@freebsd=2Eorg mailing list > >> https://lists=2Efreebsd=2Eorg/mailman/listinfo/freebsd-net > >> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd=2Eorg" > >=20 >=20 > --=20 > Regards! > _______________________________________________ > freebsd-net@freebsd=2Eorg mailing list > https://lists=2Efreebsd=2Eorg/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd=2Eorg"