From owner-freebsd-net@freebsd.org Tue Nov 26 05:55:36 2019 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 5177C1AA059 for ; Tue, 26 Nov 2019 05:55:36 +0000 (UTC) (envelope-from artem@viklenko.net) Received: from alf.viklenko.net (alf.viklenko.net [IPv6:2001:470:71:d72::61]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "www.viklenko.net", Issuer "Let's Encrypt Authority X3" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 47MY6H0SJRz4Qcs for ; Tue, 26 Nov 2019 05:55:34 +0000 (UTC) (envelope-from artem@viklenko.net) Received: from [IPv6:2001:470:71:d72:38db:9ac0:7b61:3ca2] ([IPv6:2001:470:71:d72:38db:9ac0:7b61:3ca2]) (authenticated bits=0) by alf.viklenko.net (8.15.2/8.15.2) with ESMTPSA id xAQ5tBgO032153 (version=TLSv1.2 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 26 Nov 2019 07:55:15 +0200 (EET) (envelope-from artem@viklenko.net) Subject: Re: How to remove watchdog? To: bsd-lists@BSDforge.com, freebsd-net@freebsd.org References: From: Artem Viklenko Message-ID: Date: Tue, 26 Nov 2019 07:55:11 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.1.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: uk-UA Content-Transfer-Encoding: 8bit X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.6.2 (alf.viklenko.net [IPv6:2001:470:71:d72:0:0:0:61]); Tue, 26 Nov 2019 07:55:15 +0200 (EET) X-Rspamd-Queue-Id: 47MY6H0SJRz4Qcs X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.64 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[viklenko.net:s=alf-mail]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[viklenko.net:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[viklenko.net,reject]; IP_SCORE(-1.64)[ipnet: 2001:470::/32(-4.64), asn: 6939(-3.51), country: US(-0.05)]; FROM_EQ_ENVFROM(0.00)[]; SUBJECT_ENDS_QUESTION(1.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Nov 2019 05:55:36 -0000 Hi! I have several small boxes with realtek nics acting as a router/firewall. Also had same issues. FreeBSD driver didn't work at least for me so I switched to Realtek's driver. But after some time traffic stops passing my routers. Did some investigation and found that the issue is 9k mbufs. As far as I understand more traffic you push more issues with 9k mbufs appears due to memory fragmentation. You can check it with 'vmstat -z | grep mbuf'. So I decided to do wery dirty hack - I've changed Jumbo_Frame_9k to Jumbo_Frame_4k in the if_re.c from Realtek's latest 1.95 driver. It comiles and work on FreeBSD 10.x and 11.x and now it just works (vendor says that driver is for older versions of the FreeBSD OS). And no more issues. ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP mbuf_packet: 256, 2362080, 2, 1263, 2054916, 0, 0 mbuf: 256, 2362080, 514, 1776,3460790080, 0, 0 mbuf_cluster: 2048, 369076, 1265, 31, 154081, 0, 0 mbuf_jumbo_page: 4096, 184537, 513, 294,1592339809, 0, 0 mbuf_jumbo_9k: 9216, 54677, 0, 0, 0, 0, 0 mbuf_jumbo_16k: 16384, 30756, 0, 0, 0, 0, 0 Now driver use mbuf_jumbo_page not mbuf_jumbo_9k and no fails. I'm ok with mtu 1500 in my environment and I don't know if mtu 9000 will work with this change. But at least it is stable now even after 100 days of uptime (just rebooted after upgraded to 11.3-RELEASE-p5). Hope this helps. 26.11.19 02:44, Chris пише: > Or at least make it non fatal. > OK here's the story; I'm experimenting with a multiport NIC (re(4)) > as we hope to start using multiport 10G NICs. > Any of the re's we've used in the past have been very stable, which > is why I picked the one I did for this experiment. This one has been > performing rock solid for some 4 to 6 mos, under full time use. That > is until the last week. Where we're seeing: > watchdog timeout > repeated frequently. Which is ultimately fatal. ifconfig up/down will > not resuscitate it. Nor will service ifconfig restart, or plugging/ > unplugging the cable(s). Bouncing the server is the only cure. Which > is unacceptable. Any, and All suggestions, or insight into the matter > GREATLY appreciated. Note; while this is an old 11.1, we're not planning > to up this box until we can confirm this can be cured. :) > > Details follow: > 11.1-STABLE r327867 amd64 > > watchdog timeout > watchdog timeout > watchdog timeout > watchdog timeout > watchdog timeout > watchdog timeout > watchdog timeout > watchdog timeout > watchdog timeout > watchdog timeout > watchdog timeout > watchdog timeout > > rc.conf(5) > ifconfig_re0="inet AA.BBB.CC.XX netmask 255.255.255.0 rxcsum txcsum tso4" > ifconfig_re1="inet AA.BBB.CC.WW netmask 255.255.255.0 rxcsum txcsum tso4" > ifconfig_re1_alias0="inet AA.BBB.CC.ZZ netmask 255.255.255.0" > > ifconfig(8) > re0: flags=8843 metric 0 mtu 1500 >     options=8219b >     ether 00:13:3b:0f:13:44 >     hwaddr 00:13:3b:0f:13:44 >     inet6 fe80::213:3bff:fe0f:1344%re0 prefixlen 64 scopeid 0x1 >     inet AA.BBB.CC.XX netmask 0xffffff00 broadcast 24.113.41.255 >     nd6 options=23 >     media: Ethernet autoselect (1000baseT ) >     status: active > re1: flags=8843 metric 0 mtu 1500 >     options=8219b >     ether 00:13:3b:0f:13:45 >     hwaddr 00:13:3b:0f:13:45 >     inet AA.BBB.CC.WW netmask 0xffffff00 broadcast 24.113.41.255 >     inet AA.BBB.CC.ZZ netmask 0xffffff00 broadcast 24.113.41.255 >     inet6 fe80::213:3bff:fe0f:1345%re1 prefixlen 64 scopeid 0x2     nd6 > options=23 >     media: Ethernet autoselect (1000baseT ) >     status: active > lo0: flags=8049 metric 0 mtu 16384 >     options=600003 >     inet6 ::1 prefixlen 128     inet6 fe80::1%lo0 prefixlen 64 scopeid > 0x3     inet 127.0.0.1 netmask 0xff000000     nd6 > options=21 >     groups: lo > pciconf(8) > re0@pci0:5:0:0:    class=0x020000 card=0x012310ec chip=0x816810ec > rev=0x07 hdr=0x00 >    vendor     = 'Realtek Semiconductor Co., Ltd.' >    device     = 'RTL8111/8168/8411 PCI Express Gigabit Ethernet > Controller' >    class      = network >    subclass   = ethernet > re1@pci0:6:0:0:    class=0x020000 card=0x012310ec chip=0x816810ec > rev=0x07 hdr=0x00 >    vendor     = 'Realtek Semiconductor Co., Ltd.' >    device     = 'RTL8111/8168/8411 PCI Express Gigabit Ethernet > Controller' >    class      = network >    subclass   = ethernet > > Thanks again! > > --Chris > > > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" -- Regards!