Date: Sun, 6 Oct 2019 13:50:14 -0700 From: Doug Hardie <bc979@lafn.org> To: =?utf-8?Q?Morgan_Wesstr=C3=B6m?= <freebsd-database@pp.dyndns.biz> Cc: FreeBSD Questions <freebsd-questions@freebsd.org> Subject: Re: Intermittent connectivity loss with em(4) Message-ID: <94B563F6-55C4-46BC-BD79-5CC2AD86E6C1@mail.sermon-archive.info> In-Reply-To: <ba24db9b-6a34-76d4-7c95-4c2e64d0a5a0@pp.dyndns.biz> References: <ba24db9b-6a34-76d4-7c95-4c2e64d0a5a0@pp.dyndns.biz>
next in thread | previous in thread | raw e-mail | index | archive | help
> On 6 October 2019, at 12:34, Morgan Wesstr=C3=B6m = <freebsd-database@pp.dyndns.biz> wrote: >=20 > I'd appreciate some help with the following problem because I'm = probably blind to the obvious from all my trouble shooting. >=20 > I have a small Supermicro Atom-based ITX-board with 3x Intel 82574L = NICs (2 on mainboard, 1 in PCIe slot) and I experience intermittent = network connectivity loss every few minutes. The machine is currently = running FreeBSD 12.0-RELEASE-p10. I loop a single ping once per minute = to dns.google (8.8.8.8) and output looks like this: >=20 > THINGS I'VE TESTED WITHOUT RESOLVING THE PROBLEM >=20 > - Tried all three NICs in the computer but they all show the same = behaviour. > - Tried a different ping target. > - Tried the LiveCD environment from the older FreeBSD 11.3-RELEASE = memstick. > - Disabled MSI-X interrupts. > - Additionally disabled MSI interrupts. > - Recompiled em(4) and enabled DEBUG_INIT, DEBUG_IOCTL and DEBUG_HW = but this only generates a few more message in dmesg during boot. There = is nothing shown in dmesg or otherwise when connectivity is lost. >=20 > THING'S I'VE TRIED THAT SEEMINGLY RESOLVES OR ALLEVIATES THE PROBLEM >=20 > - Booting the system on Linux (4.19 kernel) with just a simple command = prompt and running the same ping test does _NOT_ show any connectivity = loss. At least not during the hour or so I tested. To me this rules out = any hardware related problems as well as the network connection itself. > - Generating small amounts of network traffic on the interface (like = an ssh session) seems to reduce the problem. I can then run the ping = test for maybe 30 minutes without loss of connectivity in FreeBSD but = eventually it fails too. >=20 > I really need help to understand what's going on here. I have a gut = feeling some power saving is playing a trick on me but = hw.em.smart_pwr_down is set to 0 default and I have no indication of any = power saving function kicking in. How can I debug what's going on in = em(4)? Am I just stupid and missing something obvious here? I just tried the same thing using 12.0-RELEASE-p9. No drops. mail# ping -i 60 8.8.8.8 PING 8.8.8.8 (8.8.8.8): 56 data bytes 64 bytes from 8.8.8.8: icmp_seq=3D0 ttl=3D54 time=3D10.528 ms 64 bytes from 8.8.8.8: icmp_seq=3D1 ttl=3D54 time=3D11.403 ms 64 bytes from 8.8.8.8: icmp_seq=3D2 ttl=3D54 time=3D10.351 ms 64 bytes from 8.8.8.8: icmp_seq=3D3 ttl=3D54 time=3D10.787 ms 64 bytes from 8.8.8.8: icmp_seq=3D4 ttl=3D54 time=3D12.280 ms 64 bytes from 8.8.8.8: icmp_seq=3D5 ttl=3D54 time=3D10.038 ms 64 bytes from 8.8.8.8: icmp_seq=3D6 ttl=3D54 time=3D10.640 ms 64 bytes from 8.8.8.8: icmp_seq=3D7 ttl=3D54 time=3D11.090 ms 64 bytes from 8.8.8.8: icmp_seq=3D8 ttl=3D54 time=3D10.615 ms 64 bytes from 8.8.8.8: icmp_seq=3D9 ttl=3D54 time=3D8.972 ms 64 bytes from 8.8.8.8: icmp_seq=3D10 ttl=3D54 time=3D10.016 ms 64 bytes from 8.8.8.8: icmp_seq=3D11 ttl=3D54 time=3D11.097 ms 64 bytes from 8.8.8.8: icmp_seq=3D12 ttl=3D54 time=3D13.035 ms 64 bytes from 8.8.8.8: icmp_seq=3D13 ttl=3D54 time=3D11.251 ms 64 bytes from 8.8.8.8: icmp_seq=3D14 ttl=3D54 time=3D9.588 ms 64 bytes from 8.8.8.8: icmp_seq=3D15 ttl=3D54 time=3D10.649 ms 64 bytes from 8.8.8.8: icmp_seq=3D16 ttl=3D54 time=3D9.965 ms 64 bytes from 8.8.8.8: icmp_seq=3D17 ttl=3D54 time=3D9.900 ms 64 bytes from 8.8.8.8: icmp_seq=3D18 ttl=3D54 time=3D11.253 ms 64 bytes from 8.8.8.8: icmp_seq=3D19 ttl=3D54 time=3D9.440 ms 64 bytes from 8.8.8.8: icmp_seq=3D20 ttl=3D54 time=3D11.544 ms 64 bytes from 8.8.8.8: icmp_seq=3D21 ttl=3D54 time=3D11.068 ms 64 bytes from 8.8.8.8: icmp_seq=3D22 ttl=3D54 time=3D10.196 ms 64 bytes from 8.8.8.8: icmp_seq=3D23 ttl=3D54 time=3D12.446 ms 64 bytes from 8.8.8.8: icmp_seq=3D24 ttl=3D54 time=3D10.467 ms 64 bytes from 8.8.8.8: icmp_seq=3D25 ttl=3D54 time=3D11.100 ms 64 bytes from 8.8.8.8: icmp_seq=3D26 ttl=3D54 time=3D9.746 ms 64 bytes from 8.8.8.8: icmp_seq=3D27 ttl=3D54 time=3D10.627 ms 64 bytes from 8.8.8.8: icmp_seq=3D28 ttl=3D54 time=3D10.339 ms 64 bytes from 8.8.8.8: icmp_seq=3D29 ttl=3D54 time=3D10.166 ms 64 bytes from 8.8.8.8: icmp_seq=3D30 ttl=3D54 time=3D10.055 ms 64 bytes from 8.8.8.8: icmp_seq=3D31 ttl=3D54 time=3D9.303 ms 64 bytes from 8.8.8.8: icmp_seq=3D32 ttl=3D54 time=3D11.240 ms 64 bytes from 8.8.8.8: icmp_seq=3D33 ttl=3D54 time=3D11.407 ms 64 bytes from 8.8.8.8: icmp_seq=3D34 ttl=3D54 time=3D10.930 ms 64 bytes from 8.8.8.8: icmp_seq=3D35 ttl=3D54 time=3D10.166 ms 64 bytes from 8.8.8.8: icmp_seq=3D36 ttl=3D54 time=3D11.400 ms 64 bytes from 8.8.8.8: icmp_seq=3D37 ttl=3D54 time=3D10.946 ms ^C --- 8.8.8.8 ping statistics --- 38 packets transmitted, 38 packets received, 0.0% packet loss round-trip min/avg/max/stddev =3D 8.972/10.685/13.035/0.846 ms I have had issues with em nics in the past and don't use them anymore. = However, none of the issues were as blatant as you are seeing. The dmsg = log is only written to during boot. Messages during operation are = written to /var/log/messages. Check there to see if there is anything = that correlates with the outages. -- Doug
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?94B563F6-55C4-46BC-BD79-5CC2AD86E6C1>