Date: Fri, 12 Jun 2015 18:07:37 +0200 From: Cs <bimmer@field.hu> To: Christopher Forgeron <csforgeron@gmail.com> Cc: FreeBSD Net <freebsd-net@freebsd.org> Subject: Re: FreeBSD 10.1-REL - network unaccessible after high traffic Message-ID: <557B03C9.4000509@field.hu> In-Reply-To: <CAB2_NwCgEvmMxqmAotO1USsipXOSaGkwK3Uu%2BiVbKd9_bn%2BLWg@mail.gmail.com> References: <374339249.53058039.1433681874571.JavaMail.root@uoguelph.ca> <55744F28.5000402@field.hu> <CAB2_NwA-D7bH47=Qkf9QLF3=mZOQBVo81bUsQzQr02W9U4vHMA@mail.gmail.com> <557AB1BB.60502@field.hu> <CAB2_NwA9i-wMXGH2%2BcP9SWxDMNomFRjoVP25hsGWaTDGjBxFTw@mail.gmail.com> <557AD10D.5070205@field.hu> <CAB2_NwAeD43tSwWO3LGuniRMNZ3TVupOuLWj3aUm228jLT2y1A@mail.gmail.com> <557AD2FA.103@field.hu> <CAB2_NwCgEvmMxqmAotO1USsipXOSaGkwK3Uu%2BiVbKd9_bn%2BLWg@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
I'll take your advice and give it a shot, thanks :) 2015.06.12. 14:43 keltezéssel, Christopher Forgeron írta: > Ah, but the 'why' will come later, after we know for sure what the > 'what' is in your problem. > > I'm just pointing out the problems that I'm having, as yours sound > similar. Once the box runs out of memory, all sorts of interesting > things can happen. Perhaps that's not your case, but it's quite possible. > > Setup a remote terminal, do the copy again, and send in the last few > lines of 'vmstat 5' after it's locked up, perhaps I can help. > > On Fri, Jun 12, 2015 at 9:39 AM, Cs <bimmer@field.hu > <mailto:bimmer@field.hu>> wrote: > > but why is that machine runs fine except the network if it's > memory related? swap didn't increased before the network outage. > > > 2015.06.12. 14:37 keltezéssel, Christopher Forgeron írta: >> rsycn burns memory - I'd say you have a good chance you're >> running out of mem before it's replenished. >> >> For vmstat 5 - Don't run it on console. Connect via a second box >> with ssh, and run it there - That way it's the last thing on the >> ssh terminal screen when the box dies, and you'll have your proof. >> >> On Fri, Jun 12, 2015 at 9:31 AM, Cs <bimmer@field.hu >> <mailto:bimmer@field.hu>> wrote: >> >> machine has been restarted before I could check the "vmstat >> 5" output. Yep, it's rsync. Anyway I disabled the backup >> transfer it'll solve, but I can't really accept this for >> solution. >> >> >> 2015.06.12. 14 <tel:2015.06.12.%2014>:29 keltezéssel, >> Christopher Forgeron írta: >> >> Well, even at low speed it could drop due to memory from >> what I've seen. >> >> What was the last line from vmstat 5 before it locked up? >> >> I find that the em driver isn't crap, but there is a >> deeper problem inside >> of FreeBSD that is being exposed now - For me it's due to >> faster network >> connections. >> >> Are you using rsync to move the files? >> >> On Fri, Jun 12, 2015 at 7:17 AM, Cs <bimmer@field.hu >> <mailto:bimmer@field.hu>> wrote: >> >> it seems it's not memory related. Server just died a >> few minutes ago >> during transporting the backup (400GB) around 800Mbps >> speed.. >> will disable remote backup, it's a shame that em >> driver is such a crap. >> >> >> 2015.06.08. 5:01 keltezéssel, Christopher Forgeron írta: >> >> You know what helped me: >> >> 'vmstat 5' >> >> Leave that running. If the last thing on the >> console after a crash/hang is >> vmstat showing 8k of memory left, then you're in >> the same problem-park as >> me. >> >> My 10.1 96GiB RAM box is chewing ~8 GiB of RAM in >> less than 5 seconds, and >> then crashing/panicking/hanging. >> >> There's others with this issues if you search for >> it; a sysctl >> to vm.v_free_min to double or triple that value >> may help, but first let us >> know if that's what is bonking your sever. >> >> >> >> On Sun, Jun 7, 2015 at 11:03 AM, Cs >> <bimmer@field.hu <mailto:bimmer@field.hu>> wrote: >> >> ok, just lowered it to 1500 but please also >> note that it was on 1500 for >> >> 2 >> years >> >> 2015.06.07. 14 <tel:2015.06.07.%2014>:57 >> keltezéssel, Rick Macklem írta: >> >> Since disabling TSO didn't help, you could >> try dropping to 1500mtu >> >> on both interfaces. Some people run into >> problems when 9K jumbo clusters >> fragment the kernel address space used to >> allocate mbufs. >> >> Good luck with it, rick >> >> ----- Original Message ----- >> >> Hi All, >> >> It worked fine for two weeks but I >> had a network outage 2 days ago >> then >> today. Tried to disable rxcsum and >> txcsum after the first one, didn't >> help. Don't know what else to do it's >> a shame that I can't use this >> card >> with fbsd i REALLY don't want to >> install linux instead but my >> production >> servers outages are not welcomed by >> the customers.. >> >> 2015.05.26. 10 >> <tel:2015.05.26.%2010>:36 >> keltezéssel, Cs írta: >> >> Thanks Mark, good idea. I found >> this thread which is exactly the >> >> same >> problem as mine: >> >> >> https://forums.freebsd.org/threads/workaround-freebsd-10-1-sudden-network-down.49264/ >> >> Will see if it helps in a couple >> weeks. >> >> Regards, >> Csaba >> >> 2015.05.26. 10 >> <tel:2015.05.26.%2010>:30 >> keltezéssel, Mark Schouten írta: >> >> Oh, didn't see your lowest >> remark. Then, the next thing that >> comes >> >> past here a few times per >> week is 'Try disabling TSO'. >> >> >> Met vriendelijke groeten, >> >> -- >> Kerio Operator in de Cloud? >> https://www.kerioindecloud.nl/ >> Mark Schouten | Tuxis >> Internet Engineering >> KvK: 61527076 | >> http://www.tuxis.nl/ >> T: 0318 200208 | >> info@tuxis.nl >> <mailto:info@tuxis.nl> >> >> >> >> Van: Cs >> <bimmer@field.hu >> <mailto:bimmer@field.hu>> >> Aan: Mark Schouten >> <mark@tuxis.nl >> <mailto:mark@tuxis.nl>> >> Cc: >> <freebsd-net@freebsd.org >> <mailto:freebsd-net@freebsd.org>> >> Verzonden: 25-5-2015 11:12 >> Onderwerp: Re: FreeBSD >> 10.1-REL - network >> unaccessible after >> high >> traffic >> >> It was on 1500 for ~3 years :) >> Regards, >> Csaba >> On May 25, 2015, >> 10:30, at 10:30, Mark Schouten >> <mark@tuxis.nl >> <mailto:mark@tuxis.nl>> >> wrote: >> >> Try lowering your mtu to >> 1500, that worked miracles >> for me.. >> >> -- >> Mark Schouten >> Tuxis Internet Engineering >> mark@tuxis.nl >> <mailto:mark@tuxis.nl> / >> 0318 200208 >> >> On 25 May 2015, at >> 09:36, "Cs" >> <bimmer@field.hu >> <mailto:bimmer@field.hu>> >> wrote: >> >> Hi all, >> I have two >> FreeBSd 10.1-RELEASE >> servers connected to each >> other. >> They >> >> were connected via >> cross link, but they >> are connected to a cisco >> >> switch >> now (the problem was the >> same with cross link >> too). When >> transferring >> huge files (50-500GB >> backup files) via Gigabit >> (it is important!) >> the >> network randomly dies. >> The backup runs every >> day/week and >> sometimes the >> connection is ok for >> months sometimes it >> happens twice a week. >> When the >> network dies I can log in >> to the server via IPMI >> and use the >> console >> everything is OK, but >> can't send anything out >> on the network. >> ifconfig >> em0 down/up doesn't help >> nor netif restart. The >> problem never >> occured >> when I used 100Mbit >> connection between them, >> but it was 3com NIC >> (xl), >> gigabit adapter is Intel >> (em0). When I limit the >> transfer rate >> (rsync >> bandwith limit or ipfw >> pipe) the problem is much >> more rare. >> >> I tried to set >> these tuning parameters >> on both servers with >> >> different >> >> buffer size but >> nothing helped: >> >> # cat /etc/sysctl.conf >> >> security.bsd.see_other_uids=0 >> net.inet.tcp.recvspace=512000 >> net.route.netisr_maxqlen=2048 >> kern.ipc.nmbclusters=1310720 >> net.inet.tcp.sendbuf_max=16777216 >> net.inet.tcp.recvbuf_max=16777216 >> kern.ipc.soacceptqueue=32768 >> # cat >> /boot/loader.conf >> geom_mirror_load="YES" # >> RAID1 disk driver >> (see gmirror(8)) >> ipfw_load="YES" >> net.inet.ip.fw.default_to_accept=1 >> kern.maxusers=4096 >> accf_data_load="YES" >> The duplex >> settings are >> identical on both >> servers. >> Server A: >> em1: >> flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> >> metric 0 >> mtu >> >> 9000 >> >> >> options=4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,VLAN_HWTSO> >> >> >> ether >> 00:25:90:24:52:66 >> >> inet >> x.x.x.x netmask >> 0xfffffe00 broadcast >> x.x.x.x >> nd6 >> options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> >> media: >> Ethernet autoselect >> (1000baseT <full-duplex>) >> status: active >> Server B: >> em0: >> flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> >> metric 0 >> mtu >> >> 9000 >> >> >> options=4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,VLAN_HWTSO> >> >> >> ether >> 00:30:48:dd:fe:3e >> >> inet >> x.x.x.x netmask >> 0xfffffe00 broadcast >> x.x.x.x >> nd6 >> options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> >> media: >> Ethernet autoselect >> (1000baseT <full-duplex>) >> status: active >> Today I tried to >> set mtu to 9000 but >> in tcpdump I see that >> during >> scp >> >> it is still 1500: >> >> x.x.x.x.222 > >> x.x.x.x.37612: Flags [.], >> cksum 0xb6ee >> >> (incorrect -> >> >> 0xda6f), seq 35749, >> ack 113701596, win >> 7986, options [nop,nop,TS >> >> val >> 3103966325 >> <tel:3103966325> ecr >> 853712893], length 0 >> >> 09:27:33.912354 IP (tos >> 0x8, ttl 64, id 1028, >> offset 0, flags >> >> [DF], >> >> proto TCP (6), >> length 1500) >> >> 09:27:33.912358 IP (tos >> 0x8, ttl 64, id 1029, >> offset 0, flags >> >> [DF], >> >> proto TCP (6), >> length 1500) >> >> Any ideas? Thanks >> guys! >> >> _______________________________________________ >> freebsd-net@freebsd.org >> <mailto:freebsd-net@freebsd.org> >> mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-net >> To unsubscribe, send >> any mail to >> >> >> "freebsd-net-unsubscribe@freebsd.org >> <mailto:freebsd-net-unsubscribe@freebsd.org>" >> >> _______________________________________________ >> >> freebsd-net@freebsd.org >> <mailto:freebsd-net@freebsd.org> >> mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-net >> To unsubscribe, send any mail to >> "freebsd-net-unsubscribe@freebsd.org >> <mailto:freebsd-net-unsubscribe@freebsd.org>" >> >> >> _______________________________________________ >> >> freebsd-net@freebsd.org >> <mailto:freebsd-net@freebsd.org> >> mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-net >> To unsubscribe, send any mail to >> "freebsd-net-unsubscribe@freebsd.org >> <mailto:freebsd-net-unsubscribe@freebsd.org>" >> >> _______________________________________________ >> >> freebsd-net@freebsd.org >> <mailto:freebsd-net@freebsd.org> >> mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-net >> To unsubscribe, send any mail to >> "freebsd-net-unsubscribe@freebsd.org >> <mailto:freebsd-net-unsubscribe@freebsd.org>" >> >> _______________________________________________ >> >> freebsd-net@freebsd.org >> <mailto:freebsd-net@freebsd.org> mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-net >> To unsubscribe, send any mail to >> "freebsd-net-unsubscribe@freebsd.org >> <mailto:freebsd-net-unsubscribe@freebsd.org>" >> >> _______________________________________________ >> >> freebsd-net@freebsd.org >> <mailto:freebsd-net@freebsd.org> mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-net >> To unsubscribe, send any mail to >> "freebsd-net-unsubscribe@freebsd.org >> <mailto:freebsd-net-unsubscribe@freebsd.org>" >> >> _______________________________________________ >> freebsd-net@freebsd.org >> <mailto:freebsd-net@freebsd.org> mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-net >> To unsubscribe, send any mail to >> "freebsd-net-unsubscribe@freebsd.org >> <mailto:freebsd-net-unsubscribe@freebsd.org>" >> >> _______________________________________________ >> freebsd-net@freebsd.org <mailto:freebsd-net@freebsd.org> >> mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-net >> To unsubscribe, send any mail to >> "freebsd-net-unsubscribe@freebsd.org >> <mailto:freebsd-net-unsubscribe@freebsd.org>" >> >> >> _______________________________________________ >> freebsd-net@freebsd.org <mailto:freebsd-net@freebsd.org> >> mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-net >> To unsubscribe, send any mail to >> "freebsd-net-unsubscribe@freebsd.org >> <mailto:freebsd-net-unsubscribe@freebsd.org>" >> >> > >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?557B03C9.4000509>