Date: Mon, 5 Sep 2016 10:14:59 -0600 From: Warner Losh <imp@bsdimp.com> To: Slawa Olhovchenkov <slw@zxy.spb.ru> Cc: hiren panchasara <hiren@strugglingcoder.info>, FreeBSD-STABLE Mailing List <freebsd-stable@freebsd.org> Subject: Re: 11.0 stuck on high network load Message-ID: <CANCZdfoOqVGxiytKRhpFgT6N1EbKyP3qqyj2QQzTbUSPzAQW0A@mail.gmail.com> In-Reply-To: <20160905074348.GE34394@zxy.spb.ru> References: <20160904215739.GC22212@zxy.spb.ru> <20160905014612.GA42393@strugglingcoder.info> <20160905074348.GE34394@zxy.spb.ru>
next in thread | previous in thread | raw e-mail | index | archive | help
On Mon, Sep 5, 2016 at 1:43 AM, Slawa Olhovchenkov <slw@zxy.spb.ru> wrote: > On Sun, Sep 04, 2016 at 06:46:12PM -0700, hiren panchasara wrote: > >> On 09/05/16 at 12:57P, Slawa Olhovchenkov wrote: >> > I am try using 11.0 on Dual E5-2620 (no X2APIC). >> > Under high network load and may be addtional conditional system go to >> > unresponsible state -- no reaction to network and console (USB IPMI >> > emulation). INVARIANTS give to high overhad. Is this exist some way to >> > debug this? >> >> Can you panic it from console to get to db> to get backtrace and other >> info when it goes unresponsive? > > no > no reaction So the canonical 'ipmitool chassis power diag' doesn't send an NMI to get you to the debugger? I've seen this at Netflix on one variant of our flash offload box with a Intel e5-2697v2 running with the Chelsio driver. We're working around it by having fewer receive threads than CPUs in the system. The only way the boxes would come back was with watchdog. The load was streaming video > ~36Gbps out 4 lagged 10G ports. Console is totally unresponsive as well. This is on our FreeBSD-10 stable based fork. >From my debugging, we go from totally fine as far as I can tell from ps, etc in the moments leading to the hang to being totally wedged. It seems a very sudden-onset condition. Sound at all familiar? Warner
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CANCZdfoOqVGxiytKRhpFgT6N1EbKyP3qqyj2QQzTbUSPzAQW0A>