From owner-freebsd-current@freebsd.org Mon Jun 24 20:10:45 2019 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 72A0415D804D; Mon, 24 Jun 2019 20:10:45 +0000 (UTC) (envelope-from ler@lerctr.org) Received: from thebighonker.lerctr.org (ns-b.lerctr.org [IPv6:2001:470:1f0f:3ad::53:2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "*.lerctr.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BDA5D86454; Mon, 24 Jun 2019 20:10:43 +0000 (UTC) (envelope-from ler@lerctr.org) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lerctr.org; s=ler2019; h=Message-ID:References:In-Reply-To:Subject:Cc:To:From:Date: Content-Transfer-Encoding:Content-Type:MIME-Version:Sender:Reply-To: Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender: Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe: List-Subscribe:List-Post:List-Owner:List-Archive; bh=x+wbJkUK/sGjpSc2wFBAAyrR9hCRVyMKaA7GAQa8E9A=; b=jtOOU9EroMjsjD3FDhaHz1EKiB x6tn2H12jL2Awo95ahWxnClF3QeXRetDmdiTxXYIudORz9t8ouQRFMV0Q1M7ZAbDNKE145RGwjkiB HuzAgws6Ut+fbSi0R37oaoqrt2+NnCjurTXl47hCeTMqPkEeSdQKUkZFFRUXOrPXTTfYvLm7J5C1c cPWUC4Dzxd+Yrv+RdXRgvB6FmfWs1KWJLBnPWw1+f5x9aZdCoAD6qbqzTHiSjy07y9uGlHtvsDy2I o6cwLjtUG29m6vdSXRWVqV3/S33s8nRxTIBBIZgImBtGTbVjvHSiDHgiUzZtQd3gzVYqu894YBDNt a3Wp83mA==; Received: from thebighonker.lerctr.org ([2001:470:1f0f:3ad:bb:dcff:fe50:d900]:51547 helo=webmail.lerctr.org) by thebighonker.lerctr.org with esmtpsa (TLSv1.3:TLS_AES_256_GCM_SHA384:256) (Exim 4.92 (FreeBSD)) (envelope-from ) id 1hfVIW-000BNV-Sm; Mon, 24 Jun 2019 15:10:40 -0500 Received: from 2600:1700:210:b180:4c24:fd26:d30c:ae75 by webmail.lerctr.org with HTTP (HTTP/1.1 POST); Mon, 24 Jun 2019 15:10:40 -0500 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Date: Mon, 24 Jun 2019 15:10:40 -0500 From: Larry Rosenman To: "Andrey V. Elsukov" Cc: freebsd-current@freebsd.org, freebsd-net@freebsd.org Subject: Re: ng_snd_item: Panic? In-Reply-To: References: <20190624183200.hu4vzocjsopjsnnz@ler-imac.local> Message-ID: X-Sender: ler@lerctr.org User-Agent: Roundcube Webmail/1.3.9 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Jun 2019 20:10:45 -0000 On 06/24/2019 3:01 pm, Andrey V. Elsukov wrote: > 24.06.2019 21:32, Larry Rosenman пишет: >> Got 2 of these today, and I have cores.... >> Ideas? >> r349200. >> >> Unread portion of the kernel message buffer: >> panic: ng_snd_item: 42 != 1414 >> cpuid = 10 >> time = 1561382494 >> KDB: stack backtrace: >> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame >> 0xfffffe012628d400 >> vpanic() at vpanic+0x19d/frame 0xfffffe012628d450 >> panic() at panic+0x43/frame 0xfffffe012628d4b0 >> ng_snd_item() at ng_snd_item+0x477/frame 0xfffffe012628d4f0 >> ng_ether_output() at ng_ether_output+0x5e/frame 0xfffffe012628d520 >> ether_output() at ether_output+0x473/frame 0xfffffe012628d5c0 >> arpintr() at arpintr+0xfe3/frame 0xfffffe012628d780 >> netisr_dispatch_src() at netisr_dispatch_src+0x89/frame >> 0xfffffe012628d7f0 >> ether_demux() at ether_demux+0x137/frame 0xfffffe012628d820 >> ng_ether_rcv_upper() at ng_ether_rcv_upper+0x95/frame >> 0xfffffe012628d840 >> ng_apply_item() at ng_apply_item+0xf1/frame 0xfffffe012628d8c0 >> ng_snd_item() at ng_snd_item+0x2ab/frame 0xfffffe012628d900 >> ng_apply_item() at ng_apply_item+0xf1/frame 0xfffffe012628d980 >> ng_snd_item() at ng_snd_item+0x2ab/frame 0xfffffe012628d9c0 >> ng_ether_input() at ng_ether_input+0x4c/frame 0xfffffe012628d9f0 >> ether_nh_input() at ether_nh_input+0x2cd/frame 0xfffffe012628da40 >> netisr_dispatch_src() at netisr_dispatch_src+0x89/frame >> 0xfffffe012628dab0 >> ether_input() at ether_input+0x48/frame 0xfffffe012628dad0 >> bce_intr() at bce_intr+0x697/frame 0xfffffe012628db50 >> ithread_loop() at ithread_loop+0x187/frame 0xfffffe012628dbb0 >> fork_exit() at fork_exit+0x84/frame 0xfffffe012628dbf0 >> fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe012628dbf0 >> --- trap 0, rip = 0, rsp = 0, rbp = 0 --- >> Uptime: 4d18h45m34s >> Dumping 24921 out of 131026 >> MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91% >> >> #5 0xffffffff828ee5b7 in ng_snd_item (item=0xfffff8021e3b4d80, >> flags=0) >> at /usr/src/sys/netgraph/ng_base.c:2252 > > It looks like you use some netgraph based ethernet interface. > The system got received ARP request and is going to send the reply, > but somehow mbuf with this ARP request has initialized m_next pointer, > thus it is considered as a chain of mbufs. > > in_arpinput() reuses received mbuf to construct the reply, but it > doesn't check that an mbut is a chain. It just sets m_len and sends it. > Then since you have INVARIANTS in your kernel, the netgraph code check > the actual length of the chain, and it doesn't match to m_len. It > panics. so, is this a bug? Timing race? Other? [I] ➜ sudo ngctl list There are 7 total nodes: Name: bce0 Type: ether ID: 00000001 Num hooks: 2 Name: bce1 Type: ether ID: 00000002 Num hooks: 0 Name: bce2 Type: ether ID: 00000003 Num hooks: 0 Name: bce3 Type: ether ID: 00000004 Num hooks: 0 Name: netflow Type: netflow ID: 00000006 Num hooks: 3 Name: Type: ksocket ID: 00000007 Num hooks: 1 Name: ngctl3705 Type: socket ID: 00000008 Num hooks: 0 ler in ~ at borg [I] ➜ > >> #6 0xffffffff82900c2e in ng_ether_output (ifp=, >> mp=0xfffffe012628d578) at /usr/src/sys/netgraph/ng_ether.c:294 >> #7 0xffffffff805b1e43 in ether_output (ifp=, >> m=0xfffff81f59eefb00, dst=0xfffffe012628d740, ro=) >> at /usr/src/sys/net/if_ethersubr.c:430 >> #8 0xffffffff805cb3e3 in in_arpinput (m=) >> at /usr/src/sys/netinet/if_ether.c:1152 >> #9 arpintr (m=0xfffff81f59eefb00) at >> /usr/src/sys/netinet/if_ether.c:749 >> #10 0xffffffff805bcf89 in netisr_dispatch_src (proto=4, >> source=, m=) at >> /usr/src/sys/net/netisr.c:1123 >> #11 0xffffffff805b22d7 in ether_demux (ifp=0xfffff8012c902000, >> m=) at /usr/src/sys/net/if_ethersubr.c:913 >> #12 0xffffffff82901045 in ng_ether_rcv_upper (hook=, >> item=) at /usr/src/sys/netgraph/ng_ether.c:741 >> #13 0xffffffff828ee6e1 in ng_apply_item (node=0xfffff81054f43400, >> item=0xfffff8021e3b4d80, rw=0) at >> /usr/src/sys/netgraph/ng_base.c:2403 >> #14 0xffffffff828ee3eb in ng_snd_item (item=0xfffff8021e3b4d80, >> flags=0) >> at /usr/src/sys/netgraph/ng_base.c:2320 >> #15 0xffffffff828ee6e1 in ng_apply_item (node=0xfffff8012c2d3e00, >> item=0xfffff8021e3b4d80, rw=0) at >> /usr/src/sys/netgraph/ng_base.c:2403 >> #16 0xffffffff828ee3eb in ng_snd_item (item=0xfffff8021e3b4d80, >> flags=0) >> at /usr/src/sys/netgraph/ng_base.c:2320 >> #17 0xffffffff82900cbc in ng_ether_input (ifp=, >> mp=0xfffffe012628da18) at /usr/src/sys/netgraph/ng_ether.c:255 >> #18 0xffffffff805b34fd in ether_input_internal >> (ifp=0xfffff8012c902000, >> m=0xfffff81f59eefb00) at /usr/src/sys/net/if_ethersubr.c:654 >> #19 ether_nh_input (m=) at >> /usr/src/sys/net/if_ethersubr.c:735 >> #20 0xffffffff805bcf89 in netisr_dispatch_src (proto=5, >> source=, m=) at >> /usr/src/sys/net/netisr.c:1123 >> #21 0xffffffff805b26f8 in ether_input (ifp=0xfffff8012c902000, m=0x0) >> at /usr/src/sys/net/if_ethersubr.c:823 >> #22 0xffffffff8273c7f7 in bce_rx_intr (sc=) >> at /usr/src/sys/dev/bce/if_bce.c:6848 >> #23 bce_intr (xsc=0xfffffe01665c2000) at >> /usr/src/sys/dev/bce/if_bce.c:8017 >> #24 0xffffffff8047e0e7 in intr_event_execute_handlers (p=> out>, >> ie=) at /usr/src/sys/kern/kern_intr.c:1148 >> #25 ithread_execute_handlers (p=, ie=) >> at /usr/src/sys/kern/kern_intr.c:1161 >> #26 ithread_loop (arg=) at >> /usr/src/sys/kern/kern_intr.c:1241 >> #27 0xffffffff8047ac74 in fork_exit ( >> callout=0xffffffff8047df60 , arg=0xfffff8012c883100, >> frame=0xfffffe012628dc00) at /usr/src/sys/kern/kern_fork.c:1056 >> #28 >> (kgdb) >> -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 E-Mail: ler@lerctr.org US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106