Date: Tue, 12 Jul 2016 06:25:36 -0400 From: Donald Baud <donaldbaud@yahoo.com> To: freebsd-net@freebsd.org Subject: Re: kernel panic with netgraph and mpd5.8 Message-ID: <2c786d19-7ba2-10ad-89a0-9a78abfe7146@yahoo.com> In-Reply-To: <3b3c7a0a-35d4-93c5-c283-1b03f627ac38@yahoo.com> References: <58413371.417578.1468131588156.JavaMail.yahoo.ref@mail.yahoo.com> <58413371.417578.1468131588156.JavaMail.yahoo@mail.yahoo.com> <5782BAA4.2090606@gmail.com> <3b3c7a0a-35d4-93c5-c283-1b03f627ac38@yahoo.com>
next in thread | previous in thread | raw e-mail | index | archive | help
>> On 2016-07-10 10:49, Donald Baud via freebsd-net wrote: >>> Hi I'm running an l2tp lns through mpd5.8 and it's been crashing >>> twice in 24h. >>> This is a new project replacing a cisco 7206, 700-sessions 800mbit/s >>> >>> >>> I am not familiar with troubleshooting kernel panic's, >>> >>> I am suspecting that the crash is happening inside the netgraph >>> module because the crash is happening at the >>> >>> instruction pointer = 0x20:0xffffffff81c38283 >>> I included the 2 two crash logs. I need some help to to figure out >>> what to do next. >>> >>> -Dbaud >>> >>> >> On 7/10/16 5:14 PM, Hooman Fazaeli wrote: >> - Upgrade to mpd 5 (/usr/ports/net/mpd5) >> - Try below workarounds: >> https://lists.freebsd.org/pipermail/freebsd-bugs/2014-June/056548.html >> https://lists.freebsd.org/pipermail/freebsd-bugs/2014-June/056549.html >> https://lists.freebsd.org/pipermail/freebsd-net/2014-June/038954.html >> >> > On 7/10/16 8:43 PM, Donald Baud via freebsd-net wrote: > - I'm already using the latest mpd5: > > mpd5 --version > Version 5.8 (root@101amd64-quarterly-job-15 12:36 5-Jun-2016) > > - I had already reviewed those links you mentioned. > Here is a summary of the main suggestions in them. > * Add a "sleep 1" to up-down interface events. > * Revert to RELENG8 or 9 > * boost net.graph sysctl/loader.conf > net.graph.maxdata=262140 # /boot/loader.conf > net.graph.maxalloc=262140 # /boot.loader.conf > > I was using the following tunings > net.graph.maxdgram=524288 (via sysctl.conf default=20480) > net.graph.recvspace=524288 (via sysctl.conf default=20480) > net.graph.maxdata=65536 (via loader.conf default=4096 ) > net.graph.maxalloc=65536 (via loader.conf default=4096 ) > > I am suspecting that the panic might be caused by a too high maxdata > and maxalloc values: > I reduced the value to 20480, I'll report back if that will reduce the > occurence of kernel panics. > >> vmstat -z | head -1 ; vmstat -z | grep -i graph > ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP > NetGraph items: 72, 20491, 2, 1672,467166841, 0, 0 > NetGraph data items: 72, 20491, 0, 1643,1240166475, 0, 0 > > The server crashed again this morning. It looks like it crashes somewhere in the netgraph.ko module Could someone please help me troubleshoot this issue, it crashes around the same location instruction pointer = 0x20:0xffffffff81c3828d The crash happens at random times not necessarily under heavy load. - using plain GENERIC kernel 10.3-RELEASE-p4 FreeBSD 10.3-RELEASE-p4 #0: Sat May 28 12:23:44 UTC 2016 root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64 - # kldstat Id Refs Address Size Name 1 32 0xffffffff80200000 17bc6a8 kernel 2 2 0xffffffff81c11000 114db ipfw.ko 3 1 0xffffffff81c23000 d32f dummynet.ko 4 1 0xffffffff81c31000 3831 ng_socket.ko 5 8 0xffffffff81c35000 ba02 netgraph.ko 6 1 0xffffffff81c41000 2b99 ng_mppc.ko 7 1 0xffffffff81c44000 80c rc4.ko 8 1 0xffffffff81c45000 23dc vmmemctl.ko 9 1 0xffffffff81c48000 397d ng_l2tp.ko 10 1 0xffffffff81c4c000 4b04 ng_ksocket.ko 11 1 0xffffffff81c51000 17d6 ng_tee.ko 12 1 0xffffffff81c53000 40d2 ng_iface.ko 13 1 0xffffffff81c58000 5829 ng_ppp.ko 14 1 0xffffffff81c5e000 18b1 ng_tcpmss.ko - /etc/rc.conf mpd_enable="YES" quagga_daemons="zebra ospfd" devd_enable="NO" ipv6_network_interfaces="none" ip6addrctl_enable="NO" - /etc/sysctl.conf net.inet.ip.fastforwarding=1 hw.intr_storm_threshold=40000 net.graph.maxdgram=524288 net.graph.recvspace=524288 - /boot/loader.conf net.graph.maxdata=20480 net.graph.maxalloc=20480 - grep kernel: /var/log/messages Jul 12 04:18:05 mybox syslogd: kernel boot file is /boot/kernel/kernel Jul 12 04:18:05 mybox kernel: Jul 12 04:18:05 mybox kernel: Jul 12 04:18:05 mybox kernel: Fatal trap 9: general protection fault while in kernel mode Jul 12 04:18:05 mybox kernel: cpuid = 0; apic id = 00 Jul 12 04:18:05 mybox kernel: instruction pointer = 0x20:0xffffffff81c3828d Jul 12 04:18:05 mybox kernel: stack pointer = 0x28:0xfffffe0174da8380 Jul 12 04:18:05 mybox kernel: frame pointer = 0x28:0xfffffe0174da83c0 Jul 12 04:18:05 mybox kernel: code segment = base 0x0, limit 0xfffff, type 0x1b Jul 12 04:18:05 mybox kernel: = DPL 0, pres 1, long 1, def32 0, gran 1 Jul 12 04:18:05 mybox kernel: processor eflags = interrupt enabled, resume, IOPL = 0 Jul 12 04:18:05 mybox kernel: current process = 659 (ng_queue3) Jul 12 04:18:05 mybox kernel: trap number = 9 Jul 12 04:18:05 mybox kernel: panic: general protection fault Jul 12 04:18:05 mybox kernel: cpuid = 0 Jul 12 04:18:05 mybox kernel: KDB: stack backtrace: Jul 12 04:18:05 mybox kernel: #0 0xffffffff8098e390 at kdb_backtrace+0x60 Jul 12 04:18:05 mybox kernel: #1 0xffffffff80951066 at vpanic+0x126 Jul 12 04:18:05 mybox kernel: #2 0xffffffff80950f33 at panic+0x43 Jul 12 04:18:05 mybox kernel: #3 0xffffffff80d55f7b at trap_fatal+0x36b Jul 12 04:18:05 mybox kernel: #4 0xffffffff80d55bfd at trap+0x77d Jul 12 04:18:05 mybox kernel: #5 0xffffffff80d3b8d2 at calltrap+0x8 Jul 12 04:18:05 mybox kernel: #6 0xffffffff81c49606 at ng_l2tp_rcvdata_lower+0x946 Jul 12 04:18:05 mybox kernel: #7 0xffffffff81c370ca at ng_apply_item+0x21a Jul 12 04:18:05 mybox kernel: #8 0xffffffff81c36d1a at ng_snd_item+0x38a Jul 12 04:18:05 mybox kernel: #9 0xffffffff81c4d3e2 at ng_ksocket_incoming2+0x2f2 Jul 12 04:18:05 mybox kernel: #10 0xffffffff81c36f62 at ng_apply_item+0xb2 Jul 12 04:18:05 mybox kernel: #11 0xffffffff81c38d39 at ngthread+0x1b9 Jul 12 04:18:05 mybox kernel: #12 0xffffffff8091a4ea at fork_exit+0x9a Jul 12 04:18:05 mybox kernel: #13 0xffffffff80d3be0e at fork_trampoline+0xe Jul 12 04:18:05 mybox kernel: Uptime: 1d19h53m59s Jul 12 04:18:05 mybox kernel: Dumping 560 out of 6119 MB:..3%..12%..23%..32%..43%..52%..63%..72%..83%..92% Jul 12 04:18:05 mybox kernel: Dump complete Jul 12 04:18:05 mybox kernel: [...]
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?2c786d19-7ba2-10ad-89a0-9a78abfe7146>