From owner-freebsd-net@freebsd.org Tue Jul 12 10:28:33 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 09D22B91F87 for ; Tue, 12 Jul 2016 10:28:33 +0000 (UTC) (envelope-from donaldbaud@yahoo.com) Received: from nm13-vm0.bullet.mail.bf1.yahoo.com (nm13-vm0.bullet.mail.bf1.yahoo.com [98.139.213.79]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C3F701D07 for ; Tue, 12 Jul 2016 10:28:32 +0000 (UTC) (envelope-from donaldbaud@yahoo.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1468319130; bh=s5T5qPp2Jv8hhamb19udnqNLjYlo9RKdo6jE9fsFSjY=; h=Subject:To:References:From:Date:In-Reply-To:From:Subject; b=C5LAPrLX0SMzxZJnXX3A4RmWZ3rKx1Ygr0B0o5S/17uHIH3RgfDggodnzlRRisDDfW4ob4hQs2KjvnFIwng7DytiCaW2MlLKYHVEF5dewPknVUgi4DhZAIZUWIMH1ozVEHehcE3sTGo+FlNFZWU5soPLGo0cTlwamRUugNRzvCWmFwsE1LyFjnoql2AjACLAx0/fKG3OEjC58Stl3eDIAIigCFDW5/X8uLOWSyuurGcHeuJOgxQpP/jzc2whl/DNtcJfxdose5xlwixPFC0ZZN7QQoZaHwYbPFd4eDeaaLQrxNc5a0EWPuoCpVpc9HpJP08DQwz/VywnHqQC61oWJA== Received: from [66.196.81.171] by nm13.bullet.mail.bf1.yahoo.com with NNFMP; 12 Jul 2016 10:25:30 -0000 Received: from [68.142.230.76] by tm17.bullet.mail.bf1.yahoo.com with NNFMP; 12 Jul 2016 10:25:30 -0000 Received: from [127.0.0.1] by smtp233.mail.bf1.yahoo.com with NNFMP; 12 Jul 2016 10:25:30 -0000 X-Yahoo-Newman-Id: 751801.52722.bm@smtp233.mail.bf1.yahoo.com X-Yahoo-Newman-Property: ymail-3 X-YMail-OSG: J9ZSep0VM1lzvDeejmjdp0oZf09RVQdUY35aiOxN7uLXSOC E89rq9YTdXd9QCZhRP..e7XiXpoGOTGMeGQVrQbq95lBKkUNVAsLkzGbAf2R wHvNe0aCJFXyVotn0LOUMcPk2QG8ur3Mdsntk9uBsCpa0JJZAe6nRunUTQs2 guL9fHkOCqtQ2zoZwcjbhg4czuKDnQsFBb_eejvmOqE_B1VakeNEl8Ci0_pO jLXSZwfCaQY26OTQmEtNwihofAVw.u_Ijiq9ZyPgI.iXNw8.F.5VYzPvY.vj oLPDwUAUpUSiT2dmzSOYdwOoQLY3.9qiPpJ2NrzfWUGqp_q9qE4V4R9exFVL VeI5ndKZO8mxBHhjcbxgkq6579oXh8Hn1WGBcGq1n7r9KSgq_Ib3pcLDF7Uo Cr3qCyNave98ZUy7GfhZhepAmyDC6AeLJNnAm_QcUo9VV_P3RFFRPNMJHDir fZ7ZkgofAqpiE5xJQJU0SRddOOlCMUNmvoyisqKaudNdRKaQGq5QEiRwjMOK UDo.sQ.e3yrvQrRaKOqrhscf_hIsD.iDsxuttChPKyT47FJyX.m7A1A-- X-Yahoo-SMTP: fRRiyyGswBB1GYfBhw8frTz_QYRdpxY- Subject: Re: kernel panic with netgraph and mpd5.8 To: freebsd-net@freebsd.org References: <58413371.417578.1468131588156.JavaMail.yahoo.ref@mail.yahoo.com> <58413371.417578.1468131588156.JavaMail.yahoo@mail.yahoo.com> <5782BAA4.2090606@gmail.com> <3b3c7a0a-35d4-93c5-c283-1b03f627ac38@yahoo.com> From: Donald Baud Message-ID: <2c786d19-7ba2-10ad-89a0-9a78abfe7146@yahoo.com> Date: Tue, 12 Jul 2016 06:25:36 -0400 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:45.0) Gecko/20100101 Thunderbird/45.2.0 MIME-Version: 1.0 In-Reply-To: <3b3c7a0a-35d4-93c5-c283-1b03f627ac38@yahoo.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Jul 2016 10:28:33 -0000 >> On 2016-07-10 10:49, Donald Baud via freebsd-net wrote: >>> Hi I'm running an l2tp lns through mpd5.8 and it's been crashing >>> twice in 24h. >>> This is a new project replacing a cisco 7206, 700-sessions 800mbit/s >>> >>> >>> I am not familiar with troubleshooting kernel panic's, >>> >>> I am suspecting that the crash is happening inside the netgraph >>> module because the crash is happening at the >>> >>> instruction pointer = 0x20:0xffffffff81c38283 >>> I included the 2 two crash logs. I need some help to to figure out >>> what to do next. >>> >>> -Dbaud >>> >>> >> On 7/10/16 5:14 PM, Hooman Fazaeli wrote: >> - Upgrade to mpd 5 (/usr/ports/net/mpd5) >> - Try below workarounds: >> https://lists.freebsd.org/pipermail/freebsd-bugs/2014-June/056548.html >> https://lists.freebsd.org/pipermail/freebsd-bugs/2014-June/056549.html >> https://lists.freebsd.org/pipermail/freebsd-net/2014-June/038954.html >> >> > On 7/10/16 8:43 PM, Donald Baud via freebsd-net wrote: > - I'm already using the latest mpd5: > > mpd5 --version > Version 5.8 (root@101amd64-quarterly-job-15 12:36 5-Jun-2016) > > - I had already reviewed those links you mentioned. > Here is a summary of the main suggestions in them. > * Add a "sleep 1" to up-down interface events. > * Revert to RELENG8 or 9 > * boost net.graph sysctl/loader.conf > net.graph.maxdata=262140 # /boot/loader.conf > net.graph.maxalloc=262140 # /boot.loader.conf > > I was using the following tunings > net.graph.maxdgram=524288 (via sysctl.conf default=20480) > net.graph.recvspace=524288 (via sysctl.conf default=20480) > net.graph.maxdata=65536 (via loader.conf default=4096 ) > net.graph.maxalloc=65536 (via loader.conf default=4096 ) > > I am suspecting that the panic might be caused by a too high maxdata > and maxalloc values: > I reduced the value to 20480, I'll report back if that will reduce the > occurence of kernel panics. > >> vmstat -z | head -1 ; vmstat -z | grep -i graph > ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP > NetGraph items: 72, 20491, 2, 1672,467166841, 0, 0 > NetGraph data items: 72, 20491, 0, 1643,1240166475, 0, 0 > > The server crashed again this morning. It looks like it crashes somewhere in the netgraph.ko module Could someone please help me troubleshoot this issue, it crashes around the same location instruction pointer = 0x20:0xffffffff81c3828d The crash happens at random times not necessarily under heavy load. - using plain GENERIC kernel 10.3-RELEASE-p4 FreeBSD 10.3-RELEASE-p4 #0: Sat May 28 12:23:44 UTC 2016 root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64 - # kldstat Id Refs Address Size Name 1 32 0xffffffff80200000 17bc6a8 kernel 2 2 0xffffffff81c11000 114db ipfw.ko 3 1 0xffffffff81c23000 d32f dummynet.ko 4 1 0xffffffff81c31000 3831 ng_socket.ko 5 8 0xffffffff81c35000 ba02 netgraph.ko 6 1 0xffffffff81c41000 2b99 ng_mppc.ko 7 1 0xffffffff81c44000 80c rc4.ko 8 1 0xffffffff81c45000 23dc vmmemctl.ko 9 1 0xffffffff81c48000 397d ng_l2tp.ko 10 1 0xffffffff81c4c000 4b04 ng_ksocket.ko 11 1 0xffffffff81c51000 17d6 ng_tee.ko 12 1 0xffffffff81c53000 40d2 ng_iface.ko 13 1 0xffffffff81c58000 5829 ng_ppp.ko 14 1 0xffffffff81c5e000 18b1 ng_tcpmss.ko - /etc/rc.conf mpd_enable="YES" quagga_daemons="zebra ospfd" devd_enable="NO" ipv6_network_interfaces="none" ip6addrctl_enable="NO" - /etc/sysctl.conf net.inet.ip.fastforwarding=1 hw.intr_storm_threshold=40000 net.graph.maxdgram=524288 net.graph.recvspace=524288 - /boot/loader.conf net.graph.maxdata=20480 net.graph.maxalloc=20480 - grep kernel: /var/log/messages Jul 12 04:18:05 mybox syslogd: kernel boot file is /boot/kernel/kernel Jul 12 04:18:05 mybox kernel: Jul 12 04:18:05 mybox kernel: Jul 12 04:18:05 mybox kernel: Fatal trap 9: general protection fault while in kernel mode Jul 12 04:18:05 mybox kernel: cpuid = 0; apic id = 00 Jul 12 04:18:05 mybox kernel: instruction pointer = 0x20:0xffffffff81c3828d Jul 12 04:18:05 mybox kernel: stack pointer = 0x28:0xfffffe0174da8380 Jul 12 04:18:05 mybox kernel: frame pointer = 0x28:0xfffffe0174da83c0 Jul 12 04:18:05 mybox kernel: code segment = base 0x0, limit 0xfffff, type 0x1b Jul 12 04:18:05 mybox kernel: = DPL 0, pres 1, long 1, def32 0, gran 1 Jul 12 04:18:05 mybox kernel: processor eflags = interrupt enabled, resume, IOPL = 0 Jul 12 04:18:05 mybox kernel: current process = 659 (ng_queue3) Jul 12 04:18:05 mybox kernel: trap number = 9 Jul 12 04:18:05 mybox kernel: panic: general protection fault Jul 12 04:18:05 mybox kernel: cpuid = 0 Jul 12 04:18:05 mybox kernel: KDB: stack backtrace: Jul 12 04:18:05 mybox kernel: #0 0xffffffff8098e390 at kdb_backtrace+0x60 Jul 12 04:18:05 mybox kernel: #1 0xffffffff80951066 at vpanic+0x126 Jul 12 04:18:05 mybox kernel: #2 0xffffffff80950f33 at panic+0x43 Jul 12 04:18:05 mybox kernel: #3 0xffffffff80d55f7b at trap_fatal+0x36b Jul 12 04:18:05 mybox kernel: #4 0xffffffff80d55bfd at trap+0x77d Jul 12 04:18:05 mybox kernel: #5 0xffffffff80d3b8d2 at calltrap+0x8 Jul 12 04:18:05 mybox kernel: #6 0xffffffff81c49606 at ng_l2tp_rcvdata_lower+0x946 Jul 12 04:18:05 mybox kernel: #7 0xffffffff81c370ca at ng_apply_item+0x21a Jul 12 04:18:05 mybox kernel: #8 0xffffffff81c36d1a at ng_snd_item+0x38a Jul 12 04:18:05 mybox kernel: #9 0xffffffff81c4d3e2 at ng_ksocket_incoming2+0x2f2 Jul 12 04:18:05 mybox kernel: #10 0xffffffff81c36f62 at ng_apply_item+0xb2 Jul 12 04:18:05 mybox kernel: #11 0xffffffff81c38d39 at ngthread+0x1b9 Jul 12 04:18:05 mybox kernel: #12 0xffffffff8091a4ea at fork_exit+0x9a Jul 12 04:18:05 mybox kernel: #13 0xffffffff80d3be0e at fork_trampoline+0xe Jul 12 04:18:05 mybox kernel: Uptime: 1d19h53m59s Jul 12 04:18:05 mybox kernel: Dumping 560 out of 6119 MB:..3%..12%..23%..32%..43%..52%..63%..72%..83%..92% Jul 12 04:18:05 mybox kernel: Dump complete Jul 12 04:18:05 mybox kernel: [...]