Date: Sun, 12 Jan 2014 23:33:36 +0200 From: Konstantin Belousov <kostikbel@gmail.com> To: sbruno@freebsd.org Cc: "freebsd-stable@freebsd.org" <freebsd-stable@freebsd.org> Subject: Re: panic: stable/10 with Debugging enabled in kern_cons.c:500 Message-ID: <20140112213336.GF59496@kib.kiev.ua> In-Reply-To: <1389561037.1395.10.camel@powernoodle.corp.yahoo.com> References: <1389478327.46758.7.camel@powernoodle.corp.yahoo.com> <1389480297.46758.9.camel@powernoodle.corp.yahoo.com> <20140112034203.GA59496@kib.kiev.ua> <1389561037.1395.10.camel@powernoodle.corp.yahoo.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--xC0pSYFZtSbvM9LT Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sun, Jan 12, 2014 at 01:10:37PM -0800, Sean Bruno wrote: > On Sun, 2014-01-12 at 05:42 +0200, Konstantin Belousov wrote: > > On Sat, Jan 11, 2014 at 02:44:57PM -0800, Sean Bruno wrote: > > > On Sat, 2014-01-11 at 14:12 -0800, Sean Bruno wrote: > > > > I can't imagine that I'm the first person to run with the following > > > > debug options enabled: > > > >=20 > > > > http://www.freebsd.org/doc/en_US.ISO8859-1/books/developers-handboo= k/kerneldebug-deadlocks.html > > > >=20 > > > > But, I can't get stable/10 to post on a plain old supermicro box wi= th > > > > those debugging symbols enabled. =20 > > > >=20 > > > >=20 > > > > random: unblocking device. > > > > panic: mtx_lock_spin: recursed on non-recursive mutex cnputs_mtx > > > > @ /usr/src/sys/kern/kern_cons.c:500 > > > >=20 > > > > cpuid =3D 0 > > > > KDB: stack backtrace: > > > > db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame > > > > 0xfffffe1764e681b0 > > > > kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe1764e68260 > > > > vpanic() at vpanic+0x126/frame 0xfffffe1764e682a0 > > > > kassert_panic() at kassert_panic+0x136/frame 0xfffffe1764e68310 > > > > __mtx_lock_spin_flags() at __mtx_lock_spin_flags+0x166/frame > > > > 0xfffffe1764e68350 > > > > cnputs() at cnputs+0x32/frame 0xfffffe1764e68370 > > > > putchar() at putchar+0x13a/frame 0xfffffe1764e683f0 > > > > kvprintf() at kvprintf+0xda/frame 0xfffffe1764e684f0 > > > > vprintf() at vprintf+0x87/frame 0xfffffe1764e685c0 > > > > printf() at printf+0x43/frame 0xfffffe1764e68620 > > > > witness_checkorder() at witness_checkorder+0xa99/frame > > > > 0xfffffe1764e686b0 > > > > __mtx_lock_spin_flags() at __mtx_lock_spin_flags+0x95/frame > > > > 0xfffffe1764e686f0 > > > > uart_cnputc() at uart_cnputc+0x3b/frame 0xfffffe1764e68710 > > > > cnputc() at cnputc+0x7f/frame 0xfffffe1764e68740 > > > > cnputs() at cnputs+0x58/frame 0xfffffe1764e68760 > > > > putchar() at putchar+0x13a/frame 0xfffffe1764e687e0 > > > > kvprintf() at kvprintf+0xda/frame 0xfffffe1764e688e0 > > > > vprintf() at vprintf+0x87/frame 0xfffffe1764e689b0 > > > > printf() at printf+0x43/frame 0xfffffe1764e68a10 > > > > witness_checkorder() at witness_checkorder+0xa99/frame > > > > 0xfffffe1764e68aa0 > > > > __mtx_lock_spin_flags() at __mtx_lock_spin_flags+0x95/frame > > > > 0xfffffe1764e68ae0 > > > > msleep_spin_sbt() at msleep_spin_sbt+0x90/frame 0xfffffe1764e68b70 > > > > random_kthread() at random_kthread+0x1d0/frame 0xfffffe1764e68bb0 > > > > fork_exit() at fork_exit+0x84/frame 0xfffffe1764e68bf0 > > > > fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe1764e68bf0 > > > > --- trap 0, rip =3D 0, rsp =3D 0xfffffe1764e68cb0, rbp =3D 0 --- > > > > KDB: enter: panic > > > > [ thread pid 14 tid 100057 ] > > > > Stopped at kdb_enter+0x3e: movq $0,kdb_why > > >=20 > > >=20 > > > Witness induced. Removed the witness option and this doesn't happen > > > (obviously). Can I get someone familiar with console code to look in= to > > > this? Kind of crappy when I'm investigating a resource starvation > > > issue. > >=20 > > Add > > options WITNESS_SKIPSPIN > > to the kernel config. This is required now, unfortunately. >=20 >=20 > Confirmed. Thank you. >=20 > It looks like the machine becomes seriously unuseable with this much > debugging on. After a few hours of testing. I cannot run "svn up" any > more and the system seems sluggish. >=20 > root@redbuild03:/usr/src # svn up -r 260075 > Updating '.': > load: 2.06 cmd: svn 12480 [*kmem arena] 624.87r 0.35u 434.25s 0% 11672k > load: 2.06 cmd: svn 12480 [*kmem arena] 625.71r 0.35u 434.25s 0% 11672k > load: 2.06 cmd: svn 12480 [*kmem arena] 625.91r 0.35u 434.25s 0% 11672k > load: 2.06 cmd: svn 12480 [*kmem arena] 626.09r 0.35u 434.25s 0% 11672k > load: 2.06 cmd: svn 12480 [*kmem arena] 626.27r 0.35u 434.25s 0% 11672k > load: 2.06 cmd: svn 12480 [*kmem arena] 626.45r 0.35u 434.25s 0% 11672k > load: 2.06 cmd: svn 12480 [*kmem arena] 626.61r 0.35u 434.25s 0% 11672k > load: 2.06 cmd: svn 12480 [*kmem arena] 626.79r 0.35u 434.25s 0% 11672k > load: 2.06 cmd: svn 12480 [*kmem arena] 626.95r 0.35u 434.25s 0% 11672k > load: 2.06 cmd: svn 12480 [*kmem arena] 627.13r 0.35u 434.25s 0% 11672k >=20 >=20 > Is this a symptom of the level of debugging or am I starting to come > close to a problem that needs investigation? This is a DIAGNOSTIC option, I think. Set sysctl debug.enable_vmem_check= =3D0; I thought that the default was changed in HEAD long time ago, but apparently it was not. --xC0pSYFZtSbvM9LT Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (FreeBSD) iQIcBAEBAgAGBQJS0wowAAoJEJDCuSvBvK1BTVsP/A3ivsh3kMLrarHOtIj2t0Uj wxohFfB5/Ol5Q2GsqwJ+pc48VqjIb3yy+2+P0eOZs+tLkx0D4rFxNysZ8osfERV4 AZt/senmam17mEpP1fEkXJobOhHO1BY9+szNV2iFoEdun2QIyOwNf5gZ32gMYhmO p1CwYbS/mqKjb1lKlT4IMbALieJhs2ljhXALmqUTvtcGbTAbHmYQOuW7w76uJHse jXR45j2orZsWBO/KMmadaD8govFuqF98ofDi6a1kBL2A3OyGi94diDgVHaQgn4yR YiLo5OC09k2ZH8J+mjZqoa5OrwNh/h4FJxRHZqRDqdoGpfArNpVzoXr6svqj6io5 GEBBU1zHVbaKlhMxxBWGlKrCd0lmoWyisr0LmpzHndrWdp188lyb77ghVI/fFi3U XlU8/q7M99YnVJ4LcnE4vGk7gu2ED/ErO0p8HRFt0tkJ4gWqyNYmM+nNs6n54i9x z7y54xrcg9dbJv4Okn3CxPVcDPF1cSIaOHt3uJ4KQqKC59R8MgIAx1MSijjtw2X7 jO9OfguxaJEGqgNO5AW43NZMThkQVS3RRKV7xCyErmeWxDmxquBm0nyojmk/Fcwn P1o1TetGbxMsKVNe5y9MTDKfFlHV//iKVtallQUFIGTjcPgLGwy2rGklYoEsa8VM siZ32Cfr1VGClT1R1iUp =V85i -----END PGP SIGNATURE----- --xC0pSYFZtSbvM9LT--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20140112213336.GF59496>