Date: Thu, 18 Aug 2011 20:04:10 +0300 From: Andriy Gapon <avg@FreeBSD.org> To: Andrew Boyer <aboyer@averesystems.com> Cc: freebsd-stable@FreeBSD.org, Eugene Grosbein <egrosbein@rdtc.ru>, Vishal.Shah@netapp.com, Hans Petter Selasky <hselasky@c2i.net>, Jeremiah Lott <jlott@averesystems.com>, Steven Hartland <killing@multiplay.co.uk> Subject: Re: USB/coredump hangs in 8 and 9 Message-ID: <4E4D460A.2080100@FreeBSD.org> In-Reply-To: <DA1FD6FD-2E57-4EC4-899D-2C1CBB769456@averesystems.com> References: <DA1FD6FD-2E57-4EC4-899D-2C1CBB769456@averesystems.com>
next in thread | previous in thread | raw e-mail | index | archive | help
on 12/08/2011 22:59 Andrew Boyer said the following: > Re: panic: bufwrite: buffer is not busy??? (originally on freebsd-net) > > Re: debugging frequent kernel panics on 8.2-RELEASE (originally on freebsd-stable) > > Re: System hang in USB umass module while processing panic (originally on > freebsd-usb) > > Hello Andriy and Hans, > > Sorry for tying in so many discussions on this topic, but I think I have an > explanation for the problems we have been reporting* with hanging coredumps on > multicore systems on 8.2-RELEASE, and it has implications for Andriy's proposed > scheduler patch** and for USB. > > In today's 8.X and 9.X branches, nothing that I can find stops the other CPUs when > the kernel panics, but many parts of the locking code get disabled (grep on > 'panicstr'). The 'bufwrite: buffer is not busy???' panic is caused by the syncer > encountering an error. If that happens when it's on the dumping CPU everything > hangs. If it's running on a different CPU, it will be blocked and hidden by the > panic_cpu spinlock in panic(), and the dump continues, polling every attached > keyboard for a Ctl-C. > > But, the new 8.X USB stack relies on multithreading. (The new stack is the > variable that broke coredumps for us in the 7.1->8.2 transition, I think.) SVN > 224223 fixes a hang that would happen when dumpsys() polls the USB keyboard (IPMI > KVM, in our case). That helps, but it only gets as far as usb_process(), where it > hangs in a loop around a cv_wait() call. This is easy to reproduce by adding code > to the watchdog to break into the debugger if panicstr is set. > > I am experimenting with Andriy's patch** to stop the scheduler and it seems to be > most of the way there, stopping the CPUs and disabling the rest of locking. There > are a few places that still reference panicstr, but that's minor. These are the > changes I made to the patch: > * Changed ukbd_do_poll() to return immediately if SCHEDULER_STOPPED() is true, so > that we don't hang up in USB. ukbd_yield() locks up in DROP_GIANT(), and if you > skip ukbd_yield(), usbd_transfer_poll() locks up trying to drop mutexes. Hmm, this is a little bit unexpected. I though that with the patch all the mutex/lock operations would be skipped. Can you please check which locks give you the trouble and why? I would like to improve the patch, so that all lock operations are by-passed (whether locking or unlocking). > * Changed the call to spinlock_enter() back to critical_enter(), so that > interrupts stay enabled and the hardclock still functions. Not sure if I like this idea in general. > * Added code in the beginning of panic() to switch to CPU 0, so that we're able > to service the hardclock interrupts and so that watchdog panics get through. Also I wouldn't like switching a panic thread to a different CPU as that messes up with a lot of state and is not safe for an arbitrary context. Also, can you please clarify what you meant by "watchdog panics get through"? Do you talk about SW_WATCHDOG specifically? > This has worked 100% for me so far, although anyone using a USB keyboard or dump > device would still be out of luck. > > Thoughts? It seems like stopping all of the other CPUs is the right thing to do > on a panic (what are they doing otherwise?). Are the USB issues fixable? If > Andriy's patch get committed it might just involve short-circuiting all of the > locking in the polling path, but I haven't gotten that far yet. I bet dumping to > NFS will have the same problem. I think that no subsystem should rely on working scheduling and interrupts in post-panic world. In fact, all the code for skipping locking is just a giant hack/workaround in my opinion. Ideally, all the subsystems that can be expected to be called after panic should be aware of that and should check for that. So they should not attempt any locking or switching threads or rebinding CPUs or expect interrupts, etc. The environment should mirror early boot where we have only one CPU, only one thread, no interrupts, only polling. If you can help Hans to figure out what you is wrong with USB subsystem in this respect that would help us all. Thank you for your testing and feedback! -- Andriy Gapon
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4E4D460A.2080100>