Date: Thu, 16 Jul 2009 10:33:02 -0700 From: Nick Esborn <nick@desert.net> To: Attilio Rao <attilio@freebsd.org> Cc: freebsd-threads@freebsd.org Subject: Re: threads/136345: Recursive read rwlocks in thread A cause deadlock with write lock in thread B Message-ID: <6E6A9516-6C69-4E41-803C-FE5F126F402C@desert.net> In-Reply-To: <3bbf2fe10907160548l74de896bka609de7a9a994899@mail.gmail.com> References: <200907152140.n6FLe42l045879@freefall.freebsd.org> <3bbf2fe10907160526l2f066698qce8a5e77aee6366b@mail.gmail.com> <3bbf2fe10907160548l74de896bka609de7a9a994899@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --Apple-Mail-24-1022926530 Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit On Jul 16, 2009, at 5:48 AM, Attilio Rao wrote: > 2009/7/16 Attilio Rao <attilio@freebsd.org>: >> 2009/7/15 Nick Esborn <nick@desert.net>: >>> The following reply was made to PR threads/136345; it has been >>> noted by GNATS. >>> >>> From: Nick Esborn <nick@desert.net> >>> To: bug-followup@FreeBSD.org, >>> rink@FreeBSD.org >>> Cc: >>> Subject: Re: threads/136345: Recursive read rwlocks in thread A >>> cause deadlock with write lock in thread B >>> Date: Wed, 15 Jul 2009 14:32:38 -0700 >>> >>> This is an OpenPGP/MIME signed message (RFC 2440 and 3156) >>> --Apple-Mail-19-950902279 >>> Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes >>> Content-Transfer-Encoding: 7bit >>> >>> Even after the above patch, I still run into occasional MySQL thread >>> deadlocks, which I originally described in what is now threads/ >>> 135673. >>> >>> I also posted on freebsd-current a few days ago: >>> >>> http://lists.freebsd.org/pipermail/freebsd-current/2009-July/009328.html >>> >>> I'd be happy to collect whatever data would be helpful in tracking >>> down this deadlock. This only seems to happen under our production >>> workload, so that might make it harder to capture meaningful debug >>> data, but I'm certainly willing to try. I can also arrange for >>> developer access to the system in question, if that would help >>> significantly. >> >> So did you backport this to 7 and still experience deadlocks? >> I just committed the fix to HEAD not to STABLE branch. > > Ok, I got, you just upgraded. > Can you try the following things?: > - Upgrade to the -CURRENT of today > - Recompile the kernel with the following options: > KDB, DDB, SCHED_ULE, PREEMPTION, FULL_PREEMPTION, INVARIANT_SUPPORT, > INVARIANTS, WITNESS > - When the deadlock takes place break into DDB and please retrieve the > following info: > db> show allpcpu > db> ps > db> alltrace > db> show alllock > > - Save them with a serial console output or using the textdump(4) > format. > > (if necessary read the ddb(4) and textdump(4) before to set up the > whole system). > This would shade a light if the problem lives within the kernel or > not. > > Thanks, > Attilio > > > -- > Peace can only be achieved by understanding - A. Einstein I can definitely do the upgrade. KDB, DDB, SCHED_ULE, and PREEMPTION are already turned on. I will try FULL_PREEMPTION, INVARIANT_SUPPORT, INVARIANTS, and WITNESS, but when I first upgraded to 8.0, this server was unable to handle its workload with the INVARIANTS and WITNESS options turned on. Also, it can take a while for it to become clear that the deadlock has occurred -- usually our monitoring picks it up when replication falls behind. So it may be 15-20 minutes after the deadlock that I am able to run the above db commands. Of course the thread will still be deadlocked. Hopefully that doesn't reduce the value of the data obtained. Thanks, -nick -- nick@desert.net - all messages cryptographically signed --Apple-Mail-24-1022926530 content-type: application/pgp-signature; x-mac-type=70674453; name=PGP.sig content-description: This is a digitally signed message part content-disposition: inline; filename=PGP.sig content-transfer-encoding: 7bit -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (Darwin) iEYEARECAAYFAkpfZE4ACgkQw1bX5UNr2ACFZACdHIvw3MJUMXNBqE3oktQ5dfg0 AJwAnjYZJ0WpC1lix1ZceWa6hTfNg5YY =6IoP -----END PGP SIGNATURE----- --Apple-Mail-24-1022926530--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?6E6A9516-6C69-4E41-803C-FE5F126F402C>