Date: Mon, 26 Jan 2004 01:26:15 -0500 (EST) From: Andre Guibert de Bruet <andy@siliconlandmark.com> To: freebsd-current@freebsd.org Subject: Re: Processes blocked on ufs or getblk Message-ID: <20040126005108.Q42487@alpha.siliconlandmark.com> In-Reply-To: <20040115015136.B47506@alpha.siliconlandmark.com> References: <D90B8DC2-471E-11D8-A84C-000A95DBB47C@ca.com> <20040115015136.B47506@alpha.siliconlandmark.com>
next in thread | previous in thread | raw e-mail | index | archive | help
This message is in MIME format. The first part should be readable text, while the remaining parts are likely unreadable without MIME-aware tools. Send mail to mime@docserver.cac.washington.edu for more info. --0-198582552-1075097407=:42487 Content-Type: TEXT/PLAIN; CHARSET=US-ASCII Content-ID: <20040126012323.H42487@alpha.siliconlandmark.com> On Thu, 15 Jan 2004, Andre Guibert de Bruet wrote: > On Thu, 15 Jan 2004, Lachlan O'Dea wrote: > > > -----BEGIN PGP SIGNED MESSAGE----- > > > > I found some discussion about this in December, but I don't think > > anyone has been able to get to the bottom of it yet. The symptom is > > that processes become permanently blocked in a state of ufs or getblk. > > I can reproduce it with find at will: > > > > % ps axl | grep ufs > > 0 13225 13215 1 -4 0 1300 804 ufs D ?? 0:00.96 find > > /var -xdev -type f ( -perm -u+x -or -perm -g+x -or -perm -o+ > > 0 28778 28765 0 -4 0 1300 804 ufs D ?? 0:00.97 find > > /var -xdev -type f ( -perm -u+x -or -perm -g+x -or -perm -o+ > > 0 33017 32933 2 -4 0 1304 788 ufs D p2- 0:10.69 find > > / -name samba > > > > It has also happened several times in single user mode to makewhatis > > running at the end of installworld. > > > > System details: 5.2-RC FreeBSD 5.2-RC #1: Fri Jan 9 04:45:51 EST 2004. > > Dell PowerEdge 2500. All filesystems are on a single raid 5 volume > > using the aac driver. The box has two CPUs, but I'm currently running > > with kern.smp.disabled=1. > > > > % mount > > /dev/aacd0s1a on / (ufs, local) > > devfs on /dev (devfs, local) > > /dev/aacd0s1e on /usr (ufs, local, with quotas, soft-updates) > > /dev/aacd0s1d on /var (ufs, local, soft-updates) > > procfs on /proc (procfs, local) > > linprocfs on /usr/compat/linux/proc (linprocfs, local) > > > > I also have ACLs enabled on /usr, if that's at all relevant. > > > > The kernel has DDB and DEBUG_LOCKS. Please let me know if there's > > anything I can do to help debug this. > > > > I don't know if this is related, but another problem is that when > > shutting down, it always gives up on a bunch of buffers. I think I've > > seen over 100, but usually it's 4-10 buffers. > > I'm seeing the same thing on my desktop machine. It usually occurs while > scanning large directories and/or dealing with large collections of files > rather quickly. I came across this bug while using gqview to go through my > image collection and a second time while re-checking out my ports tree > from local cvs. The programs appear to grab an exclusive lock and anything > that tries to read or write to the directory (or get a directory listing) > gets stuck in ufs state. > > My kernel config is rather simple, GENERIC without a lot of cruft except > amr, ata, scsi, usb and pcm. I'll try to get the output of a ddb ps and a > show lockedvnods. I'm reviving this thread as I have more information that might help track this problem down. The offending process in this case is gqview but it could have been 'find /' or any other process running when there's high system load (such as daylies). >From the emails that I've gotten it appears that this bug affects users that are using either ccd or hardware raid (amr driver in my case). I've attached the output of a ddb ps and a 'show lockednods'. Every time the getblk hang rears it's ugly head, I've seen "amr0: bad slot x completed" (where x is an integer between 0 to 4) printed on the serial console. This makes me think that there's a failure mode or special state that isn't being checked with the amr driver. Perusing the code shows that the bad slot message is a result of a NULL busy command. I'm no storage driver and my VFS knowledge is somewhat limited. Anyone out there want to have a look at this? I'm willing to try out any patches on this system. I'm currently running: FreeBSD bling.home 5.2-CURRENT FreeBSD 5.2-CURRENT #1: Thu Jan 22 11:38:46 EST 2004 andy@bling.home:/usr/src/sys/i386/compile/BLING i386 Full Kernel config file is up at: http://bling.properkernel.com/BLING I'll have a boot -v up shortly at: http://bling.properkernel.com/boot-v.txt Regards, > Andre Guibert de Bruet | Enterprise Software Consultant > > Silicon Landmark, LLC. | http://siliconlandmark.com/ > --0-198582552-1075097407=:42487 Content-Type: TEXT/PLAIN; CHARSET=US-ASCII; NAME="getblk.txt" Content-Transfer-Encoding: BASE64 Content-ID: <20040126011007.Y42487@alpha.siliconlandmark.com> Content-Description: Content-Disposition: ATTACHMENT; FILENAME="getblk.txt" ZGI+IHBzDQogIHBpZCAgIHByb2MgICAgIHVhcmVhICAgdWlkICBwcGlkICBw Z3JwICBmbGFnICAgc3RhdCAgd21lc2cgICAgd2NoYW4gIGNtZA0KIDExNDAg NzcwODJhNTAgYjRkMGQwMDAgICAgMCAgICAgMSAgMTE0MCAwMDA0MDAyIFtT TFBdbmFuc2xwIDB4NjA3OTliYmNdIHJlYm9vdA0KIDEwNDMgNmFhMDhkYzAg YjRjM2MwMDAgIDUwMSAgICAgMSAgMTA0MiAwMDA0MDAwIFtTTFBdZ2V0Ymxr IDB4OTkyYmE3MjRdIGdxdmlldw0KICAgNTggNjhlYTk1MjggYjA4YmUwMDAg ICAgMCAgICAgMCAgICAgMCAwMDAwMjA0IFtTTFBdLSAweDYwN2M4MGFjXSBu ZnNpb2QgMw0KICAgNTcgNjhlYTk2ZTAgYjA4YmYwMDAgICAgMCAgICAgMCAg ICAgMCAwMDAwMjA0IFtTTFBdLSAweDYwN2M4MGE4XSBuZnNpb2QgMg0KICAg NTYgNjhlYTk4OTggYjA4YzAwMDAgICAgMCAgICAgMCAgICAgMCAwMDAwMjA0 IFtTTFBdLSAweDYwN2M4MGE0XSBuZnNpb2QgMQ0KICAgNTUgNjhlYTlhNTAg YjA4YzEwMDAgICAgMCAgICAgMCAgICAgMCAwMDAwMjA0IFtTTFBdLSAweDYw N2M4MGEwXSBuZnNpb2QgMA0KICAgNTQgNjhlYTljMDggYjA4YzIwMDAgICAg MCAgICAgMCAgICAgMCAwMDAwMjA0IFtTTFBddmxydXd0IDB4NjhlYTljMDhd IHZubHJ1DQogICA1MyA2OGVhOWRjMCBiMDhjMzAwMCAgICAwICAgICAwICAg ICAwIDAwMDAyMDQgW1NMUF1zeW5jZXIgMHg2MDc5OTU4MF0gc3luY2VyDQog ICA1MiA2OTBmMDAwMCBiMjkwMjAwMCAgICAwICAgICAwICAgICAwIDAwMDAy MDQgW1NMUF1wc2xlZXAgMHg2MDdjMTQyY10gYnVmZGFlbW9uDQogICA1MSA2 OTBmMDFiOCBiMjkwMzAwMCAgICAwICAgICAwICAgICAwIDAwMDAyMGMgW1NM UF1wZ3plcm8gMHg2MDdjZTgyOF0gcGFnZXplcm8NCiAgIDUwIDY5MGYwMzcw IGIyOTA0MDAwICAgIDAgICAgIDAgICAgIDAgMDAwMDIwNCBbU0xQXXBzbGVl cCAweDYwN2NlODgwXSB2bWRhZW1vbg0KICAgNDkgNjkwZjA1MjggYjI5MDUw MDAgICAgMCAgICAgMCAgICAgMCAwMDAwMjA0IFtTTFBdcHNsZWVwIDB4NjA3 Y2U4NmNdIHBhZ2VkYWVtb24NCiAgICA5IDY5MGYwNmUwIGIyOTA2MDAwICAg IDAgICAgIDAgICAgIDAgMDAwMDIwNCBbU0xQXS0gMHhiMjkzMGQwY10gc2No ZWRjcHUNCiAgIDQ4IDY5MGYwODk4IGIyOTRmMDAwICAgIDAgICAgIDAgICAg IDAgMDAwMDIwNCBbSVdBSVRdIHN3aTA6IHR0eTpzaW8NCiAgIDQ3IDY4ZTU1 YTUwIGIwODhiMDAwICAgIDAgICAgIDAgICAgIDAgMDAwMDIwNCBbU0xQXXVz YnRzayAweDYwNzkxYzA0XSB1c2J0YXNrDQogICA0NiA2OGU1NWMwOCBiMDg4 YzAwMCAgICAwICAgICAwICAgICAwIDAwMDAyMDQgW1NMUF11c2JldnQgMHg2 OGZjZDIxMF0gdXNiMA0KICAgIDggNjhlNTVkYzAgYjA4OGQwMDAgICAgMCAg ICAgMCAgICAgMCAwMDAwMjA0IFtTTFBdYWN0YXNrIDB4NjA4Y2IzNmNdIGFj cGlfdGFzazINCiAgICA3IDY4ZWE3MDAwIGIwODhlMDAwICAgIDAgICAgIDAg ICAgIDAgMDAwMDIwNCBbU0xQXWFjdGFzayAweDYwOGNiMzZjXSBhY3BpX3Rh c2sxDQogICAgNiA2OGVhNzFiOCBiMDg4ZjAwMCAgICAwICAgICAwICAgICAw IDAwMDAyMDQgW1NMUF1hY3Rhc2sgMHg2MDhjYjM2Y10gYWNwaV90YXNrMA0K LS1Nb3JlLS0NCiAgIDQ1IDY4ZWE3MzcwIGIwODkwMDAwICAgIDAgICAgIDAg ICAgIDAgMDAwMDIwNCBbSVdBSVRdIHN3aTc6IHRhc2sgcXVldWUNCiAgIDQ0 IDY4ZWE3NTI4IGIwODkxMDAwICAgIDAgICAgIDAgICAgIDAgMDAwMDIwNCBb SVdBSVRdIHN3aTc6IGFjcGl0YXNrcQ0KICAgNDMgNjhlYTc2ZTAgYjA4OTIw MDAgICAgMCAgICAgMCAgICAgMCAwMDAwMjA0IFtJV0FJVF0gc3dpMzogY2Ft YmlvDQogICA0MiA2OGVhNzg5OCBiMDg5MzAwMCAgICAwICAgICAwICAgICAw IDAwMDAyMDQgbmV3IFtJV0FJVF0gc3dpMjogY2FtbmV0DQogICA0MSA2OGVh N2E1MCBiMDg5NDAwMCAgICAwICAgICAwICAgICAwIDAwMDAyMDQgbmV3IFtJ V0FJVF0gc3dpNTorDQogICAgNSA2OGVhN2MwOCBiMDhiOTAwMCAgICAwICAg ICAwICAgICAwIDAwMDAyMDQgW1NMUF10cXRociAweDYwNzlhZmU4XSB0YXNr cXVldWUNCiAgIDQwIDY4ZWE3ZGMwIGIwOGJhMDAwICAgIDAgICAgIDAgICAg IDAgMDAwMDIwNCBbSVdBSVRdIHN3aTY6Kw0KICAgMzkgNjhlYTkwMDAgYjA4 YmIwMDAgICAgMCAgICAgMCAgICAgMCAwMDAwMjA0IFtTTFBdLSAweDYwNzhl OWEwXSByYW5kb20NCiAgICA0IDY4ZTRjNTI4IGIwODViMDAwICAgIDAgICAg IDAgICAgIDAgMDAwMDIwNCBbU0xQXS0gMHg2MDc5NDIyMF0gZ19kb3duDQog ICAgMyA2OGU0YzZlMCBiMDg1YzAwMCAgICAwICAgICAwICAgICAwIDAwMDAy MDQgW1NMUF0tIDB4NjA3OTQyMWNdIGdfdXANCiAgICAyIDY4ZTRjODk4IGIw ODVkMDAwICAgIDAgICAgIDAgICAgIDAgMDAwMDIwNCBbU0xQXS0gMHg2MDc5 NDIxNF0gZ19ldmVudA0KICAgMzggNjhlNGNhNTAgYjA4NWUwMDAgICAgMCAg ICAgMCAgICAgMCAwMDAwMjA0IG5ldyBbSVdBSVRdIHN3aTQ6IHZtDQogICAz NyA2OGU0Y2MwOCBiMDg1ZjAwMCAgICAwICAgICAwICAgICAwIDAwMDAyMGMg W0xPQ0sgIEdpYW50IDY5MTA5Y2MwXSBzd2k4OiB0dHk6c2lvIGNsb2NrDQog ICAzNiA2OGU0Y2RjMCBiMDg2MDAwMCAgICAwICAgICAwICAgICAwIDAwMDAy MDQgW0lXQUlUXSBzd2kxOiBuZXQNCiAgIDM1IDY4ZTU1MDAwIGIwODYxMDAw ICAgIDAgICAgIDAgICAgIDAgMDAwMDIwNCBuZXcgW0lXQUlUXSBpcnEwOiBj bGsNCiAgIDM0IDY4ZTU1MWI4IGIwODg2MDAwICAgIDAgICAgIDAgICAgIDAg MDAwMDIwNCBuZXcgW0lXQUlUXSBpcnEyMzoNCiAgIDMzIDY4ZTU1MzcwIGIw ODg3MDAwICAgIDAgICAgIDAgICAgIDAgMDAwMDIwNCBuZXcgW0lXQUlUXSBp cnEyMjoNCiAgIDMyIDY4ZTU1NTI4IGIwODg4MDAwICAgIDAgICAgIDAgICAg IDAgMDAwMDIwNCBbSVdBSVRdIGlycTIxOiBhbXIwDQogICAzMSA2OGU1NTZl MCBiMDg4OTAwMCAgICAwICAgICAwICAgICAwIDAwMDAyMDQgbmV3IFtJV0FJ VF0gaXJxMjA6DQogICAzMCA2OGU1NTg5OCBiMDg4YTAwMCAgICAwICAgICAw ICAgICAwIDAwMDAyMDQgW0lXQUlUXSBpcnExOTogZndvaGNpMSsNCi0tTW9y ZS0tDQogICAyOSA2NGY2NjFiOCBhZWUyYTAwMCAgICAwICAgICAwICAgICAw IDAwMDAyMDQgW0lXQUlUXSBpcnExODogcmwwDQogICAyOCA2NGY2NjM3MCBh ZWUyYjAwMCAgICAwICAgICAwICAgICAwIDAwMDAyMDQgW0lXQUlUXSBpcnEx NzogYXRhcGNpMSBwY20wDQogICAyNyA2NGY2NjUyOCBhZWUyYzAwMCAgICAw ICAgICAwICAgICAwIDAwMDAyMDQgW0lXQUlUXSBpcnExNjogZndvaGNpMA0K ICAgMjYgNjRmNjY2ZTAgYWVlMmQwMDAgICAgMCAgICAgMCAgICAgMCAwMDAw MjA0IFtJV0FJVF0gaXJxMTU6IGF0YTENCiAgIDI1IDY0ZjY2ODk4IGFlZTUy MDAwICAgIDAgICAgIDAgICAgIDAgMDAwMDIwNCBbSVdBSVRdIGlycTE0OiBh dGEwDQogICAyNCA2NGY2NmE1MCBhZWU1MzAwMCAgICAwICAgICAwICAgICAw IDAwMDAyMDQgbmV3IFtJV0FJVF0gaXJxMTM6DQogICAyMyA2NGY2NmMwOCBh ZWU1NDAwMCAgICAwICAgICAwICAgICAwIDAwMDAyMDQgbmV3IFtJV0FJVF0g aXJxMTI6DQogICAyMiA2NGY2NmRjMCBhZWU1NTAwMCAgICAwICAgICAwICAg ICAwIDAwMDAyMDQgbmV3IFtJV0FJVF0gaXJxMTE6DQogICAyMSA2OGU0YzAw MCBiMDg1ODAwMCAgICAwICAgICAwICAgICAwIDAwMDAyMDQgbmV3IFtJV0FJ VF0gaXJxMTA6DQogICAyMCA2OGU0YzFiOCBiMDg1OTAwMCAgICAwICAgICAw ICAgICAwIDAwMDAyMDQgbmV3IFtJV0FJVF0gaXJxOTogYWNwaTANCiAgIDE5 IDY4ZTRjMzcwIGIwODVhMDAwICAgIDAgICAgIDAgICAgIDAgMDAwMDIwNCBu ZXcgW0lXQUlUXSBpcnE4OiBydGMNCiAgIDE4IDY0ZjVkMDAwIGFlZGQ4MDAw ICAgIDAgICAgIDAgICAgIDAgMDAwMDIwNCBuZXcgW0lXQUlUXSBpcnE3OiBw cGMwDQogICAxNyA2NGY1ZDFiOCBhZWUyMTAwMCAgICAwICAgICAwICAgICAw IDAwMDAyMDQgbmV3IFtJV0FJVF0gaXJxNjoNCiAgIDE2IDY0ZjVkMzcwIGFl ZTIyMDAwICAgIDAgICAgIDAgICAgIDAgMDAwMDIwNCBuZXcgW0lXQUlUXSBp cnE1Og0KICAgMTUgNjRmNWQ1MjggYWVlMjMwMDAgICAgMCAgICAgMCAgICAg MCAwMDAwMjA0IG5ldyBbSVdBSVRdIGlycTQ6IHNpbzANCiAgIDE0IDY0ZjVk NmUwIGFlZTI0MDAwICAgIDAgICAgIDAgICAgIDAgMDAwMDIwNCBuZXcgW0lX QUlUXSBpcnEzOiBzaW8xDQogICAxMyA2NGY1ZDg5OCBhZWUyNTAwMCAgICAw ICAgICAwICAgICAwIDAwMDAyMDQgW0NQVSAwXSBpcnExOiBhdGtiZDANCiAg IDEyIDY0ZjVkYTUwIGFlZTI2MDAwICAgIDAgICAgIDAgICAgIDAgMDAwMDIw YyBbQ2FuIHJ1bl0gaWRsZTogY3B1MA0KICAgMTEgNjRmNWRjMDggYWVlMjcw MDAgICAgMCAgICAgMCAgICAgMCAwMDAwMjBjIFtDUFUgMV0gaWRsZTogY3B1 MQ0KICAgIDEgNjRmNWRkYzAgYWVlMjgwMDAgICAgMCAgICAgMCAgICAgMSAw MDA0MjAwIFtTTFBdd2FpdCAweDY0ZjVkZGMwXSBpbml0DQotLU1vcmUtLQ0K ICAgMTAgNjRmNjYwMDAgYWVlMjkwMDAgICAgMCAgICAgMCAgICAgMCAwMDAw MjA0IFtDVl1rdHJhY2UgMHg2MDc5NzdhNF0ga3RyYWNlDQogICAgMCA2MDc5 NDMyMCA2MGMxZjAwMCAgICAwICAgICAwICAgICAwIDAwMDAyMDAgW1NMUF1z Y2hlZCAweDYwNzk0MzIwXSBzd2FwcGVyDQpkYj4gIHNob3cgbG9ja2Vkdm5v ZHMNCkxvY2tlZCB2bm9kZXMNCjB4NmQ0YTFlMzg6IHRhZyB1ZnMsIHR5cGUg VlJFRywgdXNlY291bnQgMiwgd3JpdGVjb3VudCAwLCByZWZjb3VudCAyMSwg ZmxhZ3MgKFZWX09CSkJVRiksIGxvY2sgdHlwZSB1ZnM6IEVYQ0wgKGNvdW50 IDEpIGJ5IHRocmVhZCAweDZhYTA5ZTcwIChwaWQgMTA0MykNCglpbm8gMjI5 ODg3NTUsIG9uIGRldiBhbXJkMGEgKDQsIDMwKQ0KZGI+IA== --0-198582552-1075097407=:42487--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20040126005108.Q42487>