Date: Thu, 07 Aug 2003 17:06:36 -0700 From: Lars Eggert <larse@ISI.EDU> To: John Baldwin <jhb@FreeBSD.org> Cc: current <current@FreeBSD.ORG> Subject: Re: spin lock sched lock held for > 5 seconds Message-ID: <3F32E98C.6090007@isi.edu> In-Reply-To: <XFMail.20030803003123.jhb@FreeBSD.org> References: <XFMail.20030803003123.jhb@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
This is a cryptographically signed message in MIME format.
--------------ms010100080307030107020403
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Transfer-Encoding: 7bit
John Baldwin wrote:
> On 01-Aug-2003 Lars Eggert wrote:
>
>>Hi,
>>
>>got the following panic overnight running with all debugging options on
>>(WITNESS, MUTEX_DEBUG, DIAGNOSTIC, INVARIANTS; WITNESS_SKIPSPIN off):
>>
>>panic: spin lock sched lock held by 0xc658e130 for > 5 seconds
>>cpuid = 0; lapic.id = 00000000
>>Stack backtrace:
>>backtrace(c031d030,0,c031c4c5,df0dab8c,100) at backtrace+0x17
>>panic(c031c4c5,c031c62e,c658e130,c036f160,18b) at panic+0x13d
>>_mtx_lock_spin(c036f160,2,c031a229,18b,c21b2ab0) at _mtx_lock_spin+0x83
>>_mtx_lock_spin_flags(c036f160,2,c031a229,18b,df0dac0c) at
>>_mtx_lock_spin_flags+0xb9
>>statclock(df0dac00,df0dac44,c02d8a9c,0,c2198d00) at statclock+0x39
>>rtcintr(0) at rtcintr+0x4f
>>Xfastunpend8(df0dacb8,c02d1f05,8,608,c0372e60) at Xfastunpend8+0x1c
>>call_fast_unpend(8,608,c0372e60,ffc00034,0) at call_fast_unpend+0xd
>>i386_unpend(c036f160,c21b0790,df0dacd0,c01b1ae0,df0dacec) at
>>i386_unpend+0x8d
>>cpu_unpend(df0dacec,c01a2534,c036f160,1,c031c2e2) at cpu_unpend+0x2d
>>critical_exit(c036f160,1,c031c2e2,1bc,1) at critical_exit+0x2d
>>_mtx_unlock_spin_flags(c036f160,0,c031ac9f,7c,c21b2ab0) at
>>_mtx_unlock_spin_flags+0xbb
>>idle_proc(0,df0dad48,c031ab89,312,c21b2ab0) at idle_proc+0xb0
>>fork_exit(c0199972,0,df0dad48) at fork_exit+0xc3
>>fork_trampoline() at fork_trampoline+0x1a
>>--- trap 0x1, eip = 0, esp = 0xdf0dad7c, ebp = 0 ---
>>Debugger("panic")
>>timeout stopping cpus
>>Stopped at Debugger+0x4f: xchgl %ebx,in_Debugger.0
>>db>
>>
>>The machine is still in ddb, let me know if I can provide additional info.
>
> Try updating to a more recent current. I recently added some extra
> debugging here that will attempt to better show who owns the lock and
> where it was acquired.
Same panic string, but a different call chain:
spin lock sched lock held by 0xc21b2d10 for > 5 seconds
exclusive spin mutex sched lock r = 0 (0xc036efc0) locked @
/usr/src/sys/kern/kern_mutex.c:512
panic: spin lock held too long
cpuid = 3; lapic.id = 03000000
Stack backtrace:
backtrace(c031ce60,3000000,c031c305,df0d0c30,100) at backtrace+0x17
panic(c031c305,c21b2d10,c21b2d10,c036efc0,bc) at panic+0x13d
_mtx_lock_spin(c036efc0,2,c031a037,bc,c21b2720) at _mtx_lock_spin+0xb4
_mtx_lock_spin_flags(c036efc0,2,c031a037,bc,c21b2720) at
_mtx_lock_spin_flags+0xb9
hardclock_process(df0d0ca0,df0d0ce4,c02d7062,0,c0390018) at
hardclock_process+0x3c
forwarded_hardclock(0) at forwarded_hardclock+0x11
Xhardclock(df0d0d10,c01998d7,c036f000,2,c031aaad) at Xhardclock+0x52
cpu_idle(c036f000,2,c031aaad,5f,c21b2720) at cpu_idle+0x8
idle_proc(0,df0d0d48,c031a997,30e,0) at idle_proc+0x45
fork_exit(c0199892,0,df0d0d48) at fork_exit+0xc3
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xdf0d0d7c, ebp = 0 ---
Debugger("panic")
timeout stopping cpus
Stopped at Debugger+0x4f: xchgl %ebx,in_Debugger.0
db>
Machine is still in ddb, in case you'd like me to poke around some more.
Lars
--
Lars Eggert <larse@isi.edu> USC Information Sciences Institute
--------------ms010100080307030107020403
Content-Type: application/x-pkcs7-signature; name="smime.p7s"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="smime.p7s"
Content-Description: S/MIME Cryptographic Signature
MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIJtjCC
AzgwggKhoAMCAQICEGZFcrfMdPXPY3ZFhNAukQEwDQYJKoZIhvcNAQEEBQAwgdExCzAJBgNV
BAYTAlpBMRUwEwYDVQQIEwxXZXN0ZXJuIENhcGUxEjAQBgNVBAcTCUNhcGUgVG93bjEaMBgG
A1UEChMRVGhhd3RlIENvbnN1bHRpbmcxKDAmBgNVBAsTH0NlcnRpZmljYXRpb24gU2Vydmlj
ZXMgRGl2aXNpb24xJDAiBgNVBAMTG1RoYXd0ZSBQZXJzb25hbCBGcmVlbWFpbCBDQTErMCkG
CSqGSIb3DQEJARYccGVyc29uYWwtZnJlZW1haWxAdGhhd3RlLmNvbTAeFw0wMDA4MzAwMDAw
MDBaFw0wNDA4MjcyMzU5NTlaMIGSMQswCQYDVQQGEwJaQTEVMBMGA1UECBMMV2VzdGVybiBD
YXBlMRIwEAYDVQQHEwlDYXBlIFRvd24xDzANBgNVBAoTBlRoYXd0ZTEdMBsGA1UECxMUQ2Vy
dGlmaWNhdGUgU2VydmljZXMxKDAmBgNVBAMTH1BlcnNvbmFsIEZyZWVtYWlsIFJTQSAyMDAw
LjguMzAwgZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJAoGBAN4zMqZjxwklRT7SbngnZ4HF2ogZ
gpcO40QpimM1Km1wPPrcrvfudG8wvDOQf/k0caCjbZjxw0+iZdsN+kvx1t1hpfmFzVWaNRqd
knWoJ67Ycvm6AvbXsJHeHOmr4BgDqHxDQlBRh4M88Dm0m1SKE4f/s5udSWYALQmJ7JRr6aFp
AgMBAAGjTjBMMCkGA1UdEQQiMCCkHjAcMRowGAYDVQQDExFQcml2YXRlTGFiZWwxLTI5NzAS
BgNVHRMBAf8ECDAGAQH/AgEAMAsGA1UdDwQEAwIBBjANBgkqhkiG9w0BAQQFAAOBgQAxsUtH
XfkBceX1U2xdedY9mMAmE2KBIqcS+CKV6BtJtyd7BDm6/ObyJOuR+r3sDSo491BVqGz3Da1M
G7wD9LXrokefbKIMWI0xQgkRbLAaadErErJAXWr5edDqLiXdiuT82w0fnQLzWtvKPPZE6iZp
h39Ins6ln+eE2MliYq0FxjCCAzkwggKioAMCAQICAwp2bzANBgkqhkiG9w0BAQQFADCBkjEL
MAkGA1UEBhMCWkExFTATBgNVBAgTDFdlc3Rlcm4gQ2FwZTESMBAGA1UEBxMJQ2FwZSBUb3du
MQ8wDQYDVQQKEwZUaGF3dGUxHTAbBgNVBAsTFENlcnRpZmljYXRlIFNlcnZpY2VzMSgwJgYD
VQQDEx9QZXJzb25hbCBGcmVlbWFpbCBSU0EgMjAwMC44LjMwMB4XDTAzMDgwMTE3MjkyOVoX
DTA0MDczMTE3MjkyOVowVDEPMA0GA1UEBBMGRWdnZXJ0MQ0wCwYDVQQqEwRMYXJzMRQwEgYD
VQQDEwtMYXJzIEVnZ2VydDEcMBoGCSqGSIb3DQEJARYNbGFyc2VAaXNpLmVkdTCCASIwDQYJ
KoZIhvcNAQEBBQADggEPADCCAQoCggEBAMb7PuLXnwV+45vwlkgogdSijd5HVqUB14bWvoK0
MjWPnkLPMDMDEezdsMG1BPiZyNeqXlJJtEgdAK8H2Mc9/qLeJUq3CoAeD6Wrjq4QaxJBXgdS
KcGDeQAZSDgwUJS9vx9+cXJVfLyOYxJ+CLBcO/eu8PvSi17lk6oeAbrskSGDu/Xi1o2SC4Qm
l69k8xcZQEMQDodkIk/U5SJmsCRGGYdy7opHZb58yXI8eiIGp5MlgryFmmgrp1pg3OYzPOR9
zJjn7Pu1vsd97LM5hLnKrmNuYt02jLNSjr8HmpLyWCDZq4Jlfq1YgNYZZ4KOSxipia7Bxjcs
nMOsxEWiolkVVT8CAwEAAaNWMFQwKgYFK2UBBAEEITAfAgEAMBowGAIBBAQTTDJ1TXlmZkJO
VWJOSkpjZFoyczAYBgNVHREEETAPgQ1sYXJzZUBpc2kuZWR1MAwGA1UdEwEB/wQCMAAwDQYJ
KoZIhvcNAQEEBQADgYEANRaPsUtrdJzTW0AMj/EQamqxOkZnzwnPWGryqskMKIf+OKa+eaXp
zlBv8CHdffv9hrYpvzWUxk0WW+YJ2LRdd4fFiVGXZCGU60eYeZGf7Z8ORoexylJpvUuKZCE4
aPGY2/QZXDfOs1NE82Bhgltx59dpWfH2K0dxbpHslO8/IbowggM5MIICoqADAgECAgMKdm8w
DQYJKoZIhvcNAQEEBQAwgZIxCzAJBgNVBAYTAlpBMRUwEwYDVQQIEwxXZXN0ZXJuIENhcGUx
EjAQBgNVBAcTCUNhcGUgVG93bjEPMA0GA1UEChMGVGhhd3RlMR0wGwYDVQQLExRDZXJ0aWZp
Y2F0ZSBTZXJ2aWNlczEoMCYGA1UEAxMfUGVyc29uYWwgRnJlZW1haWwgUlNBIDIwMDAuOC4z
MDAeFw0wMzA4MDExNzI5MjlaFw0wNDA3MzExNzI5MjlaMFQxDzANBgNVBAQTBkVnZ2VydDEN
MAsGA1UEKhMETGFyczEUMBIGA1UEAxMLTGFycyBFZ2dlcnQxHDAaBgkqhkiG9w0BCQEWDWxh
cnNlQGlzaS5lZHUwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDG+z7i158FfuOb
8JZIKIHUoo3eR1alAdeG1r6CtDI1j55CzzAzAxHs3bDBtQT4mcjXql5SSbRIHQCvB9jHPf6i
3iVKtwqAHg+lq46uEGsSQV4HUinBg3kAGUg4MFCUvb8ffnFyVXy8jmMSfgiwXDv3rvD70ote
5ZOqHgG67JEhg7v14taNkguEJpevZPMXGUBDEA6HZCJP1OUiZrAkRhmHcu6KR2W+fMlyPHoi
BqeTJYK8hZpoK6daYNzmMzzkfcyY5+z7tb7HfeyzOYS5yq5jbmLdNoyzUo6/B5qS8lgg2auC
ZX6tWIDWGWeCjksYqYmuwcY3LJzDrMRFoqJZFVU/AgMBAAGjVjBUMCoGBStlAQQBBCEwHwIB
ADAaMBgCAQQEE0wydU15ZmZCTlViTkpKY2RaMnMwGAYDVR0RBBEwD4ENbGFyc2VAaXNpLmVk
dTAMBgNVHRMBAf8EAjAAMA0GCSqGSIb3DQEBBAUAA4GBADUWj7FLa3Sc01tADI/xEGpqsTpG
Z88Jz1hq8qrJDCiH/jimvnml6c5Qb/Ah3X37/Ya2Kb81lMZNFlvmCdi0XXeHxYlRl2QhlOtH
mHmRn+2fDkaHscpSab1LimQhOGjxmNv0GVw3zrNTRPNgYYJbcefXaVnx9itHcW6R7JTvPyG6
MYID1TCCA9ECAQEwgZowgZIxCzAJBgNVBAYTAlpBMRUwEwYDVQQIEwxXZXN0ZXJuIENhcGUx
EjAQBgNVBAcTCUNhcGUgVG93bjEPMA0GA1UEChMGVGhhd3RlMR0wGwYDVQQLExRDZXJ0aWZp
Y2F0ZSBTZXJ2aWNlczEoMCYGA1UEAxMfUGVyc29uYWwgRnJlZW1haWwgUlNBIDIwMDAuOC4z
MAIDCnZvMAkGBSsOAwIaBQCgggIPMBgGCSqGSIb3DQEJAzELBgkqhkiG9w0BBwEwHAYJKoZI
hvcNAQkFMQ8XDTAzMDgwODAwMDYzN1owIwYJKoZIhvcNAQkEMRYEFJAERy2cam+UQ+jof+7P
TRtAEdgHMFIGCSqGSIb3DQEJDzFFMEMwCgYIKoZIhvcNAwcwDgYIKoZIhvcNAwICAgCAMA0G
CCqGSIb3DQMCAgFAMAcGBSsOAwIHMA0GCCqGSIb3DQMCAgEoMIGrBgkrBgEEAYI3EAQxgZ0w
gZowgZIxCzAJBgNVBAYTAlpBMRUwEwYDVQQIEwxXZXN0ZXJuIENhcGUxEjAQBgNVBAcTCUNh
cGUgVG93bjEPMA0GA1UEChMGVGhhd3RlMR0wGwYDVQQLExRDZXJ0aWZpY2F0ZSBTZXJ2aWNl
czEoMCYGA1UEAxMfUGVyc29uYWwgRnJlZW1haWwgUlNBIDIwMDAuOC4zMAIDCnZvMIGtBgsq
hkiG9w0BCRACCzGBnaCBmjCBkjELMAkGA1UEBhMCWkExFTATBgNVBAgTDFdlc3Rlcm4gQ2Fw
ZTESMBAGA1UEBxMJQ2FwZSBUb3duMQ8wDQYDVQQKEwZUaGF3dGUxHTAbBgNVBAsTFENlcnRp
ZmljYXRlIFNlcnZpY2VzMSgwJgYDVQQDEx9QZXJzb25hbCBGcmVlbWFpbCBSU0EgMjAwMC44
LjMwAgMKdm8wDQYJKoZIhvcNAQEBBQAEggEAppeds2qC3fEhaKugv/mBWEXE0GTNwgroI3vW
xHxt7Tf1OiF7GAfieRWk+gcGzcCEY1+/Q/oOYuOi6+JqcTk2akxxDe4F7ZqF7yhENk+l71dG
Ka4WMTN/k6IK51LQMvUZaCpaepQf2e3Da7U8xJjAx+KQ54NA2cJTpvIamofYKk2GjvHnCpzg
UsQmG4HsGSHIeWJMzFxnDsn716a0PY6IemnZ0Rh6moX8zHxyZrKNFh/gFNxKYxKYaBgne/Un
Wc8edHFpir7uNzGyEOxcfTccj2YcpCrGDwWH4lw7wAjGHQS1QtvKHo/2P9icQ8GRKK3xVuci
7CZPjyzevvUIhEydLgAAAAAAAA==
--------------ms010100080307030107020403--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?3F32E98C.6090007>