Date: Thu, 16 Nov 2006 09:24:07 +0100 From: Rink Springer <rink@FreeBSD.org> To: stable@FreeBSD.org Subject: panic(): vinvalbuf: dirty bufs: perhaps a ffs_syncvnode bug? Message-ID: <20061116082407.GB33390@rink.nu>
next in thread | raw e-mail | index | archive | help
--WhfpMioaduB5tiZL
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
Hi people,
At work, some storage is put on a SUN StorEdge 3510 FibreChannel array;
the disks are divided into two volumes, which are mounted using isp(4)
controllers. Two seperate machines each mount a single volume (this is,
box1 mounts the first volume (/dev/da1), and box2 mounts the second
volume (/dev/da2)). Both volumes are formatted using UFS2.
Over the night, we reset the shelf in order to activate its new
management IP address, causing the /dev/da[12] devices to be temporarily
unavailable. This resulted in the following panic on the rather busy
mailstorage server (the other server has minor load and was fine):
---
(da0:isp0:0:1:0): lost device
(da0:isp0:0:1:0): removing device entry
(da1:isp0:0:2:0): lost device
g_vfs_done():da1s1[WRITE(offset=3D292316823552, length=3D16384)]error =3D 6
g_vfs_done():da1s1[WRITE(offset=3D240287318016, length=3D16384)]error =3D 6
g_vfs_done():da1s1[READ(offset=3D12175362048, length=3D2048)]error =3D 6
g_vfs_done():da1s1[WRITE(offset=3D240287318016, length=3D16384)]error =3D 6
g_vfs_done():da1s1[READ(offset=3D18370689024, length=3D2048)]error =3D 6
g_vfs_done():da1s1[READ(offset=3D25829486592, length=3D512)]error =3D 6
vnode_pager_getpages: I/O read error
vm_fault: pager read error, pid 78035 (lmtpd)
g_vfs_done():da1s1[WRITE(offset=3D240287318016,
length=3D1638(da1:isp0:0:2:0): Invalidating pack
4)]error =3D 6
g_vfs_done():da1s1[READ(offset=3D13768671232, length=3D6144)]error =3D 6
g_vfs_done():da1s1[READ(offset=3D102126977024, length=3D16384)]error =3D 6
g_vfs_done():da1s1[READ(offset=3D13768671232, length=3D6144)]error =3D 6
g_vfs_dpone():da1s1[READ(offset=3D102319669248, length=3D16384)]error =3D 6a
nic: vinvalbuf: dirty bufs
cpuid =3D 2
Uptime: 54d15h48m38s
kernel trap 12 with interrupts disabled
Fatal trap 12: page fault while in kernel mode
cpuid =3D 0; apic id =3D 00
fault virtual address =3D 0x56
fault code =3D supervisor read, page not present
instruction pointer =3D 0x20:0xc0681303
stack pointer =3D 0x28:0xe8d973f0
frame pointer =3D 0x28:0xe8d973f8
code segment =3D base 0x0, limit 0xfffff, type 0x1b
=3D DPL 0, pres 1, def32 1, gran 1
processor eflags =3D resume, IOPL =3D 0
current process =3D 78066 (lmtpd)
trap number =3D 12
---
When looking at the source code of vinvalbuf(), which calls
bufobj_invalbuf(), it seems that this panic is raised after a bufobj
still contains dirty data after waiting for it to complete without
error. The code can be found at /sys/kern/vfs_subr.c
The sync routine called eventually translates to bufsync(), as in
/sys/kern/vfs_bio.c, which calls the filesystem's sync routine. It seems
as if the return status of vfs_bio_awrite() in ffs_syncvnode() is not
checked; all the other parts are checked. I believe this could provoke
this panic.
As the machine is in production use, it was instantly rebooted by a
collegue and thus I have no vmcore, backtrace or anything. I therefore
hope the information provided here is adequate.
Can someone with more FreeBSD-VFS knowledge please look at this?
Thanks!
--=20
Rink P.W. Springer - http://rink.nu
"It's you isn't it? THE BASTARD OPERATOR FROM HELL!"
"In the flesh, on the phone and in your account..." - BOFH #3
--WhfpMioaduB5tiZL
Content-Type: application/x-pkcs7-signature
Content-Disposition: attachment; filename="smime.p7s"
Content-Transfer-Encoding: base64
MIIJawYJKoZIhvcNAQcCoIIJXDCCCVgCAQExCzAJBgUrDgMCGgUAMAsGCSqGSIb3DQEHAaCC
BuIwggObMIIDBKADAgECAhAiuN7bs9pg6t3I0n6G5OOTMA0GCSqGSIb3DQEBBQUAMGIxCzAJ
BgNVBAYTAlpBMSUwIwYDVQQKExxUaGF3dGUgQ29uc3VsdGluZyAoUHR5KSBMdGQuMSwwKgYD
VQQDEyNUaGF3dGUgUGVyc29uYWwgRnJlZW1haWwgSXNzdWluZyBDQTAeFw0wNjExMDgwOTI2
NTNaFw0wNzExMDgwOTI2NTNaMIHSMREwDwYDVQQEEwhTcHJpbmdlcjEaMBgGA1UEKhMRUmlu
ayBQZXRlciBXeWNoZXIxIzAhBgNVBAMTGlJpbmsgUGV0ZXIgV3ljaGVyIFNwcmluZ2VyMRsw
GQYJKoZIhvcNAQkBFgxtYWlsQHJpbmsubnUxHzAdBgkqhkiG9w0BCQEWEHJpbmtAZnJlZWJz
ZC5vcmcxIDAeBgkqhkiG9w0BCQEWEXJpbmtAaWwuZm9udHlzLm5sMRwwGgYJKoZIhvcNAQkB
Fg1yaW5rQHN0YWNrLm5sMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAxryGDfel
YzzENX7wodkbVY1NALfaiPfNEG10YjD8ZWdK9zkN26Tc878Shbqapq0KYFD8TACGfEhKoMvo
qbf0PHAS/gNYr81Arqa9FRPUfzvtDE/cMbhvI+p7ufBITyYnPJp9MUD72iT+DohRR2ISVi3i
NAEgDuSbYYNxctnvXqU6O6EPy3mzoFPDoiOQwBfVtFrjxBbND9BUK2bjtUyGt4x8I/Vulzrt
qLPTokva+b97DHRgbCA/aLLYIrU6QoqOFJ8GrAbro/FZLYh4m1oJk3FEHVQOKkk7xzIaFmmP
QGJRL8m6nrIZFTrQ+X2wmzfLD55K/UiqbekOuMiWbY9EbwIDAQABo10wWzBLBgNVHREERDBC
gQxtYWlsQHJpbmsubnWBEHJpbmtAZnJlZWJzZC5vcmeBEXJpbmtAaWwuZm9udHlzLm5sgQ1y
aW5rQHN0YWNrLm5sMAwGA1UdEwEB/wQCMAAwDQYJKoZIhvcNAQEFBQADgYEAIfIcieRjePBA
wjZqvOdGpyPcNDnK/ubeQSTV5Y4AHWxm1sXhQxB/XrQ3RVdz1qDnBRL1AjkEBAl8e9+am4s6
D6TaSlmJeNXn6ZPJTQecisz3M+AKiMckShM3oAeUi0ktn1yNYR+hz5aQN612XT5OZRYznJVZ
kPf1DiA2RVVyz+MwggM/MIICqKADAgECAgENMA0GCSqGSIb3DQEBBQUAMIHRMQswCQYDVQQG
EwJaQTEVMBMGA1UECBMMV2VzdGVybiBDYXBlMRIwEAYDVQQHEwlDYXBlIFRvd24xGjAYBgNV
BAoTEVRoYXd0ZSBDb25zdWx0aW5nMSgwJgYDVQQLEx9DZXJ0aWZpY2F0aW9uIFNlcnZpY2Vz
IERpdmlzaW9uMSQwIgYDVQQDExtUaGF3dGUgUGVyc29uYWwgRnJlZW1haWwgQ0ExKzApBgkq
hkiG9w0BCQEWHHBlcnNvbmFsLWZyZWVtYWlsQHRoYXd0ZS5jb20wHhcNMDMwNzE3MDAwMDAw
WhcNMTMwNzE2MjM1OTU5WjBiMQswCQYDVQQGEwJaQTElMCMGA1UEChMcVGhhd3RlIENvbnN1
bHRpbmcgKFB0eSkgTHRkLjEsMCoGA1UEAxMjVGhhd3RlIFBlcnNvbmFsIEZyZWVtYWlsIElz
c3VpbmcgQ0EwgZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJAoGBAMSmPFVzVftOucqZWh5owHUE
cJ3f6f+jHuy9zfVb8hp2vX8MOmHyv1HOAdTlUAow1wJjWiyJFXCO3cnwK4Vaqj9xVsuvPAsH
5/EfkTYkKhPPK9Xzgnc9A74r/rsYPge/QIACZNenprufZdHFKlSFD0gEf6e20TxhBEAeZBly
YLf7AgMBAAGjgZQwgZEwEgYDVR0TAQH/BAgwBgEB/wIBADBDBgNVHR8EPDA6MDigNqA0hjJo
dHRwOi8vY3JsLnRoYXd0ZS5jb20vVGhhd3RlUGVyc29uYWxGcmVlbWFpbENBLmNybDALBgNV
HQ8EBAMCAQYwKQYDVR0RBCIwIKQeMBwxGjAYBgNVBAMTEVByaXZhdGVMYWJlbDItMTM4MA0G
CSqGSIb3DQEBBQUAA4GBAEiM0VCD6gsuzA2jZqxnD3+vrL7CF6FDlpSdf0whuPg2H6otnzYv
wPQcUCCTcDz9reFhYsPZOhl+hLGZGwDFGguCdJ4lUJRix9sncVcljd2pnDmOjCBPZV+V2vf3
h9bGCE6u9uo05RAaWzVNd+NWIXiC3CEZNd4ksdMdRv9dX2VPMYICUTCCAk0CAQEwdjBiMQsw
CQYDVQQGEwJaQTElMCMGA1UEChMcVGhhd3RlIENvbnN1bHRpbmcgKFB0eSkgTHRkLjEsMCoG
A1UEAxMjVGhhd3RlIFBlcnNvbmFsIEZyZWVtYWlsIElzc3VpbmcgQ0ECECK43tuz2mDq3cjS
fobk45MwCQYFKw4DAhoFAKCBsTAYBgkqhkiG9w0BCQMxCwYJKoZIhvcNAQcBMBwGCSqGSIb3
DQEJBTEPFw0wNjExMTYwODI0MDdaMCMGCSqGSIb3DQEJBDEWBBRewA9C3M92zy2bRIS43D7O
D/28ijBSBgkqhkiG9w0BCQ8xRTBDMAoGCCqGSIb3DQMHMA4GCCqGSIb3DQMCAgIAgDANBggq
hkiG9w0DAgIBQDAHBgUrDgMCBzANBggqhkiG9w0DAgIBKDANBgkqhkiG9w0BAQEFAASCAQBH
f1APZmbLSbY065tajDFmsa+KQsuhuHTrnB7hN30hAE7+MksX14re4clUN6C8M1idQAVuWUET
KQdkW5ogy1Irk1yYoUuOZJBB42yyxMdfcuizuwgKtf8BFkQJfxYwoszyE0mFMydtHf3QrYRt
cVyeYwhw/GAO/tmgojm/swM2geurS/j2EDVoEBojemLYy/9z8xKjtQGFnqLUwRarOoAVjO9p
EZPv5SpEFW4A95BLTdniWnUdKS9Kw+SzesVpMfv2w1EQdp41lRLf96cNQGGtSVSN6but6bYh
Ai2rDzZQ8r3t9KhnJqJiMU6hLiF05JsKA2S9PTaMNi1T3qr3EEkV
--WhfpMioaduB5tiZL--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20061116082407.GB33390>