Date: Tue, 6 Jul 2010 00:35:02 +0300 From: Kostik Belousov <kostikbel@gmail.com> To: Charles Sprickman <spork@bway.net> Cc: freebsd-fs@freebsd.org Subject: Re: 7.2 - ufs2 corruption Message-ID: <20100705213502.GX13238@deviant.kiev.zoral.com.ua> In-Reply-To: <alpine.OSX.2.00.1007051701020.33454@hotlap.local> References: <alpine.OSX.2.00.1007051701020.33454@hotlap.local>
next in thread | previous in thread | raw e-mail | index | archive | help
--mnRbTohuKv9GovM+ Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Jul 05, 2010 at 05:23:03PM -0400, Charles Sprickman wrote: > Howdy, >=20 > I've posted previously about this, but I'm going to give it one more shot= =20 > before I start reformatting and/or upgrading things. >=20 > I have a largish filesystem (1.3TB) that holds a few jails, the main one= =20 > being a mail server. Running 7.2/amd64 on a Dell 2970 with the mfi=20 > raid card, 6GB RAM, UFS2 (SU was enabled, I disabled it for testing to=20 > no effect) >=20 > The symptoms are as follows: >=20 > Various applications will log messages about "bad file descriptors" (imap= ,=20 > rsync backup script, quota counter): >=20 > du: > ./cur/1271801961.M21831P98582V0000005BI08E85975_0.foo.net,S=3D2824:2,S: > Bad file descriptor >=20 > The kernel also starts logging messages like this to the console: >=20 > g_vfs_done():mfid0s1e[READ(offset=3D2456998070156636160, length=3D16384)]= error=20 > =3D 5 > g_vfs_done():mfid0s1e[READ(offset=3D-7347040593908226048, length=3D16384)= ]error=20 > =3D 5 > g_vfs_done():mfid0s1e[READ(offset=3D2456998070156636160, length=3D16384)]= error=20 > =3D 5 > g_vfs_done():mfid0s1e[READ(offset=3D-7347040593908226048, length=3D16384)= ]error=20 > =3D 5 > g_vfs_done():mfid0s1e[READ(offset=3D2456998070156636160, length=3D16384)]= error=20 > =3D 5 >=20 > Note that the offsets look a bit... suspicious, especially those negative= =20 > ones. >=20 > Usually within a day or two of those "g_vfs_done()" messages showing up= =20 > the box will panic shortly after the daily run. Things are hosed up=20 > enough that it is unable to save a dump. The panic always looks like=20 > this: >=20 > panic: ufs_dirbad: /spool: bad dir ino 151699770 at offset 163920: mangle= d=20 > entry > cpuid =3D 0 > Uptime: 70d22h56m48s > Physical memory: 6130 MB > Dumping 811 MB: 796 780 764 748 732 716 700 684 668 652 636 620 604 588= =20 > 572 556 540 524 508 492 476 460 444 428 412 396 380 364 348 332 316 300= =20 > 284 > ** DUMP FAILED (ERROR 16) ** >=20 > panic: ufs_dirbad: /spool: bad dir ino 150073505 at offset 150: mangled= =20 > entry > cpuid =3D 2 > Uptime: 13d22h30m21s > Physical memory: 6130 MB > Dumping 816 MB: 801 785 769 753 737 721 705 689 > ** DUMP FAILED (ERROR 16) ** > Automatic reboot in 15 seconds - press a key on the console to abort > Rebooting... >=20 > The fs, specifically "/spool" (which is where the errors always=20 > originate), will be pretty trashed and require a manual fsck. The first= =20 > pass finds/fixes errors, but does not mark the fs clean. It can take=20 > anywhere from 2-4 passes to get a clean fs. >=20 > The box then runs fine for a few weeks or a few months until the=20 > "g_vfs_done" errors start popping up, then it's a repeat. >=20 > Are there any *known* issues with either the fs or possibly the mfi drive= r=20 > in 7.2? >=20 > My plan was to do something like this: >=20 > -shut down services and copy all of /spool off to the backups server > -newfs /spool > -copy everything back >=20 > Then if it continues, repeat the above with a 7.3 upgrade before running= =20 > newfs. >=20 > If it still continues, then just go nuts and see what 8.0 or 8.1 does.=20 > But I'd really like to avoid that. >=20 > Any tips? Show "df -i" output for the the affected filesystem. --mnRbTohuKv9GovM+ Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (FreeBSD) iEYEARECAAYFAkwyUAYACgkQC3+MBN1Mb4iinwCfc7TNNQJTl08QixNmSwrQJKLp YrEAnim9o+sJ5J7nGlBk8FWN0z64GLdv =hqg1 -----END PGP SIGNATURE----- --mnRbTohuKv9GovM+--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20100705213502.GX13238>