Date: Mon, 02 Apr 2018 15:46:32 +0000 From: bugzilla-noreply@freebsd.org To: freebsd-bugs@FreeBSD.org Subject: [Bug 227204] Combination of gmirror and enabled softupdates journalling cause slow filesystem degradation Message-ID: <bug-227204-8@https.bugs.freebsd.org/bugzilla/>
next in thread | raw e-mail | index | archive | help
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D227204 Bug ID: 227204 Summary: Combination of gmirror and enabled softupdates journalling cause slow filesystem degradation Product: Base System Version: 10.3-RELEASE Hardware: Any OS: Any Status: New Severity: Affects Only Me Priority: --- Component: kern Assignee: freebsd-bugs@FreeBSD.org Reporter: aeder@list.ru Combination of gmirror and enabled softupdates journalling cause slow filesystem degradation Hello! I'm supporting at least 10 freebsd installations in different parts of the country (remotely). All of them configured with gmirror in two-disk configuration, all filesystems with softupdates enabled, and some of them w= ith softupdates journalling enabled. Most of installations are rather old, binary updated from 8.x-RELEASE versi= on to 10.3-RELEASE At least 3 different systems after a while (years) of uptime and multiple reboots (sometimes due to power failure) get the following problem: filesys= tem unconsistency, causing=20 a) kernel panics=20 b) forever locks of processes accessing some files or file listing. In all 3 cases, I have it solved by booting in single-user mode, disabling soft-updates journalling (leaving softupdates only) and performing fsck -f = -y on all filesystems. Typical list of errors found by fsck. Most of dublicates lines omitted: =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D [root@yakutia /usr/home/der]# fsck -f -y /dev/ada0s1a ** /dev/ada0s1a ** Last Mounted on / ** Phase 1 - Check Blocks and Sizes ** Phase 2 - Check Pathnames ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts UNREF FILE I=3D5858989 OWNER=3Droot MODE=3D100600 SIZE=3D380 MTIME=3DOct 2 17:50 2017 CLEAR? yes UNREF FILE I=3D5858993 OWNER=3Droot MODE=3D100600 SIZE=3D427 MTIME=3DOct 2 17:54 2017 CLEAR? yes ** Phase 5 - Check Cyl groups FREE BLK COUNT(S) WRONG IN SUPERBLK SALVAGE? yes SUMMARY INFORMATION BAD SALVAGE? yes BLK(S) MISSING IN BIT MAPS SALVAGE? yes 149860 files, 918866 used, 31578821 free (15053 frags, 3945471 blocks, 0.0% fragmentation) ***** FILE SYSTEM MARKED CLEAN ***** ***** FILE SYSTEM WAS MODIFIED ***** [root@yakutia /usr/home/der]# fsck -f -y /dev/ada0s1d ** /dev/ada0s1d ** Last Mounted on /usr ** Phase 1 - Check Blocks and Sizes 26629536 DUP I=3D13324783 UNEXPECTED SOFT UPDATE INCONSISTENCY 26629537 DUP I=3D13324783 UNEXPECTED SOFT UPDATE INCONSISTENCY ... 26629543 DUP I=3D13324783 UNEXPECTED SOFT UPDATE INCONSISTENCY 26629544 DUP I=3D13324783 UNEXPECTED SOFT UPDATE INCONSISTENCY 26629545 DUP I=3D13324783 UNEXPECTED SOFT UPDATE INCONSISTENCY 26629546 DUP I=3D13324783 UNEXPECTED SOFT UPDATE INCONSISTENCY EXCESSIVE DUP BLKS I=3D13324783 CONTINUE? yes INCORRECT BLOCK COUNT I=3D13324783 (11200 should be 80) CORRECT? yes INCORRECT BLOCK COUNT I=3D28090681 (8 should be 0) CORRECT? yes ... INCORRECT BLOCK COUNT I=3D28100016 (8 should be 0) CORRECT? yes INCORRECT BLOCK COUNT I=3D28100017 (8 should be 0) CORRECT? yes INCORRECT BLOCK COUNT I=3D28100020 (8 should be 0) CORRECT? yes INTERNAL ERROR: dups with softupdates UNEXPECTED SOFT UPDATE INCONSISTENCY ** Phase 1b - Rescan For More DUPS 26629536 DUP I=3D13322839 UNEXPECTED SOFT UPDATE INCONSISTENCY 26629537 DUP I=3D13322839 UNEXPECTED SOFT UPDATE INCONSISTENCY ... 26629545 DUP I=3D13322839 UNEXPECTED SOFT UPDATE INCONSISTENCY ** Phase 2 - Check Pathnames DUP/BAD I=3D13322839 OWNER=3Droot MODE=3D100644 SIZE=3D946634 MTIME=3DOct 30 17:43 2017 FILE=3D/local/lib/libfreetype.a UNEXPECTED SOFT UPDATE INCONSISTENCY REMOVE? yes ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts BAD/DUP FILE I=3D13322839 OWNER=3Droot MODE=3D100644 SIZE=3D946634 MTIME=3DOct 30 17:43 2017 CLEAR? yes BAD/DUP FILE I=3D13324783 OWNER=3Droot MODE=3D100555 SIZE=3D5691264 MTIME=3DJun 26 22:35 2017 CLEAR? yes ... UNREF FILE I=3D28098948 OWNER=3Droot MODE=3D100644 SIZE=3D0 MTIME=3DOct 23 20:32 2017 RECONNECT? yes ZERO LENGTH DIR I=3D28098949 OWNER=3Droot MODE=3D40755 SIZE=3D0 MTIME=3DApr 2 23:33 2018 CLEAR? yes UNREF FILE I=3D28098950 OWNER=3Droot MODE=3D100644 SIZE=3D0 MTIME=3DNov 13 07:12 2016 RECONNECT? yes UNREF FILE I=3D28098951 OWNER=3Droot MODE=3D100644 SIZE=3D0 MTIME=3DNov 13 07:12 2016 RECONNECT? yes ... ... UNREF FILE I=3D28100020 OWNER=3Droot MODE=3D100644 SIZE=3D0 MTIME=3DApr 2 23:33 2018 RECONNECT? yes UNREF FILE I=3D28100049 OWNER=3Droot MODE=3D120755 SIZE=3D0 MTIME=3DApr 2 23:33 2018 RECONNECT? yes ** Phase 5 - Check Cyl groups FREE BLK COUNT(S) WRONG IN SUPERBLK SALVAGE? yes SUMMARY INFORMATION BAD SALVAGE? yes BLK(S) MISSING IN BIT MAPS SALVAGE? yes 370932 files, 2653812 used, 82406856 free (23056 frags, 10297975 blocks, 0.= 0% fragmentation) ***** FILE SYSTEM STILL DIRTY ***** ***** FILE SYSTEM WAS MODIFIED ***** ***** PLEASE RERUN FSCK ***** [root@yakutia /usr/home/der]# fsck -f -y /dev/ada0s1d ** /dev/ada0s1d ** Last Mounted on /usr ** Phase 1 - Check Blocks and Sizes ** Phase 2 - Check Pathnames ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts ** Phase 5 - Check Cyl groups 370932 files, 2653812 used, 82406856 free (23056 frags, 10297975 blocks, 0.= 0% fragmentation) ***** FILE SYSTEM MARKED CLEAN ***** =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D I can't find direct information in handbook, if journalling softupdates is = not compatible with gmirror, so decided to create this bug. I'm sure that it's not hardware problem - smartctl show nothing on those ha= rd drives, and we actually replace disks ~every 3 years using gmirror. >From the last case I can save broken filesystem image (mounts OK, but cause kernel panic if attempting to write some of the files), if it can be used to find root cause. Due to security reasons I can't give it to anyone outside = my company, but I can try to analyse it if proper questions was given. --=20 You are receiving this mail because: You are the assignee for the bug.=
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?bug-227204-8>