From owner-freebsd-questions@FreeBSD.ORG Sat Sep 17 19:54:19 2005 Return-Path: X-Original-To: freebsd-questions@freebsd.org Delivered-To: freebsd-questions@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 67F0E16A41F for ; Sat, 17 Sep 2005 19:54:19 +0000 (GMT) (envelope-from ulfma629@student.liu.se) Received: from arcadia.unit.liu.se (arcadia.unit.liu.se [130.236.230.144]) by mx1.FreeBSD.org (Postfix) with ESMTP id 52B4343D45 for ; Sat, 17 Sep 2005 19:54:17 +0000 (GMT) (envelope-from ulfma629@student.liu.se) Received: by arcadia.unit.liu.se (Postfix, from userid 102) id 7225A45E14; Sat, 17 Sep 2005 21:54:16 +0200 (MEST) Received: from liu.se (avalon.unit.liu.se [130.236.230.138]) by arcadia.unit.liu.se (Postfix) with ESMTP id 9467A45E0C for ; Sat, 17 Sep 2005 21:54:14 +0200 (MEST) Received: from [85.194.52.118] by qom.unit.liu.se (mshttpd); Sat, 17 Sep 2005 21:54:14 +0200 From: Ulf Magnusson To: freebsd-questions@freebsd.org Message-ID: Date: Sat, 17 Sep 2005 21:54:14 +0200 X-Mailer: iPlanet Messenger Express 5.2 HotFix 2.07 (built Jun 24 2005) MIME-Version: 1.0 Content-Language: en X-Accept-Language: en Priority: normal Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: 7bit X-Spam-Checker-Version: SpamAssassin 3.0.4-liu_20050621_1634 (2005-06-05) on themis.unit.liu.se X-Spam-Level: X-Spam-Status: No, score=-0.0 required=5.0 tests=FROM_ENDS_IN_NUMS, LIU_FROM_MATCHES_LIUSTUDENT autolearn=disabled version=3.0.4-liu_20050621_1634 Subject: Strange case of root filesystem corruption X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 17 Sep 2005 19:54:19 -0000 First of all, all low level diagnostic tools I've used on my drive show no signs of hardware error. The shutdown prior to when the problems started went smooth, without any error messages. Yesterday GRUB would suddenly not display the boot menu anymore. After some investigation I discovered that all files in /boot/grub had turned into symlinks to locale files, as seen in the following ufs2tools dump: Jun 22,2005 01:16:24 . Jul 16,2005 00:41:53 .. May 11,2005 18:23:10 30 menu.lst -> ../it_CH.ISO8859-1/LC_MONETARY Jun 22,2005 01:16:04 201 menu.lst~ May 11,2005 18:23:10 30 stage1 -> ../it_IT.ISO8859-1/LC_MONETARY May 11,2005 18:23:10 30 stage2 -> ../nl_BE.ISO8859-1/LC_MONETARY May 11,2005 18:23:10 30 ufs2_stage1_5 -> ../nl_NL.ISO8859-1/LC_MONETARY I could no longer log in via any of my accounts, so I tried adding a new account in single-user mode. This gave an error message about "vnodes" (or perhaps inodes, my system won't boot anymore, so I can't reproduce the error) and a kernel panic followed by a reboot. I then ran fsck, which reported a whole bunch of errors: wrong ref wounts, something about things being bad with the superblock, and about bad/missing things in general. Perhaps the strangest errors were about extraneous links to files directly under / (more on this later). I answered 'yes' to all fix/salvage questions from fsck. After that FreeBSD would not boot at all, causing a reboot before the beastie menu. Looking at filesystems on the disk with the free ufs2tools program, only the root filesystem seems to be affected by the corruption. Here's what / looks like (seems fsck was right about those extraneous links): Jan 01,1970 01:00:00 0 . Sep 16,2005 01:07:25 . Jan 01,1970 01:00:00 0 .. Sep 16,2005 01:07:25 .. Jan 01,1970 01:00:00 0 Jan 01,1970 01:00:00 0 Jan 01,1970 01:00:00 0 Jan 01,1970 01:00:00 0 Jan 01,1970 01:00:00 0 Jan 01,1970 01:00:00 0 Jan 01,1970 01:00:00 0 Jan 01,1970 01:00:00 0 Jan 01,1970 01:00:00 0 Jan 01,1970 01:00:00 0 Jan 01,1970 01:00:00 0 Jan 01,1970 01:00:00 0 Jan 01,1970 01:00:00 0 Jan 01,1970 01:00:00 0 Jan 01,1970 01:00:00 0 Jan 01,1970 01:00:00 0 Jan 01,1970 01:00:00 0 Jan 01,1970 01:00:00 0 Jan 01,1970 01:00:00 0 Jan 01,1970 01:00:00 0 Jan 01,1970 01:00:00 0 Jan 01,1970 01:00:00 0 Jan 01,1970 01:00:00 0 Jan 01,1970 01:00:00 0 Jan 01,1970 01:00:00 0 Jan 01,1970 01:00:00 0 Jan 01,1970 01:00:00 0 Jan 01,1970 01:00:00 0 Jan 01,1970 01:00:00 0 Jan 01,1970 01:00:00 0 Nov 05,2004 02:27:17 801 .cshrc Nov 05,2004 02:27:17 801 .cshrc Nov 05,2004 02:27:17 251 .profile Nov 05,2004 02:27:17 251 .profile Feb 17,2005 18:21:41 .snap Jan 01,1970 01:00:00 0 .snap May 11,2005 18:31:27 6192 COPYRIGHT May 11,2005 18:31:27 6192 COPYRIGHT May 11,2005 18:21:44 bin Jan 01,1970 01:00:00 0 bin Jan 01,1970 01:00:00 0 boot Jul 16,2005 00:41:53 boot Feb 17,2005 18:21:48 cdrom Jan 01,1970 01:00:00 0 cdrom Feb 17,2005 18:29:51 10 compat -> usr/compat Feb 17,2005 18:29:51 10 compat -> usr/compat Jan 01,1970 01:00:00 0 dev Feb 17,2005 18:21:41 dev Jan 01,1970 01:00:00 0 dist Feb 17,2005 18:21:50 dist Sep 16,2005 01:07:22 4096 entropy Sep 15,2005 21:27:31 etc Jan 01,1970 01:00:00 0 etc Mar 10,2005 21:53:07 655360 festival-synthesis-.core Mar 10,2005 21:53:07 655360 festival-synthesis-.core Feb 17,2005 18:21:42 home Jan 01,1970 01:00:00 0 home Mar 23,2005 22:03:23 ipod Jan 01,1970 01:00:00 0 ipod Jan 01,1970 01:00:00 0 lib Jul 16,2005 04:50:21 lib May 11,2005 18:21:39 libexec Jan 01,1970 01:00:00 0 libexec Sep 16,2005 00:05:35 lost+found Nov 05,2004 02:23:32 mnt Jan 01,1970 01:00:00 0 mnt Jan 01,1970 01:00:00 0 pappas May 22,2005 14:50:23 pappas Apr 10,2005 21:10:07 proc Jan 01,1970 01:00:00 0 proc May 11,2005 18:22:29 rescue Jan 01,1970 01:00:00 0 rescue Sep 10,2005 13:08:14 root Jan 01,1970 01:00:00 0 root May 29,2005 13:26:54 s Jan 01,1970 01:00:00 0 s Jan 01,1970 01:00:00 0 sbin May 11,2005 18:23:56 sbin Feb 17,2005 18:21:48 stand Jan 01,1970 01:00:00 0 stand Jun 15,2005 18:10:54 11 sys -> usr/src/sys Jun 15,2005 18:10:54 11 sys -> usr/src/sys Sep 15,2005 21:27:32 tmp Jan 01,1970 01:00:00 0 tmp Jan 01,1970 01:00:00 0 usr Feb 17,2005 18:21:45 usr Jan 01,1970 01:00:00 0 var Feb 17,2005 18:21:46 var Mar 06,2005 15:16:20 vcdrom Jan 01,1970 01:00:00 0 vcdrom Jan 01,1970 01:00:00 0 win Aug 27,2005 12:12:03 win Subfolders of / on the same filesystem are affected as well. I can't see any corruption inside files though. My problems wouldn't be so alarming if not for the fact that two friends of mine have had very similar problems (but unfortunately never told anyone about it). Their root filesystem suddenly went haywire on seemingly fine disks for no apparent reason. Some general info: I had other filesystems (stored on the same and other disks) mounted on /usr, /var, /home, and /win. I'm running FreeBSD 5.4. My friends can't remember which version they ran at the time, but they both think it was 5.x. In case they might be helpful, I'll include the contents of /etc and /boot (both show signs of corruption), as well as the output of the bsdlabel program (part of the ufs2tools package) when ran on the disk in question: /etc: Sep 15,2005 21:27:31 . Sep 16,2005 01:07:25 .. Jun 06,2005 12:49:27 0 COPYRIGHT Aug 16,2005 13:54:44 X11 Feb 17,2005 18:22:04 12 aliases -> mail/aliases Mar 30,2005 15:01:01 65536 aliases.db Nov 05,2004 02:27:16 209 amd.map Nov 05,2004 02:27:16 1234 apmd.conf Mar 12,2005 16:33:49 225 auth.conf Nov 05,2004 02:27:16 231 auth.conf~ Mar 12,2005 22:12:43 0 blocks.fb Feb 17,2005 18:22:02 bluetooth Apr 28,2005 21:09:42 7 console Nov 05,2004 02:27:16 737 crontab Nov 05,2004 02:27:16 108 csh.cshrc Nov 05,2004 02:27:16 481 csh.login Nov 05,2004 02:27:16 110 csh.logout May 19,2005 18:25:36 defaults May 11,2005 18:29:51 4846 devd.conf Nov 05,2004 02:27:16 2071 devfs.conf Aug 20,2005 18:55:00 23 dhclient-enter-hooks Aug 20,2005 18:54:57 23 dhclient-enter-hooks~ Nov 05,2004 02:27:16 267 dhclient.conf Nov 05,2004 02:27:16 6621 disktab Nov 05,2004 02:27:17 0 dumpdates Nov 05,2004 02:27:16 142 fbtab Sep 03,2005 22:43:21 509 fstab Aug 27,2005 11:26:13 511 fstab~ Nov 05,2004 02:27:16 245 ftpusers Nov 05,2004 02:27:16 5904 gettytab May 11,2005 18:26:29 gnats Aug 04,2005 16:42:42 487 group Feb 23,2005 18:52:40 443 group~ May 07,2005 20:08:15 25 host.aliases May 07,2005 20:04:04 21 host.aliases~ Feb 17,2005 19:57:39 60 host.conf Feb 18,2005 20:07:35 1107 hosts Nov 05,2004 02:27:16 3186 hosts.allow Nov 05,2004 02:27:16 111 hosts.equiv Nov 05,2004 02:27:16 99 hosts.lpd Feb 17,2005 22:51:26 1106 hosts~ Apr 10,2005 22:15:42 5378 inetd.conf Apr 10,2005 22:12:07 5380 inetd.conf~ Feb 17,2005 18:22:02 isdn May 11,2005 18:23:10 30 libfstab -> ../en_AU.ISO8859-1/LC_MONETARY Aug 27,2005 23:08:06 511 libfstab~ Feb 27,2005 22:23:18 390 libmap.conf Feb 17,2005 20:24:09 675 localtime Nov 05,2004 02:27:16 619 locate.rc Nov 05,2004 02:27:16 1847 login.access Mar 12,2005 16:29:14 6522 login.conf Mar 12,2005 16:30:16 65536 login.conf.db Nov 05,2004 02:27:16 6522 login.conf~ Nov 05,2004 02:27:16 564 mac.conf Jul 31,2005 00:36:29 mail Nov 05,2004 02:27:16 106 mail.rc Mar 23,2005 12:59:50 117 mailcap Mar 23,2005 12:57:00 117 mailcap~ Jul 20,2005 13:19:42 13206 make.conf Jul 16,2005 04:08:09 13132 make.conf.bak Mar 19,2005 11:40:53 13132 make.conf2~ Jul 20,2005 13:19:14 13206 make.conf~ Jul 20,2005 13:19:14 1087 manpath.config Jul 16,2005 04:08:09 946 manpath.config.bak Aug 04,2005 16:42:42 1884 master.passwd Mar 12,2005 17:07:09 1750 master.passwd~ Jun 06,2005 12:45:59 0 motd May 11,2005 18:16:36 1111 motd.old May 11,2005 18:27:42 mtree May 02,2005 19:44:26 153 named.conf~ Sep 15,2005 21:27:31 21 namedb -> /var/named/etc/namedb Nov 05,2004 02:27:16 783 netconfig Nov 05,2004 02:27:16 2362 netstart May 11,2005 18:30:09 13696 network.subr Nov 05,2004 02:27:16 365 networks Mar 09,2005 21:33:05 1932 newsyslog.conf Mar 09,2005 21:28:46 1935 newsyslog.conf~ Nov 05,2004 02:27:16 1701 nsmb.conf Feb 17,2005 19:57:39 113 nsswitch.conf Nov 05,2004 02:23:32 ntp Nov 05,2004 02:27:16 432 opieaccess Feb 17,2005 20:01:00 0 opiekeys Feb 17,2005 18:22:02 pam.d Aug 04,2005 16:42:42 1582 passwd Feb 22,2005 19:50:54 1394 passwd~ May 11,2005 18:30:56 5814 pccard_ether Feb 17,2005 18:22:03 periodic Mar 21,2005 11:50:51 137 periodic.conf Nov 05,2004 02:27:16 3567 pf.conf Nov 05,2004 02:27:16 22556 pf.os Nov 05,2004 02:27:16 293 phones Feb 17,2005 18:22:03 ppp Apr 23,2005 16:34:38 2152 printcap Apr 23,2005 16:32:27 2152 printcap~ May 07,2005 20:07:03 842 profile May 07,2005 20:06:58 841 profile~ May 11,2005 18:30:11 5807 protocols Aug 04,2005 16:42:42 40960 pwd.db Nov 05,2004 02:27:16 2716 rc May 11,2005 18:30:16 5298 rc.bsdextended Aug 20,2005 19:09:38 710 rc.conf Jun 18,2005 19:06:23 693 rc.conf.orig Aug 20,2005 19:09:34 710 rc.conf~ May 11,2005 18:29:47 rc.d Nov 05,2004 02:27:16 9569 rc.firewall Nov 05,2004 02:27:16 8905 rc.firewall6 Nov 05,2004 02:27:16 2165 rc.resume Nov 05,2004 02:27:16 5785 rc.sendmail May 11,2005 18:30:27 3211 rc.shutdown May 11,2005 18:30:31 30905 rc.subr Nov 05,2004 02:27:16 2209 rc.suspend Nov 05,2004 02:27:16 2392 remote Sep 15,2005 21:27:31 52 resolv.conf Aug 20,2005 18:32:44 23 resolv.conf.no Jul 15,2005 21:07:06 69 resolv.conf~ Feb 17,2005 18:22:04 13 rmt -> /usr/sbin/rmt Nov 05,2004 02:27:16 1674 rpc Nov 05,2004 02:27:16 73410 services Jun 02,2005 12:40:40 276 shells Nov 05,2004 02:23:32 skel May 11,2005 18:23:10 30 spwd.db -> ../pt_PT.ISO8859-1/LC_MONETARY Feb 17,2005 20:00:07 ssh May 11,2005 18:29:49 ssl Nov 05,2004 02:27:16 367 sysctl.conf Jun 06,2005 12:37:53 1352 syslog.conf May 02,2005 20:24:16 1344 syslog.conf~ Feb 17,2005 18:22:04 23 termcap -> /usr/share/misc/termcap Apr 28,2005 21:09:29 7 tty Mar 12,2005 17:18:17 7670 ttys Nov 05,2004 02:27:16 7658 ttys~ May 11,2005 18:30:52 2032 usbd.conf Jun 16,2005 22:31:19 0 wall_cmos_clock /boot: ul 16,2005 00:41:53 . Sep 16,2005 01:07:25 .. May 11,2005 18:23:16 7007 beastie.4th Mar 11,2005 23:53:02 6757 beastie.4th~ May 11,2005 18:23:15 8192 boot May 11,2005 18:23:15 512 boot0 May 11,2005 18:23:15 512 boot0sio May 11,2005 18:23:15 512 boot1 May 11,2005 18:23:15 7680 boot2 May 11,2005 18:23:15 1184 cdboot May 11,2005 18:23:16 defaults Nov 05,2004 02:27:17 1982 device.hints May 11,2005 18:23:16 2249 frames.4th Jun 22,2005 01:16:24 grub Jun 15,2005 18:11:05 kernel Jun 15,2005 15:59:52 kernel.old May 11,2005 18:23:16 262144 loader May 11,2005 18:23:16 7772 loader.4th Jul 16,2005 00:41:53 58 loader.conf May 11,2005 18:23:10 30 loader.conf.orig -> ../no_NO.ISO8859-1/LC_MONETARY Feb 17,2005 18:29:51 0 loader.conf~ May 11,2005 18:23:16 13893 loader.help Nov 05,2004 02:26:31 212992 loader.old Nov 05,2004 02:26:31 370 loader.rc May 11,2005 18:23:15 512 mbr Jul 16,2005 00:41:48 modules May 11,2005 18:23:16 264192 pxeboot May 11,2005 18:23:16 692 screen.4th May 11,2005 18:23:16 36440 support.4th bsdlabel output: # Drive 0, Slice 1: type: ESDI disk: label: flags: bytes/sector: 512 sectors/track: 63 tracks/cylinder: 255 sectors/cylinder: 16065 cylinders: 4865 sectors/unit: 78165360 rpm: 3600 interleave: 1 trackskew: 0 cylinderskew: 0 headswitch: 0 # milliseconds track-to-track seek: 0 # milliseconds drivedata: 0 8 partitions: # size offset fstype [fsize bsize bps/cpg] a: 1048576 0 4.2BSD 0 0 0 b: 3145728 1048576 swap c: 78164624 0 unused 0 0 # "raw" part, don't edit d: 2097152 4194304 4.2BSD 0 0 0 e: 62914560 6291456 4.2BSD 0 0 0 f: 8957952 69206016 4.2BSD 0 0 0 I haven't lost any important data because of these problems, but others might.. /Ulf