From owner-freebsd-questions@FreeBSD.ORG Wed Feb 18 06:04:41 2004 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id F2DC716A4CE; Wed, 18 Feb 2004 06:04:40 -0800 (PST) Received: from mail018.syd.optusnet.com.au (mail018.syd.optusnet.com.au [211.29.132.72]) by mx1.FreeBSD.org (Postfix) with ESMTP id E093643D2D; Wed, 18 Feb 2004 06:04:39 -0800 (PST) (envelope-from tfrank@optushome.com.au) Received: from marvin.home.local (c211-28-241-189.eburwd5.vic.optusnet.com.au [211.28.241.189])i1IE4U521727; Thu, 19 Feb 2004 01:04:31 +1100 Received: by marvin.home.local (Postfix, from userid 1001) id 46BF2223; Thu, 19 Feb 2004 01:04:29 +1100 (EST) Date: Thu, 19 Feb 2004 01:04:29 +1100 From: Tony Frank To: Danny Carroll Message-ID: <20040218140429.GG289@marvin.home.local> References: <00ed01c3f559$d7f1ac20$8052260a@capgemini.nl> <20040217223827.GM33797@wantadilla.lemis.com> <1077093269.15aypd5fp76s@mailsrv.dannysplace.net> <20040218111653.GB289@marvin.home.local> <1077106518.yyrhxqeampw@mailsrv.dannysplace.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1077106518.yyrhxqeampw@mailsrv.dannysplace.net> User-Agent: Mutt/1.4.2i cc: Tony Frank cc: freebsd-questions@freebsd.org Subject: Re: Help with Vinum disk crash... X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Feb 2004 14:04:41 -0000 Hi, On Wed, Feb 18, 2004 at 01:15:18PM +0100, Danny Carroll wrote: > Quoting Tony Frank : > > So you have a subdisk down which means Vinum can still read from the plex > > but has to manually calculate the missing subdisk data. > But I assume it cant write till I replace it.. I believe it should function for both read & write in degraded mode. Any reads will have to be rebuilt hence there will be a big hit in performance. > > Typically I see this problem when trying to mount filesystem with incorrect > > type. > > Was your filesystem ufs? If not you probably need to specify the type to > > mount > > command using -t parameter. See "man 8 mount" for details. > > > > Have you tried running fsck against the volume? > > > > Assuming ufs filesystem, I'd suggest starting with: > > > > fsck -n -t ufs /dev/vinum/data > > > > Note the -n option tells fsck not to correct any errors but will give you an > > indication about what is going on. > fsck -n -t ufs did not work... Seems like fsck does not know -t ... > You dont want to see the output from that... Nearly every inode has problems.. I got confused with mount there. :) > I ran it without the -n and the first entry looks like this: > > UNKNOWN FILE TYPE I=188225^C[12:39 PM root@guard:/usr/home/danny]#fsck > /dev/vinum/data > ** /dev/vinum/data > ** Last Mounted on /usr/jails/ftp/data > ** Phase 1 - Check Blocks and Sizes > UNKNOWN FILE TYPE I=29 > Well -n is useful in that it makes no changes... > > There are extra things you can try (recover using alternate super-block) but > > perhaps wait and see the results first? > How do I read an alternative superblock wih a vinum drive??? superblock is a filesystem function - so you run fsck with -b option as so: fsck -b x where x is the replacement superblock. Typically 32 is a valid replacement (says this in fsck man page) so you might want to try "fsck -n -b 32 /dev/vinum/data" The newfs output will give you a list of the superblocks created. You can 'rerun' newfs with: newfs -N -v /dev/vinum/data Be sure to include the -N as otherwise it will overwrite your volume. If 32 wont work you can try a higher number > > Another option would be to force the particular subdisk down and try the > > above steps again. > Something like: > > vinum down data.p0.s0 ??? The command would be "vinum stop data.p0.s0" In my case I had to do "vinum stop -f data.p0.s0" In your case if you have replaced the faulty drive you should be able to run 'vinum start data.p0.s0' to have the subdisk rebuilt in the background. This will take a long time for your listed volume sizes. However this should not have any impact to your situation. The volume is working at the Vinum level (although degraded) The problem here is that the data on the vinum volume appears to be somehow corrupt. If the fsck option with alternate superblock doesn't help I think the only option is a restore of data. If you do that I would recommend rebuilding the vinum configuration. Arrange a single vinum partition per disk and then use multiple subdisks to share the storage amongst different plexes. Here's output of a scenario I just now ran on my test box: I got a lot of errors with fsck which are strange to me as I have only two files on the filesystem. More investigation ongoing (probably tomorrow as it's now late) Throughout the various activities vinum gives full access to the volume with only one subdisk lost. raider# vinum stop data.p0.s0 Can't stop data.p0.s0: Device busy (16) raider# vinum lv -r data V data State: up Plexes: 1 Size: 16 GB P data.p0 R5 State: up Subdisks: 5 Size: 16 GB S data.p0.s0 State: up PO: 0 B Size: 4298 MB S data.p0.s1 State: up PO: 492 kB Size: 4298 MB S data.p0.s2 State: up PO: 984 kB Size: 4298 MB S data.p0.s3 State: up PO: 1476 kB Size: 4298 MB S data.p0.s4 State: up PO: 1968 kB Size: 4298 MB raider# vinum stop -f data.p0.s0 vinum: data.p0.s0 is down by force vinum: data.p0 is degraded raider# vinum lv -r data V data State: up Plexes: 1 Size: 16 GB P data.p0 R5 State: degraded Subdisks: 5 Size: 16 GB S data.p0.s0 State: down PO: 0 B Size: 4298 MB S data.p0.s1 State: up PO: 492 kB Size: 4298 MB S data.p0.s2 State: up PO: 984 kB Size: 4298 MB S data.p0.s3 State: up PO: 1476 kB Size: 4298 MB S data.p0.s4 State: up PO: 1968 kB Size: 4298 MB raider# ll /data total 1046000 -rw-r--r-- 1 root wheel 510709760 Feb 17 22:45 objtest.tar -rw-r--r-- 1 root wheel 559839232 Feb 17 22:40 office2k.iso raider# umount /data vinum: data.p0.s0 is obsolete by force raider# vinum lv -r data V data State: up Plexes: 1 Size: 16 GB P data.p0 R5 State: degraded Subdisks: 5 Size: 16 GB S data.p0.s0 State: obsolete PO: 0 B Size: 4298 MB S data.p0.s1 State: up PO: 492 kB Size: 4298 MB S data.p0.s2 State: up PO: 984 kB Size: 4298 MB S data.p0.s3 State: up PO: 1476 kB Size: 4298 MB S data.p0.s4 State: up PO: 1968 kB Size: 4298 MB raider# mount /data raider# mount /dev/vinum/root on / (ufs, local) /dev/vinum/tmp on /tmp (ufs, local, soft-updates) /dev/vinum/usr on /usr (ufs, local, soft-updates) /dev/vinum/var on /var (ufs, local, soft-updates) procfs on /proc (procfs, local) /dev/vinum/data on /data (ufs, local, soft-updates) raider# umount /data raider# newfs -N -v /dev/vinum/data Warning: Block size and bytes per inode restrict cylinders per group to 22. Warning: 1856 sector(s) in last cylinder unallocated /dev/vinum/data: 35211456 sectors in 8597 cylinders of 1 tracks, 4096 sectors 17193.1MB in 391 cyl groups (22 c/g, 44.00MB/g, 10944 i/g) super-block backups (for fsck -b #) at: 32, 90144, 180256, 270368, 360480, 450592, 540704, 630816, 720928, 811040, 901152, 991264, 1081376, 1171488, 1261600, 1351712, 1441824, 1531936, 1622048, 1712160, 1802272, 1892384, 1982496, 2072608, 2162720, 2252832, 2342944, 2433056, 2523168, 2613280, 2703392, 2793504, 2883616, 2973728, 3063840, 3153952, 3244064, 3334176, 3424288, 3514400, 3604512, 3694624, 3784736, 3874848, 3964960, 4055072, 4145184, 4235296, 4325408, 4415520, 4505632, [... cut the rest ...] raider# fsck -n /dev/vinum/data ** /dev/vinum/data (NO WRITE) ** Last Mounted on /data ** Phase 1 - Check Blocks and Sizes PARTIALLY ALLOCATED INODE I=98742 UNEXPECTED SOFT UPDATE INCONSISTENCY CLEAR? no PARTIALLY ALLOCATED INODE I=1276460 UNEXPECTED SOFT UPDATE INCONSISTENCY CLEAR? no PARTIALLY ALLOCATED INODE I=2903795 UNEXPECTED SOFT UPDATE INCONSISTENCY CLEAR? no PARTIALLY ALLOCATED INODE I=3207148 UNEXPECTED SOFT UPDATE INCONSISTENCY CLEAR? no PARTIALLY ALLOCATED INODE I=3455276 UNEXPECTED SOFT UPDATE INCONSISTENCY CLEAR? no UNKNOWN FILE TYPE I=3455277 UNEXPECTED SOFT UPDATE INCONSISTENCY CLEAR? no UNKNOWN FILE TYPE I=3672558 UNEXPECTED SOFT UPDATE INCONSISTENCY CLEAR? no PARTIALLY ALLOCATED INODE I=3760776 UNEXPECTED SOFT UPDATE INCONSISTENCY CLEAR? no ** Phase 2 - Check Pathnames ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts BAD/DUP DIR I=3455277 OWNER=root MODE=40000 SIZE=17592228466688 MTIME=Jan 1 10:00 1970 CLEAR? no BAD/DUP FILE I=3672558 OWNER=root MODE=100000 SIZE=17592227827712 MTIME=Apr 4 07:47 1971 CLEAR? no ** Phase 5 - Check Cyl groups FREE BLK COUNT(S) WRONG IN SUPERBLK SALVAGE? no SUMMARY INFORMATION BAD SALVAGE? no ALLOCATED INODE 3455277 MARKED FREE BLK(S) MISSING IN BIT MAPS SALVAGE? no ALLOCATED INODE 3672558 MARKED FREE 3 files, 1046001 used, 16018560 free (8 frags, 2002319 blocks, 0.0% fragmentation) raider# mount /data raider# cd /data raider# find . -ls 2 2 drwxr-xr-x 2 root wheel 512 Feb 18 13:19 . 3 998000 -rw-r--r-- 1 root wheel 510709760 Feb 17 22:45 ./objtest.tar 4 1094000 -rw-r--r-- 1 root wheel 559839232 Feb 17 22:40 ./office2k.iso raider# This needs some more investigation - but the vinum part appears to work. Hope it helps, Tony