Date: Thu, 19 Feb 2004 01:04:29 +1100 From: Tony Frank <tfrank@optushome.com.au> To: Danny Carroll <danny@dannysplace.net> Cc: freebsd-questions@freebsd.org Subject: Re: Help with Vinum disk crash... Message-ID: <20040218140429.GG289@marvin.home.local> In-Reply-To: <1077106518.yyrhxqeampw@mailsrv.dannysplace.net> References: <00ed01c3f559$d7f1ac20$8052260a@capgemini.nl> <20040217223827.GM33797@wantadilla.lemis.com> <1077093269.15aypd5fp76s@mailsrv.dannysplace.net> <20040218111653.GB289@marvin.home.local> <1077106518.yyrhxqeampw@mailsrv.dannysplace.net>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi,
On Wed, Feb 18, 2004 at 01:15:18PM +0100, Danny Carroll wrote:
> Quoting Tony Frank <tfrank@optushome.com.au>:
> > So you have a subdisk down which means Vinum can still read from the plex
> > but has to manually calculate the missing subdisk data.
> But I assume it cant write till I replace it..
I believe it should function for both read & write in degraded mode.
Any reads will have to be rebuilt hence there will be a big hit in performance.
> > Typically I see this problem when trying to mount filesystem with incorrect
> > type.
> > Was your filesystem ufs? If not you probably need to specify the type to
> > mount
> > command using -t parameter. See "man 8 mount" for details.
> >
> > Have you tried running fsck against the volume?
> >
> > Assuming ufs filesystem, I'd suggest starting with:
> >
> > fsck -n -t ufs /dev/vinum/data
> >
> > Note the -n option tells fsck not to correct any errors but will give you an
> > indication about what is going on.
> fsck -n -t ufs did not work... Seems like fsck does not know -t ...
> You dont want to see the output from that... Nearly every inode has problems..
I got confused with mount there. :)
> I ran it without the -n and the first entry looks like this:
>
> UNKNOWN FILE TYPE I=188225^C[12:39 PM root@guard:/usr/home/danny]#fsck
> /dev/vinum/data
> ** /dev/vinum/data
> ** Last Mounted on /usr/jails/ftp/data
> ** Phase 1 - Check Blocks and Sizes
> UNKNOWN FILE TYPE I=29
>
Well -n is useful in that it makes no changes...
> > There are extra things you can try (recover using alternate super-block) but
> > perhaps wait and see the results first?
> How do I read an alternative superblock wih a vinum drive???
superblock is a filesystem function - so you run fsck with -b option as so:
fsck -b x
where x is the replacement superblock.
Typically 32 is a valid replacement (says this in fsck man page) so you might want
to try "fsck -n -b 32 /dev/vinum/data"
The newfs output will give you a list of the superblocks created.
You can 'rerun' newfs with:
newfs -N -v /dev/vinum/data
Be sure to include the -N as otherwise it will overwrite your volume.
If 32 wont work you can try a higher number
> > Another option would be to force the particular subdisk down and try the
> > above steps again.
> Something like:
>
> vinum down data.p0.s0 ???
The command would be "vinum stop data.p0.s0"
In my case I had to do "vinum stop -f data.p0.s0"
In your case if you have replaced the faulty drive you should be
able to run 'vinum start data.p0.s0' to have the subdisk rebuilt in the
background.
This will take a long time for your listed volume sizes.
However this should not have any impact to your situation.
The volume is working at the Vinum level (although degraded)
The problem here is that the data on the vinum volume appears to be
somehow corrupt.
If the fsck option with alternate superblock doesn't help I think
the only option is a restore of data.
If you do that I would recommend rebuilding the vinum configuration.
Arrange a single vinum partition per disk and then use multiple subdisks
to share the storage amongst different plexes.
Here's output of a scenario I just now ran on my test box:
I got a lot of errors with fsck which are strange to me as I have
only two files on the filesystem.
More investigation ongoing (probably tomorrow as it's now late)
Throughout the various activities vinum gives full access to the
volume with only one subdisk lost.
raider# vinum stop data.p0.s0
Can't stop data.p0.s0: Device busy (16)
raider# vinum lv -r data
V data State: up Plexes: 1 Size: 16 GB
P data.p0 R5 State: up Subdisks: 5 Size: 16 GB
S data.p0.s0 State: up PO: 0 B Size: 4298 MB
S data.p0.s1 State: up PO: 492 kB Size: 4298 MB
S data.p0.s2 State: up PO: 984 kB Size: 4298 MB
S data.p0.s3 State: up PO: 1476 kB Size: 4298 MB
S data.p0.s4 State: up PO: 1968 kB Size: 4298 MB
raider# vinum stop -f data.p0.s0
vinum: data.p0.s0 is down by force
vinum: data.p0 is degraded
raider# vinum lv -r data
V data State: up Plexes: 1 Size: 16 GB
P data.p0 R5 State: degraded Subdisks: 5 Size: 16 GB
S data.p0.s0 State: down PO: 0 B Size: 4298 MB
S data.p0.s1 State: up PO: 492 kB Size: 4298 MB
S data.p0.s2 State: up PO: 984 kB Size: 4298 MB
S data.p0.s3 State: up PO: 1476 kB Size: 4298 MB
S data.p0.s4 State: up PO: 1968 kB Size: 4298 MB
raider# ll /data
total 1046000
-rw-r--r-- 1 root wheel 510709760 Feb 17 22:45 objtest.tar
-rw-r--r-- 1 root wheel 559839232 Feb 17 22:40 office2k.iso
raider# umount /data
vinum: data.p0.s0 is obsolete by force
raider# vinum lv -r data
V data State: up Plexes: 1 Size: 16 GB
P data.p0 R5 State: degraded Subdisks: 5 Size: 16 GB
S data.p0.s0 State: obsolete PO: 0 B Size: 4298 MB
S data.p0.s1 State: up PO: 492 kB Size: 4298 MB
S data.p0.s2 State: up PO: 984 kB Size: 4298 MB
S data.p0.s3 State: up PO: 1476 kB Size: 4298 MB
S data.p0.s4 State: up PO: 1968 kB Size: 4298 MB
raider# mount /data
raider# mount
/dev/vinum/root on / (ufs, local)
/dev/vinum/tmp on /tmp (ufs, local, soft-updates)
/dev/vinum/usr on /usr (ufs, local, soft-updates)
/dev/vinum/var on /var (ufs, local, soft-updates)
procfs on /proc (procfs, local)
/dev/vinum/data on /data (ufs, local, soft-updates)
raider# umount /data
raider# newfs -N -v /dev/vinum/data
Warning: Block size and bytes per inode restrict cylinders per group to 22.
Warning: 1856 sector(s) in last cylinder unallocated
/dev/vinum/data: 35211456 sectors in 8597 cylinders of 1 tracks, 4096 sectors
17193.1MB in 391 cyl groups (22 c/g, 44.00MB/g, 10944 i/g)
super-block backups (for fsck -b #) at:
32, 90144, 180256, 270368, 360480, 450592, 540704, 630816, 720928, 811040,
901152, 991264, 1081376, 1171488, 1261600, 1351712, 1441824, 1531936, 1622048,
1712160, 1802272, 1892384, 1982496, 2072608, 2162720, 2252832, 2342944,
2433056, 2523168, 2613280, 2703392, 2793504, 2883616, 2973728, 3063840,
3153952, 3244064, 3334176, 3424288, 3514400, 3604512, 3694624, 3784736,
3874848, 3964960, 4055072, 4145184, 4235296, 4325408, 4415520, 4505632,
[... cut the rest ...]
raider# fsck -n /dev/vinum/data
** /dev/vinum/data (NO WRITE)
** Last Mounted on /data
** Phase 1 - Check Blocks and Sizes
PARTIALLY ALLOCATED INODE I=98742
UNEXPECTED SOFT UPDATE INCONSISTENCY
CLEAR? no
PARTIALLY ALLOCATED INODE I=1276460
UNEXPECTED SOFT UPDATE INCONSISTENCY
CLEAR? no
PARTIALLY ALLOCATED INODE I=2903795
UNEXPECTED SOFT UPDATE INCONSISTENCY
CLEAR? no
PARTIALLY ALLOCATED INODE I=3207148
UNEXPECTED SOFT UPDATE INCONSISTENCY
CLEAR? no
PARTIALLY ALLOCATED INODE I=3455276
UNEXPECTED SOFT UPDATE INCONSISTENCY
CLEAR? no
UNKNOWN FILE TYPE I=3455277
UNEXPECTED SOFT UPDATE INCONSISTENCY
CLEAR? no
UNKNOWN FILE TYPE I=3672558
UNEXPECTED SOFT UPDATE INCONSISTENCY
CLEAR? no
PARTIALLY ALLOCATED INODE I=3760776
UNEXPECTED SOFT UPDATE INCONSISTENCY
CLEAR? no
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
BAD/DUP DIR I=3455277 OWNER=root MODE=40000
SIZE=17592228466688 MTIME=Jan 1 10:00 1970
CLEAR? no
BAD/DUP FILE I=3672558 OWNER=root MODE=100000
SIZE=17592227827712 MTIME=Apr 4 07:47 1971
CLEAR? no
** Phase 5 - Check Cyl groups
FREE BLK COUNT(S) WRONG IN SUPERBLK
SALVAGE? no
SUMMARY INFORMATION BAD
SALVAGE? no
ALLOCATED INODE 3455277 MARKED FREE
BLK(S) MISSING IN BIT MAPS
SALVAGE? no
ALLOCATED INODE 3672558 MARKED FREE
3 files, 1046001 used, 16018560 free (8 frags, 2002319 blocks, 0.0% fragmentation)
raider# mount /data
raider# cd /data
raider# find . -ls
2 2 drwxr-xr-x 2 root wheel 512 Feb 18 13:19 .
3 998000 -rw-r--r-- 1 root wheel 510709760 Feb 17 22:45 ./objtest.tar
4 1094000 -rw-r--r-- 1 root wheel 559839232 Feb 17 22:40 ./office2k.iso
raider#
This needs some more investigation - but the vinum part appears to work.
Hope it helps,
Tony
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20040218140429.GG289>