Date: Thu, 18 Mar 2004 11:51:12 -0800 (PST) From: Dmitry Kazarov <kazarov@willcom.ru> To: freebsd-gnats-submit@FreeBSD.org Subject: kern/64433: vinum: mirrored volume stopped responding when one of it's subdisk failed to write Message-ID: <200403181951.i2IJpChd075261@www.freebsd.org> Resent-Message-ID: <200403182000.i2IK0X45004020@freefall.freebsd.org>
next in thread | raw e-mail | index | archive | help
>Number: 64433
>Category: kern
>Synopsis: vinum: mirrored volume stopped responding when one of it's subdisk failed to write
>Confidential: no
>Severity: critical
>Priority: high
>Responsible: freebsd-bugs
>State: open
>Quarter:
>Keywords:
>Date-Required:
>Class: sw-bug
>Submitter-Id: current-users
>Arrival-Date: Thu Mar 18 12:00:33 PST 2004
>Closed-Date:
>Last-Modified:
>Originator: Dmitry Kazarov
>Release: FreeBSD 4.9 STABLE
>Organization:
MIDIKON
>Environment:
FreeBSD serv.willcom.ru 4.9-STABLE FreeBSD 4.9-STABLE #1: Wed Mar 17 10:06:14 MSK 2004 root@serv.willcom.ru:/var/obj/usr/src/sys/SERV i386
>Description:
Disk /dev/da1 (Ava) is known to have some surface defects.
Dist /dev/da0 (Rex) has no defects.
At the time of daily periodic scripts (3:01 am) alot of write ops failed on plex located on Ava (da1) and whole (!) volume became unavailable and server stopped responding because it's a /var file system.
Server was revived by hardware reset (at 10:07 am). Kernel was alive (consoles switched, network worked), but Ctrl-Alt-Del did not worked.
# vinum printconfig
drive Rex device /dev/da0s1h
drive Ava device /dev/da1s1h
volume root
volume usr
volume var
volume home
plex name root.Rex org concat vol root
plex name usr.Rex org concat vol usr
plex name var.Rex org concat vol var
plex name home.Rex org concat vol home
plex name root.Ava org concat vol root
plex name usr.Ava org concat vol usr
plex name var.Ava org concat vol var
plex name home.Ava org concat vol home
sd name var.Ava.s1 drive Ava plex var.Ava len 4192256s driveoffset 1024s plexoffset 14680064s
sd name root.Rex.s0 drive Rex plex root.Rex len 524288s driveoffset 4193280s plexoffset 0s
sd name usr.Rex.s0 drive Rex plex usr.Rex len 8388608s driveoffset 4717568s plexoffset 0s
sd name var.Rex.s0 drive Rex plex var.Rex len 14680064s driveoffset 13106176s plexoffset 0s
sd name home.Rex.s0 drive Rex plex home.Rex len 8150078s driveoffset 27786240s plexoffset 0s
sd name var.Rex.s1 drive Rex plex var.Rex len 4192256s driveoffset 1024s plexoffset 14680064s
sd name root.Ava.s0 drive Ava plex root.Ava len 524288s driveoffset 4193280s plexoffset 0s
sd name usr.Ava.s0 drive Ava plex usr.Ava len 8388608s driveoffset 4717568s plexoffset 0s
sd name var.Ava.s0 drive Ava plex var.Ava len 14680064s driveoffset 13106176s plexoffset 0s
sd name home.Ava.s0 drive Ava plex home.Ava len 8150078s driveoffset 27786240s plexoffset 0s
>From /var/log/messages
Mar 18 03:02:34 serv /kernel: vinum: var.Ava.s0 is stale by force
Mar 18 03:02:34 serv /kernel: vinum: var.Ava is corrupt
Mar 18 03:02:34 serv /kernel: fatal :var.Ava.s0 write error, block 20775744 for 4096 bytes
Mar 18 03:02:34 serv /kernel: var.Ava.s0: user buffer block 7669568 for 4096 bytes
Mar 18 03:02:34 serv /kernel: fatal :var.Ava.s0 write error, block 21861376 for 4096 bytes
Mar 18 03:02:34 serv /kernel: var.Ava.s0: user buffer block 8755200 for 4096 bytes
Mar 18 03:02:34 serv /kernel: fatal :var.Ava.s0 write error, block 13106192 for 2048 bytes
Mar 18 03:02:34 serv /kernel: var.Ava.s0: user buffer block 16 for 2048 bytes
Mar 18 03:02:34 serv /kernel: fatal :var.Ava.s0 write error, block 13111808 for 2048 bytes
Mar 18 03:02:34 serv /kernel: var.Ava.s0: user buffer block 5632 for 2048 bytes
Mar 18 03:02:34 serv /kernel: fatal :var.Ava.s0 write error, block 13476452 for 4096 bytes
Mar 18 03:02:34 serv /kernel: var.Ava.s0: user buffer block 370276 for 4096 bytes
Mar 18 03:02:34 serv /kernel: fatal :var.Ava.s0 write error, block 14575680 for 16384 bytes
Mar 18 03:02:34 serv /kernel: var.Ava.s0: user buffer block 1469504 for 16384 bytes
Mar 18 03:02:34 serv /kernel: fatal :var.Ava.s0 write error, block 14576384 for 16384 bytes
Mar 18 03:02:34 serv /kernel: var.Ava.s0: user buffer block 1470208 for 16384 bytes
Mar 18 03:02:34 serv /kernel: fatal :var.Ava.s0 write error, block 14575776 for 16384 bytes
Mar 18 03:02:34 serv /kernel: var.Ava.s0: user buffer block 1469600 for 16384 bytes
Mar 18 03:02:34 serv /kernel: fatal :var.Ava.s0 write error, block 21860864 for 2048 bytes
Mar 18 03:02:34 serv /kernel: var.Ava.s0: user buffer block 8754688 for 2048 bytes
Mar 18 03:02:34 serv /kernel: fatal :var.Ava.s0 write error, block 13476364 for 2048 bytes
Mar 18 03:02:35 serv /kernel: var.Ava.s0: user buffer block 370188 for 2048 bytes
Mar 18 03:02:35 serv /kernel: fatal :var.Ava.s0 write error, block 14572704 for 8192 bytes
Mar 18 03:02:35 serv /kernel: var.Ava.s0: user buffer block 1466528 for 8192 bytes
Mar 18 03:02:35 serv /kernel: fatal :var.Ava.s0 write error, block 14573568 for 16384 bytes
Mar 18 03:02:35 serv /kernel: var.Ava.s0: user buffer block 1467392 for 16384 bytes
Mar 18 03:02:35 serv /kernel: fatal :var.Ava.s0 write error, block 14573888 for 16384 bytes
Mar 18 03:02:35 serv /kernel: var.Ava.s0: user buffer block 1467712 for 16384 bytes
Mar 18 03:02:35 serv /kernel: fatal :var.Ava.s0 write error, block 14571072 for 16384 bytes
Mar 18 03:02:35 serv /kernel: var.Ava.s0: user buffer block 1464896 for 16384 bytes
Mar 18 03:02:35 serv /kernel: fatal :var.Ava.s0 write error, block 14570208 for 16384 bytes
Mar 18 03:02:35 serv /kernel: var.Ava.s0: user buffer block 1464032 for 16384 bytes
Mar 18 03:02:35 serv /kernel: fatal :var.Ava.s0 write error, block 14577408 for 16384 bytes
Mar 18 03:02:35 serv /kernel: var.Ava.s0: user buffer block 1471232 for 16384 bytes
Mar 18 03:02:35 serv /kernel: fatal :var.Ava.s0 write error, block 14577312 for 16384 bytes
Mar 18 03:02:35 serv /kernel: var.Ava.s0: user buffer block 1471136 for 16384 bytes
Mar 18 03:02:35 serv /kernel: fatal :var.Ava.s0 write error, block 14577056 for 32768 bytes
Mar 18 03:02:35 serv /kernel: var.Ava.s0: user buffer block 1470880 for 32768 bytes
Mar 18 03:02:35 serv /kernel: fatal :var.Ava.s0 write error, block 14573696 for 32768 bytes
Mar 18 03:02:35 serv /kernel: var.Ava.s0: user buffer block 1467520 for 32768 bytes
Mar 18 03:02:35 serv /kernel: fatal :var.Ava.s0 write error, block 14573856 for 16384 bytes
Mar 18 03:02:35 serv /kernel: var.Ava.s0: user buffer block 1467680 for 16384 bytes
Mar 18 03:02:35 serv /kernel: fatal :var.Ava.s0 write error, block 14572864 for 16384 bytes
Mar 18 03:02:35 serv /kernel: var.Ava.s0: user buffer block 1466688 for 16384 bytes
Mar 18 03:02:35 serv /kernel: fatal :var.Ava.s0 write error, block 14575360 for 16384 bytes
Mar 18 03:02:35 serv /kernel: var.Ava.s0: user buffer block 1469184 for 16384 bytes
Mar 18 03:02:35 serv /kernel: fatal :var.Ava.s0 write error, block 14573440 for 16384 bytes
Mar 18 03:02:35 serv /kernel: var.Ava.s0: user buffer block 1467264 for 16384 bytes
Mar 18 03:02:35 serv /kernel: fatal :var.Ava.s0 write error, block 14576320 for 16384 bytes
Mar 18 03:02:35 serv /kernel: var.Ava.s0: user buffer block 1470144 for 16384 bytes
Mar 18 03:02:35 serv /kernel: fatal :var.Ava.s0 write error, block 22660312 for 2048 bytes
Mar 18 03:02:35 serv /kernel: var.Ava.s0: user buffer block 9554136 for 2048 bytes
Mar 18 03:02:35 serv /kernel: fatal :var.Ava.s0 write error, block 24048832 for 2048 bytes
Mar 18 03:02:35 serv /kernel: var.Ava.s0: user buffer block 10942656 for 2048 bytes
Mar 18 03:02:35 serv /kernel: fatal :var.Ava.s0 write error, block 24048604 for 2048 bytes
Mar 18 03:02:35 serv /kernel: var.Ava.s0: user buffer block 10942428 for 2048 bytes
Mar 18 03:02:35 serv /kernel: fatal :var.Ava.s0 write error, block 20786780 for 2048 bytes
Mar 18 03:02:35 serv /kernel: var.Ava.s0: user buffer block 7680604 for 2048 bytes
Mar 18 03:02:35 serv /kernel: fatal :var.Ava.s0 write error, block 20776960 for 4096 bytes
Mar 18 03:02:35 serv /kernel: var.Ava.s0: user buffer block 7670784 for 4096 bytes
Mar 18 03:02:35 serv /kernel: fatal :var.Ava.s0 write error, block 24050620 for 2048 bytes
Mar 18 03:02:35 serv /kernel: var.Ava.s0: user buffer block 10944444 for 2048 bytes
Mar 18 03:02:35 serv /kernel: fatal :var.Ava.s0 write error, block 24051488 for 8192 bytes
Mar 18 03:02:35 serv /kernel: var.Ava.s0: user buffer block 10945312 for 8192 bytes
Mar 18 03:02:35 serv /kernel: fatal :var.Ava.s0 write error, block 20779584 for 6144 bytes
Mar 18 03:02:35 serv /kernel: var.Ava.s0: user buffer block 7673408 for 6144 bytes
Mar 18 03:02:35 serv /kernel: fatal :var.Ava.s0 write error, block 21860968 for 12288 bytes
Mar 18 03:02:35 serv /kernel: var.Ava.s0: user buffer block 8754792 for 12288 bytes
Mar 18 03:02:35 serv /kernel: fatal :var.Ava.s0 write error, block 21884828 for 2048 bytes
Mar 18 03:02:35 serv /kernel: var.Ava.s0: user buffer block 8778652 for 2048 bytes
Mar 18 03:02:35 serv /kernel: fatal :var.Ava.s0 write error, block 21881920 for 16384 bytes
Mar 18 03:02:35 serv /kernel: var.Ava.s0: user buffer block 8775744 for 16384 bytes
Mar 18 03:02:35 serv /kernel: fatal :var.Ava.s0 write error, block 22611296 for 16384 bytes
Mar 18 03:02:35 serv /kernel: var.Ava.s0: user buffer block 9505120 for 16384 bytes
Mar 18 03:02:38 serv /kernel: fatal :var.Ava.s0 write error, block 20775648 for 16384 bytes
Mar 18 03:02:38 serv /kernel: var.Ava.s0: user buffer block 7669472 for 16384 bytes
Mar 18 10:07:31 serv /kernel: vinum: root.Ava.s0 is stale by force
Mar 18 10:07:31 serv /kernel: vinum: root.Ava is faulty
Mar 18 10:07:31 serv /kernel: fatal :root.Ava.s0 write error, block 4578400 for 16384 bytes
Mar 18 10:07:31 serv /kernel: root.Ava.s0: user buffer block 385120 for 16384 bytes
Mar 18 10:07:31 serv /kernel: vinum: usr.Ava.s0 is crashed by force
Mar 18 10:07:31 serv /kernel: vinum: usr.Ava is faulty
Mar 18 10:07:31 serv /kernel: fatal:usr.Ava.s0 read error, block 11783340 for 2048 bytes
Mar 18 10:07:31 serv /kernel: usr.Ava.s0: user buffer block 7065772 for 2048 bytes
Mar 18 10:07:31 serv /kernel: Copyright (c) 1992-2003 The FreeBSD Project.
>How-To-Repeat:
>Fix:
>Release-Note:
>Audit-Trail:
>Unformatted:
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200403181951.i2IJpChd075261>