From owner-freebsd-questions@FreeBSD.ORG Thu Jun 30 13:37:57 2005 Return-Path: X-Original-To: freebsd-questions@freebsd.org Delivered-To: freebsd-questions@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E0BC516A41F for ; Thu, 30 Jun 2005 13:37:57 +0000 (GMT) (envelope-from gjbailey@gmail.com) Received: from wproxy.gmail.com (wproxy.gmail.com [64.233.184.201]) by mx1.FreeBSD.org (Postfix) with ESMTP id 8620D43D4C for ; Thu, 30 Jun 2005 13:37:57 +0000 (GMT) (envelope-from gjbailey@gmail.com) Received: by wproxy.gmail.com with SMTP id i21so90136wra for ; Thu, 30 Jun 2005 06:37:56 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:message-id:date:from:reply-to:to:subject:in-reply-to:mime-version:content-type:references; b=aSLnOaFKVtYXLjkA7ybUlJCm2rtRCp+stjj9Emw+kXJUQFNeCIJfnbvF75AzdUrvldS4s+2T28qACUpUeusTiiscMA5umw3sJFtoeQGYaS4qN6mKsktp/uaRh852dqRzIsbRTdkxoW4pioK0UqH8vwQ7xz2Z/EP5RzuUbgPBq9I= Received: by 10.54.115.4 with SMTP id n4mr348835wrc; Thu, 30 Jun 2005 06:37:56 -0700 (PDT) Received: by 10.54.82.12 with HTTP; Thu, 30 Jun 2005 06:37:56 -0700 (PDT) Message-ID: <48a5f32a05063006376728d52b@mail.gmail.com> Date: Thu, 30 Jun 2005 15:37:56 +0200 From: Gareth Bailey To: freebsd-questions In-Reply-To: <48a5f32a0506300250402d4514@mail.gmail.com> Mime-Version: 1.0 References: <48a5f32a0506300250402d4514@mail.gmail.com> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: Re: Vinum subdisk crash X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Gareth Bailey List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 30 Jun 2005 13:37:58 -0000 Hello everyone, Just to add, I had to type out the above message manually since I can't get= =20 access to anything with the crashed subdisk on /usr. With regard to Greg's requests for information when reporting vinum problem= s=20 as stated on vinumvm.org 's website, I can provide the= =20 following info: What problems are you having? My usr.p0.s0 subdisk reports a 'crashed' status Which version of FreeBSD are you running? 4.10 Stable Have you made any changes to the system sources, including Vinum? No Supply the output of the *vinum list* command. If you can't start Vinum,=20 supply the on-disk configuration, as described below. If you can't start=20 Vinum, then (and only then) send a copy of the configuration file. I can't get anything off the system, and its too long to type out! (I have= =20 the same layout as in the Van Valzah article.) Supply an *extract* of the Vinum history file. Unless you have explicitly= =20 renamed it, it will be */var/log/vinum_history*. This file can get very big= ;=20 please limit it to the time around when you have the problems. Each line=20 contains a timestamp at the beginning, so you will have no difficulty in=20 establishing which data is of relevance. I will summarise the tail of vinum_history (doesn't seem to provide any=20 interesting info): 30 Jun 2005 ***vinum started*** 30 Jun 2005 list 30 Jun 2005 ***vinum started*** 30 Jun 2005 dumpconfig Supply an *extract* of the file */var/log/messages*. Restrict the extract t= o=20 the same time frame as the history file. Again, each line contains a=20 timestamp at the beginning, so you will have no difficulty in establishing= =20 which data is of relevance. Again, I will summarise the tail contents of messages: 30 Jun server /kernel: ad0s1h: hard error reading fsbn 59814344 of=20 29126985-29127080 (ad0s1 bn 59814344; cn 3723 tn 69 sn 2) trying PIO mode 30 Jun server /kernel: ad0s1h: hard error reading fsbn 59814344 of=20 29126985-29127080 (ad0s1 bn 59814344; cn 3723 tn 69 sn 2) status=3D59 error= =3D40 30 Jun server /kernel: vinum: usr.p0.s0 is crashed by force 30 Jun server /kernel: vinum: usr.p0 is faulty 30 Jun server /kernel: vinum: usr is down 30 Jun server /kernel: fatal:usr.p0.s0 read error, block 29126985 for 49152= =20 bytes 30 Jun server /kernel: usr.p0.s0: user buffer block 28102720 for 49152 byte= s If you have a crash, please supply a backtrace from the dump analysis as=20 discussed below under Kernel Panics. Please don't delete the crash dump; it= =20 may be needed for further analysis. I'm not sure if a kernel panic occurred? I hope this information helps, and that someone can give me some advice!=20 Cheers, Gareth On 6/30/05, Gareth Bailey wrote: >=20 > Hello, >=20 > It appears that one of the vinum subdisks on my server has crashed. On=20 > rebooting I get the following message: >=20 >=20 > <-- start message --> >=20 > Warning: defective objects >=20 > V usr State:down Plexes:2 Size:37GB > P usr.p0 C State:faulty Subdisks:1 Size:37GB > P usr.p1 C State:faulty Subdisks:1 Size:37GB > S usr.p0.s0 State:crashed P0:0 B Size:37GB > S usr.p1.s0 State:stale P0:0 B Size:37GB >=20 > [some fsck messages] > Can't open /dev/vinum/usr: Input/output error > [some more fsck messages] > THE FOLLOWING FILE SYSTEM HAD AN UNEXPEXTED INCONSISTENCY: > /dev/vinum/usr (/usr) > Automatic file system checj failed . . . help! > Enter full pathname of shell.... >=20 > <-- end message --> >=20 > I have a straight forward configuration based on the "Bootstrapping Vinum= :=20 > A Foundation for Reliable Servers" article by Robert Van Valzah. > What could have caused this? The disks are pretty new. Please advise on= =20 > the quickest route to getting our server back online. >=20 > Much appreciated, >=20 > Gareth Bailey