From owner-freebsd-fs@FreeBSD.ORG Sun May 11 05:55:03 2003 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D1B6937B401 for ; Sun, 11 May 2003 05:55:03 -0700 (PDT) Received: from dsl-mail.kamp.net (mail.kamp-dsl.de [195.62.99.42]) by mx1.FreeBSD.org (Postfix) with SMTP id 0EB8143FB1 for ; Sun, 11 May 2003 05:55:02 -0700 (PDT) (envelope-from me@farid-hajji.de) Received: (qmail 5228 invoked by uid 505); 11 May 2003 12:55:03 -0000 Received: from me@farid-hajji.de by dsl-mail by uid 502 with qmail-scanner-1.14 (spamassassin: 2.43. Clear:. Processed in 0.707024 secs); 11 May 2003 12:55:03 -0000 Received: from unknown (HELO reverse-213-146-115-183.dialin.kamp-dsl.de) (213.146.115.183) by dsl-mail.kamp.net with SMTP; 11 May 2003 12:55:02 -0000 From: Farid Hajji To: freebsd-fs@freebsd.org Date: Sun, 11 May 2003 14:55:27 +0200 User-Agent: KMail/1.5.1 References: <200305100055.37190.me@farid-hajji.de> <20030510144132.GA4214@schweikhardt.net> In-Reply-To: <20030510144132.GA4214@schweikhardt.net> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200305111455.28175.me@farid-hajji.de> cc: schweikh@schweikhardt.net Subject: Re: Are snapshots always consistent? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: me@farid-hajji.de List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 11 May 2003 12:55:04 -0000 Hello Jens, > # I'm somewhat confused about snapshots. > # > # Are snapshots (e.g. created with dump -L) supposed to > # be always in a consistent state? > # > # What happens if a snapshot is taken, while background > # fsck is fixing a filesystem? > > I'm no expert in file systems, but AFAIK the bg fsck mostly looks for > unreferenced blocks and puts them in the free block bitmap again, while > dump reads files (contents) and does not store block bitmaps and other > fs meta information. So, these operations should not interfere with > one another. dump saves the contents of inodes. This is more than just file contents and fsck operates on inodes. So there is potential for breakage if fsck is in the process of modifying inodes (e.g. block counts). > Regards, > > Jens -- Farid Hajji -- Unix Systems and Network Management. http://www.farid-hajji.net/address.html Quoth the Raven, "Nevermore." --Edgar Allan Poe. From owner-freebsd-fs@FreeBSD.ORG Sun May 11 10:34:41 2003 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 513BC37B401; Sun, 11 May 2003 10:34:41 -0700 (PDT) Received: from blueyonder.co.uk (pcow025o.blueyonder.co.uk [195.188.53.125]) by mx1.FreeBSD.org (Postfix) with ESMTP id 7C1D043FBD; Sun, 11 May 2003 10:34:39 -0700 (PDT) (envelope-from schweikh@schweikhardt.net) Received: from mail pickup service by blueyonder.co.uk with Microsoft SMTPSVC; Sun, 11 May 2003 18:19:39 +0100 Received: from mx2.freebsd.org ([216.136.204.119]) by blueyonder.co.uk with Microsoft SMTPSVC(5.5.1877.757.75); Sat, 10 May 2003 15:42:23 +0100 Received: from hub.freebsd.org (hub.freebsd.org [216.136.204.18]) by mx2.freebsd.org (Postfix) with ESMTP id 2BFEF574BD; Sat, 10 May 2003 07:41:46 -0700 (PDT) (envelope-from owner-freebsd-current@freebsd.org) Received: from hub.freebsd.org (localhost [127.0.0.1]) by hub.freebsd.org (Postfix) with ESMTP id A480437B409; Sat, 10 May 2003 07:41:45 -0700 (PDT) Delivered-To: freebsd-current@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9FFE937B401; Sat, 10 May 2003 07:41:34 -0700 (PDT) Received: from bremen.shuttle.de (bremen.shuttle.de [194.95.249.251]) by mx1.FreeBSD.org (Postfix) with ESMTP id B1B1743FE0; Sat, 10 May 2003 07:41:33 -0700 (PDT) (envelope-from schweikh@schweikhardt.net) Received: from bremen.shuttle.de (localhost [127.0.0.1]) by bremen.shuttle.de (Postfix) with ESMTP id 0F10A17D44; Sat, 10 May 2003 16:41:32 +0200 (CEST) Received: (from uucp@localhost)h4AEfVC7011365; Sat, 10 May 2003 16:41:31 +0200 Received: (from schweikh@localhost) by hal9000.schweikhardt.net (8.12.9/8.12.9) id h4AEfW1C004768; Sat, 10 May 2003 16:41:32 +0200 (CEST) (envelope-from schweikh) Date: Sat, 10 May 2003 16:41:32 +0200 From: Jens Schweikhardt To: Farid Hajji Message-ID: <20030510144132.GA4214@schweikhardt.net> References: <200305100055.37190.me@farid-hajji.de> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <200305100055.37190.me@farid-hajji.de> User-Agent: Mutt/1.4.1i X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Sender: owner-freebsd-current@freebsd.org Errors-To: owner-freebsd-current@freebsd.org cc: freebsd-fs@freebsd.org cc: freebsd-current@freebsd.org Subject: Re: Are snapshots always consistent? X-BeenThere: freebsd-fs@freebsd.org Reply-To: freebsd-fs@freebsd.org List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 11 May 2003 17:34:41 -0000 Farid, On Sat, May 10, 2003 at 12:55:37AM +0200, Farid Hajji wrote: # I'm somewhat confused about snapshots. # # Are snapshots (e.g. created with dump -L) supposed to # be always in a consistent state? # # What happens if a snapshot is taken, while background # fsck is fixing a filesystem? I'm no expert in file systems, but AFAIK the bg fsck mostly looks for unreferenced blocks and puts them in the free block bitmap again, while dump reads files (contents) and does not store block bitmaps and other fs meta information. So, these operations should not interfere with one another. # If snapshots are not always consistent, how can one # be sure about dump -L output? Is it better to take # a snapshot first, fsck it, and _then_ dump the # snapshot without -L (how?)? The most appropriate place to ask this is probably freebsd-fs@ (in Cc: and Reply-To:). Regards, Jens -- Jens Schweikhardt http://www.schweikhardt.net/ SIGSIG -- signature too long (core dumped) _______________________________________________ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" From owner-freebsd-fs@FreeBSD.ORG Sun May 11 12:00:47 2003 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 60A3037B404; Sun, 11 May 2003 12:00:47 -0700 (PDT) Received: from blueyonder.co.uk (pcow025o.blueyonder.co.uk [195.188.53.125]) by mx1.FreeBSD.org (Postfix) with ESMTP id 9DC7A43FE9; Sun, 11 May 2003 12:00:45 -0700 (PDT) (envelope-from schweikh@schweikhardt.net) Received: from mail pickup service by blueyonder.co.uk with Microsoft SMTPSVC; Sun, 11 May 2003 18:47:59 +0100 Received: from mx2.freebsd.org ([216.136.204.119]) by blueyonder.co.uk with Microsoft SMTPSVC(5.5.1877.757.75); Sat, 10 May 2003 17:42:50 +0100 Received: from hub.freebsd.org (hub.freebsd.org [216.136.204.18]) by mx2.freebsd.org (Postfix) with ESMTP id B78CF574AC; Sat, 10 May 2003 07:41:45 -0700 (PDT) (envelope-from owner-freebsd-current@freebsd.org) Received: from hub.freebsd.org (localhost [127.0.0.1]) by hub.freebsd.org (Postfix) with ESMTP id E5BFB37B405; Sat, 10 May 2003 07:41:44 -0700 (PDT) Delivered-To: freebsd-current@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9FFE937B401; Sat, 10 May 2003 07:41:34 -0700 (PDT) Received: from bremen.shuttle.de (bremen.shuttle.de [194.95.249.251]) by mx1.FreeBSD.org (Postfix) with ESMTP id B1B1743FE0; Sat, 10 May 2003 07:41:33 -0700 (PDT) (envelope-from schweikh@schweikhardt.net) Received: from bremen.shuttle.de (localhost [127.0.0.1]) by bremen.shuttle.de (Postfix) with ESMTP id 0F10A17D44; Sat, 10 May 2003 16:41:32 +0200 (CEST) Received: (from uucp@localhost)h4AEfVC7011365; Sat, 10 May 2003 16:41:31 +0200 Received: (from schweikh@localhost) by hal9000.schweikhardt.net (8.12.9/8.12.9) id h4AEfW1C004768; Sat, 10 May 2003 16:41:32 +0200 (CEST) (envelope-from schweikh) Date: Sat, 10 May 2003 16:41:32 +0200 From: Jens Schweikhardt To: Farid Hajji Message-ID: <20030510144132.GA4214@schweikhardt.net> References: <200305100055.37190.me@farid-hajji.de> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <200305100055.37190.me@farid-hajji.de> User-Agent: Mutt/1.4.1i X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Sender: owner-freebsd-current@freebsd.org Errors-To: owner-freebsd-current@freebsd.org cc: freebsd-fs@freebsd.org cc: freebsd-current@freebsd.org Subject: Re: Are snapshots always consistent? X-BeenThere: freebsd-fs@freebsd.org Reply-To: freebsd-fs@freebsd.org List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 11 May 2003 19:00:47 -0000 Farid, On Sat, May 10, 2003 at 12:55:37AM +0200, Farid Hajji wrote: # I'm somewhat confused about snapshots. # # Are snapshots (e.g. created with dump -L) supposed to # be always in a consistent state? # # What happens if a snapshot is taken, while background # fsck is fixing a filesystem? I'm no expert in file systems, but AFAIK the bg fsck mostly looks for unreferenced blocks and puts them in the free block bitmap again, while dump reads files (contents) and does not store block bitmaps and other fs meta information. So, these operations should not interfere with one another. # If snapshots are not always consistent, how can one # be sure about dump -L output? Is it better to take # a snapshot first, fsck it, and _then_ dump the # snapshot without -L (how?)? The most appropriate place to ask this is probably freebsd-fs@ (in Cc: and Reply-To:). Regards, Jens -- Jens Schweikhardt http://www.schweikhardt.net/ SIGSIG -- signature too long (core dumped) _______________________________________________ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" From owner-freebsd-fs@FreeBSD.ORG Mon May 12 15:32:48 2003 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id CFCBB37B401; Mon, 12 May 2003 15:32:48 -0700 (PDT) Received: from dynamic.hydro.washington.edu (dynamic.hydro.washington.edu [128.95.246.166]) by mx1.FreeBSD.org (Postfix) with ESMTP id 20D1943FE0; Mon, 12 May 2003 15:32:48 -0700 (PDT) (envelope-from penglish@hydro.washington.edu) Received: from dynamic.hydro.washington.edu (localhost [127.0.0.1]) h4CMWdTP047626; Mon, 12 May 2003 15:32:39 -0700 (PDT) (envelope-from penglish@hydro.washington.edu) Received: from localhost (penglish@localhost)h4CMWdBL047623; Mon, 12 May 2003 15:32:39 -0700 (PDT) (envelope-from penglish@hydro.washington.edu) X-Authentication-Warning: dynamic.hydro.washington.edu: penglish owned process doing -bs Date: Mon, 12 May 2003 15:32:39 -0700 (PDT) From: Paul English To: Chuck Swiger In-Reply-To: <3EBD4405.8060406@mac.com> Message-ID: <20030512145220.K47130-100000@dynamic.hydro.washington.edu> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: freebsd-fs@freebsd.org cc: freebsd-questions@freebsd.org Subject: Filesystem directory structure recovery - competition for Beer of the Month Club X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 May 2003 22:32:49 -0000 Hi Everyone, At Chuck's suggestion I'm opening up a competition for a 6 month subscription to Beer of the Month club. The Judge: That's me. A reasonably competent sysadmin. The Goal: Be the first one to send me the most helpful information on how to recover the directory structure of a messed-up filesystem. The Parameters: I had a disk failure. Unfortunately, I also discovered that my backup server (Legato Networker under Solaris) had run out of licenses. And tried to email me, but failed to inform me on the Networker screen that I keep open at all times. (I'm not bitter!) And I also discovered the email delivery wasn't working properly on that machine. So - I have no backups of this system. I sent my disk to a data recovery company. They were able to recover "all but 2%" of the data, but they did it in a raw mode and copied it to a new disk and sent that to me. I promptly copied that to yet another disk (using dd) so that I wouldn't mess up what I paid them so much money for. They're still trying to recover the directory structure for me, and if they beat you all the competition is off since they won't be giving me a refund! I am only interested in one separate partition on this disk with my user's data on it. So I tried to mount that partition, only to find that the superblock was bad. I used newfs -N to find out where the backup superblocks were, and found one (several actually) that were fine. Using fsck -b to restore it, then fsck wants to make LOTS of changes. If I say "yes" to everything, *all* of the data on the disk (15GB) ends up in lost+found. Some of the directory structure is preserved, but there 145 inode-numbered directories in the top level. If I say "no" to everything (except replacement of the bad superblock), I can mount it read-only (or rw with -f), but there is nothing there when I do an ls: #ls /mnt > ls: /mnt: Bad file descriptor I have not tried doing a mixture of yes's and no's to fsck's questions, since it asks a *lot* of questions about this filesystem, and I don't know which ones to say yes or no to in any case. That would be useful information. I would like to hand my user back his directory with all of those 145 inode-numbered directories in their proper places in the filesystem with their proper names. Without the names, places would still be helpful and vice versa. Paul From owner-freebsd-fs@FreeBSD.ORG Tue May 13 14:06:36 2003 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2FE9837B401; Tue, 13 May 2003 14:06:36 -0700 (PDT) Received: from mail.allcaps.org (allcaps.org [216.240.173.16]) by mx1.FreeBSD.org (Postfix) with ESMTP id 7E7F643F3F; Tue, 13 May 2003 14:06:35 -0700 (PDT) (envelope-from bsder@allcaps.org) Received: from mail.allcaps.org (localhost [127.0.0.1]) by mail.allcaps.org (Postfix) with ESMTP id 14A1A92FAF; Tue, 13 May 2003 17:09:44 -0400 (EDT) Received: from localhost (bsder@localhost)h4DL9hNQ002005; Tue, 13 May 2003 14:09:44 -0700 X-Authentication-Warning: mail.allcaps.org: bsder owned process doing -bs Date: Tue, 13 May 2003 14:09:43 -0700 (PDT) From: "Andrew P. Lentvorski, Jr." To: Robert Watson In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: Don Lewis cc: alfred@FreeBSD.org cc: fs@FreeBSD.org Subject: Kernel support for NFS locking (was: Re: rpc.lockd spinning; much breakage) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 May 2003 21:06:36 -0000 On Tue, 13 May 2003, Robert Watson wrote: > > Andrew P. Lentvorski wrote: > > > > One problem is that FreeBSD doesn't allocate enough fields in its local > > lock structure to distinguish external identifiers in the locks (all > > locks look like they are owned by the rpc.lockd user). Consequently, > > rpc.lockd has to maintain its own state as to who has what locks. > > If this isn't a user/kernel ABI/API problem, it should be easily solvable, > and we should do that following 5.1-RELEASE. This opens up a several cans of worms: 1) Is this going to suck down too much memory? rpc.lockd identifiers can be really long 2) Should the kernel have some mechanism for handling locking Notifiying processes which are waiting for a blocked lock is more efficient as a kernel callback rather than process polling. Is kqueue/kevent sufficient? 3) Should the kernel have some mechanism for modular handling of locking associated with an FS? ie. the ability to pass out NFS locks on say an SMB mounted FS 4) Should duct tape be the solution to NFSv2 and NFSv3 and folks should just move to NFSv4? 5) What is the status of NFSv4? 6) What other FS's need adjustments to the FreeBSD file lock strcutures? I'm certainly not qualified to answer these questions, so I'm going to transfer this message over to fs@freebsd.org in the hope that someone better qualified can provide more insight. -a From owner-freebsd-fs@FreeBSD.ORG Fri May 16 10:33:54 2003 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id CA33237B401 for ; Fri, 16 May 2003 10:33:54 -0700 (PDT) Received: from sbcs.cs.sunysb.edu (sbcs.sunysb.edu [130.245.1.15]) by mx1.FreeBSD.org (Postfix) with ESMTP id E015343F75 for ; Fri, 16 May 2003 10:33:53 -0700 (PDT) (envelope-from harikesa@cs.sunysb.edu) Received: from compserv2 (compserv2 [130.245.1.58]) by sbcs.cs.sunysb.edu (8.12.3/8.12.3) with ESMTP id h4GHUmXY021427 for ; Fri, 16 May 2003 13:30:48 -0400 (EDT) Date: Fri, 16 May 2003 13:30:50 -0400 (EDT) From: Harikesavan Krishnan X-X-Sender: harikesa@compserv2 To: freebsd-fs@freebsd.org Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Subject: Problem with Flags on fist X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 May 2003 17:33:55 -0000 Hi, We're working on porting fistgen (stackable templates) from freebsd 4.x to FreeBSD 5.0. I'm facing a small problem while renaming inside any of my stackable file systems (such as cryptfs, rot13fs, etc.). To achieve full stacking transparency, we keep a componentname structure in our layer, as well as the lower layer. We have some code that tries to check the cnp->cn_flags fields in the two layers. We found that the flags differ in both 4.x and 5.x: we're not sure why, or whether this is a bug. It doesn't cause anything bad that we've noticed, and we ran a large compile, fsx, bonnie, etc. repeatedly. Here's the code snippet: static int cryptfs_rename(ap) struct vop_rename_args *ap; { vnode_t *thisfdvp = ap->a_fdvp; vnode_t *thistdvp = ap->a_tdvp; // some code follows thisfcnp = ap->a_fcnp; lowerfcnp = cryptfs_new_cnp((thisfdvp), thisfcnp); ap->a_fcnp = lowerfcnp; error = cryptfs_bypass((struct vop_generic_args *) ap); if ((thisfcnp->cn_flags & PARAMASK) != (lowerfcnp->cn_flags & PARAMASK)) { printk("%s: FLAGS CHANGED fthis:0x%x flower:0x%x", __FUNCTION__, (int)thisfcnp->cn_flags, (int) lowerfcnp->cn_flags); } ... } The flag values of the upper and lower layers are as follows: thisfcnp->cn_flags = 0x209410 which is: (PDIRUNLOCK | ISLASTCN | SAVESTART | HASBUF ) lowerfcnp->cn_flags = 0x940c which is: (ISLASTCN | SAVESTART | HASBUF) These are the same flag values that we've noticed when running stackable f/s based on our 4.x templates. However, in fbsd 4.x, the flags always match, and the above printk "FLAGS CHANGED" is never printed. Once we ported the templates to 5.0, this printk started showing up. We did notice that the value of PARAMASX in 5.0 is different. Fortunately, the bug is quite reproducible: mount the f/s, and rename a file inside of it (same result whether the file existed or not). So our questions are: 1. Is it ok to leave the code as is, or is there something wrong we're doing that may bite us later? 2. what could be causing the change in flags? Are we doing something wrong in ->rename, or somewhere else we're not setting something right? 3. any advise on how to fix the code? BTW, this is the last issue we have left before we can release a new version of fistgen that includes fully working, brand new, templates for freebsd 4.x and 5.0. Thanks, Hari and Erez.