From owner-freebsd-fs@FreeBSD.ORG Thu Jun 19 15:16:01 2003 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 79C9E37B401 for ; Thu, 19 Jun 2003 15:16:01 -0700 (PDT) Received: from mail.ic.sunysb.edu (mail.ic.sunysb.edu [129.49.1.4]) by mx1.FreeBSD.org (Postfix) with ESMTP id 7DAE943FDD for ; Thu, 19 Jun 2003 15:16:00 -0700 (PDT) (envelope-from mzubair@ic.sunysb.edu) Received: from postal.ic.sunysb.edu (mail [129.49.1.4]) by mail.ic.sunysb.edu (8.12.9/8.12.9) with SMTP id h5JMFvgh021888 for ; Thu, 19 Jun 2003 18:15:59 -0400 (EDT) Received: from smtp.ic.sunysb.edu ([129.49.1.24]) by postal.ic.sunysb.edu (NAVGW 2.5.2.9) with SMTP id M2003061918155831694 for ; Thu, 19 Jun 2003 18:15:58 -0400 Received: from sparky.ic.sunysb.edu (sparky.ic.sunysb.edu [129.49.1.3]) by smtp.ic.sunysb.edu (8.12.9/8.12.9) with ESMTP id h5JMFxP4021896 for ; Thu, 19 Jun 2003 18:15:59 -0400 (EDT) Received: from localhost (mzubair@localhost) by sparky.ic.sunysb.edu (8.12.9/8.12.9) with ESMTP id h5JMFxD2017897 for ; Thu, 19 Jun 2003 18:15:59 -0400 (EDT) Date: Thu, 19 Jun 2003 18:15:59 -0400 (EDT) From: Mohammad Nayyer Zubair To: freebsd-fs@freebsd.org Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Subject: ideas about a unioning file system X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Jun 2003 22:16:01 -0000 Hi. We are currently designing a stackable, fan-out file system, similar in principle to the freebsd unionfs filesystem, at the Files and Storage Lab at Stony Brook University. We intend to use the FiST (http://www1.cs.columbia.edu/~ezk/research/fist/) stackable templates. Has anyone extensively used freebsd unionfs? From a system/network administrator or from a kernel developer standpoint, what do you like about it and what you dont like about it? How should a unioning filesystem should behave? What specific features would you like it to have? Out of the previous efforts at a unioning file system like the Sun's TFS, 3DFS, Plan 9 and FreeBSD unionfs itself, which fs do you think came close to an ideal unioning file system? I would really appreciate if you can share any thoughts, ideas, comments on designing such a file system. Mohammad Nayyer Zubair From owner-freebsd-fs@FreeBSD.ORG Thu Jun 19 20:07:16 2003 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 28A9537B401 for ; Thu, 19 Jun 2003 20:07:16 -0700 (PDT) Received: from HAL9000.homeunix.com (ip114.bella-vista.sfo.interquest.net [66.199.86.114]) by mx1.FreeBSD.org (Postfix) with ESMTP id 6F87A43F85 for ; Thu, 19 Jun 2003 20:07:15 -0700 (PDT) (envelope-from das@FreeBSD.ORG) Received: from HAL9000.homeunix.com (localhost [127.0.0.1]) by HAL9000.homeunix.com (8.12.9/8.12.9) with ESMTP id h5K37DJa085151; Thu, 19 Jun 2003 20:07:13 -0700 (PDT) (envelope-from das@FreeBSD.ORG) Received: (from das@localhost) by HAL9000.homeunix.com (8.12.9/8.12.9/Submit) id h5K37C70085150; Thu, 19 Jun 2003 20:07:12 -0700 (PDT) (envelope-from das@FreeBSD.ORG) Date: Thu, 19 Jun 2003 20:07:12 -0700 From: David Schultz To: Mohammad Nayyer Zubair Message-ID: <20030620030712.GA85079@HAL9000.homeunix.com> Mail-Followup-To: Mohammad Nayyer Zubair , freebsd-fs@freebsd.org References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: cc: freebsd-fs@FreeBSD.ORG Subject: Re: ideas about a unioning file system X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Jun 2003 03:07:16 -0000 On Thu, Jun 19, 2003, Mohammad Nayyer Zubair wrote: > > Hi. > > We are currently designing a stackable, fan-out file system, similar in > principle to the freebsd unionfs filesystem, at the Files and Storage > Lab at Stony Brook University. > > We intend to use the FiST (http://www1.cs.columbia.edu/~ezk/research/fist/) stackable templates. > > Has anyone extensively used freebsd unionfs? From a system/network > administrator or from a kernel developer standpoint, what do you like > about it and what you dont like about it? Unionfs in FreeBSD has a number of implementation issues. Many of these stem out of locking problems and the fact that VFS layering doesn't really work right in FreeBSD. These are fixable details, and perhaps FiST addresses some of these. I think the cleanest way to do a union-like implementation is at the block level, with writable (``snap-off'') snapshots. This approach avoids ugly situations where file or directory renames require massive amounts of copying. Writable snapshots also have very clean semantics from the point of view of determining when a particular part of the upper layer is disassociated from modifications that may occur in the lower layer. Specifically, after the snapshot is made, changes in the lower layer are never reflected in the upper layer. With unionfs, by contrast, changes in the lower layer are reflected in the upper layer, but only for files that haven't already been modified in the upper layer. This can result in inconsistencies across files. On the other hand, in certain situations, the semantics offered by unionfs can be very useful. For example, an administrator could upgrade the userland of multiple virtual machines based on union-mounted filesystems while allowing local modifications to persist. So writable snapshots are not a drop-in replacement for something more like unionfs. Perhaps there are better alternatives that I'm unaware of. I've heard various mutterings about views, but I don't know offhand how to interpret updatable views outside of a relational database context... From owner-freebsd-fs@FreeBSD.ORG Fri Jun 20 01:51:00 2003 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4602037B401 for ; Fri, 20 Jun 2003 01:51:00 -0700 (PDT) Received: from mail-2.zoominternet.net (mail-2.zoominternet.net [63.67.120.22]) by mx1.FreeBSD.org (Postfix) with SMTP id 0907A43FB1 for ; Fri, 20 Jun 2003 01:50:59 -0700 (PDT) (envelope-from dmmiller@cvzoom.net) Received: (qmail 10313 invoked from network); 20 Jun 2003 08:50:55 -0000 Received: from acs-24-239-219-252.zoominternet.net (HELO cvzoom.net) ([24.239.219.252]) (envelope-sender ) by mail-2.zoominternet.net (qmail-ldap-1.03) with SMTP for ; 20 Jun 2003 08:50:55 -0000 Message-ID: <3EF2CAF1.4080605@cvzoom.net> Date: Fri, 20 Jun 2003 04:50:57 -0400 From: Donn Miller User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.3) Gecko/20030611 X-Accept-Language: en-us, en MIME-Version: 1.0 To: David Schultz References: In-Reply-To: X-Enigmail-Version: 0.74.3.0 X-Enigmail-Supports: pgp-inline, pgp-mime Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit cc: fs@freebsd.org Subject: Re: ideas about a unioning file system X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Jun 2003 08:51:00 -0000 David Schultz wrote: > Unionfs in FreeBSD has a number of implementation issues. Many of > these stem out of locking problems and the fact that VFS layering > doesn't really work right in FreeBSD. These are fixable details, > and perhaps FiST addresses some of these. That's one thing I was curious about. Is there a need to redo the VFS layer in FreeBSD? Someone mentioned it as a possible to-do item. If so, would it make it easier to port new filesystems to FreeBSD? From owner-freebsd-fs@FreeBSD.ORG Fri Jun 20 01:55:52 2003 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id B3DE237B401; Fri, 20 Jun 2003 01:55:52 -0700 (PDT) Received: from critter.freebsd.dk (critter.freebsd.dk [212.242.86.163]) by mx1.FreeBSD.org (Postfix) with ESMTP id 927FA43FDD; Fri, 20 Jun 2003 01:55:51 -0700 (PDT) (envelope-from phk@phk.freebsd.dk) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.12.9/8.12.9) with ESMTP id h5K8tm67007881; Fri, 20 Jun 2003 10:55:49 +0200 (CEST) (envelope-from phk@phk.freebsd.dk) To: Donn Miller From: "Poul-Henning Kamp" In-Reply-To: Your message of "Fri, 20 Jun 2003 04:50:57 EDT." <3EF2CAF1.4080605@cvzoom.net> Date: Fri, 20 Jun 2003 10:55:48 +0200 Message-ID: <7880.1056099348@critter.freebsd.dk> cc: David Schultz cc: fs@freebsd.org Subject: Re: ideas about a unioning file system X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Jun 2003 08:55:53 -0000 In message <3EF2CAF1.4080605@cvzoom.net>, Donn Miller writes: >David Schultz wrote: > >> Unionfs in FreeBSD has a number of implementation issues. Many of >> these stem out of locking problems and the fact that VFS layering >> doesn't really work right in FreeBSD. These are fixable details, >> and perhaps FiST addresses some of these. > >That's one thing I was curious about. Is there a need to redo the VFS >layer in FreeBSD? Someone mentioned it as a possible to-do item. If >so, would it make it easier to port new filesystems to FreeBSD? There are certainly deficiencies in our VFS layer, but nothing compared to our other challenges, so unless some external factor makes an up/down/side-grade a good idea, I think we should focus on making it work as it is. The majority of the problems we have in this area can be nailed down to only two issues: 1. The very tricky semantics of rename(2) 2. read/write/mmap coherency for stacked filesystems. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From owner-freebsd-fs@FreeBSD.ORG Fri Jun 20 07:32:41 2003 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4CBE737B404 for ; Fri, 20 Jun 2003 07:32:41 -0700 (PDT) Received: from laptop.tenebras.com (laptop.tenebras.com [66.92.188.18]) by mx1.FreeBSD.org (Postfix) with SMTP id 1ED6C43F85 for ; Fri, 20 Jun 2003 07:32:38 -0700 (PDT) (envelope-from kudzu@tenebras.com) Received: (qmail 23808 invoked from network); 20 Jun 2003 14:32:37 -0000 Received: from sapphire.tenebras.com (HELO tenebras.com) (192.168.188.241) by 0 with SMTP; 20 Jun 2003 14:32:37 -0000 Message-ID: <3EF31B05.5070004@tenebras.com> Date: Fri, 20 Jun 2003 07:32:37 -0700 From: Michael Sierchio User-Agent: Mozilla/5.0 (X11; U; Linux i386; en-US; rv:1.3.1) Gecko/20030425 X-Accept-Language: en-us, zh-tw, zh-cn, fr, en, de-de MIME-Version: 1.0 To: Poul-Henning Kamp References: <7880.1056099348@critter.freebsd.dk> In-Reply-To: <7880.1056099348@critter.freebsd.dk> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit cc: Donn Miller cc: David Schultz cc: fs@freebsd.org Subject: Re: ideas about a unioning file system X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Jun 2003 14:32:41 -0000 Poul-Henning Kamp wrote: > The majority of the problems we have in this area can be nailed down > to only two issues: > 1. The very tricky semantics of rename(2) > 2. read/write/mmap coherency for stacked filesystems. This is of particular interest to me. While Poul-Henning has done some very interesting work on encryption at the GEOM level, I'm trying to work with ACL filesystems and stacking crypto filesystems that manage keys with one per file. Imagine the coherency problems when pages don't match up because of IVs for CBC block mode encryption, and imagine the headache for (un-)locking pages at different levels. Making it easy for FIST and cryptfs (Erez Zadok's stuff) would be very helpful. (I.e. gee, I'd rather not have to do this on Linux). From owner-freebsd-fs@FreeBSD.ORG Fri Jun 20 09:52:37 2003 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 10CCD37B401; Fri, 20 Jun 2003 09:52:37 -0700 (PDT) Received: from filer.fsl.cs.sunysb.edu (filer.fsl.cs.sunysb.edu [130.245.126.2]) by mx1.FreeBSD.org (Postfix) with ESMTP id 25EA343FA3; Fri, 20 Jun 2003 09:52:36 -0700 (PDT) (envelope-from ezk@fsl.cs.sunysb.edu) Received: from agora.fsl.cs.sunysb.edu (IDENT:0h+vpBlfae501dvKc9K/QAWuuCLUGN6g@agora.fsl.cs.sunysb.edu [130.245.126.12])h5KGqWcq011988; Fri, 20 Jun 2003 12:52:32 -0400 Received: from agora.fsl.cs.sunysb.edu (IDENT:P62RTwWEB5BsiaHypKi54rzsQahp96A/@localhost.localdomain [127.0.0.1]) h5KGqZif007703; Fri, 20 Jun 2003 12:52:35 -0400 Received: (from ezk@localhost) by agora.fsl.cs.sunysb.edu (8.12.8/8.12.8/Submit) id h5KGqYSp007699; Fri, 20 Jun 2003 12:52:34 -0400 Date: Fri, 20 Jun 2003 12:52:34 -0400 Message-Id: <200306201652.h5KGqYSp007699@agora.fsl.cs.sunysb.edu> From: Erez Zadok To: David Schultz In-reply-to: Your message of "Thu, 19 Jun 2003 20:07:12 PDT." <20030620030712.GA85079@HAL9000.homeunix.com> X-MailKey: Erez Zadok cc: freebsd-fs@freebsd.org cc: Mohammad Nayyer Zubair Subject: Re: ideas about a unioning file system X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Jun 2003 16:52:37 -0000 In message <20030620030712.GA85079@HAL9000.homeunix.com>, David Schultz writes: > On Thu, Jun 19, 2003, Mohammad Nayyer Zubair wrote: [...] > Unionfs in FreeBSD has a number of implementation issues. Many of > these stem out of locking problems and the fact that VFS layering > doesn't really work right in FreeBSD. These are fixable details, > and perhaps FiST addresses some of these. We're looking to find out general problems that people have faced with the existing stackable union mounts in fbsd. We'd like to know more than just implementation problems (which presumably are "easy" to solve): are there any fundamental design problems with union mounts that may require OS/VFS changes or a redesign of how unioning works? Our fan-out unionfs, for example, is the first step in exploring general fan-out file systems and their issues (e.g., partial failures in replicated f/s). > I think the cleanest way to do a union-like implementation is at > the block level, with writable (``snap-off'') snapshots. This > approach avoids ugly situations where file or directory renames > require massive amounts of copying. That certainly sounds like a viable aproach, but it seems like there are already block-level snapshotting systems out there (mostly commercial). The reasons we want to stick w/ stackable are several: 1. So we can mount it on top of any other file system, not just a block-based one. Also avoid breaking the lower-level f/s format. 2. So we can port unionfs (or any other fan-out) f/s to the other platforms on which we have FiST templates (fbsd, linux, and solaris). Cheers, Erez. From owner-freebsd-fs@FreeBSD.ORG Fri Jun 20 09:57:31 2003 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id DADF437B401; Fri, 20 Jun 2003 09:57:31 -0700 (PDT) Received: from filer.fsl.cs.sunysb.edu (filer.fsl.cs.sunysb.edu [130.245.126.2]) by mx1.FreeBSD.org (Postfix) with ESMTP id F184243F75; Fri, 20 Jun 2003 09:57:30 -0700 (PDT) (envelope-from ezk@fsl.cs.sunysb.edu) Received: from agora.fsl.cs.sunysb.edu (IDENT:4OyeuAqs5YZwiS10WJqDKTl4CTYbImr3@agora.fsl.cs.sunysb.edu [130.245.126.12])h5KGvQcq012016; Fri, 20 Jun 2003 12:57:27 -0400 Received: from agora.fsl.cs.sunysb.edu (IDENT:fLjPHy4sV/097bPgUteop/ynAcmmD36r@localhost.localdomain [127.0.0.1]) h5KGvTif007746; Fri, 20 Jun 2003 12:57:29 -0400 Received: (from ezk@localhost) by agora.fsl.cs.sunysb.edu (8.12.8/8.12.8/Submit) id h5KGvPDs007736; Fri, 20 Jun 2003 12:57:25 -0400 Date: Fri, 20 Jun 2003 12:57:25 -0400 Message-Id: <200306201657.h5KGvPDs007736@agora.fsl.cs.sunysb.edu> From: Erez Zadok To: Michael Sierchio In-reply-to: Your message of "Fri, 20 Jun 2003 07:32:37 PDT." <3EF31B05.5070004@tenebras.com> X-MailKey: Erez Zadok cc: Donn Miller cc: Poul-Henning Kamp cc: David Schultz cc: fs@freebsd.org Subject: Re: ideas about a unioning file system X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Jun 2003 16:57:32 -0000 In message <3EF31B05.5070004@tenebras.com>, Michael Sierchio writes: > Poul-Henning Kamp wrote: > > > The majority of the problems we have in this area can be nailed down > > to only two issues: > > 1. The very tricky semantics of rename(2) > > 2. read/write/mmap coherency for stacked filesystems. > > This is of particular interest to me. While Poul-Henning has > done some very interesting work on encryption at the GEOM > level, I'm trying to work with ACL filesystems and stacking > crypto filesystems that manage keys with one per file. Yup, coherency is certainly a difficult problem. We have only begun to look at it, and implemented some simple solutions (for NCryptfs). We'll be exploring something more like Heidemann's work (SOSP'95) in the future. > Imagine the coherency problems when pages don't match up > because of IVs for CBC block mode encryption, and imagine > the headache for (un-)locking pages at different levels. Those of you doing crypto f/s, might be interested in reading our paper on NCryptfs, which was just presented at Usenix: http://www.fsl.cs.sunysb.edu/papers/ > Making it easy for FIST and cryptfs (Erez Zadok's stuff) would > be very helpful. (I.e. gee, I'd rather not have to do this > on Linux). BTW, we've completed a port of the fist templates to 4.x (all versions) and 5.0. We're planning to release a new fistgen RSN. Erez.