From owner-freebsd-fs Wed Jun 17 04:10:57 1998 Return-Path: Received: (from majordom@localhost) by hub.freebsd.org (8.8.8/8.8.8) id EAA25098 for freebsd-fs-outgoing; Wed, 17 Jun 1998 04:10:57 -0700 (PDT) (envelope-from owner-freebsd-fs@FreeBSD.ORG) Received: from parkplace.cet.co.jp (parkplace.cet.co.jp [202.32.64.1]) by hub.freebsd.org (8.8.8/8.8.8) with ESMTP id EAA24981; Wed, 17 Jun 1998 04:10:48 -0700 (PDT) (envelope-from michaelh@cet.co.jp) Received: from localhost (michaelh@localhost) by parkplace.cet.co.jp (8.8.8/CET-v2.2) with SMTP id LAA05194; Wed, 17 Jun 1998 11:09:48 GMT Date: Wed, 17 Jun 1998 20:09:48 +0900 (JST) From: Michael Hancock To: "Alton, Matthew" cc: "'FreeBSD-fs@FreeBSD.ORG'" , "Smallie, Scott" , FreeBSD Hackers Subject: Re: Filesystem Development Toolkit In-Reply-To: <31B3F0BF1C40D11192A700805FD48BF9017765D7@STLABCEXG011> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-fs@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Matthew, The project sounds interesting. You might want to refer to John Heidemann's work done while he was at UCLA. You can find his pages at I think http://www.isi.edu/~johnh. His approach is radically different and potentially a lot less work for future FS implementors. You could define a standard vop interface and map it into FreeBSD and the vop layers of other Unix implementations. Basically it allows you to leverage off of existing code. In FreeBSD a partially implemented framework exists, but it needs to be cleaned up. There are 2 major problems with it now: 1) There are ref counting and locking layering violations in the code. I've cleaned these up for everything except vop_rename, vop_mknod, and vop_symlink so far. 2) There is no object coherence management done. This is a more serious problem that is inherent in any stacked design where you want to cache objects, data, and attributes in different layers that represent a file. This will require taking a hard look at the top half of the kernel code where calls are made to things like vop_rdwr, mmap, vop_{put|get}pages and the implementation themselves to properly design it. In a layered environment you need to make sure that operations are either done on the same vnode of a file or you need a cache_mgr to manage coherence between the cached objects hanging off of all the vnodes that represent the file. One reason that I'd like to see a user-space layer implemented is that it would represent an extra requirement in the design of the solution to the problems in 2) above. i.e. Instead of putting in VM calls here and there you would be forced to think of a cleaner solution, otherwise you will have to implement a lot of weird system calls to emulate those VM calls. If you implemented a cache_mgr then you could reduce the number of system calls you would need to implement and use in your user-land emulation of the kernel APIs. With a cache_mgr implementation, vop_putpages and vop_getpages would take 2 vnode pointers. One would be a caching vp and the other would be the paging vp and these could potentially be in different layers of the stack. The cache_mgr would ensure that the correct vps are passed to the operations. For example, if you implemented a compression or encryption layer on top of ufs, you would likely want the clear text cached for performance, but you would want to ensure that the ufs pager were used to actually write to the media. Then at night when a backup is running against the compressed files, the ufs layer would be the caching layer. I wish I had the time to work on something like this, but I don't. As far as people that can contribute to this the following list is what I can come up with off the top of my head. Poul Henning-Kamp, Julian Elisher, David Greenman, Bruce Evans, Peter Wemm, Doug Rabson, Tor Egge, Luoqi Chen, and Simon Shapiro. I mention Simon Shapiro because he has been working on a DLM and many of the concepts are similar but generalized to not just locks, but also data pages and file attributes. It's interesting to note that John Heidemann's experimental versions did not include locks and was not distributed. It just used the existing vnode locking implementation and interfaced with other distributed protocols such as NFS, etc. Of course, if some lurker out there were to take on this challenge that would be pretty cool too. Regards, Mike Hancock To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-fs" in the body of the message From owner-freebsd-fs Wed Jun 17 04:20:12 1998 Return-Path: Received: (from majordom@localhost) by hub.freebsd.org (8.8.8/8.8.8) id EAA27669 for freebsd-fs-outgoing; Wed, 17 Jun 1998 04:20:12 -0700 (PDT) (envelope-from owner-freebsd-fs@FreeBSD.ORG) Received: from parkplace.cet.co.jp (parkplace.cet.co.jp [202.32.64.1]) by hub.freebsd.org (8.8.8/8.8.8) with ESMTP id EAA27545; Wed, 17 Jun 1998 04:19:53 -0700 (PDT) (envelope-from michaelh@cet.co.jp) Received: from localhost (michaelh@localhost) by parkplace.cet.co.jp (8.8.8/CET-v2.2) with SMTP id LAA05224; Wed, 17 Jun 1998 11:18:55 GMT Date: Wed, 17 Jun 1998 20:18:55 +0900 (JST) From: Michael Hancock To: "Alton, Matthew" cc: "'FreeBSD-fs@FreeBSD.ORG'" , "Smallie, Scott" , FreeBSD Hackers Subject: Re: Filesystem Development Toolkit In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-fs@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org On Wed, 17 Jun 1998, Michael Hancock wrote: > Poul Henning-Kamp, Julian Elisher, David Greenman, Bruce Evans, Peter > Wemm, Doug Rabson, Tor Egge, Luoqi Chen, and Simon Shapiro. > > I mention Simon Shapiro because he has been working on a DLM and many of > the concepts are similar but generalized to not just locks, but also data > pages and file attributes. It's interesting to note that John Heidemann's > experimental versions did not include locks and was not distributed. It > just used the existing vnode locking implementation and interfaced with > other distributed protocols such as NFS, etc. > > Of course, if some lurker out there were to take on this challenge that > would be pretty cool too. Oh, and how could I forget, Terry Lambert would also have a lot of ideas in this area as well or he might just yak and be a general pain in the ass. ;-) which is about all I have time to do these days. Regards, Mike Hancock To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-fs" in the body of the message From owner-freebsd-fs Wed Jun 17 11:20:58 1998 Return-Path: Received: (from majordom@localhost) by hub.freebsd.org (8.8.8/8.8.8) id LAA07036 for freebsd-fs-outgoing; Wed, 17 Jun 1998 11:20:58 -0700 (PDT) (envelope-from owner-freebsd-fs@FreeBSD.ORG) Received: from gershwin.tera.com (gershwin.tera.com [207.224.230.28]) by hub.freebsd.org (8.8.8/8.8.8) with ESMTP id LAA07016; Wed, 17 Jun 1998 11:20:53 -0700 (PDT) (envelope-from kline@tao.thought.org) Received: from tao.thought.org (tao.tera.com [207.108.223.55]) by gershwin.tera.com (8.8.8/8.8.8) with ESMTP id LAA20318; Wed, 17 Jun 1998 11:20:10 -0700 (PDT) Received: (from kline@localhost) by tao.thought.org (8.8.8/8.7.3) id LAA06262; Wed, 17 Jun 1998 11:19:58 -0700 (PDT) From: Gary Kline Message-Id: <199806171819.LAA06262@tao.thought.org> Subject: Re: Filesystem Development Toolkit In-Reply-To: from Michael Hancock at "Jun 17, 98 08:18:55 pm" To: michaelh@cet.co.jp (Michael Hancock) Date: Wed, 17 Jun 1998 11:19:58 -0700 (PDT) Cc: Matthew.Alton@anheuser-busch.com, FreeBSD-fs@FreeBSD.ORG, Scott.Smallie@anheuser-busch.com, Hackers@FreeBSD.ORG Organization: <> thought.org: public access uNix in service... <> X-Mailer: ELM [version 2.4ME+ PL32 (25)] MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: owner-freebsd-fs@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org According to Michael Hancock: > On Wed, 17 Jun 1998, Michael Hancock wrote: > > > Poul Henning-Kamp, Julian Elisher, David Greenman, Bruce Evans, Peter > > Wemm, Doug Rabson, Tor Egge, Luoqi Chen, and Simon Shapiro. > > > > I mention Simon Shapiro because he has been working on a DLM and many of > > the concepts are similar but generalized to not just locks, but also data > > pages and file attributes. It's interesting to note that John Heidemann's > > experimental versions did not include locks and was not distributed. It > > just used the existing vnode locking implementation and interfaced with > > other distributed protocols such as NFS, etc. > > > > Of course, if some lurker out there were to take on this challenge that > > would be pretty cool too. > > Oh, and how could I forget, Terry Lambert would also have a lot of ideas > in this area as well or he might just yak and be a general pain in the > ass. ;-) which is about all I have time to do these days. > > If Terry's FS-based Unicode support would fit into this, it'd be interesting. So far I'm working on localizations via the locale catalogs. This may be a short-term solution and a broader, global solution may be a FS with wchar_t support. Any thoughts; or am I too far off-course? gary -- Gary D. Kline kline@tao.thought.org Public service uNix To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-fs" in the body of the message From owner-freebsd-fs Wed Jun 17 13:20:48 1998 Return-Path: Received: (from majordom@localhost) by hub.freebsd.org (8.8.8/8.8.8) id NAA28021 for freebsd-fs-outgoing; Wed, 17 Jun 1998 13:20:48 -0700 (PDT) (envelope-from owner-freebsd-fs@FreeBSD.ORG) Received: from smtp02.primenet.com (daemon@smtp02.primenet.com [206.165.6.132]) by hub.freebsd.org (8.8.8/8.8.8) with ESMTP id NAA28013; Wed, 17 Jun 1998 13:20:47 -0700 (PDT) (envelope-from tlambert@usr01.primenet.com) Received: (from daemon@localhost) by smtp02.primenet.com (8.8.8/8.8.8) id NAA02618; Wed, 17 Jun 1998 13:20:36 -0700 (MST) Received: from usr01.primenet.com(206.165.6.201) via SMTP by smtp02.primenet.com, id smtpd002582; Wed Jun 17 13:20:31 1998 Received: (from tlambert@localhost) by usr01.primenet.com (8.8.5/8.8.5) id NAA26615; Wed, 17 Jun 1998 13:20:23 -0700 (MST) From: Terry Lambert Message-Id: <199806172020.NAA26615@usr01.primenet.com> Subject: Re: Filesystem Development Toolkit To: michaelh@cet.co.jp (Michael Hancock) Date: Wed, 17 Jun 1998 20:20:23 +0000 (GMT) Cc: Matthew.Alton@anheuser-busch.com, FreeBSD-fs@FreeBSD.ORG, Scott.Smallie@anheuser-busch.com, Hackers@FreeBSD.ORG In-Reply-To: from "Michael Hancock" at Jun 17, 98 08:09:48 pm X-Mailer: ELM [version 2.4 PL25] MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: owner-freebsd-fs@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org [ ... NTFS ... ] > In FreeBSD a partially implemented framework exists, but it needs to be > cleaned up. There are 2 major problems with it now: > > 1) There are ref counting and locking layering violations in the code. > I've cleaned these up for everything except vop_rename, vop_mknod, and > vop_symlink so far. > > 2) There is no object coherence management done. This is a more serious > problem that is inherent in any stacked design where you want to cache > objects, data, and attributes in different layers that represent a file. I can not emphasize the interactions with an NTFS implementation that results from these two errors enough. Linux has a read-only NTFS, and will not be able to implement an actual read-write NTFS without a major overhaul of their VFS/VOP interface. FreeBSD also needs an overhaul, but it is somewhat less dire. At one point in time I had a loaner PPC that I was doing a FreeBSD port on; this port suffered when I had to look for paying work, and returned the machine. The machine was running AIX, and I has the PPC booting (via PPCBug, not OpenBoot) into a single user system. I was able to mount and access an AIX JFS, which has many of the same issues as NTFS, or SGI's XFS (or, to a lesser extent, Veritas's VXFS -- lesser because VXFS's directory management code was SVR4 UFS derived). The main isue here is that VOP_ABORTOP is not used correctly in FreeBSD's VFS interface. It is used to recover cn_pnbuf's allocated by the caller (ie: it is not reflexive). For file systems which are, in effect, transaction interfaces, such as journalling or log stuctured FS's, the VOP_ABORTOP needs to, in fact, actually be capable of aborting a transaction that spans a number of VOP calls. Such as VOP_LOOKUP calls that result in the allocation of a directory entry slot for later use by a rename, mknod, or link operation, or slot locks for use by unlink and rename. Add to this that VOP_ABORTOP is not called in the correct places because of self-freeing the cn_pnbuf by operations that should tag the operation as complete. VOP_ABORTOP needs to be VOP_TRANSACT, needs to take a "BEGIN", "COMMIT", or "ABORT". The "BEGIN" needs to return a transaction ID to the caller, which is provided as a context argument to the other VOP's ... in most cases, this transaction identifier will be an opaque reference to a proc pointer; for NTFS, it needs to be a real transaction, where the proc pointer is a member. In other words, this is about as FS specific as TFS's vnodes. > This will require taking a hard look at the top half of the kernel code > where calls are made to things like vop_rdwr, mmap, vop_{put|get}pages and > the implementation themselves to properly design it. In a layered > environment you need to make sure that operations are either done on the > same vnode of a file or you need a cache_mgr to manage coherence between > the cached objects hanging off of all the vnodes that represent the file. Yes. But one of the main benefits of FreeBSD over NetBSD is the unification of the coherency model. I really think that the code sould be using macrotized VM calls, such that coherency can be enforced automatically in a unified VM, and manually in a non-unified or partially unified (like NetBSD's UVM) VM system. The real issue here is portability of the FS code without dependency on the host kernel implementation. > One reason that I'd like to see a user-space layer implemented is that it > would represent an extra requirement in the design of the solution to the > problems in 2) above. i.e. Instead of putting in VM calls here and there > you would be forced to think of a cleaner solution, otherwise you will > have to implement a lot of weird system calls to emulate those VM calls. Right. The coherency model for such an inteface requires that the interface be reflexive. For a user space implementation, the FS can replace the VM macro references with a wrapper to proxy pseudo VM obects into and out of the kernel. This is another major impetus to making the interfaces reflexive, such that if you call something with a buffer you allocate, then *you* expected to perform the deallocation. You can't proxy a deallocation of a cn_pnbuf in user space if you allocated the thing in the kernel. The one exception allowed is locks, which are objects which are "held". They are in a different abstraction domain. Locks are already abstract in that they are opaque, and you must use operators against them. You can operate on an abstract interface using proxies all you want, and never break anything. Path name buffers and vnodes need to be similary abstract. For vnodes, this means that a proxied interface must manage the data portion of the vnodes itself, as an abstract object. This is somewhat different than the current FreeBSD model, though there are non-integrated patches (not by me) that clean this up somewhat. You would then need to macrotize vnode allocations and releases from a common pool the same way, to allow the pool to be proxied into user space. Right now, FreeBSD is not in a good position to support user space FS developement. Some work is needed to make intermediate stacking layer developement possible. For bottom level device access, which is what a local media FS, like NTFS or FFS have to do, there are over 120 kernel interfaces being imported. That's 120 interfaces to proxy. Some of these are as simple as each and every FS calling the same kernel interface using a address of a local opject ("inode") to implement an abstract data reference (vnode->inode->ref), instead of hanging the reference off the vnode directly (vnode->ref). A direct hanging would allow the kernel interface to use the default VOP's (which breaks layer collapse, and are therefore evil), or, better, put the common calls in common code, and invert the call to make it veto-based. This would preserve the ability to collapse null VOPs in N layers to one layer of function calls, without having to implement null layer stub functions for coherency (this is currently what Tor Egge's patches to null FS do to workaround the interface problems -- the nullfs isn't very "null" after that). Either approach, however, reduces the number of kernel interfaces an FS must consume to be a fullimplementation, and therefore reduces the number of interfaces which must be proied to user space, and back again. > If you implemented a cache_mgr then you could reduce the number of system > calls you would need to implement and use in your user-land emulation of > the kernel APIs. This is a proxy gateway as a single system call, with a large number of proxies. It would be much better to reduce the number of calls that must be proxied, or at least some combination of both (clearly, there must be a proxy gateway of some kind). Terry Lambert terry@lambert.org --- Any opinions in this posting are my own and not those of my present or previous employers. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-fs" in the body of the message From owner-freebsd-fs Wed Jun 17 13:33:32 1998 Return-Path: Received: (from majordom@localhost) by hub.freebsd.org (8.8.8/8.8.8) id NAA00310 for freebsd-fs-outgoing; Wed, 17 Jun 1998 13:33:32 -0700 (PDT) (envelope-from owner-freebsd-fs@FreeBSD.ORG) Received: from smtp03.primenet.com (daemon@smtp03.primenet.com [206.165.6.133]) by hub.freebsd.org (8.8.8/8.8.8) with ESMTP id NAA00288; Wed, 17 Jun 1998 13:33:22 -0700 (PDT) (envelope-from tlambert@usr01.primenet.com) Received: (from daemon@localhost) by smtp03.primenet.com (8.8.8/8.8.8) id NAA19956; Wed, 17 Jun 1998 13:33:16 -0700 (MST) Received: from usr01.primenet.com(206.165.6.201) via SMTP by smtp03.primenet.com, id smtpd019928; Wed Jun 17 13:33:13 1998 Received: (from tlambert@localhost) by usr01.primenet.com (8.8.5/8.8.5) id NAA27289; Wed, 17 Jun 1998 13:33:07 -0700 (MST) From: Terry Lambert Message-Id: <199806172033.NAA27289@usr01.primenet.com> Subject: Re: Filesystem Development Toolkit To: michaelh@cet.co.jp (Michael Hancock) Date: Wed, 17 Jun 1998 20:33:07 +0000 (GMT) Cc: Matthew.Alton@anheuser-busch.com, FreeBSD-fs@FreeBSD.ORG, Scott.Smallie@anheuser-busch.com, Hackers@FreeBSD.ORG In-Reply-To: from "Michael Hancock" at Jun 17, 98 08:18:55 pm X-Mailer: ELM [version 2.4 PL25] MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: owner-freebsd-fs@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > Oh, and how could I forget, Terry Lambert would also have a lot of ideas > in this area as well or he might just yak and be a general pain in the > ass. ;-) which is about all I have time to do these days. It's always annoying when someone tells you your existing abstractions don't match your design documents, isn't it.? It's even more annoying when you don't have time to review the changes proposed to fix this with a suffient eye towrads the big picture that the changes actually make it in... }B-). Meanwhile, I still have a system with an OS image from June of last year that can support user space FS developement (but which would have a hell of a time with VM proxie, which would need a total rewrite) and a rudimentary GFS/JFS implementaiton that can read AIX disks enough to load /bin/sh. 8-P. I'm still willing to code if you're willing to commit. I guess FreeBSD -current can be though of as considerably stable at this point, so now is probably a good time to do this. Terry Lambert terry@lambert.org --- Any opinions in this posting are my own and not those of my present or previous employers. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-fs" in the body of the message From owner-freebsd-fs Wed Jun 17 13:56:28 1998 Return-Path: Received: (from majordom@localhost) by hub.freebsd.org (8.8.8/8.8.8) id NAA04823 for freebsd-fs-outgoing; Wed, 17 Jun 1998 13:56:28 -0700 (PDT) (envelope-from owner-freebsd-fs@FreeBSD.ORG) Received: from smtp02.primenet.com (daemon@smtp02.primenet.com [206.165.6.132]) by hub.freebsd.org (8.8.8/8.8.8) with ESMTP id NAA04774; Wed, 17 Jun 1998 13:56:15 -0700 (PDT) (envelope-from tlambert@usr01.primenet.com) Received: (from daemon@localhost) by smtp02.primenet.com (8.8.8/8.8.8) id NAA18054; Wed, 17 Jun 1998 13:56:06 -0700 (MST) Received: from usr01.primenet.com(206.165.6.201) via SMTP by smtp02.primenet.com, id smtpd018027; Wed Jun 17 13:56:04 1998 Received: (from tlambert@localhost) by usr01.primenet.com (8.8.5/8.8.5) id NAA28313; Wed, 17 Jun 1998 13:55:58 -0700 (MST) From: Terry Lambert Message-Id: <199806172055.NAA28313@usr01.primenet.com> Subject: Re: Filesystem Development Toolkit To: kline@tao.thought.org (Gary Kline) Date: Wed, 17 Jun 1998 20:55:58 +0000 (GMT) Cc: michaelh@cet.co.jp, Matthew.Alton@anheuser-busch.com, FreeBSD-fs@FreeBSD.ORG, Scott.Smallie@anheuser-busch.com, Hackers@FreeBSD.ORG In-Reply-To: <199806171819.LAA06262@tao.thought.org> from "Gary Kline" at Jun 17, 98 11:19:58 am X-Mailer: ELM [version 2.4 PL25] MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: owner-freebsd-fs@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > If Terry's FS-based Unicode support would fit into this, > it'd be interesting. So far I'm working on localizations > via the locale catalogs. This may be a short-term solution > and a broader, global solution may be a FS with wchar_t > support. > > Any thoughts; or am I too far off-course? Unicode cn_pnbuf code is part of the problem for the NTFS, which, like VFAT, has multiple namespaces which must be kept coherent, and whose coherency can't be implemented via late-binding. The bigger problem here is that nameidata structure is not treated as relatively opaque, except for the name spaces an FS is interested in accessing. Because the cn_pnbuf is freed at random locations in the kernel, this dictates implementation for VOP's which utilize the nameidata, such as VOP_LOOKUP, VOP_RENAME, VOP_LINK, VOP_CREATE, VOP_UNLINK, etc.. In order to be able to deal with both the Unicode and the DOS code page based 8.3 name at the same time, the path needs to be broken up into a parsed path structure, wherein seperate components are grouped. For the initial case, where we are still passing an 8 bit string to those system calls that take paths, the easies conversionis a direct mapping to code page 0 (ISO 8859-1) in the Unicode 16/8 set (or the ISO 10646 32/8 set). System calls that operate purely on Unicode objects can come later, and the legacy support can be pushed into libc (open( 8bit:8859-1, ) -> uopen( 16bit:Unicode, )) later. POSIX compatability is an issue that can be dealt with in the library. Alternately, a wchar_t encoding could have a "magic" introducer that is prepended to strings, and ignored except when the strings are passed to system calls (open( char *) vs. open( wchar_t *) prototpyes, as in c++ namespace overlaoding). Maybe "0wchar_t0 r e a l s t r i n g" for strings declared "wide". (_W"realstring"). It doesn't really matter; it depend on who you want to take the hit (I prefer hitting th old code). Terry Lambert terry@lambert.org --- Any opinions in this posting are my own and not those of my present or previous employers. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-fs" in the body of the message From owner-freebsd-fs Wed Jun 17 15:16:50 1998 Return-Path: Received: (from majordom@localhost) by hub.freebsd.org (8.8.8/8.8.8) id PAA20212 for freebsd-fs-outgoing; Wed, 17 Jun 1998 15:16:50 -0700 (PDT) (envelope-from owner-freebsd-fs@FreeBSD.ORG) Received: from parkplace.cet.co.jp (parkplace.cet.co.jp [202.32.64.1]) by hub.freebsd.org (8.8.8/8.8.8) with ESMTP id PAA20170; Wed, 17 Jun 1998 15:16:41 -0700 (PDT) (envelope-from michaelh@cet.co.jp) Received: from localhost (michaelh@localhost) by parkplace.cet.co.jp (8.8.8/CET-v2.2) with SMTP id WAA08666; Wed, 17 Jun 1998 22:15:23 GMT Date: Thu, 18 Jun 1998 07:15:23 +0900 (JST) From: Michael Hancock To: Terry Lambert cc: Matthew.Alton@anheuser-busch.com, FreeBSD-fs@FreeBSD.ORG, Scott.Smallie@anheuser-busch.com, Hackers@FreeBSD.ORG Subject: Re: Filesystem Development Toolkit In-Reply-To: <199806172020.NAA26615@usr01.primenet.com> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-fs@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Ok, there are 3 problems: 1) Refcounting and locking layering violations remaining in 3 vops. 2) cn_path freeing layering violations. 3) vnode object coherence. 1 and 2 are in the same class of complexity, mostly grunt work but still requires someone reasonably knowledgeable of the bsd file systems. A solution to 3 that allows you to take advantage of caching at different layers is the tough one. An easier solution is to never cache, but always proxy to the right object. Heidemann has a nice paper on the tough solution, but it's hard to verify that a hands free approach to locking and distributed operations is correct. Which is why I'd like to look at Simon's DLM. Regards, Mike Hancock To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-fs" in the body of the message From owner-freebsd-fs Wed Jun 17 16:23:54 1998 Return-Path: Received: (from majordom@localhost) by hub.freebsd.org (8.8.8/8.8.8) id QAA01139 for freebsd-fs-outgoing; Wed, 17 Jun 1998 16:23:54 -0700 (PDT) (envelope-from owner-freebsd-fs@FreeBSD.ORG) Received: from gatewayb.anheuser-busch.com (gatewayb.anheuser-busch.com [151.145.250.253]) by hub.freebsd.org (8.8.8/8.8.8) with SMTP id QAA01126; Wed, 17 Jun 1998 16:23:51 -0700 (PDT) (envelope-from Matthew.Alton@anheuser-busch.com) Received: by gatewayb.anheuser-busch.com; id SAA09265; Wed, 17 Jun 1998 18:22:03 -0500 Received: from stlabcexg003.anheuser-busch.com( 151.145.101.158) by gatewayb via smap (V2.1) id xma009156; Wed, 17 Jun 98 18:21:43 -0500 Received: by STLABCEXG003 with Internet Mail Service (5.5.1960.3) id ; Wed, 17 Jun 1998 18:19:57 -0500 Message-ID: <31B3F0BF1C40D11192A700805FD48BF9017765EF@STLABCEXG011> From: "Alton, Matthew" To: "'johnh@isi.edu'" Cc: "'FreeBSD-fs@FreeBSD.ORG'" , "'Hackers@FreeBSD.ORG'" , "Smallie, Scott" Subject: Stackable filesystems and SunOS 4.1.1 Date: Wed, 17 Jun 1998 18:21:37 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.1960.3) Content-Type: text/plain Sender: owner-freebsd-fs@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Sun Microsystems has apparently freed the 4.1.1 binary distribution. It is available from: http://doener.unix-ag.uni-kl.de/ It is conceivable that Sun may be persuaded to release the sources for a fee a la SCO's "Ancient Versions" source license. If so, we FreeBeasties could benifit from the mature SunOS 4.1.1 stackable filesystem code. Maybe we should pester them. Matthew Alton Computer Services - UNIX Systems Administration (314)632-6644 matthew.alton@anheuser-busch.com alton@plantnet.com To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-fs" in the body of the message From owner-freebsd-fs Wed Jun 17 18:57:12 1998 Return-Path: Received: (from majordom@localhost) by hub.freebsd.org (8.8.8/8.8.8) id SAA23245 for freebsd-fs-outgoing; Wed, 17 Jun 1998 18:57:12 -0700 (PDT) (envelope-from owner-freebsd-fs@FreeBSD.ORG) Received: from smtp03.primenet.com (daemon@smtp03.primenet.com [206.165.6.133]) by hub.freebsd.org (8.8.8/8.8.8) with ESMTP id SAA23240; Wed, 17 Jun 1998 18:57:10 -0700 (PDT) (envelope-from tlambert@usr09.primenet.com) Received: (from daemon@localhost) by smtp03.primenet.com (8.8.8/8.8.8) id SAA23554; Wed, 17 Jun 1998 18:57:04 -0700 (MST) Received: from usr09.primenet.com(206.165.6.209) via SMTP by smtp03.primenet.com, id smtpd023462; Wed Jun 17 18:56:55 1998 Received: (from tlambert@localhost) by usr09.primenet.com (8.8.5/8.8.5) id SAA07446; Wed, 17 Jun 1998 18:56:49 -0700 (MST) From: Terry Lambert Message-Id: <199806180156.SAA07446@usr09.primenet.com> Subject: Re: Stackable filesystems and SunOS 4.1.1 To: Matthew.Alton@anheuser-busch.com (Alton, Matthew) Date: Thu, 18 Jun 1998 01:56:49 +0000 (GMT) Cc: johnh@isi.edu, FreeBSD-fs@FreeBSD.ORG, Hackers@FreeBSD.ORG, Scott.Smallie@anheuser-busch.com In-Reply-To: <31B3F0BF1C40D11192A700805FD48BF9017765EF@STLABCEXG011> from "Alton, Matthew" at Jun 17, 98 06:21:37 pm X-Mailer: ELM [version 2.4 PL25] MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: owner-freebsd-fs@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > Sun Microsystems has apparently freed the 4.1.1 binary distribution. It is > available from: > http://doener.unix-ag.uni-kl.de/ > > It is conceivable that Sun may be persuaded to release the sources for a fee > a la SCO's "Ancient Versions" source license. If so, we FreeBeasties could > benifit from the mature SunOS 4.1.1 stackable filesystem code. Maybe we > should pester them. This distrubtion is *only* for Sun3 hardware. The distributor admits to taking the risk upon himself of distributing the binary images without the legal consent of Sun Microsystems. The SunOS code contains a VFS implementation, but does not contain a stacking vnode architecture (even though Rosental's stacking research was under the auspices of Sun Microsystems). I think you are reading too much into SunOS's "union" and "translucent" mount capabilities, which are less related to a stacking architecture than they are to special one-off implementation. Terry Lambert terry@lambert.org --- Any opinions in this posting are my own and not those of my present or previous employers. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-fs" in the body of the message From owner-freebsd-fs Thu Jun 18 09:38:35 1998 Return-Path: Received: (from majordom@localhost) by hub.freebsd.org (8.8.8/8.8.8) id JAA02121 for freebsd-fs-outgoing; Thu, 18 Jun 1998 09:38:35 -0700 (PDT) (envelope-from owner-freebsd-fs@FreeBSD.ORG) Received: from uni4nn.gn.iaf.nl (osmium.gn.iaf.nl [193.67.144.12]) by hub.freebsd.org (8.8.8/8.8.8) with SMTP id JAA02088; Thu, 18 Jun 1998 09:38:31 -0700 (PDT) (envelope-from wilko@yedi.iaf.nl) Received: by uni4nn.gn.iaf.nl with UUCP id AA18577 (5.67b/IDA-1.5); Thu, 18 Jun 1998 18:21:48 +0200 Received: (from wilko@localhost) by yedi.iaf.nl (8.8.8/8.6.12) id IAA08043; Thu, 18 Jun 1998 08:49:14 +0200 (CEST) From: Wilko Bulte Message-Id: <199806180649.IAA08043@yedi.iaf.nl> Subject: Re: Stackable filesystems and SunOS 4.1.1 In-Reply-To: <31B3F0BF1C40D11192A700805FD48BF9017765EF@STLABCEXG011> from "Alton, Matthew" at "Jun 17, 98 06:21:37 pm" To: Matthew.Alton@anheuser-busch.com (Alton, Matthew) Date: Thu, 18 Jun 1998 08:49:14 +0200 (CEST) Cc: johnh@isi.edu, FreeBSD-fs@FreeBSD.ORG, Hackers@FreeBSD.ORG, Scott.Smallie@anheuser-busch.com X-Organisation: Private FreeBSD site - Arnhem, The Netherlands X-Pgp-Info: PGP public key at 'finger wilko@freefall.freebsd.org' X-Mailer: ELM [version 2.4ME+ PL38 (25)] Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: owner-freebsd-fs@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org As Alton, Matthew wrote... > Sun Microsystems has apparently freed the 4.1.1 binary distribution. It is > available from: > http://doener.unix-ag.uni-kl.de/ They have not freed it. They are presumably looking the other way. That is all. The 4.1.1 mentioned in the URL is Sun3 and Sun3x only, so 680[23]0 CPUs. Wilko _ ______________________________________________________________________ | / o / / _ Bulte email: wilko @ yedi.iaf.nl |/|/ / / /( (_) Arnhem, The Netherlands WWW: http://www.tcja.nl ______________________________________________ Powered by FreeBSD __________ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-fs" in the body of the message From owner-freebsd-fs Fri Jun 19 08:51:55 1998 Return-Path: Received: (from majordom@localhost) by hub.freebsd.org (8.8.8/8.8.8) id IAA07066 for freebsd-fs-outgoing; Fri, 19 Jun 1998 08:51:55 -0700 (PDT) (envelope-from owner-freebsd-fs@FreeBSD.ORG) Received: from gatewayb.anheuser-busch.com (gatewayb.anheuser-busch.com [151.145.250.253]) by hub.freebsd.org (8.8.8/8.8.8) with SMTP id IAA07044; Fri, 19 Jun 1998 08:51:47 -0700 (PDT) (envelope-from Matthew.Alton@anheuser-busch.com) Received: by gatewayb.anheuser-busch.com; id KAA02860; Fri, 19 Jun 1998 10:50:00 -0500 Received: from stlabcexg003.anheuser-busch.com( 151.145.101.158) by gatewayb via smap (V2.1) id xma002855; Fri, 19 Jun 98 10:49:34 -0500 Received: by STLABCEXG003 with Internet Mail Service (5.5.1960.3) id ; Fri, 19 Jun 1998 10:47:48 -0500 Message-ID: <31B3F0BF1C40D11192A700805FD48BF9017765F1@STLABCEXG011> From: "Alton, Matthew" To: "'Terry Lambert'" Cc: johnh@isi.edu, FreeBSD-fs@FreeBSD.ORG, Hackers@FreeBSD.ORG, "Smallie, Scott" Subject: RE: Stackable filesystems and SunOS 4.1.1 Date: Fri, 19 Jun 1998 10:48:51 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.1960.3) Content-Type: text/plain Sender: owner-freebsd-fs@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org > -----Original Message----- > From: Terry Lambert [SMTP:tlambert@primenet.com] > Sent: Wednesday, June 17, 1998 8:57 PM > To: Alton, Matthew > Cc: johnh@isi.edu; FreeBSD-fs@FreeBSD.ORG; Hackers@FreeBSD.ORG; Smallie, > Scott > Subject: Re: Stackable filesystems and SunOS 4.1.1 > > > > Sun Microsystems has apparently freed the 4.1.1 binary distribution. It is > > available from: > > http://doener.unix-ag.uni-kl.de/ > > > > It is conceivable that Sun may be persuaded to release the sources for a fee > > a la SCO's "Ancient Versions" source license. If so, we FreeBeasties could > > benifit from the mature SunOS 4.1.1 stackable filesystem code. Maybe we > > should pester them. [Alton, Matthew] I apologize for being hopelessy vague and incomplete here. I was thinking about John Heidemann's work. His web page says that his primary development platform was SunOS 4.1.1 and that there is "unsatisfactory" code in Free- & Net- BSD. I downloaded the necessarily incomplete (license issues)code from his site and was pining away for the rest. > This distrubtion is *only* for Sun3 hardware. > > The distributor admits to taking the risk upon himself of > distributing the binary images without the legal consent of Sun > Microsystems. [Alton, Matthew] I believe that Sun blessed this recently. The web page thanks Herr Somebody of Germany Sun for making these images available. [Alton, Matthew] Anyway, my Filesystem Development Odyssey (sp?) has now led me through sev- eral Dante-esque levels of La Inferno. I remain fixated upon the goal of obtaining as platform-independent a set of semantics as I can lay hands on. Is fleshing out the Heidemann work the Right Thing? To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-fs" in the body of the message From owner-freebsd-fs Sat Jun 20 17:25:41 1998 Return-Path: Received: (from majordom@localhost) by hub.freebsd.org (8.8.8/8.8.8) id RAA01810 for freebsd-fs-outgoing; Sat, 20 Jun 1998 17:25:41 -0700 (PDT) (envelope-from owner-freebsd-fs@FreeBSD.ORG) Received: from parkplace.cet.co.jp (parkplace.cet.co.jp [202.32.64.1]) by hub.freebsd.org (8.8.8/8.8.8) with ESMTP id RAA01746; Sat, 20 Jun 1998 17:25:32 -0700 (PDT) (envelope-from michaelh@cet.co.jp) Received: from localhost (michaelh@localhost) by parkplace.cet.co.jp (8.8.8/CET-v2.2) with SMTP id AAA05452; Sun, 21 Jun 1998 00:24:10 GMT Date: Sun, 21 Jun 1998 09:24:10 +0900 (JST) From: Michael Hancock To: "Alton, Matthew" cc: "'Terry Lambert'" , johnh@isi.edu, FreeBSD-fs@FreeBSD.ORG, Hackers@FreeBSD.ORG, "Smallie, Scott" Subject: RE: Stackable filesystems and SunOS 4.1.1 In-Reply-To: <31B3F0BF1C40D11192A700805FD48BF9017765F1@STLABCEXG011> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-fs@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.org Matthew, I think it is definitely worthwhile to review John's work. The approach you suggested earlier can easily be made portable, but it is a very very large project. It reminds me of how Oracle or Sybase are implemented as large systems with OS-like features. Just implementing the name space subsystem is not a trivial undertaking. Have a look at namei() and you'll understand what I'm talking about. Portability and not reinventing the wheel is a lot of what John's work is about. Regards, Mike To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-fs" in the body of the message